Some tests I have been running agree with your findings for Spectra performance etc.
Also, I believe I have found some interesting information in regards to verity
collection performance. It seems that EVERY time you do any kind of update, insert,
optimize, even purge to a verity index your index files sizes increases some. I am
venturing a guess that Verity stores a transaction log or some such overhead and thus
the "bloating". Optimization seems to repair most of this "bloating", but it does not
seem to be a 100% fix
For example, I started with shiny new set of verity collections on my test server.
Their (all Spectra related collections) total file size was 1.01 megs. After running
some test indexing, then purging all the indexes (either via cf admin or by code) the
collection size ended up at 1.12, then 1.23, then 1.42, then 1.53, then 1.64, then
1.73.
Also, I found that passing a lObjectIDs list is the only way to do multiple index
update (not only for speed as you mention but to minimize the verity "bloating" as
well. For example: In one test scenario running each index update by itself took 37
minutes and made my collections total 45.7 megs. Optimizing all the collections took
the collections down to 1.85 megs. However, passing lObjectIDs and making 1
cfa_typeIndexKeyUpdate and 1 cfa_metaDataIndexUpdate call took under 3 minutes and
made the collections total 1.61 megs.
The above findings lead me to believe that even with very strict optimization of your
collections sooner or later you are going to need to completely delete them (not
purge) and recreate and rebuild them from the ground up.
Jeremy Petersen
-----Original Message-----
From: Michiel Boland [mailto:[EMAIL PROTECTED]]
Sent: Saturday, July 15, 2000 7:19 PM
To: [EMAIL PROTECTED]
Subject: More verity musings
Ok I have been fiddling with verity a bit more. I'm developing an
application that needs to import a, possibly large, number of content
objects.
My import routine first called cfa_typeIndexUpdate for each individual
imported object. A single typeIndexUpdate takes 2 seconds (*) on a good
collection and gradually degrades to 6 seconds after about 100 objects. I
don't need to tell you that this gets unbearably slow. Updating the type
index for all new objects (141 total) at the end, using
cfa_typeIndexUpdate with the objectIDs as lObjectIDs parameter of the
import took just 6.7 seconds.
I then proceeded to metadata indexing. From my very recent experience with
type indexing I decided not to update the metadata index for each
individual object but just call cfa_metadataindexupdate at the end of the
import with the list of new objectIDs as lObjectIDs paramter.
Each object gets a few keywords, so I end up with 141 objects and about
the same number of keywords at the end of the import. To my horror I found
that cfa_metadataindexupdate took about 145 seconds!
I then replaced it with cfa_metadataindexall which took just 15.5 seconds.
(Which still is a bit slow, but luckily the import will be done by a bot,
so the user will not get bored or hit stop by accident. :) It turns out
that cfa_metadataindexupdate does a 'CFINDEX ACTION="DELETE"' for every
object in its lObjectIDs argument (twice even.) Obviously this does not
work if you have a very large number of objects.
In the end I just used cfa_typeIndex at the end of the import in
combination with cfa_metadataIndexAll; there does not appear to be any
significant speed difference between cfa_typeIndexUpdate with a large
number of objects and just cfa_typeIndex.
Morale: (someone correct me if I'm wrong)
- if you insert a new contentobject, use cfa_typeindexupdate and
cfa_metadataindexupdate;
- if you insert a bunch of contentobjects, use cfa_typeindex (or
cfa_typeindexupdate with your objects as lObjectIDs parameter) and
cfa_metadataindexall; DO NOT use cfa_metadataindexupdate with a large
number of object ids;
Cheers
Michiel
(*) testing was done on a spare pentium 350 with 128MB RAM and a single
IDE disk with Spectra 1.0.1, CF451 and MSSQL7.
--
Michiel Boland <[EMAIL PROTECTED]>
Digital Valley Internet Professionals
Plantsoen 17, Wageningen, The Netherlands
Phone: +31 317 465555, Fax: +31 317 460276
------------------------------------------------------------------------------
To Unsubscribe visit
http://www.houseoffusion.com/index.cfm?sidebar=lists&body=lists/spectra_talk or send a
message to [EMAIL PROTECTED] with 'unsubscribe' in the body.
------------------------------------------------------------------------------
To Unsubscribe visit
http://www.houseoffusion.com/index.cfm?sidebar=lists&body=lists/spectra_talk or send a
message to [EMAIL PROTECTED] with 'unsubscribe' in the body.
