On 9/3/07, Babu, Lokesh <[EMAIL PROTECTED]> wrote: > One more question, As you said, Full text builds an index of data, so > I hope you have done some memory analysis too, could you please tell > me the memory usage based on your analysis.
That's really something that you'll wan to do with your specific goals in mind and with a representative dataset. The memory usage is adequate for my primary customer (Google Gears), but they're running in a desktop context, so YMMV. The code has an optimization which collects multiple records within a transaction to make updates more efficient. This uses approximately 1M of memory for a buffer. It can be changed by tweaking kPendingThreshold in the source. 0 is an acceptable value to turn off the buffering. The overall memory usage should be proportional to what you throw at it. So if you insert large documents, it will use memory proportional to the number of terms in that document. There are no current plans to address this (it seems sensible to me to expect large documents to require more memory to process than small documents). If you query for popular terms, it will use memory proportional to the number of hits those terms have in the index. I do hope to address this over time, because it is expected that users will insert lots of documents. Hmm. What else? When you have a lot of documents in the index, the current code may touch a lot of data when doing segment merges. This isn't fts memory, per se, but it is being used by sqlite, so it needs consideration. Segment merges happen at powers-of-16 boundaries, so 16, 256, 4k, 64k, 1m, etc. The 64k-doc merges are pretty reasonable in desktop apps, perhaps less so for embedded apps. The 1m-doc merges aren't even reasonable for desktop apps, and this is something I'm working on for the next version. -scott ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
