On 05/19/2015 05:54 PM, Artem wrote:
>> On 05/19/2015 03:35 PM, Artem wrote:
>>> Hi!
>>>
>>> And what about result of our conversation?
>>> Can developers increase this limitations for using all memory that user
>>> have?
>> Hi Artem,
>> The conclusion was that although the first problem encountered is the
>> massive allocation that FTS tries to make, fixing that won't actually
>> improve things much. SQLite is hard-coded to limit the size of blobs to
>> a bit under 1GiB. So even if we could merge these big doclists in
>> memory, we're pretty close to the limit of what can be stored in the
>> database anyway.
>> The 1GiB limit can be raised to 2GiB by recompiling SQLite with
>> different options. But any further than that would require a redesign of
>> the file-format.
>> So to fix this, we would need to change the format FTS4 uses to store
>> the index within the database. Some sort of extension to allow it to
>> spread the largest doclists across two or more database records. And
>> unfortunately I suspect that will remain a low priority for the
>> foreseeable future.
>> I've tested FTS5 with really large databases, and it seems to work. It's
>> not actually released yet though.
>> Regards,
>> Dan.
> Thank you very much, Dan. Now I understand the situation.
> Can you please tell me, when FTS5 will be released? Maybe I can test
> it on my data before official release?
Hi Artem,
I'm not sure exactly when fts5 will be released. Testing with a large
data set would be extremely helpful though. Grab the "fts5" branch from
fossil and run [make sqlite3.c] to create an amalgamation file that
includes fts5. Build with -DSQLITE_ENABLE_FTS5.
For large data sets, I think it will help to run a command like:
INSERT INTO tbl(tbl, rank) VALUES('pgsz', 4040);
after creating the FTS5 table (where "tbl" is the name of the new
table). More here:
https://sqlite.org/fts5.html#section_5_7
Thanks,
Dan.
