> Personally I don't see it as a bug. A limitation, yes. A different > algorithm that requires less ram would remove / change the limit.
> I'll be trying some sample data tomorrow (if time permits) to see if I can > come up with any ideas. > I described an approach in a previous email where I divided my FTS index > into 53 partitions. It means running more queries, but maybe something like > that could serve as a temporary solution. What queries (partitions) do you mean? I tried to create FTS with LIMIT clause like: INSERT INTO pages_fts(title, body) SELECT title, body FROM pages LIMIT 0,1000000 INSERT INTO pages_fts(title, body) SELECT title, body FROM pages LIMIT 1000000,2000000 etc It was OK until 600 mln rows, after 600 mln rows it failed with Segmentation Fault error (on Ubuntu). > On Apr 29, 2015 1:56 AM, "Artem" <devspec at yandex.ru> wrote: >> So, is that a real bug of SQLIte and how to fix it? >> I really need to create huge FTS-indexes like that, >> I have 32GB of memory for that. >> >> Many thanks for your attention. >> >> > On 04/29/2015 05:27 AM, Artem wrote: >> >> ????????????, Tim. >> >> >> >> ?? ?????? 29 ?????? 2015 ?., 1:21:00: >> >> >> >>> On 28 Apr 2015 at 23:14, Artem <devspec at yandex.ru> wrote: >> >>>>> How about trying the sqlite3.exe command line utility. put your >> >>>>> sql for that operation in a text file, launch the program, open >> >>>>> the database, then read in the sql file with the .read command. >> >>>>> If the error occurs, then possibly sqlite3. if not then it is >> >>>>> probably something else. >> >>>> I tried it and failed. >> >>>> >> >>>> Console log: >> >>>> >> >>>> f:\Suggests\test>sqlite3.exe single.db >> >>>> SQLite version 3.8.9 2015-04-08 12:16:33 >> >>>> Enter ".help" for usage hints. >> >>>> sqlite> .read test.sql >> >>>> Error: near line 1: out of memory >> >>> That's not a segfault, though, is it. >> >> When I did the same in linux version of SQLite - I saw >> >> the "Segmentation Fault" error. >> >> > Maybe something to do with the optimistic allocation strategy Linux >> > uses. Perhaps malloc() returned non-NULL but then a segfault occurred >> > when it first tried to access the pages. From the man-page: >> >> > By default, Linux follows an optimistic memory allocation >> > strategy. This means that when malloc() returns non-NULL >> > there is no guarantee that the memory really is available. >> >> > Or maybe the OOM killer took out the process. Or something. >> >> > Dan. >> >> >> > _______________________________________________ >> > sqlite-users mailing list >> > sqlite-users at mailinglists.sqlite.org >> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >> >> >> >> -- >> ? ?????????, >> Artem mailto:devspec at yandex.ru >> >> _______________________________________________ >> sqlite-users mailing list >> sqlite-users at mailinglists.sqlite.org >> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >> > _______________________________________________ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
