On Tue, Apr 28, 2015 at 1:05 PM, Dan Kennedy <danielk1977 at gmail.com> wrote:
> I just took a quick glance at the FTS code. As I said, it has been a couple >>> years, but this looks like the malloc that was failing for me at the >>> time: >>> http://www.sqlite.org/cgi/src/artifact/81f9ed55ad586148?ln=2473 >>> >> >> That one is allocating enough space for the doclist associated with a >> single term. Doclists are between say 2 and 12 bytes in size for each >> instance of a term in the document set. So they can get quite large for >> common terms ("a", "the" etc.). And the OP does have over a billion >> documents. So I guess if there is a fairly common term in there, that >> allocation could be too large for the OS to satisfy. >> > > Or, really, 32-bit overflow resulting in a negative value being passed to > sqlite3_malloc() causing the OOM report. Huh. In looking at the code some more, it could have been in segment merging code in fts3_write.c @ http://www.sqlite.org/cgi/src/artifact/4f005f78592a1447 ... I'll try to generate some test data and see if I can get you some real information vs hazy recollections. -- Scott Robison
