On 11/25/2014 02:53 AM, Marcin Sobieszczanski wrote:
Hi
I work with sqlite files that have a few gigabytes of simple data.
Almost all of the data sits in one table that has 9 non-null integer
columns (including row_id, and one int64 column) plus 3 additional
string or int columns (additional columns are filled with nulls
mostly). The table has tens of millions of rows. (I need to
cross-correlate this data.)
Recently, I upgraded to 3.8.7.2 and I've notice that CREATE INDEX on a
table's 3 columns hangs in the new version of sqlite (last OK
versions: 3.8.4 and 3.8.6).
I located the code in sqlite3VdbeSorterWrite: there is only one loop there:
while( nNew < nMin ) nNew = nNew*2;
In my case nNew == 0 (inspected with gdb; but without symbols). I
believe the multiplication in the line above overflown:
int nNew = pSorter->nMemory * 2;
Last strace lines show series of re-allocations:
mremap(0x7f8241fff000, 33558528, 67112960, MREMAP_MAYMOVE) = 0x7f81b7fff000
...
mremap(0x7f819fffd000, 268439552, 536875008, MREMAP_MAYMOVE) = 0x7f817fffc000
mremap(0x7f817fffc000, 536875008, 1073745920, MREMAP_MAYMOVE) = 0x7f813fffb000
The next mremap would have tried to allocate 2147487744 bytes
(2147487744=2^31+4Ki, extra 4Ki from glibc), I guess, but int is not
capable to hold more than 2Gi.
Is this a known problem?
It is now.
Do you have a large cache-size configured?
Thanks,
Dan.
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
