I'm testing new code, and my latest trial run ended with a segmentation
fault after about 5 hours.
I'm running Python 3.5 and its standard sqlite3 module On Xubuntu 16.04.3
LTS. The code is short -- about 300 lines.
This particular program is merging two databases. The result has reached
25 GB, roughly 1/3 of what I expect of the final result (over 100M rows).
The filesystem is a RAID with 2+ TB free. The machine is a Core i7 with 32
GB RAM and 0 swap has been used since the last reboot. Nothing else much
is running on this machine except some idle terminal and browser windows.
Here's my prime suspect: I'm using WAL, and the journal is 543 MB. I
hadn't given it much thought, but could this be more than the software
really wants to deal with? I'm going to try doing occasional commits
(every 100K inserts/updates perhaps,) but I'd like some help:
1. If I'm on the right track, tell me so I can stop worrying and proceed
2. If I'm on the wrong track, help me figure out how to debug the problem.
I can probably find out what particular part of the merge it had reached,
but it's going to take quite a while. I'm pretty good with GDB but I have
no idea how to explore a running Python program.
The project is a hobby, so there's nothing proprietary, and I can post any
information that would help.
word of the year: *kakistocracy*
sqlite-users mailing list