I'm working on a hobby project, but the data has gotten a bit out of hand. I thought I'd put it in a real database rather than flat ASCII files.
I've got a problem set of about 1 billion game positions and 187GB to work on (no, I won't have to solve them all) that took about 4 hours for a generator program just to write. I wrote code to turn them into something SQLite could import. Actually, it's import, build a non-primary index, and alter table to add a column, all in sqlite3. The database was growing for about 1-1/2 days. Then its journal disappeared, the file size dropped to zero, but sqlite3 is still running 100% CPU time, now for a total of 3800+ minutes (63+ hours). The database is still locked, but I have no idea what sqlite3 is doing, or if it will ever stop. All partitions still have lots of space left (most of this is running in a RAID partition of 11 TiB). Here's what I gave to sqlite3 on my Linux system: time sqlite3 qubic.db <<EOF BEGIN EXCLUSIVE TRANSACTION; DROP TABLE IF EXISTS qmoves; CREATE TABLE qmoves ( qfrom CHAR(64), qmove INT, qto CHAR(64), qweight INT, PRIMARY KEY (qfrom, qmove) ON CONFLICT ROLLBACK ); CREATE INDEX IF NOT EXISTS qmoves_by_dest ON qmoves ( qto, qweight ); CREATE TABLE IF NOT EXISTS qposn ( qposn CHAR(64) PRIMARY KEY ON CONFLICT ROLLBACK, qmaxval INT, qmove INT, qminval INT, qstatus INT ); .separator " " .import am.all qmoves ALTER TABLE qmoves ADD COLUMN qstatus INT DEFAULT NULL; .schema COMMIT TRANSACTION; EOF Any clues, hints, or advice? -- #define QUESTION ((bb) || (!bb)) /* Shakespeare */ _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
