Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On 4 Aug 2016, at 4:00am, Kevin O'Gorman wrote: > I've got a problem set of about 1 billion game positions and 187GB to work > on (no, I won't have to solve them all) that took about 4 hours for a > generator program just to write. I wrote code to turn them into

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

Hello Kevin, I'd write a utility to do it instead of using the command line tool then add logging to the program in order to note progress. I like the idea of chopping it into smaller parts too. "ON CONFLICT ROLLBACK" You're doing one large transaction and if it rolls back it'll have

Re: [sqlite] core dump when writing the DB in the middle of the long read

More detailed core dump info. (gdb) bt #0 0x2c673d5c in memset () from /lib32/libc.so.6 #1 0x2c7b0fa8 in sqlite3VdbeHalt () from /ovn/lib/libsqlite3.so.3 #2 0x2c7b1904 in sqlite3VdbeReset () from /ovn/lib/libsqlite3.so.3 #3 0x2c7b55a8 in sqlite3_reset () from /ovn/lib/libsqlite3.so.3 #4

[sqlite] Tcl [db eval varname ...] non-array varname silently fails

If the array-name in a [db eval arrayname ...] command isn't actually an array, sqlite silently fails. The script is evaluated but the variable doesn't contain values from the query: package require sqlite3 sqlite3 db :memory: db eval { create table t (f) ; insert

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

Temp Files Have you checked how much storage is available to the temporary file locations? The temporary file locations are different depending on the OS, build, VFS and PRAGMA settings. See the last section "5.0 Temporary File Storage Locations" of: https://www.sqlite.org/tempfiles.html The

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

A lot of speculation here. I am certainly no SQLite expert. Your input has 1g positions, taking 187gb, so averaging 187b/position. From your CREATE TABLE, it looks like to get that size most of your qfrom and qto are fairly long strings. I'm assuming there are a great many duplications in

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On 8/4/16, Wade, William wrote: > > I believe that with SQLite, if you don't specify WITHOUT ROWID your "real" > record order is based on rowid, Correct > > In principle, indices can be created by writing the needed information > (index key, record position) in the original

[sqlite] Education Opportunity Alert: The SQL Primer

Hi all, Let me begin by thanking everyone for the feedback on the YouTube video effort! For those who would like to revisit our relatively comprehensive update to a professional introduction to SQL & SQLite, please feel free to share & enjoy =) this latest:

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On Thu, Aug 4, 2016 at 5:05 PM, Kevin O'Gorman wrote: > 3. Positions are 64 bytes always, so your size guesses are right. They are > in no particular order. I like the suggestion of a separate position > table, because they're going to appear in multiple qmove records,

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

Lots of answers, so thanks all around. Some more info: 1. All partitions have at least 3 GB free, and it's not changing. /tmp is 3 TiB and empty. 2. I have a RAID partition, for size, but no RAID controller. As a hobby project, I don't have spare parts, and I fear the results of a failure of a

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On Thu, Aug 4, 2016 at 5:29 PM, Dominique Devienne wrote: > [...] you also force SQLite's SQL parser to parse a huge amount of text. > [...] > OK, maybe not the SQL parser, depends what you write out and the .import mode (I guess, didn't look into the details). But for sure

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

The metric for feasability is coding ease, not runtime. I'm the bottleneck, not the machine, at least at this point. As for adding rows, it will be about like this time: a billion or so at a time. But there's no need to save the old data. Each round can be separate except for a persistent

Re: [sqlite] Education Opportunity Alert: The SQL Primer

Thanks! =) Sent from my iPhone > On Aug 4, 2016, at 5:38 PM, jungle Boogie wrote: > >> On 4 August 2016 at 08:14, R.A. Nagy wrote: >> Comments & suggestion for improvement be both respected, as well as >> appreciated here, as usual! > > WOW!

Re: [sqlite] Education Opportunity Alert: The SQL Primer

Enjoy yourself - by all means, do! Sent from my iPhone > On Aug 4, 2016, at 1:37 PM, Stephen Chrzanowski wrote: > > *sits back and waits for the people to cringe about Sequal or Ess-Que-El > and smirks* > >> On Thu, Aug 4, 2016 at 11:14 AM, R.A. Nagy

Re: [sqlite] Countdown hold. Was: Testing begins on the SQLite "pi" release

On 8/2/16, Richard Hipp wrote: > Testing for the SQLite "pi" release is on hold pending the resolution > of a design question. The countdown to the "pi" release is now resuming. The release candidate is the current trunk version of Fossil, which is also in the "Prerelease

Re: [sqlite] Education Opportunity Alert: The SQL Primer

On 4 August 2016 at 08:14, R.A. Nagy wrote: > Comments & suggestion for improvement be both respected, as well as > appreciated here, as usual! WOW! Such great energy and enthusiasm in the primer video! I'm looking forward to the rest. Best, sean -- --- inum:

Re: [sqlite] Education Opportunity Alert: The SQL Primer

*sits back and waits for the people to cringe about Sequal or Ess-Que-El and smirks* On Thu, Aug 4, 2016 at 11:14 AM, R.A. Nagy wrote: > Hi all, > > Let me begin by thanking everyone for the feedback on the YouTube video > effort! > > For those who would like to revisit our

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On 2016/08/04 5:56 PM, Kevin O'Gorman wrote: On Thu, Aug 4, 2016 at 8:29 AM, Dominique Devienne wrote: It's even less dense than that. Each character has only 3 possible values, and thus it's pretty easy to compress down to 2 bits each, for a 16 byte blob. It's just

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On 2016/08/04 5:05 PM, Kevin O'Gorman wrote: Lots of answers, so thanks all around. Some more info: 1. All partitions have at least 3 GB free, and it's not changing. /tmp is 3 TiB and empty. 2. I have a RAID partition, for size, but no RAID controller. As a hobby project, I don't have

Re: [sqlite] newbie has waited days for a DB build to complete. what's up with this.

On Thu, Aug 4, 2016 at 8:29 AM, Dominique Devienne wrote: > On Thu, Aug 4, 2016 at 5:05 PM, Kevin O'Gorman > wrote: > > > 3. Positions are 64 bytes always, so your size guesses are right. They > are > > in no particular order. I like the