Re: [sqlite] Simple read-only program very slow

2017-12-04 Thread Keith Medcalf
>All of my machines are 64-bit, and run 64-bit Linux kernels. Python3 >is a 64-bit executable according to 'file'. >I re-ran the whole thing (slightly modified to also do a test on >pmain) on my big dual-Xeon (32 hyperthreads, 256GB! RAM ) from >System76. In spite of having about half the CPU

Re: [sqlite] Simple read-only program very slow

2017-12-04 Thread Kevin O'Gorman
order for the entire btree structures to be cached in > >RAM -- you > >> probably want to make it as big as you can. > >> > >> --- > >> The fact that there's a Highway to Hell but only a Stairway to > >Heaven says > >> a lot about anticipated traffic volume. &g

Re: [sqlite] Simple read-only program very slow

2017-12-03 Thread Keith Medcalf
robably want to make it as big as you can. >> >> --- >> The fact that there's a Highway to Hell but only a Stairway to >Heaven says >> a lot about anticipated traffic volume. >> >> >> >-----Original Message- >> >From: sqlite-users [mailto:sqlite-u

Re: [sqlite] Simple read-only program very slow

2017-12-03 Thread Richard Rousselot
affic volume. > > > >-Original Message- > >From: sqlite-users [mailto:sqlite-users- > >boun...@mailinglists.sqlite.org] On Behalf Of Kevin O'Gorman > >Sent: Saturday, 25 November, 2017 20:14 > >To: sqlite-users > >Subject: [sqlite] Simple read-only pr

Re: [sqlite] Simple read-only program very slow

2017-11-30 Thread Keith Medcalf
man >Sent: Saturday, 25 November, 2017 20:14 >To: sqlite-users >Subject: [sqlite] Simple read-only program very slow > >I'm pretty new at SQLite, so this may seem obvious to you. Be kind. >I'm using Python on Ubuntu Linux 16.04 LTS, and the sqlite that is >built >into Python. The da

Re: [sqlite] Simple read-only program very slow

2017-11-30 Thread Kevin O'Gorman
On Sun, Nov 26, 2017 at 12:02 AM, Clemens Ladisch wrote: > Kevin O'Gorman wrote: > > I wrote a super simple program to the read the file and count how many > > records are already there. I got impatient waiting for it so I killed > > the process and added an output of one

Re: [sqlite] Simple read-only program very slow

2017-11-30 Thread Kevin O'Gorman
On Sun, Nov 26, 2017 at 1:39 AM, Simon Slavin wrote: > > > On 26 Nov 2017, at 3:13am, Kevin O'Gorman wrote: > > > > I've got a database of some 100 million records, and a file of just over > > 300 thousand that I want represented in it. I wanted

Re: [sqlite] Simple read-only program very slow

2017-11-26 Thread Simon Slavin
On 26 Nov 2017, at 3:13am, Kevin O'Gorman wrote: > > I've got a database of some 100 million records, and a file of just over > 300 thousand that I want represented in it. I wanted to check how much > difference it was going to make, so I wrote a super > simple

Re: [sqlite] Simple read-only program very slow

2017-11-26 Thread Clemens Ladisch
Kevin O'Gorman wrote: > I wrote a super simple program to the read the file and count how many > records are already there. I got impatient waiting for it so I killed > the process and added an output of one dot (".") per 1000 records. It > went very fast for what I estimate was around 200 dots

[sqlite] Simple read-only program very slow

2017-11-25 Thread Kevin O'Gorman
I'm pretty new at SQLite, so this may seem obvious to you. Be kind. I'm using Python on Ubuntu Linux 16.04 LTS, and the sqlite that is built into Python. The database is using WAL. I've got a database of some 100 million records, and a file of just over 300 thousand that I want represented in