On Sat, Jul 14, 2012 at 03:17:07PM +0100, Simon Slavin wrote:
On 14 Jul 2012, at 3:12pm, Udi Karni uka...@gmail.com wrote:
I know
nothing about writing DB engines - so I don't know whether adding a 2nd
parallel process adds 10K or 10M to the code base.
You've reached the limit of
You are right. True Parallel Query can get very complicated. I was hoping
for something very limited for starters - for example -
- only 2 processes
- only for simple full scans where the block range can be divided in two
- only when there is no ORDER/GROUP BY where sub results from the 2 threads
From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on
behalf of Udi Karni [uka...@gmail.com]
Sent: Wednesday, August 01, 2012 2:25 PM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] Windows I/O (was: Initial read speed greater than
subsequent)
You
On 1 Aug 2012, at 8:25pm, Udi Karni uka...@gmail.com wrote:
- only 2 processes
- only for simple full scans where the block range can be divided in two
- only when there is no ORDER/GROUP BY where sub results from the 2 threads
have to be combined
Premature optimization ? SQLite, just by
On Wed, Aug 1, 2012 at 2:43 PM, Black, Michael (IS)
michael.bla...@ngc.com wrote:
You may be interested in this article:
http://www.drdobbs.com/parallel/multithreaded-file-io/220300055?pgno=2
Mutli-threaded reading of multiple files (which is basically what you're
talking about by splitting
On Fri, 13 Jul 2012 21:40:19 -0700, Udi Karni uka...@gmail.com wrote:
It very easily handles billion row/100GB tables - multi-table joins, etc. -
it just chugs for a while because it's single threaded, and will gobble up
memory and swap - but it gets the job done.
If SQLite memory usage causes
On 14 Jul 2012, at 5:40am, Udi Karni uka...@gmail.com wrote:
It very easily handles billion row/100GB tables - multi-table joins, etc. -
it just chugs for a while because it's single threaded, and will gobble up
memory and swap - but it gets the job done.
... though you can, of course, do
Thank you all for your advice. I use the standard Shell compiled for 64-bit
Windows - so it's single threaded - and the only way to multi-thread is to
run 2 concurrent processes on 2 concurrent Sqlite DBs - assuming there
would be no conflict with the master tables - since they are read only -
On 14 Jul 2012, at 2:31pm, Udi Karni uka...@gmail.com wrote:
(4) Get enough RAM to contain all the data and swap
This may be the best solution to the problem but it's disappointing. You
really shouldn't need to do this. Computers and caching algorithms should be
doing better to help you.
You are right. Disk IO - even with SSD - is typically the bottleneck.
Running multiple threads would only make sense if all the data is in RAM -
:memory: DB or a giant RAM drive (remains to be tested of course and
pricey).
However - I noticed that when fully copying a master table into a :memory:
On 14 Jul 2012, at 3:12pm, Udi Karni uka...@gmail.com wrote:
I know
nothing about writing DB engines - so I don't know whether adding a 2nd
parallel process adds 10K or 10M to the code base.
You've reached the limit of what I know about parallelization. I hope someone
else can chime in.
Windows is really atrociously bad at I/O. Windows has the same basic model of
how to perform I/O as a 6 year-old. Scratch that, the six year old could
probably understand I/O better than whoever wrote/designed the crap in Windows
that passes for I/O routines.
Anyway, make sure that you have
Thanks. More RAM would clearly be helpful - but first I need a bigger
machine that can take it. For some reason - the home line of PC is
typically capped at 16GB or so. I'll Need more of a workstation to go
higher and experiment with the settings you suggested.
On Fri, Jul 13, 2012 at 8:35 PM,
I know the newer versions of Windows are fantastically bloated (and slower
every version), but what are you running that uses more than 16 GB of committed
memory?
Thanks. More RAM would clearly be helpful - but first I need a bigger
machine that can take it. For some reason - the home line
Experimenting with Data Warehouse - which should really be run on a more
mainstream DB. Sqlite was supposed to be just for piloting and testing -
but it's such an incredible little database engine - it's hard to let it go
- so I try big things on it just for kicks - delaying the inevitable.
It
15 matches
Mail list logo
