On Wed, Aug 1, 2012 at 2:43 PM, Black, Michael (IS)
wrote:
> You may be interested in this article:
> http://www.drdobbs.com/parallel/multithreaded-file-io/220300055?pgno=2
>
> Mutli-threaded reading of multiple files (which is basically what you're
> talking about by splitting a file in half) is
On 1 Aug 2012, at 8:25pm, Udi Karni wrote:
> - only 2 processes
> - only for simple full scans where the block range can be divided in two
> - only when there is no ORDER/GROUP BY where sub results from the 2 threads
> have to be combined
Premature optimization ? SQLite, just by itself with de
on Systems
From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on
behalf of Udi Karni [uka...@gmail.com]
Sent: Wednesday, August 01, 2012 2:25 PM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] Windows I/O (was: Initial read speed greater than
subsequent
You are right. True Parallel Query can get very complicated. I was hoping
for something very limited for starters - for example -
- only 2 processes
- only for simple full scans where the block range can be divided in two
- only when there is no ORDER/GROUP BY where sub results from the 2 threads
On Sat, Jul 14, 2012 at 03:17:07PM +0100, Simon Slavin wrote:
>
> On 14 Jul 2012, at 3:12pm, Udi Karni wrote:
>
> > I know
> > nothing about writing DB engines - so I don't know whether adding a 2nd
> > parallel process adds 10K or 10M to the code base.
>
> You've reached the limit of what I kn
On 14 Jul 2012, at 3:12pm, Udi Karni wrote:
> I know
> nothing about writing DB engines - so I don't know whether adding a 2nd
> parallel process adds 10K or 10M to the code base.
You've reached the limit of what I know about parallelization. I hope someone
else can chime in.
Simon.
You are right. Disk IO - even with SSD - is typically the bottleneck.
Running multiple threads would only make sense if all the data is in RAM -
:memory: DB or a giant RAM drive (remains to be tested of course and
pricey).
However - I noticed that when fully copying a master table into a :memory:
On 14 Jul 2012, at 2:31pm, Udi Karni wrote:
> (4) Get enough RAM to contain all the data and swap
This may be the best solution to the problem but it's disappointing. You
really shouldn't need to do this. Computers and caching algorithms should be
doing better to help you.
> In light of th
Thank you all for your advice. I use the standard Shell compiled for 64-bit
Windows - so it's single threaded - and the only way to multi-thread is to
run 2 concurrent processes on 2 concurrent Sqlite DBs - assuming there
would be no conflict with the master tables - since they are read only -
I'll
On 14 Jul 2012, at 5:40am, Udi Karni wrote:
> It very easily handles billion row/100GB tables - multi-table joins, etc. -
> it just chugs for a while because it's single threaded, and will gobble up
> memory and swap - but it gets the job done.
... though you can, of course, do your SQLite call
On Fri, 13 Jul 2012 21:40:19 -0700, Udi Karni wrote:
>It very easily handles billion row/100GB tables - multi-table joins, etc. -
>it just chugs for a while because it's single threaded, and will gobble up
>memory and swap - but it gets the job done.
If SQLite memory usage causes your system to
Experimenting with Data Warehouse - which should really be run on a more
"mainstream" DB. Sqlite was supposed to be just for piloting and testing -
but it's such an incredible little database engine - it's hard to let it go
- so I try big things on it just for kicks - delaying the inevitable.
It v
I know the newer versions of Windows are fantastically bloated (and slower
every version), but what are you running that uses more than 16 GB of committed
memory?
> Thanks. More RAM would clearly be helpful - but first I need a bigger
> machine that can take it. For some reason - the "home" lin
Thanks. More RAM would clearly be helpful - but first I need a bigger
machine that can take it. For some reason - the "home" line of PC is
typically capped at 16GB or so. I'll Need more of a workstation to go
higher and experiment with the settings you suggested.
On Fri, Jul 13, 2012 at 8:35 PM, K
Windows is really atrociously bad at I/O. Windows has the same basic model of
how to perform I/O as a 6 year-old. Scratch that, the six year old could
probably understand I/O better than whoever wrote/designed the crap in Windows
that passes for I/O routines.
Anyway, make sure that you have
15 matches
Mail list logo