Re: [sqlite] SQLite for single user data manipulation

2014-04-19 Thread big stone
Hi Nick, >"I just can't find any input for people like me who just want a data manipulation tool for data that's too big to read into RAM?" Here is what I found, that may interest you : >From Wes Mc Kinney : (about medium datas handling) - slides :

Re: [sqlite] SQLite for single user data manipulation

2014-04-16 Thread danap
> Hello 'Liters! > > I'd like to move to SQLite from Postgres, but have two quick questions. > > I'm a social scientist looking to manipulate a large dataset (5 billion > transactions, 700gb). I do not need multiple connections, and will only > ever run one query at a time. > > I started in

Re: [sqlite] SQLite for single user data manipulation

2014-04-16 Thread Stephan Beal
On Wed, Apr 16, 2014 at 6:13 PM, RSmith wrote: > > On 2014/04/16 03:39, Nick Eubank wrote: > >> I started in Postgres, but discovered that in Windows one cannot increase >> some of the per-query RAM memory caps above 2gb (I know -- I would love to >> switch operating systems,

Re: [sqlite] SQLite for single user data manipulation

2014-04-16 Thread RSmith
On 2014/04/16 03:39, Nick Eubank wrote: I started in Postgres, but discovered that in Windows one cannot increase some of the per-query RAM memory caps above 2gb (I know -- I would love to switch operating systems, but it's beyond my control). So I'm thinking of moving to SQLite. Firstly,

Re: [sqlite] SQLite for single user data manipulation

2014-04-16 Thread Simon Slavin
On 16 Apr 2014, at 4:42pm, Dominique Devienne wrote: > On Wed, Apr 16, 2014 at 3:41 PM, Simon Slavin wrote: >> I would recommend that you download the SQLite shell tool and get to know it >> a little. Not only is it useful in creating a SQLite

Re: [sqlite] SQLite for single user data manipulation

2014-04-16 Thread Dominique Devienne
On Wed, Apr 16, 2014 at 3:41 PM, Simon Slavin wrote: > I would recommend that you download the SQLite shell tool and get to know it > a little. Not only is it useful in creating a SQLite database from an SQL > command dump or CSV files, but it allows you to test whether

Re: [sqlite] SQLite for single user data manipulation

2014-04-16 Thread Simon Slavin
On 16 Apr 2014, at 2:39am, Nick Eubank wrote: > -- am I going to have problems using all 16gb of ram on my Windows 8 > machine for data manipulations if I switch to SQLite? Or will SQLite set me > free? If you're talking about per-query memory cap, then that won't figure

[sqlite] SQLite for single user data manipulation

2014-04-16 Thread Nick Eubank
Hello 'Liters! I'd like to move to SQLite from Postgres, but have two quick questions. I'm a social scientist looking to manipulate a large dataset (5 billion transactions, 700gb). I do not need multiple connections, and will only ever run one query at a time. I started in Postgres, but