[sqlite] Best way to temporarily store data before processing

2015-04-17 Thread Jonathan Moules
-bounces at mailinglists.sqlite.org [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Petite Abeille Sent: Wednesday, April 15, 2015 6:24 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Best way to temporarily store data before processing > On Apr 14, 2015, at 11

[sqlite] Best way to temporarily store data before processing

2015-04-15 Thread Petite Abeille
> On Apr 14, 2015, at 11:40 AM, Jonathan Moules > wrote: > > Options that have come to mind (probably missed a lot): I personally use temp tables, e.g. 'create temporary table if not exists foo?, coupled with 'pragma temp_store = memory?, and drop/create them as necessary, e.g. 'drop table

[sqlite] Best way to temporarily store data before processing

2015-04-15 Thread Scott Hess
On Tue, Apr 14, 2015 at 1:10 PM, David Cotter wrote: > does this awesome sounding temp table then merge instantly when you commit it > even if it?s huge? Since I have no idea what "merge" means in this context, it's hard to answer your question. -scott

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread David Cotter
does this awesome sounding temp table then merge instantly when you commit it even if it?s huge? > On Apr 14, 2015, at 1:01 PM, Scott Hess wrote: > > On Tue, Apr 14, 2015 at 9:37 AM, Jim Callahan > wrote: >> My recollection is that SQLite has a "temp" or "tmp" namespace available >> for

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread f-prot
Database' Subject: Re: [sqlite] Best way to temporarily store data before processing Thanks for the thoughts, useful to know. In relation to your question Joseph, historically the scripts were written to only put the raw data into the database and then the Views read it, but that was too slow. So now

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Jonathan Moules
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Joseph T. Sent: Tuesday, April 14, 2015 12:17 PM To: General Discussion of SQLite Database Subject: Re: [sqlite] Best way to temporarily store data before processing Mr. Moules, why not skip the raw tables entirely? Or failing that a separate

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Scott Hess
On Tue, Apr 14, 2015 at 9:37 AM, Jim Callahan wrote: > My recollection is that SQLite has a "temp" or "tmp" namespace available > for intermediate tables -- it was on my todo list, but I never got around > to exploring that option. CREATE TEMP TABLE ... works very well for things like this. If

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Jim Callahan
?There are too many unknowns to give a definitive answer to your question. Any solution you chose is going to create some stress somewhere (in your "Prepared Tables" database, RAM, the file system and so forth. Just from my own limited experience. It is very easy while hacking to CREATE TABLE and

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Hick Gunter
For native SQLite tables, DROP TABLE is much faster than DELETE FROM. -Urspr?ngliche Nachricht- Von: Jonathan Moules [mailto:J.Moules at hrwallingford.com] Gesendet: Dienstag, 14. April 2015 11:40 An: 'sqlite-users at mailinglists.sqlite.org' Betreff: [sqlite] Best way to temporarily

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Jonathan Moules
Hi List, I'm wondering if anyone can offer me a "best practice" way of doing this. I'm doing some log analysis using Python/SQLite. Python parses a log file and splits the raw data from each line in the log into one of about 40 tables in an SQLite database (I'll call them Raw Tables). Once a

[sqlite] Best way to temporarily store data before processing

2015-04-14 Thread Joseph T.
Mr. Moules, why not skip the raw tables entirely? Or failing that a separate in memory db would probably be the best and quickest option. Sent from my Samsung Epic? 4G TouchJonathan Moules wrote:Hi List, I'm wondering if anyone can offer me a "best practice" way of doing this. I'm doing