That's clever. I'm thinking how I'll go about this .. essentially the filename is devised by time splits. I could do what you're saying but then I can't move the file. Basically, the good part is that our warehousing department (which is what this is used for) can look at order inventories every 5 minutes (that's the current split). This way I'll have to somehow stop the writes for a few seconds, copy the file into a new file and then start writing again.
Not sure if this will work out, but I'm going to keep in my head. On Sat, May 3, 2014 at 6:06 AM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 3 May 2014, at 1:59pm, Hayden Livingston <halivings...@gmail.com> > wrote: > > > My workload is quite straightforward, I have a multi-threaded application > > that logs to this file but from within any single thread at any given > time. > > So from SQLite's perspective only one person will be writing to this > > database and nobody will ever read from it. > > > > The reading happens after the entire database is written to. > > > > Is this a use case for SQLite? It seems like the benefit I get from > SQLite > > is that I can do SQL on my previously textual data and optionally add > > indexes. > > The benefit you'd get from SQL is the ability to look up a specific entry > easily rather than having to look through an entire file to find the entry > you want. > > > HOWEVER, this is my catch -- I sometimes have the need to query two > > databases at the same time. In fact, sometimes I might need to query > 100's > > of these small databases. Would you just recommend querying the 100's in > > parallel and then joining them in memory or something? > > Had you considered merging them all into one big database ? You currently > have some sort of variable that you use to devise a filename. Turn that > into a column name which is part of a UNIQUE key. Then, when you want to > run a query all your data is in one big database and you can query every > 'database' at the same time and return the answers in a single list. > > You should be able to get enough data together to test relatively easily. > Write an importer that takes your existing text files and adds their data > into one big SQLite database. Then you can try numerous SELECT queries > using the SQLite shell tool (without having to write different software > each time) and see what indexes would be useful and how fast they'd give > you results. > > Simon. > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
