Gerry; I trashed the email I was going to send. You had the same line of thought as me in regards to chopping the file on a per-day basis, but, what made me trash it was any auto-numbered PKs that would be a hassle in new files, unless that information was put into the new DB upon creation.
I agree that when files get large, a revamp of how the data needs to be stored has to be examined when the new consideration of how to backup the sucker becomes more of a logistical nightmare. With MySQL, data replication like that is somewhat a breeze. With SQLite, the convenience and portability applications granted by SQLite die when files become too large to deal with for backup purposes. On Fri, May 6, 2016 at 10:40 AM, Gerry Snyder <mesmerizerfan at gmail.com> wrote: On 5/6/2016 5:32 AM, Stephan Buchert wrote: > >> We are using Sqlite for data from satellite Earth observations. It >> works very well. Thanks to everybody contributing to Sqlite, uppermost >> Dr. Hipp. >> >> The largest database file has now grown to about 180 GB. >> > > One feature of SQLite -- the whole database in one file -- is normally an > advantage but becomes less so when the file is huge. > > > I need to have >> copies of the files at at least two different places. The databases are >> updated regularly as new data from the satellites become available. >> > > Others have suggested keeping track of changes using a logging file. That > allows all the SELECT statements to remain unchanged, while complicating > the input side of things. > > Another approach could be to have new observations go into a separate file > (one day's worth, or whatever makes sense). This much smaller file could be > shipped to the other site(s) and then merged into each copy of the main > database. Almost no changes to the input logic, but every SELECT would have > to use a JOIN. No idea how painful process that would be. > Gerry > > _______________________________________________ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >
