Yes, but each input record also contains a timestamp that can be used to identify the relevant table.
On 9 August 2011 14:43, Igor Tandetnik <itandet...@mvps.org> wrote: > Jaco Breitenbach <jjbreitenb...@gmail.com> wrote: > > I am building an application that filters out duplicate input data by > > generating an MD5 hash of each input, and implicitly comparing that > against > > a set of keys already stored in the SQLite database by doing an insert > into > > a unique-indexed table. If the insert fails, a duplicate is assumed, > > otherwise the new unique key is stored, and the input processed. > > > > The problem that I'm facing, is that I would ultimately need to process > > 1,000,000,000 records a day, with history to be kept for up to 128 days. > I > > am currently creating a new data file per day, with hourly tables. > > Doesn't that defeat the point? Wouldn't that only guarantee uniqueness > within the last hour? > -- > Igor Tandetnik > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
