Thanks Eduardo. Answers inline JOE>>
----- Original Message ----- From: "Eduardo Morras" <emorr...@yahoo.es> To: sqlite-users@sqlite.org Sent: Saturday, November 22, 2014 9:20:46 PM Subject: Re: [sqlite] Using Sqlite3 as a Change Time Recording Data Store in Glusterfs On Fri, 21 Nov 2014 14:01:39 -0500 (EST) Joseph Fernandes <josfe...@redhat.com> wrote: > Hi There, > > 1) We are trying to use sqlite3 in Glusterfs as a Change Time > Recording Data Store i.e using sqlite3 db to record any modification > attempts that happens on the glusterfs per file inode, So that it can > be used to indicate the hotness of the file in the cluster. > > 2) We have developed a prototype that inserts/updates inode records > in sqlite3 db(one instance of the db file per glusterfs storage unit > or as we call it a brick) to record the modification that are > happening on the inode. These insert/updates are in the IO path of > the file i.e any data or metadata change time will be record inline, > w.r.t the IO, in the DB. > > 3) We only save the modification time on the to the db, so that > we can easily query for "What files have change during a specific > period of time" or "What files have not change during a specific > period of time". The query is done by a scanner that runner on each > storage unit or brick on a regular interval and chooses HOT or COLD > files as suggested by the DB for data maintenance operations. > > 4) Therefore, we are looking at a datastore that can give us a very > quick write(almost zero latency, as the recording is done inline > w.r.t file IO) and that as good data querying facilities(Slight > latency in the read is fine but the fresh of that record data should > be spot on). > > 5) Please find our DB setting, > Journal mode : WAL > SYNC MODE : NORMAL > Cache Size : 1000 - 4096 Pages (Default Page Size) > AutoCheck-Pointing: 1000 - 1000000 Pages > > We wanted to known the following > 1) How could we improve the performance on the write side so that we > have minimal latency? I'll wait for the ota extension Mr. Hipps is developing. It will (If I Understood Correctly) reorder the commit operations to get faster IO throughput. Some tricks: a)If you use Sqlite in single thread, compile it without thread support. JOE>> We are running it in multithread mode, as the database will be fed by multiple File IO thread. b)Adjust the page size to fit in a hard disk cluster size (8KB IIRC) or the multiply of it that allows integer number of rows fits in. c)Disable autovacuum JOE>> Sure will try this out in my tests > 2) Will ther be any write performance hit when the number of records > in the DB increase? > > Thanks in advance > > ~Joe --- --- Eduardo Morras <emorr...@yahoo.es> _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
