As any Filesystem glusterfs also has a atime,mtime and ctime. But we are trying to optimize the search of file that are "HOT" or "COLD" using sqlite3, As a filesystem walk would kill the scanners(which are doing data maintenance)
~Joe ----- Original Message ----- From: "Keith Medcalf" <kmedc...@dessus.com> To: "General Discussion of SQLite Database" <sqlite-users@sqlite.org> Sent: Saturday, November 22, 2014 9:02:30 PM Subject: Re: [sqlite] Using Sqlite3 as a Change Time Recording Data Store in Glusterfs Your Glusterfs does not store modification times? Have you considered adding the modification time attribute to the inode directly? --- Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. Sometimes theory and practice are combined: nothing works and no one knows why. >-----Original Message----- >From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users- >boun...@sqlite.org] On Behalf Of Joseph Fernandes >Sent: Friday, 21 November, 2014 12:02 >To: sqlite-users@sqlite.org >Subject: [sqlite] Using Sqlite3 as a Change Time Recording Data Store in >Glusterfs > >Hi There, > >1) We are trying to use sqlite3 in Glusterfs as a Change Time Recording >Data Store >i.e using sqlite3 db to record any modification attempts that happens on >the glusterfs >per file inode, So that it can be used to indicate the hotness of the >file in the cluster. > >2) We have developed a prototype that inserts/updates inode records in >sqlite3 db(one instance of the >db file per glusterfs storage unit or as we call it a brick) to record >the modification >that are happening on the inode. These insert/updates are in the IO path >of the file i.e any data or >metadata change time will be record inline, w.r.t the IO, in the DB. > >3) We only save the modification time on the to the db, so that >we can easily query for "What files have change during a specific period >of time" or >"What files have not change during a specific period of time". The query >is done by a scanner >that runner on each storage unit or brick on a regular interval and >chooses HOT or COLD files as suggested >by the DB for data maintenance operations. > >4) Therefore, we are looking at a datastore that can give us a very quick >write(almost zero latency, >as the recording is done inline w.r.t file IO) and that as good data >querying >facilities(Slight latency in the read is fine but the fresh of that >record data should be spot on). > >5) Please find our DB setting, >Journal mode : WAL >SYNC MODE : NORMAL >Cache Size : 1000 - 4096 Pages (Default Page Size) >AutoCheck-Pointing: 1000 - 1000000 Pages > >We wanted to known the following >1) How could we improve the performance on the write side so that we have >minimal latency? >2) Will ther be any write performance hit when the number of records in >the DB increase? > >Thanks in advance > >~Joe >_______________________________________________ >sqlite-users mailing list >sqlite-users@sqlite.org >http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
