On Friday, 27 September, 2019 17:00, Roman Fleysher wrote: >I am using SQLite over GPFS distributed file system. I was told it >honestly implements file locking. I never experienced corruption. But it >is slow in the sense that when many jobs from many compute nodes try to >access the same database things slow down considerably.
Yes. This is how "correct" locking would be implemented. If would be so slow as to be virtually unusable. >I suspect, from the point of view of file system, there is lots of >pressure to develop fast grabbing of a lock and slow release. I think >this is because the key to fast network file system in general is making >it as independent as possible, thus distributed. No, it is because the filesystem server must be queried on every lock operation, and before that can return to the caller, each and every client must be contacted and "made consistent". This means that as more clients "open" the same file, the overhead of each operation becomes orders of magnitude greater. In order to "get around" this issue most network filesystems use so-called "optimistic" locking protocols. This basically means that the software prays and goes ahead anyway without ensuring consistency amongst clients and simply corrupts the file in cases where the prayer does not work. For the normal case this usually works adequately. For in-place record updates, however, it usually does not. >To try to speed up locking I combined SQLite with FLoM, distributed file >lock manager. It is client/server application. I don't think FLoM does what you think it does. >From experience, it seems that because SQLite still requests file locks, >the performance increase is not that big. I wonder if there is a way to >disable SQLite's internal file locking mechanism. I know this seems >strange to ask. But FLoM should be able to do it faster over many compute >nodes. >Or, perhaps the right way is for me to combine SQLIte with simple >client/server code to create light mySQL, mySQLite? Yes, this is the correct way to do it. If multi-access to a single shared file worked properly then there would never have been any need to invent client/server database architecture. The fact that is was invented at the same time as shared filesystems indicates that the KISS solution (a shared filesystem) was not robust. _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
