Check out the alternate VFS implementations (linked earlier), and also the available parameters that you can pass to the sqlite3_open_v2 interface when using URI mode, such as perhaps using the standard unix vfs but passing the nolock option to disable all file locking operations (but not the change detection). Since you are serializing access at the application level, perhaps that might be useful for you to try.
https://sqlite.org/c3ref/open.html https://sqlite.org/uri.html#coreqp >-----Original Message----- >From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> On >Behalf Of Roman Fleysher >Sent: Friday, 27 September, 2019 21:49 >To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> >Subject: Re: [sqlite] disable file locking mechanism over the network > >I agree that overhead might grow exponentially with number of nodes >trying to access the database. But then I do not understand why FLoM can >not provide solution. It can enforce sequential access thus reducing the >overhead. > >And yes, I was trying the KISS approach: rely on GPFS to do the >synchronization instead of me programming client/server. > >Thank you, > >Roman > >________________________________________ >From: sqlite-users [sqlite-users-boun...@mailinglists.sqlite.org] on >behalf of Keith Medcalf [kmedc...@dessus.com] >Sent: Friday, September 27, 2019 11:12 PM >To: SQLite mailing list >Subject: Re: [sqlite] disable file locking mechanism over the network > >On Friday, 27 September, 2019 17:00, Roman Fleysher wrote: > >>I am using SQLite over GPFS distributed file system. I was told it >>honestly implements file locking. I never experienced corruption. But it >>is slow in the sense that when many jobs from many compute nodes try to >>access the same database things slow down considerably. > >Yes. This is how "correct" locking would be implemented. If would be so >slow as to be virtually unusable. > >>I suspect, from the point of view of file system, there is lots of >>pressure to develop fast grabbing of a lock and slow release. I think >>this is because the key to fast network file system in general is making >>it as independent as possible, thus distributed. > >No, it is because the filesystem server must be queried on every lock >operation, and before that can return to the caller, each and every >client must be contacted and "made consistent". This means that as more >clients "open" the same file, the overhead of each operation becomes >orders of magnitude greater. In order to "get around" this issue most >network filesystems use so-called "optimistic" locking protocols. This >basically means that the software prays and goes ahead anyway without >ensuring consistency amongst clients and simply corrupts the file in >cases where the prayer does not work. For the normal case this usually >workthe link you sent in item 2,s adequately. For in-place record >updates, however, it usually does not. > >>To try to speed up locking I combined SQLite with FLoM, distributed file >>lock manager. It is client/server application.the link you sent in item >2, > >I don't think FLoM does what you think it does. > >>From experience, it seems that because SQLite still requests file locks, >>the performance increase is not that big. I wonder if there is a way to >>disable SQLite's internal file locking mechanism. I know this seems >>strange to ask. But FLoM should be able to do it faster over many >compute >>nodes. > >>Or, perhaps the right way is for me to combine SQLIte with simple >>client/server code to create light mySQL, mySQLite? > >Yes, this is the correct way to do it. If multi-access to a single >shared file worked properly then there would never have been any need to >invent client/server database architecture. The fact that is was >invented at the same time as shared filesystems indicates that the KISS >solution (a shared filesystem) was not robust. > > > >_______________________________________________ >sqlite-users mailing list >sqlite-users@mailinglists.sqlite.org >https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailingl >ists.sqlite.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsqlite- >users&data=02%7C01%7Croman.fleysher%40einstein.yu.edu%7Ca315faff6fdc4 >5dfc70408d743c1a8b0%7C04c70eb48f2648079934e02e89266ad0%7C1%7C1%7C63705237 >1361975178&sdata=dhrR%2FbKGqPEmk6sLv3JGouW%2BdO%2BaAK2WyP97LLYQlCU%3D >&reserved=0 >_______________________________________________ >sqlite-users mailing list >sqlite-users@mailinglists.sqlite.org >http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
