Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-16 Thread Rowan Worth
On 15 August 2018 at 14:12, Wout Mertens wrote: > On Tue, Aug 14, 2018 at 6:28 PM Rowan Worth wrote: > > > FWIW in the building I work in we have 20-30 users hitting around a dozen > > SQLite DBs 8 hours a day 5 days a week, with all DBs served over nfs. > > > Multiple writers? I presume you

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-15 Thread Rowan Worth
On 15 August 2018 at 13:57, Wout Mertens wrote: > For the interested: > > In NixOS (http://nixos.org), a very interesting Linux distribution, the > entire OS (libraries, binaries, shared files, up to and including > configuration files) is composed out of "build products" that are addressed > by

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-15 Thread Clemens Ladisch
Simon Slavin wrote: > On 14 Aug 2018, at 3:09pm, Clemens Ladisch wrote: >> However, there are other file >> operations that are properly synchronized, e.g., it is not possible for two >> NFS clients to create a directory with the same name. > > You are correct. But there's still a problem with

[sqlite] Safe sqlite over remote filesystem?

2018-08-15 Thread wout.mertens
I know it's a bit of an annoying and thankless task, but visiting the github issues and PRs every week or so (at least twice a month) and making sure they progress, is a great way of gaining adoption… So far I have unfortunately not needed Bedrock myself (it would be overkill), but I just got a

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-15 Thread David Barrett
Ah, to clarify, we're very, very actively developing on Bedrock every single day. In fact, we're about to roll it out to our new 3-datacenter, 6-server cluster of 384-core, 3TB RAM, 100Gbps fiber-connected machines! All of Expensify is powered by it, so it's been battle tested with over a decade

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-15 Thread Wout Mertens
On Tue, Aug 14, 2018 at 4:10 PM Clemens Ladisch wrote: > So as long as all > programs that access the database cooperate, they can switch to a different > locking implementation, such as the unix-dotfile VFS: > > https://www.sqlite.org/vfs.html#standard_unix_vfses > > Note: this makes all

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-15 Thread Wout Mertens
On Tue, Aug 14, 2018 at 6:28 PM Rowan Worth wrote: > FWIW in the building I work in we have 20-30 users hitting around a dozen > SQLite DBs 8 hours a day 5 days a week, with all DBs served over nfs. Multiple writers? I presume you use WAL mode? > Erm, I got a bit carried away. My point is,

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Wout Mertens
On Tue, Aug 14, 2018 at 6:13 PM Alek Paunov wrote: > I am curious, Did you considered adapting writing in your use-case to > Bedrock? AFAIK, you can read from Bedrock instance DBs safely without > further adaptation. > Right, Bedrock is amazing, but in this particular use-case the only

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Keith Medcalf
ite-users- >boun...@mailinglists.sqlite.org] On Behalf Of Dennis Clarke >Sent: Tuesday, 14 August, 2018 13:10 >To: sqlite-users@mailinglists.sqlite.org >Subject: Re: [sqlite] Safe sqlite over remote filesystem? > > >Read all of this repeatedly. Excellent post. > >> >> But if your n

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Dennis Clarke
Read all of this repeatedly. Excellent post. But if your nfs solution is configured not to lie, to honour lock and sync Had to pop up here briefly. I ran into a number of problems with nfs clients of various types wherein the most brutal would be VMware ESXi hosts. Running backend network

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Simon Slavin
On 14 Aug 2018, at 3:09pm, Clemens Ladisch wrote: > However, there are other file > operations that are properly synchronized, e.g., it is not possible for two > NFS clients to create a directory with the same name. You are correct. But there's still a problem with simulating a mutex system.

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Rowan Worth
FWIW in the building I work in we have 20-30 users hitting around a dozen SQLite DBs 8 hours a day 5 days a week, with all DBs served over nfs. The number of corruptions I've seen in the last 5 years which nfs *might* be responsible for is *very low*. The number of corruptions where nfs was

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Alek Paunov
On 2018-08-14 16:07, Wout Mertens wrote: Is there a way to use safely sqlite in this situation, perhaps by using extra lock files or some other additional mechanism? One solution I can think of involves sending all writes through a single master, via files describing changes and lots of

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Clemens Ladisch
Wout Mertens wrote: > I know that multi-writer sqlite and NFS don't play well with each other. > > However, I wonder if some constraints could be added that would make this > situation safe. NFS locking implementations tend to be bad. However, there are other file operations that are properly

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Stephen Chrzanowski
Have you read through this document: https://www.sqlite.org/howtocorrupt.html On Tue, Aug 14, 2018 at 9:46 AM, Wout Mertens wrote: > In the meantime I found this discussion from 2011 about sqlite on CIFS: > http://sqlite.1065341.n5.nabble.com/How-to-make-SQLite- >

Re: [sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Wout Mertens
In the meantime I found this discussion from 2011 about sqlite on CIFS: http://sqlite.1065341.n5.nabble.com/How-to-make-SQLite-run-safely-on-CIFS-mounted-file-system-tt37415.html#none Basically using any networked filesystem as a backing store for sqlite is madness? I imagine not much about that

[sqlite] Safe sqlite over remote filesystem?

2018-08-14 Thread Wout Mertens
Idle musing again, I'm pretty bad at dropping thoughts that are not immediately applicable to me, sorry. I know that multi-writer sqlite and NFS don't play well with each other. However, I wonder if some constraints could be added that would make this situation safe. My problem space is that of