Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 10:39 AM, Richard Hipp wrote: > I say that a filesystem is an eventually-consistent key/value database. Yes! > The keys are the filenames and the values are all big BLOBs, specifically > the file content. Filesystems also have a hierarchical keyspace,

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Fri, Sep 12, 2014 at 7:21 PM, Richard Hipp wrote: > On Fri, Sep 12, 2014 at 8:07 PM, Simon Slavin wrote: >> one thing that annoys me about SQLite is that it needs to make a >> journal file which isn't part of the database file. Why ? Why can't it >>

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Fri, Sep 12, 2014 at 6:47 PM, James K. Lowden wrote: > On Fri, 12 Sep 2014 19:38:53 +0100 > Simon Slavin wrote: > >> I don't think it can be done by trying to build it on top of an >> existing file system. I think we need a file system (volume

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, 13 Sep 2014 15:43:00 -0400 Richard Hipp wrote: > There were often restrictions on the permitted values for block > sizes. And you couldn't ask the operating system to tell you whether > a file was text or binary or sequential or random-access or what its > block-size

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 2:53 PM, Howard Chu wrote: > Scott Robison wrote: > >> On Sat, Sep 13, 2014 at 2:24 PM, Howard Chu wrote: >> >> Scott Robison wrote: >>> >>> A couple of academic thoughts. >>> 1. If one wanted to embed the journal within the

Re: [sqlite] presentation about ordering and atomicity of filesystems

Scott Robison wrote: On Sat, Sep 13, 2014 at 2:24 PM, Howard Chu wrote: Scott Robison wrote: A couple of academic thoughts. 1. If one wanted to embed the journal within the database, would it be adequate to reserve a specific page as the "root" page of the journal, then

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 2:24 PM, Howard Chu wrote: > Scott Robison wrote: > >> A couple of academic thoughts. >> > >> 1. If one wanted to embed the journal within the database, would it be >> adequate to reserve a specific page as the "root" page of the journal, >> then >>

Re: [sqlite] presentation about ordering and atomicity of filesystems

Scott Robison wrote: On Fri, Sep 12, 2014 at 6:21 PM, Richard Hipp wrote: On Fri, Sep 12, 2014 at 8:07 PM, Simon Slavin wrote: one thing that annoys me about SQLite is that it needs to make a journal file which isn't part of the database file. Why

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 9:16pm, Howard Chu wrote: > Scott Robison wrote: > >> At the >> very least (and I suspect more) Commodore 8-bit DOS (which was embedded >> within a smart drive with its very own dedicated CPU & RAM) supported >> (essentially) sequential byte stream files (no

Re: [sqlite] presentation about ordering and atomicity of filesystems

Scott Robison wrote: On Sat, Sep 13, 2014 at 1:43 PM, Richard Hipp wrote: Decades ago, files came in all kinds of varieties and permutations. Details varied from one OS to the next. But it was common to have a distinction between text files and binary files (with different

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 1:43 PM, Richard Hipp wrote: > Decades ago, files came in all kinds of varieties and permutations. > Details varied from one OS to the next. But it was common to have a > distinction between text files and binary files (with different APIs for >

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 8:43pm, Richard Hipp wrote: > On Sat, Sep 13, 2014 at 2:46 PM, Simon Slavin wrote: > >> I would dispute that. [snip] > > Roger is correct. > > [description] Thanks for the correction. Apologies to Roger for doubting him. > You

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 2:46 PM, Simon Slavin wrote: > > > > Before Unix came along it was quite common for files to be structured, > > managed by the operating system and to be record based with file apis > > working that way. Unix turned files (and similar) into

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Fri, Sep 12, 2014 at 6:21 PM, Richard Hipp wrote: > On Fri, Sep 12, 2014 at 8:07 PM, Simon Slavin > wrote: > > > > one thing that annoys me about SQLite is that it needs to make a > > journal file which isn't part of the database file. Why ? Why

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 7:01pm, Luuk wrote: > On 13-9-2014 19:45, Simon Slavin wrote: > >> What I want is the entire volume as a database. It would be something like >> >> Key Example value >> --- - >>

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 7:05pm, Roger Binns wrote: > Ever hear of Windows and Transactional NTFS :-) > > http://msdn.microsoft.com/en-us/magazine/cc163388.aspx > > It turns out that adding transactions indiscriminately doesn't magically > make everything better and lots of

Re: [sqlite] presentation about ordering and atomicity of filesystems

Scott Robison wrote: On Sat, Sep 13, 2014 at 9:39 AM, Richard Hipp wrote: I say that a filesystem is an eventually-consistent key/value database. The keys are the filenames and the values are all big BLOBs, specifically the file content. Filesystems also have a hierarchical

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 9:39 AM, Richard Hipp wrote: > I say that a filesystem is an eventually-consistent key/value database. > The keys are the filenames and the values are all big BLOBs, specifically > the file content. Filesystems also have a hierarchical keyspace, which is

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 12/09/14 17:07, Simon Slavin wrote: > Programmers don't expect file services to support transactions because file > services have never supported transactions. Ever hear of Windows and Transactional NTFS :-) http://msdn.microsoft.com/en-us/magazine/cc163388.aspx It turns out that adding

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13-9-2014 19:45, Simon Slavin wrote: On 13 Sep 2014, at 6:16pm, Tim Streater wrote: On 13 Sep 2014 at 16:39, Richard Hipp wrote: I say that a filesystem is an eventually-consistent key/value database. The keys are the filenames and the values are

Re: [sqlite] presentation about ordering and atomicity of filesystems

Simon Slavin wrote: That's not what I meant. That's the file as a database. What I want is the entire volume as a database. That's exactly what I pointed you to before. The thesis is pretty enlightening too. http://www.fsl.cs.sunysb.edu/docs/kbdbfs-msthesis/ -- -- Howard Chu CTO,

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 6:16pm, Tim Streater wrote: > On 13 Sep 2014 at 16:39, Richard Hipp wrote: > >> I say that a filesystem is an eventually-consistent key/value database. >> The keys are the filenames and the values are all big BLOBs, specifically >>

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014 at 16:39, Richard Hipp wrote: > I say that a filesystem is an eventually-consistent key/value database. > The keys are the filenames and the values are all big BLOBs, specifically > the file content. Filesystems also have a hierarchical keyspace, which is > an

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, Sep 13, 2014 at 11:01 AM, James K. Lowden wrote: > I used "database" because that was his word. Whether or not a > filesystem is a database depends on the definition. I would say that > the files are data, but the filesystem is not a DBMS, partly for > reasons

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Sat, 13 Sep 2014 01:07:59 +0100 Simon Slavin wrote: > > Implement a block-transaction store on the device > > itself: no inodes, no directories, just writeable blocks managed in > > transactions. Build your DBMS on that. > > That would be ... erm ... perhaps a new

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 1:21am, Richard Hipp wrote: > On Fri, Sep 12, 2014 at 8:07 PM, Simon Slavin wrote: > >> one thing that annoys me about SQLite is that it needs to make a >> journal file which isn't part of the database file. Why ? Why can't it >>

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Fri, Sep 12, 2014 at 8:07 PM, Simon Slavin wrote: > > one thing that annoys me about SQLite is that it needs to make a > journal file which isn't part of the database file. Why ? Why can't it > just write the journal to the database file it already has open ? This >

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 12:47am, James K. Lowden wrote: > I asked your question: why not add transactions to FFS? > > His answer: that's the province of a database. A file system is a database. It accepts data from a user. It stores it away for later retrieval. It

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 13 Sep 2014, at 12:47am, James K. Lowden wrote: > The filesystem guys I've talked to all consider transactions to be a > high-level construct hardly ever needed by most applications. They're > interested in raw speed and a consistent file structure (metadata) >

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Fri, 12 Sep 2014 19:38:53 +0100 Simon Slavin wrote: > I don't think it can be done by trying to build it on top of an > existing file system. I think we need a file system (volume format, > drivers, etc.) built from the ground up with > atomicity/ACID/transactions in

Re: [sqlite] presentation about ordering and atomicity of filesystems

hi! Thanks for the note, and the interest. I'm cc'ing Thanu into this discussion -- he and a few other grad students did all the heavy lifting and can share the details of what they've found. As I recall, he has already spoken with some of the SQLite gang a bit about the purported "problem"

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Fri, Sep 12, 2014 at 1:18 PM, Howard Chu wrote: > Nico Williams wrote: >> On ZFS datasets with sync disabled fsync() functions as osync(), as a >> write >> barrier without durability and without the associated penalty. The >> obvious >> problem is that really do need osync()

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 12 Sep 2014, at 7:08pm, Nico Williams wrote: > On ZFS datasets with sync disabled fsync() functions as osync(), as a write > barrier without durability and without the associated penalty. The obvious > problem is that really do need osync() and fsync(); just one or

Re: [sqlite] presentation about ordering and atomicity of filesystems

Nico Williams wrote: On ZFS datasets with sync disabled fsync() functions as osync(), as a write barrier without durability and without the associated penalty. The obvious problem is that really do need osync() and fsync(); just one or the other is not a reasonable compromise. Write barriers

Re: [sqlite] presentation about ordering and atomicity of filesystems

On ZFS datasets with sync disabled fsync() functions as osync(), as a write barrier without durability and without the associated penalty. The obvious problem is that really do need osync() and fsync(); just one or the other is not a reasonable compromise. Nico --

Re: [sqlite] presentation about ordering and atomicity of filesystems

Richard Hipp wrote: On Thu, Sep 11, 2014 at 5:49 PM, Kees Nuyt wrote: Hi all, Today I bumped into a presentation about ordering and atomicity of filesystems that might interest you. https://www.youtube.com/watch?v=YvchhB1-Aws The Application/Storage Interface: After All

Re: [sqlite] presentation about ordering and atomicity of filesystems

Simon Slavin wrote: On 12 Sep 2014, at 3:18am, Scott Robison wrote: That was an interesting 90 minutes. Indeed. Thanks to Kees for posting it. Though I was surprised he didn't mention the term 'ACID' explicitly. I'm still of the opinion that we need an actual

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Thu, Sep 11, 2014 at 5:49 PM, Kees Nuyt wrote: > > Hi all, > > Today I bumped into a presentation about ordering and atomicity > of filesystems that might interest you. > > https://www.youtube.com/watch?v=YvchhB1-Aws > > The Application/Storage Interface: After All These

Re: [sqlite] presentation about ordering and atomicity of filesystems

On 12 Sep 2014, at 3:18am, Scott Robison wrote: > That was an interesting 90 minutes. Indeed. Thanks to Kees for posting it. Though I was surprised he didn't mention the term 'ACID' explicitly. I'm still of the opinion that we need an actual transactional file

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Thu, Sep 11, 2014 at 4:09 PM, Kees Nuyt wrote: > On Thu, 11 Sep 2014 23:49:22 +0200, Kees Nuyt > wrote: > > > Today I bumped into a presentation about ordering and atomicity > > of filesystems that might interest you. > > > >

Re: [sqlite] presentation about ordering and atomicity of filesystems

On Thu, 11 Sep 2014 23:49:22 +0200, Kees Nuyt wrote: > Today I bumped into a presentation about ordering and atomicity > of filesystems that might interest you. > > https://www.youtube.com/watch?v=YvchhB1-Aws Compliments for sqlite at 43:20 .. 43:59