Re: Polished FileSystem API proposal
If the backend implementation used something like git's data store then duplicate data would automatically be stored only once without any security implications. The keys are the literal sha1 of the values. If two websites had the same file tree containing the same files, it would be the same tree object in the storage. But only sites who have a reference to the hash would have access to it. Also I like the level of fs support that git's filesystem has. There are trees, files, executable files, and symlinks. (there are also gitlinks used for submodules, but let's ignore those for now) On Tue, Nov 5, 2013 at 12:19 PM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Oct 31, 2013 at 2:12 AM, Brian Stell bst...@google.com wrote: There could be *dozens* of copies of exactly the same a Javascript library, shared CSS, or web font in the FileSystem. Check out the cache part of https://github.com/slightlyoff/ServiceWorker/ Combined with a smart implementation that will do exactly what you want. And avoid all the issues of an actual cross-origin file system API. -- http://annevankesteren.nl/
Re: Polished FileSystem API proposal
On Tue, Nov 5, 2013 at 11:45 AM, Tim Caswell t...@creationix.com wrote: If the backend implementation used something like git's data store then duplicate data would automatically be stored only once without any security implications. The keys are the literal sha1 of the values. If two websites had the same file tree containing the same files, it would be the same tree object in the storage. But only sites who have a reference to the hash would have access to it. Also I like the level of fs support that git's filesystem has. There are trees, files, executable files, and symlinks. (there are also gitlinks used for submodules, but let's ignore those for now) Sounds like IndexedDB is a better fit than a filesystem for this use case. Note that the use case for the filesystem API isn't storing files, IDB is perfectly capable of doing that. The use case for the filesystem API is to satisfy people that want a true filesystem with directories etc so that they can: * Sync to a server-side file system. For example when doing web development and want to deploy a website. * Use hierarchical filesystem: URLs. * Support in-place editing of large files. * Because filesystems are familiar. A simple key-value storage, where the values happen to be files, doesn't need a filesystem API. / Jonas
Re: Polished FileSystem API proposal
On Nov 6, 2013 3:50 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Nov 5, 2013 at 11:45 AM, Tim Caswell t...@creationix.com wrote: If the backend implementation used something like git's data store then duplicate data would automatically be stored only once without any security implications. The keys are the literal sha1 of the values. If two websites had the same file tree containing the same files, it would be the same tree object in the storage. But only sites who have a reference to the hash would have access to it. Also I like the level of fs support that git's filesystem has. There are trees, files, executable files, and symlinks. (there are also gitlinks used for submodules, but let's ignore those for now) Sounds like IndexedDB is a better fit than a filesystem for this use case. I'm not saying that I want a filesystem so I can implement git. I'm saying that git is a great way to implement a filesystem and that git's abilities are a great baseline for web filesystems. Note that the use case for the filesystem API isn't storing files, IDB is perfectly capable of doing that. The use case for the filesystem API is to satisfy people that want a true filesystem with directories etc so that they can: * Sync to a server-side file system. For example when doing web development and want to deploy a website. * Use hierarchical filesystem: URLs. * Support in-place editing of large files. * Because filesystems are familiar. Yes. I completely agree. Personally in my projects, I wish there was to create custom filesystems backed by JavaScript code that are exported to hierarchical URLs. It would work somewhat like server-side web development where a js function handles the request for a file. This is basically what fuse does for Linux. I would love to be able to create a custom union filesystem that has layers. One would be in-memory and another would be backed by a local bit database. It would all be implemented in pure user space js. A simple key-value storage, where the values happen to be files, doesn't need a filesystem API. Speaking of, I currently have a git implementation that uses IndexedDB as the data store. I don't call IDB simple by any means. I don't need transactions or indexes. The leveldb interface is much closer. To what I need to implement git. / Jonas So in summary: - The filesystem interface could export symlinks, files, trees, and executable bit. (Basically feature parity with git trees) - using a git style backing store (the content addressable part) is a great way to store data from multiple domains without duplicating storage or introducing security issues around sharing data. - I really want a fuse-like way to create virtual file systems where I implement the back end using user space code. - I want a key/value system that's simple like local storage, but async and supports binary values. Sorry for conflating so many topics in one email. They are all related and I'm new to this list. I'm not sure of the proper procedure for creating new topics. - Tim Caswell
Re: Polished FileSystem API proposal
Yes. I completely agree. Personally in my projects, I wish there was to create custom filesystems backed by JavaScript code that are exported to hierarchical URLs. It would work somewhat like server-side web development where a js function handles the request for a file. This is basically what fuse does for Linux. I would love to be able to create a custom union filesystem that has layers. One would be in-memory and another would be backed by a local bit database. It would all be implemented in pure user space js. Maybe you would be interested on ServiceWorkers https://github.com/slightlyoff/ServiceWorker They are currently being implemented on Firefox and Chrome. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
On Wed, Nov 6, 2013 at 8:46 AM, pira...@gmail.com pira...@gmail.com wrote: Yes. I completely agree. Personally in my projects, I wish there was to create custom filesystems backed by JavaScript code that are exported to hierarchical URLs. It would work somewhat like server-side web development where a js function handles the request for a file. This is basically what fuse does for Linux. I would love to be able to create a custom union filesystem that has layers. One would be in-memory and another would be backed by a local bit database. It would all be implemented in pure user space js. Maybe you would be interested on ServiceWorkers https://github.com/slightlyoff/ServiceWorker They are currently being implemented on Firefox and Chrome. That's very interesting and useful, but I don't think it fits the same use case I was talking about. I want the ability to create some object that exports an URL that I can put in an iframe. Then all requests from that iframe for resources will dynamically call my javascript code. I could implement the same logic that a server-side application does, but from local code in my browser. My particular use case is I'm developing an IDE to teach programming and would love to generate dynamic file systems that are then rendered in iframes or new windows. Having a better offline store than appcache is indeed useful, but I believe it's orthogonal to what I'm looking for here. Like Jonas said, Use hierarchical filesystem: URLs is exactly what I'm looking for and currently I can only do this in chrome with their html5 fs api. But instead of creating a temporary filesystem and writing files, I'd much rather if my code was called for every file request and I could dynamically generate the files on demand. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
That's very interesting and useful, but I don't think it fits the same use case I was talking about. I want the ability to create some object that exports an URL that I can put in an iframe. Then all requests from that iframe for resources will dynamically call my javascript code. I could implement the same logic that a server-side application does, but from local code in my browser. That's just the purpose of ServiceWorker :-) Only that from your message, I suspect you are asking about having the same functionality but only on the current session or maybe also only when the page is open, deleting it on reload. I don't know of anything like to this, the most similar ones would be FirefoxOS Browser API or Chrome FileSystem API, but nothing as powerful as ServiceWorker, sorry :-( They are talking about implementing the Fetch specification, maybe you would write them about allowing to be used someway the ServiceWorker functionality on a per-session basis, I find legitimate your proposition... -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
There are multiple interesting ideas being discussed 1. Mapping files to persistent URLs. 2. Sharing persistent URLs between different origins. 3. Using the ServiceWorker [1] to redirect URL requests (and possibly manage it's own cache / files) 4. De-duping file copies using a Git like scheme. 1. Mapping files to persistent URLs. == There are some things that could be considered 'ok' to be slow: images, video startup, list of unread emails. There are some things that are very time sensitive such as the initial page layout. This means that things like the CSS required for layout and the fonts required for layout need to be there before the body begins laying out; ie: they need to be available in the head section. If they are not then the page will irritatingly flash as the parts arrive. Unless I'm missing something, this means that objects retrieved by promises/callbacks are not fast enough. For these needed for layout objects, persistent URLs are very attractive as they can be retrieved quickly and in the head section. These persistent URLs can be implemented in a filesystem or IndexedDB or other mechanism. A ServiceWorker could remap the URL but the data would still need to be available locally (eg, on disk). 2. Sharing persistent URLs between different origins. == Right now, any interesting object that is used by different origins (even from the same organization) must be downloaded and stored once per origin. Imagine if Linux required glib to be separately stored for every executable. This is how the web works today. Shareable persistent URLs would allow for a single copy to be download and shared across origins. Like shared libraries the user of the shared object has to trust the providing origin and only the providing origin should be able to write the data. 3. Using the ServiceWorker to redirect URL requests (and possibly manage it's own cache / files) == The ServiceWorker provides a way for an web page to redirect URLs. This is a very attractive feature for applications that are offline (or have an unreliable connection). The redirected URL could be to a completely different URL or to data managed by the ServiceWorker itself; eg, the ServiceWorker could use the FileSystem API to store data and redirect URLs to that data. Hopefully, this redirection will be fast; eg, fast enough for 'needed for layout' objects. Each ServiceWorker is origin specific: they are not shared across domains, and they are completely isolated from the browser's HTTP cache [2]. I take this to imply that the ServiceWorker has no ability to provide persistent URLs to other origins. 4. De-duping file copies using a Git like scheme. == My sense is everyone likes the idea of avoiding storing redundant data and Git's use of the SHA1 message digest as the filename is 'good enough'. This is a low security risk mechanism that is good for storage efficiency. The most benefit occurs when the storage mechanism (eg, FileSystem, IndexedDB) applies this across origins. Like sharing across origins it gets the benefit of avoiding duplicates but it does not address the multiple downloads issue. Multiple downloads are probably okay for smallish files but could be an issue for larger files such as 20Mbyte Chinese fonts, large Javascript libraries, etc. My wild guess is that because this is a 'good thing to do' but not 'a critical thing to do', its odds of getting implemented are poor. Brian Stell Notes: [1] https://github.com/slightlyoff/ServiceWorker [2] https://github.com/slightlyoff/ServiceWorker/blob/master/caching.md On Wed, Nov 6, 2013 at 8:28 AM, pira...@gmail.com pira...@gmail.com wrote: That's very interesting and useful, but I don't think it fits the same use case I was talking about. I want the ability to create some object that exports an URL that I can put in an iframe. Then all requests from that iframe for resources will dynamically call my javascript code. I could implement the same logic that a server-side application does, but from local code in my browser. That's just the purpose of ServiceWorker :-) Only that from your message, I suspect you are asking about having the same functionality but only on the current session or maybe also only when the page is open, deleting it on reload. I don't know of anything like to this, the most similar ones would be FirefoxOS Browser API or Chrome FileSystem API, but nothing as powerful as ServiceWorker, sorry :-( They are talking about implementing the Fetch specification, maybe you would write them about allowing to be used someway the ServiceWorker functionality on a per-session basis, I find legitimate your proposition... -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes,
Re: Polished FileSystem API proposal
On Wed, Oct 30, 2013 at 7:19 PM, pira...@gmail.com pira...@gmail.comwrote: What you are asking for could be fixed with redirects, that it's the HTTP equivalent of filesystems symbolic links :-) Is your suggestion that Google consolidate all its domains into one? These are widely separated servers (internet wise) with widely separated teams with widely separated schedules. In addition to different teams/schedules separate domains are important to internet load balancing: * Search, www.google.com, gets around 2 trillion searches per day [1]. * During Christmas YouTube got 1.6 million requests per second [2] * GMail has nearly 1/2 a billion active users per month [3] Do you really want that redirected to one domain? Brian Notes [1] http://www.statisticbrain.com/google-searches/ [2] http://www.youtube.com/watch?v=Jq-VMZK1KGk [3] http://venturebeat.com/2012/06/28/gmail-hotmail-yahoo-email-users/ 2013/10/31 Brian Stell bst...@google.com: In Request for feedback: Filesystem API [1] it says This filesystem would be origin-specific. This post discusses limited readonly sharing of filesystem resources between origins. To improve web site / application performance I'm interested in caching static [2] resources (eg, Javascript libraries, common CSS, fonts) in the filesystem and accessing them thru persistent URLs. So, what is the issue? I'd like to avoid duplication. Consider the following sites: they are all from a single organization but have different specific origins; * https://mail.google.com/ * https://plus.google.com/ * https://sites.google.com/ * ... At google there are *dozens* of these origins [3]. Even within a single page there are iframes from different origins. (There are other things that lead to different origins but for this post I'm ignoring them [4].) There could be *dozens* of copies of exactly the same a Javascript library, shared CSS, or web font in the FileSystem. What I'm suggesting is: * a filesystem's persistent URLs by default be read/write only for the same origin * the origin be able to allow other origins to access its files (readonly) by persistent URL I'm not asking-for nor suggesting API file access but others may express opinions on this. Brian Stell PS: Did I somehow miss info on same-origin in the spec [7]? Notes: [1] http://lists.w3.org/Archives/Public/public-script-coord/2013JulSep/0379.html [2] I'm also assuming immutability would be handled similar to gstatic.com [6] where different versions of a file have a different path/filename; eg, * V8: http://gstatic.com/fonts/roboto/v8/2UX7WLTfW3W8TclTUvlFyQ.woff * V9: http://gstatic.com/fonts/roboto/v9/2UX7WLTfW3W8TclTUvlFyQ.woff [3] Here are some of Google's origins: https://accounts.google.com https://blogsearch.google.com https://books.google.com https://chrome.google.com https://cloud.google.com https://code.google.com https://csi.gstatic.com https://developers.google.com https://docs.google.com https://drive.google.com https://earth.google.com https://fonts.googleapis.com https://groups.google.com https://mail.google.com https://maps.google.com https://news.google.com https://www.panoramio.com https://picasa.google.com https://picasaweb.google.com https://play.google.com https://productforums.google.com https://plus.google.com/ https://research.google.com https://support.google.com https://sites.google.com https://ssl.gstatic.com https://translate.google.com https://tables.googlelabs.com https://talkgadget.google.com https://themes.googleusercontent.com/ https://www.blogger.com https://www.google.com https://www.gstatic.com https://www.orcut.com https://www.youtube.com My guess is that there are more. I believe the XXX.blogspot.com origins belong to Google but I'm not an authority on this. [4] These are also different top level domains: * https://www.google.nl * https://www.google.co.jp Wikipedia lists about 200 of these [5] but since users tend to stick to one I'm ignoring them for this posting. I'm also ignoring http vs https (eg, http://www.google.com) and with/without leading www (eg, https://google.com) since they redirect. [5] http://en.wikipedia.org/wiki/List_of_Google_domains [6] http://wiki.answers.com/Q/What_is_gstatic [7] http://w3c.github.io/filesystem-api/Overview.html -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
On Thu, Oct 31, 2013 at 2:12 AM, Brian Stell bst...@google.com wrote: There could be *dozens* of copies of exactly the same a Javascript library, shared CSS, or web font in the FileSystem. Check out the cache part of https://github.com/slightlyoff/ServiceWorker/ Combined with a smart implementation that will do exactly what you want. And avoid all the issues of an actual cross-origin file system API. -- http://annevankesteren.nl/
Re: Polished FileSystem API proposal
I like Git's model :-) This would de-dup the file storage but won't it require downloading it for every domain (when the data is not lingering in HTTP cache)? On Tue, Nov 5, 2013 at 11:45 AM, Tim Caswell t...@creationix.com wrote: If the backend implementation used something like git's data store then duplicate data would automatically be stored only once without any security implications. The keys are the literal sha1 of the values. If two websites had the same file tree containing the same files, it would be the same tree object in the storage. But only sites who have a reference to the hash would have access to it. Also I like the level of fs support that git's filesystem has. There are trees, files, executable files, and symlinks. (there are also gitlinks used for submodules, but let's ignore those for now) On Tue, Nov 5, 2013 at 12:19 PM, Anne van Kesteren ann...@annevk.nlwrote: On Thu, Oct 31, 2013 at 2:12 AM, Brian Stell bst...@google.com wrote: There could be *dozens* of copies of exactly the same a Javascript library, shared CSS, or web font in the FileSystem. Check out the cache part of https://github.com/slightlyoff/ServiceWorker/ Combined with a smart implementation that will do exactly what you want. And avoid all the issues of an actual cross-origin file system API. -- http://annevankesteren.nl/
Re: Polished FileSystem API proposal
+1 to symbolic links, they have almost the same functionality that hard links and are more secure and flexible (they are usually just plain text files...). El 30/10/2013 01:42, Brendan Eich bren...@mozilla.com escribió: Hard links are peculiar to Unix filesystems. Not interoperable across all OSes. Symbolic links, OTOH... /be Brian Stell mailto:bst...@google.com October 29, 2013 4:53 PM I meant eg, V1/dir1/file1, V2/dir1/file1.
Re: Polished FileSystem API proposal
Good points! I was thinking of the logical functioning and hadn't considered the implementation. My understanding is that the UA will map from the filename to an actual file using some kind of database. My assumption was the logical idea of a link would happen in that layer. On Wed, Oct 30, 2013 at 1:14 AM, pira...@gmail.com pira...@gmail.comwrote: +1 to symbolic links, they have almost the same functionality that hard links and are more secure and flexible (they are usually just plain text files...). El 30/10/2013 01:42, Brendan Eich bren...@mozilla.com escribió: Hard links are peculiar to Unix filesystems. Not interoperable across all OSes. Symbolic links, OTOH... /be Brian Stell mailto:bst...@google.com October 29, 2013 4:53 PM I meant eg, V1/dir1/file1, V2/dir1/file1.
Re: Polished FileSystem API proposal
On most unix OSes, symbolic links are build using plain text files with just the path where they point inside and no more data, and later the OS identify them as a link instead a text file just by some special file flags, no more. On Windows, the direct access (.lnk) has a somewhat similar functionality (files that has the location of other files), and there was some discussions on the Wine and some Fat FS mail lists to use them to mimic real symbolic links. Hard links are more dificult to implement, since they are real link to a file, and usually this needs to have a counter of how many references are pointing to them so it doesn't get accidentally deleted, and this need support on the filesystem itself, while as I told you before, symbolic links can be mimic in several ways on a higher layer. 2013/10/30 Brian Stell bst...@google.com: Good points! I was thinking of the logical functioning and hadn't considered the implementation. My understanding is that the UA will map from the filename to an actual file using some kind of database. My assumption was the logical idea of a link would happen in that layer. On Wed, Oct 30, 2013 at 1:14 AM, pira...@gmail.com pira...@gmail.com wrote: +1 to symbolic links, they have almost the same functionality that hard links and are more secure and flexible (they are usually just plain text files...). El 30/10/2013 01:42, Brendan Eich bren...@mozilla.com escribió: Hard links are peculiar to Unix filesystems. Not interoperable across all OSes. Symbolic links, OTOH... /be Brian Stell mailto:bst...@google.com October 29, 2013 4:53 PM I meant eg, V1/dir1/file1, V2/dir1/file1. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
What you are asking for could be fixed with redirects, that it's the HTTP equivalent of filesystems symbolic links :-) 2013/10/31 Brian Stell bst...@google.com: In Request for feedback: Filesystem API [1] it says This filesystem would be origin-specific. This post discusses limited readonly sharing of filesystem resources between origins. To improve web site / application performance I'm interested in caching static [2] resources (eg, Javascript libraries, common CSS, fonts) in the filesystem and accessing them thru persistent URLs. So, what is the issue? I'd like to avoid duplication. Consider the following sites: they are all from a single organization but have different specific origins; * https://mail.google.com/ * https://plus.google.com/ * https://sites.google.com/ * ... At google there are *dozens* of these origins [3]. Even within a single page there are iframes from different origins. (There are other things that lead to different origins but for this post I'm ignoring them [4].) There could be *dozens* of copies of exactly the same a Javascript library, shared CSS, or web font in the FileSystem. What I'm suggesting is: * a filesystem's persistent URLs by default be read/write only for the same origin * the origin be able to allow other origins to access its files (readonly) by persistent URL I'm not asking-for nor suggesting API file access but others may express opinions on this. Brian Stell PS: Did I somehow miss info on same-origin in the spec [7]? Notes: [1] http://lists.w3.org/Archives/Public/public-script-coord/2013JulSep/0379.html [2] I'm also assuming immutability would be handled similar to gstatic.com [6] where different versions of a file have a different path/filename; eg, * V8: http://gstatic.com/fonts/roboto/v8/2UX7WLTfW3W8TclTUvlFyQ.woff * V9: http://gstatic.com/fonts/roboto/v9/2UX7WLTfW3W8TclTUvlFyQ.woff [3] Here are some of Google's origins: https://accounts.google.com https://blogsearch.google.com https://books.google.com https://chrome.google.com https://cloud.google.com https://code.google.com https://csi.gstatic.com https://developers.google.com https://docs.google.com https://drive.google.com https://earth.google.com https://fonts.googleapis.com https://groups.google.com https://mail.google.com https://maps.google.com https://news.google.com https://www.panoramio.com https://picasa.google.com https://picasaweb.google.com https://play.google.com https://productforums.google.com https://plus.google.com/ https://research.google.com https://support.google.com https://sites.google.com https://ssl.gstatic.com https://translate.google.com https://tables.googlelabs.com https://talkgadget.google.com https://themes.googleusercontent.com/ https://www.blogger.com https://www.google.com https://www.gstatic.com https://www.orcut.com https://www.youtube.com My guess is that there are more. I believe the XXX.blogspot.com origins belong to Google but I'm not an authority on this. [4] These are also different top level domains: * https://www.google.nl * https://www.google.co.jp Wikipedia lists about 200 of these [5] but since users tend to stick to one I'm ignoring them for this posting. I'm also ignoring http vs https (eg, http://www.google.com) and with/without leading www (eg, https://google.com) since they redirect. [5] http://en.wikipedia.org/wiki/List_of_Google_domains [6] http://wiki.answers.com/Q/What_is_gstatic [7] http://w3c.github.io/filesystem-api/Overview.html -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
I meant eg, V1/dir1/file1, V2/dir1/file1.
Re: Polished FileSystem API proposal
Hard links are peculiar to Unix filesystems. Not interoperable across all OSes. Symbolic links, OTOH... /be Brian Stell mailto:bst...@google.com October 29, 2013 4:53 PM I meant eg, V1/dir1/file1, V2/dir1/file1.
Re: Polished FileSystem API proposal
Hi Jonas, I notice that one of the common Linux file APIs, link[1], is not in you API. I don't see this as a first pass requirement but I certainly expect that applications will want to be able to have trees that represent a particular version of their immutable files; eg, V1/dir1/file1, V1/dir1/file1. It would possible to copy the unchanged files but that would double the storage size. Could you kindly share your thoughts on having a link API? Thanks, Brian Stell Notes: [1] http://linux.die.net/man/2/link
Re: Polished FileSystem API proposal
Hell yes!!! +1000 to add an interface to inotify :-D I'm concerned that you're taking this API too far. What's next - mmaping a file to a byte array? It would be cool... :-P Ok, let's get serious. Maybe inotify would be too high level API and I agree it would be moved to a later version, but I find it a really useful feature. in a per-domain sandboxed filesystem maybe it would not be so important (except if it's accessed from several pages on the same domain), but if it's a sandboxed mountpoint defined by the user on a folder on the computer filesystem and he has direct access to it, if he modify the contect of this sandboxed folder/mountpoint the webapp wouldn't know about that modifications, and for my personal use case I'm really interested on this feature (in fact, I proposed it some time ago, not remember if on this list or on Chrome...). -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
On Sat, Jul 13, 2013 at 2:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
It was also pointed out that we should address multi-file locking too.
One of the options is to make openRead() and openWrite() take a sequence.
interface Directory {
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
OpenWriteOptions options);
PromisesequenceFileHandle openRead(sequence(DOMString or File)
files);
PromisesequenceFileHandleWritable openWrite(sequence(DOMString or
File) files, OpenWriteOptions options);
}
So, this works with the current proposal, Jonas has a more complex solution
that probably requires bigger changes in the proposed file system API.
I'll let Jonas to describe it.
Jan
Re: Polished FileSystem API proposal
And here's an example:
// Copy the 1th byte of file1.bin to file2.bin
navigator.getFilesystem().then(function(root) {
return root.openWrite([file1.bin, file2.bin]);
}).then(function(handles) {
return handles[0].read(1);
}).then(function(buffer) {
return handles[1].write(buffer);
});
On Mon, Jul 22, 2013 at 8:18 PM, Jan Varga jan.va...@gmail.com wrote:
On Sat, Jul 13, 2013 at 2:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
It was also pointed out that we should address multi-file locking too.
One of the options is to make openRead() and openWrite() take a sequence.
interface Directory {
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
OpenWriteOptions options);
PromisesequenceFileHandle openRead(sequence(DOMString or File)
files);
PromisesequenceFileHandleWritable openWrite(sequence(DOMString or
File) files, OpenWriteOptions options);
}
So, this works with the current proposal, Jonas has a more complex
solution that probably requires bigger changes in the proposed file system
API.
I'll let Jonas to describe it.
Jan
Re: Polished FileSystem API proposal
On Mon, Jul 22, 2013 at 11:18 AM, Jan Varga jan.va...@gmail.com wrote:
On Sat, Jul 13, 2013 at 2:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
It was also pointed out that we should address multi-file locking too.
One of the options is to make openRead() and openWrite() take a sequence.
interface Directory {
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
OpenWriteOptions options);
PromisesequenceFileHandle openRead(sequence(DOMString or File)
files);
PromisesequenceFileHandleWritable openWrite(sequence(DOMString or
File) files, OpenWriteOptions options);
}
So, this works with the current proposal, Jonas has a more complex solution
that probably requires bigger changes in the proposed file system API.
I'll let Jonas to describe it.
First of all, I'm not sure that multi-file-locking is something that
we need to solve in this API at all. At least not yet. The google API
doesn't support locking of any type (as far as I can tell), so I think
starting with locking on a single-file basis is a better and simpler
place to start.
The main concern I have with this approach is that it creates a very
specific API which solves a pretty narrow problem of copying data
between two files. It doesn't support copying data between a server
and a file, or between indexedDB and a file.
I can think of two more generic solutions that will solve these problems:
A) Adding something like inotify.
I.e. add the ability to get notifications about changes to part of a
filesystem. This way a page could create a lock-file when it wants to
prevent others from accessing a particular part of the filesystem.
If the lockfile already exists, it could register to be notified when
the lockfile is removed. Once the file is removed it would create the
file and start accessing the files in whatever way it wants.
This is how multi-process access to application data is often handled
in filesystems today.
Adding something like inotify would also enable use cases like having
a worker synchronize a filesystem to a server. Other parts of the
application could simply access the filesystem directly and do
whatever modifications it wants. Those notifications will
automatically be noticed and synchronized to the server by the worker.
B) Add a generic cross-window lock mechanism.
Basically an API for asynchronously requesting a lock. Once the lock
becomes available the caller would be notified. When the caller is
done using the lock, he/she calls a function to explicitly release the
lock, thus enabling other callers to get notified that they are now
holding the lock.
A lock is never automatically released by the platform on some timeout
or similar. The only time the lock is forcefully released is when the
user closes the page that created the lock.
It is the application's responsibility to determine when it is
appropriate to grab the lock. I.e. there is no platform connection
between a lock and the resources that it protects. This way a lock
could represent anything from the whole filesystem, a couple of IDB
database and some server resources, to just parts of a file.
Darin Fisher has proposed something similar in the past as I recall
it. Though the details might not exactly match the above.
Pros of A
- Solves more use cases than simply locking-related ones. We've
discussed adding similar observer mechanisms to IDB because the same
non-locking-related usecases have come up there.
- Enables building something like solution B on top of it.
Cons of A
- More complicated to use correctly. For example it's important to
register for the notification before checking if the lock file is
already created. Otherwise there's a risk that the lock file is
removed between the time when the page checks if its there and the
notification is registered.
Pros of B
- Easier to use than A
Cons of B
- Doesn't help with use cases other than locking. I.e. to enable one
window to notice when another window modified some data, you would
either have to use polling, or create custom signaling mechanisms and
make sure to use those whenever something in the filesystem is
modified.
/ Jonas
Re: Polished FileSystem API proposal
Adding something like inotify would also enable use cases like having a worker synchronize a filesystem to a server. Other parts of the application could simply access the filesystem directly and do whatever modifications it wants. Those notifications will automatically be noticed and synchronized to the server by the worker. Hell yes!!! +1000 to add an interface to inotify :-D
Re: Polished FileSystem API proposal
2013/7/22 pira...@gmail.com pira...@gmail.com: Adding something like inotify would also enable use cases like having a worker synchronize a filesystem to a server. Other parts of the application could simply access the filesystem directly and do whatever modifications it wants. Those notifications will automatically be noticed and synchronized to the server by the worker. Hell yes!!! +1000 to add an interface to inotify :-D I'm concerned that you're taking this API too far. What's next - mmaping a file to a byte array? It was supposed to be simple, sandboxed API for storing data in file-like structures - let's maybe get this thing working first? /Janusz Majnert
Re: Polished FileSystem API proposal
On Tue, Jul 23, 2013 at 4:48 AM, Jonas Sicking jo...@sicking.cc wrote:
On Mon, Jul 22, 2013 at 11:18 AM, Jan Varga jan.va...@gmail.com wrote:
On Sat, Jul 13, 2013 at 2:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
It was also pointed out that we should address multi-file locking too.
One of the options is to make openRead() and openWrite() take a sequence.
interface Directory {
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
OpenWriteOptions options);
PromisesequenceFileHandle openRead(sequence(DOMString or File)
files);
PromisesequenceFileHandleWritable openWrite(sequence(DOMString or
File) files, OpenWriteOptions options);
}
So, this works with the current proposal, Jonas has a more complex
solution
that probably requires bigger changes in the proposed file system API.
I'll let Jonas to describe it.
First of all, I'm not sure that multi-file-locking is something that
we need to solve in this API at all. At least not yet. The google API
doesn't support locking of any type (as far as I can tell), so I think
starting with locking on a single-file basis is a better and simpler
place to start.
The main concern I have with this approach is that it creates a very
specific API which solves a pretty narrow problem of copying data
between two files. It doesn't support copying data between a server
and a file, or between indexedDB and a file.
I can think of two more generic solutions that will solve these problems:
A) Adding something like inotify.
I.e. add the ability to get notifications about changes to part of a
filesystem. This way a page could create a lock-file when it wants to
prevent others from accessing a particular part of the filesystem.
If the lockfile already exists, it could register to be notified when
the lockfile is removed. Once the file is removed it would create the
file and start accessing the files in whatever way it wants.
This is how multi-process access to application data is often handled
in filesystems today.
Adding something like inotify would also enable use cases like having
a worker synchronize a filesystem to a server. Other parts of the
application could simply access the filesystem directly and do
whatever modifications it wants. Those notifications will
automatically be noticed and synchronized to the server by the worker.
B) Add a generic cross-window lock mechanism.
Basically an API for asynchronously requesting a lock. Once the lock
becomes available the caller would be notified. When the caller is
done using the lock, he/she calls a function to explicitly release the
lock, thus enabling other callers to get notified that they are now
holding the lock.
A lock is never automatically released by the platform on some timeout
or similar. The only time the lock is forcefully released is when the
user closes the page that created the lock.
It is the application's responsibility to determine when it is
appropriate to grab the lock. I.e. there is no platform connection
between a lock and the resources that it protects. This way a lock
could represent anything from the whole filesystem, a couple of IDB
database and some server resources, to just parts of a file.
Darin Fisher has proposed something similar in the past as I recall
it. Though the details might not exactly match the above.
Pros of A
- Solves more use cases than simply locking-related ones. We've
discussed adding similar observer mechanisms to IDB because the same
non-locking-related usecases have come up there.
- Enables building something like solution B on top of it.
Cons of A
- More complicated to use correctly. For example it's important to
register for the notification before checking if the lock file is
already created. Otherwise there's a risk that the lock file is
removed between the time when the page checks if its there and the
notification is registered.
Pros of B
- Easier to use than A
Cons of B
- Doesn't help with use cases other than locking. I.e. to enable one
window to notice when another window modified some data, you would
either have to use polling, or create custom signaling mechanisms and
make sure to use those whenever something in the filesystem is
modified.
I vote for B, it's more generic, simpler to use and can be implemented
without dependency to the FileSystem API (thus it'd have higher probability
to get implemented more quickly by multiple vendors).
Separately from the locking issue, inotify-like feature (e.g. FileWatcher
and DirectoryWatcher) has been also requested in Chrome version's FS API,
and I can imagine the feature would get some popularity. But my personal
feeling is it might be too rich for 'simple' sandboxed file-storage API (at
least for version 1).
/
Re: Polished FileSystem API proposal
On Mon, Jul 22, 2013 at 3:40 PM, Janusz Majnert jmajn...@gmail.com wrote: 2013/7/22 pira...@gmail.com pira...@gmail.com: Adding something like inotify would also enable use cases like having a worker synchronize a filesystem to a server. Other parts of the application could simply access the filesystem directly and do whatever modifications it wants. Those notifications will automatically be noticed and synchronized to the server by the worker. Hell yes!!! +1000 to add an interface to inotify :-D I'm concerned that you're taking this API too far. What's next - mmaping a file to a byte array? It was supposed to be simple, sandboxed API for storing data in file-like structures - let's maybe get this thing working first? FWIW in many other storage APIs, there has been an established need to enable observing modifications of that storage area. localStorage had this feature from the beginning. In IndexedDB discussions between Google and Mozilla determined that both had received requests for this. Not sure if Microsoft has received similar feedback. Even storage OS-level storage APIs, i.e. filesystems, generally have this functionality. So I think it makes a lot of sense to add it to this filesystem too. That said. I'm happy to punt on this for now. As long as we are ok with punting on support for multi-file locking. / Jonas
Re: Polished FileSystem API proposal
On 15/07/13 23:26, Kinuko Yasuda wrote: OTOH one limitation I could think of in not having JS object is it'll disallow a possible future API expansion for sending a 'Directory' object to another app by postMessage. (It's another popular request we get in Chrome) Isn't a Directory object just a path? I mean, would you send the list of files and their content or just the path to the directory? If you literally want to pass the Directory object, I am not sure how passing the path is different except that you would have to know if this is temporary or permanent storage. -- Mounir
Re: Polished FileSystem API proposal
On Thu, Jul 18, 2013 at 4:52 PM, Mounir Lamouri mou...@lamouri.fr wrote: Isn't a Directory object just a path? I mean, would you send the list of files and their content or just the path to the directory? If you literally want to pass the Directory object, I am not sure how passing the path is different except that you would have to know if this is temporary or permanent storage. If you use a sandboxed system like Caja you could imagine passing the Directory and thereby not exposing any of the files higher in the hierarchy. However, Caja could probably also work around this somehow given different primitives. -- http://annevankesteren.nl/
Re: Polished FileSystem API proposal
Or simply if we make it possible to pass a Directory through postMessage then you can open another website in a sandboxed iframe and pass it a Directory and let it modify its contents, without having to grant access to other parts of the sandboxed filesystem. / Jonas On Thu, Jul 18, 2013 at 5:02 PM, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jul 18, 2013 at 4:52 PM, Mounir Lamouri mou...@lamouri.fr wrote: Isn't a Directory object just a path? I mean, would you send the list of files and their content or just the path to the directory? If you literally want to pass the Directory object, I am not sure how passing the path is different except that you would have to know if this is temporary or permanent storage. If you use a sandboxed system like Caja you could imagine passing the Directory and thereby not exposing any of the files higher in the hierarchy. However, Caja could probably also work around this somehow given different primitives. -- http://annevankesteren.nl/
Re: Polished FileSystem API proposal
Glad to see this proposal has a new draft.
On Sat, Jul 13, 2013 at 9:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters
parameters);
};
interface Directory {
readonly attribute DOMString name;
So neither File nor Directory has 'path' attribute but only exposes 'name'.
It feels a bit inconvenient but is it intentional?
File object is meant to be a snapshot and becomes invalid after
modification is made.
Will Directory follow the same model?
Say, if a Directory is acquired then moved (renamed) to another name, does
the Directory object keep functioning or become invalid?
PromiseFile createFile(DOMString path, MakeFileOptions options);
PromiseDirectory createDirectory(DOMString path);
Promise(File or Directory) get(DOMString path);
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
One of the major requests we've gotten in Chrome's FS API for potentially
long-running copy/remove operations is the ability to get progress and to
abort the operation.
Does it make sense to make them return AbortableProgressPromise when they
run recursively?
It's possible that an app creates tons of files in a directory and then
copies the entire directory to another, which could take long time
(depending on implementation).
(Also +1 to have a separate method for recursiveRemove or removeDepp)
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
Can this enumeration/stream be stopped halfway? If a directory contains
thousands of files callers may not want to keep the disk spinning until the
enumeration reaches the end.
};
interface FileHandle
{
readonly attribute FileOpenMode mode;
readonly attribute boolean active;
attribute long long? location;
PromiseFile getFile();
AbortableProgressPromiseArrayBuffer read(unsigned long long size);
AbortableProgressPromiseDOMString readText(unsigned long long
size, optional DOMString encoding = utf-8);
void abort();
};
interface FileHandleWritable : FileHandle
{
AbortableProgressPromisevoid write((DOMString or ArrayBuffer or
ArrayBufferView or Blob) value);
Promisevoid setSize(optional unsigned long long size);
Promisevoid flush();
};
partial interface URL {
static DOMString? getPersistentURL(File file);
}
// WebIDL cruft that's largely transparent
enum PersistenceType { temporary, persistent };
dictionary FilesystemParameters {
PersistenceType storage = temporary;
};
dictionary MakeFileOptions {
boolean overwriteIfExists = false;
The term 'overwrite' feels a bit confusing as it could mean 'overwriting a
file without truncating'.
If what it really means is truncate-if-exists can we rename this to
'truncateIfExists' (or { ifExists: truncate }) so that it becomes clearer
it works just like O_TRUNC?
(DOMString or Blob or ArrayBuffer or ArrayBufferView) data;
};
enum CreateMode { createifneeded, dontcreate }
enum DeleteMode { recursive, nonrecursive }
dictionary DestinationDict {
Directory dir;
DOMString name;
};
enum FileOpenMode { read, write, append };
So this API introduces 2 classes: Directory and FileHandle. Directory
allows manipulation of the files and directories stored inside that
directory. FileHandle represents an exclusively opened file and allows
manipulation of the file contents.
The behavior is hopefully mostly obvious. A few general comments:
The functions on Directory that accept DOMString arguments for
filenames allow names like path/to/file. If the function creates a
file, then it creates the intermediate directories. Such paths are
always interpreted as relative
Re: Polished FileSystem API proposal
On Tue, Jul 16, 2013 at 11:29 AM, Mounir Lamouri mou...@lamouri.fr wrote:
Hi,
I am not a big fan of the Directory approach of this proposal. It puts
the API between a high level, object oriented API and a low level API.
It is unfortunately not really high level because you have to use the
Directory for most operations and the File objects can't be subject to
any action.
I don't have a strong opinion on this but I kinda agree that introducing a
'Directory' object may not be definitely necessary in this API.
Associating a JS object to a file or a directory could easily result in a
stale object, and 'snapshot' nature often confuses developers. Looking at
other API examples, Node.js's FileSystem API purely works on path names and
it looks much simpler. http://nodejs.org/api/fs.html
OTOH one limitation I could think of in not having JS object is it'll
disallow a possible future API expansion for sending a 'Directory' object
to another app by postMessage. (It's another popular request we get in
Chrome)
I designed an API that looks very similar to the one proposed here but
based around DOMString instead of File and Directory. This API is meant
to be low level and more basic/simple. It gives room for JS libraries to
come up with their own high level object oriented API design and adds
very little overhead for simple use case.
This API is built around a FileSystem interface that exposes all the
methods needed to manipulate the virtual filesystem. Getting a
FileSystem object would be done with:
navigator.getFileSystem({temporary, permanent}).
An alternative would be to have navigator.filesystem that would return a
FileSystem object and navigator.requestPermanentFileSystem() that could
be used to make the navigator.filesystem object different (ie. permanent
vs temporary). That approach would make using temporary storage more
straightforward but could make using permanent storage a bit more painful.
This API is a bit less polished than the initial proposal (there were
less eyes on it) and contrary to the initial proposal, it is trying to
be very simple regarding the behaviour. For example, .remove() would
remove the directory/file and all sub-directories. The consumer of the
API is expected to check if the directory is empty before calling the
method. The API is re-using FileHandle and FileHandleWritable.
There are a couple of functions there only to make developers' life
easier assuming they would make a common use case simpler. Removing them
is definitely an option.
The interface below describes this alternative proposal. The comments
should give a rough explanation of what each method is doing.
enum FileSystemType {
temporary,
persistent
};
partial interface Navigator {
FileSystem getFileSystem(optional FileSystemType type = temporary);
};
interface FileSystem {
// Those two methods will create the directory/file if it does not
// exist but will keep there content as is. The returned value will
// be whether something as been created.
Promiseboolean createDir(DOMString path);
Promiseboolean createFile(DOMString path);
// Can be used to rename or move. Whether for a file or directory.
Promisevoid move(DOMString source, DOMString destination);
// Mostly to make developer's life easier.
Promisevoid copy(DOMString source, DOMString destination);
// Removes the file if the path is a file or the sub-tree if the path
// is a directory.
Promiseboolean remove(DOMString path);
// The next three functions would reject if the path is a directory.
PromiseFileHandle read(DOMString path);
PromiseFileHandleWritable write(DOMString path);
// This is for convenience so we can easily write a Blob.
Promisevoid write(DOMString path, Blob data);
// TODO: we could add append().
// For convenience. Could be expressed as:
// .read(path).then(f) { return f.getFile(); }
PromiseFile getFile(DOMString path);
EventStreamDOMString enumerate(DOMString path, optional boolean deep
= false);
// Those two methods are trivial. They could be merged into one
// method that would return a tri-state enum: file, directory,
// notfound.
Promiseboolean isDirectory(DOMString path);
Promiseboolean exists(DOMString path);
// TODO: we could add something like .childrenCount that would return
// how many files/directory a directory contains. Would be 0 for a
// file. This could also be .isEmpty(). But that might be
// mis-interpreted for files.
};
Thanks,
--
Mounir
On 12/07/13 17:31, Jonas Sicking wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding
Re: Polished FileSystem API proposal
On Mon, Jul 15, 2013 at 11:02 PM, Kinuko Yasuda kin...@chromium.org wrote:
Glad to see this proposal has a new draft.
On Sat, Jul 13, 2013 at 9:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters
parameters);
};
interface Directory {
readonly attribute DOMString name;
So neither File nor Directory has 'path' attribute but only exposes 'name'.
It feels a bit inconvenient but is it intentional?
It's something that I intended to add but forgot.
Keeping .name as just containing the leafname is probably the right thing to do.
So we could introduce a .path property which contains the path within
the filesystem. The full filename would be .path + .name.
And doing exactly the same for both Directory and File objects seems
like a good thing.
File object is meant to be a snapshot and becomes invalid after modification
is made.
Will Directory follow the same model?
Say, if a Directory is acquired then moved (renamed) to another name, does
the Directory object keep functioning or become invalid?
Good question. I think the most sane way to implement the Directory
object is to internally keep a full path. Whenever an operation is to
be performed, we check if that full path still exists. If it doesn't,
the operation fails.
PromiseFile createFile(DOMString path, MakeFileOptions options);
PromiseDirectory createDirectory(DOMString path);
Promise(File or Directory) get(DOMString path);
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
One of the major requests we've gotten in Chrome's FS API for potentially
long-running copy/remove operations is the ability to get progress and to
abort the operation.
Does it make sense to make them return AbortableProgressPromise when they
run recursively?
Yeah, I think that's a good idea. We have to figure out exactly what
the progress information would look like, but that seems very doable.
It's possible that an app creates tons of files in a directory and then
copies the entire directory to another, which could take long time
(depending on implementation).
(Also +1 to have a separate method for recursiveRemove or removeDepp)
Done. Though I'd still like to remove the non-recursive variant. Is it
really useful?
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
Can this enumeration/stream be stopped halfway? If a directory contains
thousands of files callers may not want to keep the disk spinning until the
enumeration reaches the end.
EventStream is something that still needs to be defined. And shouldn't
be a filesystem specific interface. But yes, it definitely needs to be
stoppable halfway. Maybe we'll need something like
AbortableEventStream, or PausableEventStream, or some such.
The FileHandle class automatically closes itself as soon as the page
stops posting further calls to .read/.readBinary/.write to it. This
happens once the last Promise returned from one of those operations
has been resolved, without further calls to .read/.readBinary/.write
having happened.
Can the same FileHandle be reused after it's closed, or one need to create a
new FileHandler to start another sequence of read/write?
Once it's closed, i.e. once the last accept/reject callback has
happened and no new operations were started, the FileHandle becomes
useless. You have to call openRead/openWrite again to start another
sequence.
/ Jonas
Re: Polished FileSystem API proposal
On Tue, Jul 16, 2013 at 8:56 AM, Eric U er...@google.com wrote:
On Mon, Jul 15, 2013 at 5:58 PM, Jonas Sicking jo...@sicking.cc wrote:
On Mon, Jul 15, 2013 at 4:47 PM, Eric U er...@google.com wrote:
What's the purpose of DestinationDict? It seems redundant.
There's basically three potential ways to specify a copy/move destination:
Specify a new name within the current directory.
Specify a new directory but keep the current name.
Specify a new directory as well as a new name within that directory.
We obviously don't have to support them all, for example you could
imagine requiring a two-step process in some cases. But we decided to
try to support all three as a single operation for now.
Coming up with a syntax for this is somewhat tricky. In C++ you'd
simply have three overloads that take the different combinations, but
that doesn't feel very javascripty.
Another is to use optional arguments like:
move(..., Directory? destDir, optional DOMString destName);
would work. That means that the three options would look like:
dir.move(srcfile, null, newName);
dir.move(srcfile, destDir);
dir.move(srcfile, destDir, newName);
dir.move(srcfile, dir/and/new/name);
Granted, that doesn't help you if you're holding a Directory object
and don't have a way to generate a relative path to it. But perhaps
that's not a big lack? You can always munge paths yourself.
Obviously we went with the way you'd see as iffy.
Unfortunately the first one there looks pretty iffy. So instead we
went with a union which results in syntax like:
dir.move(srcfile, newName);
dir.move(srcfile, destDir);
dir.move(srcfile, { dir: destDir. name: newName });
In this syntax, could you say it this way?
dir.move(srcfile, { dir: dirName. name: newName });
No, but you could say
dir.move(srcfile, dirName/newName);
Or you can use .get() to grab the directory object named dirName and use
that.
So what can't you do without the DestinationDict? You can rename the
file without moving it to a new dir, you can move it to a new dir
without renaming it, and you can rename and move it using
dirName/newName as the second argument.
Only if the new destination directory is a subdir of the current directory.
/ Jonas
You have MakeFileOptions.overwriteIfExists for createFile, but
CreateMode { createIfNeeded | dontcreate } for openWrite; it feels
like those should share a set of flags. The options should probably
be {createIfNeeded, dontCreate, failIfExists}, where failIfExists
implies createIfNeeded and is the standard POSIX paradigm
[O_CREAT|O_EXCL]. It's useful for coordination between multiple pages
in the same origin, or the same page loaded twice.
So you're basically proposing (modulo naming):
createFile(name, { mode: createIfNeeded });
Creates the file if it doesn't exist. Overwrites it if it does.
Is the idea that createFile will overwrite an existing file's
contents, even if you don't supply any, e.g. nuking a file down to 0
bytes?
Yes.
That might be a bit surprising. Would you require
MakeFileOptions.data?
In the current proposal the function is called createFile and if you
want it to overwrite what's there you have to explicitly opt in to
that using
createFile(name, { overwriteIfExists: true });
so I don't think the current proposal is that surprising. Especially
since creating files usually will either error or remove what's there.
I'd like to keep that clarity.
createFile(name, { mode: dontCreate });
Error if the file doesn't exist. Overwrites it if it does.
I'm not sure why you'd ever call it this way. Perhaps disallow it for
now? Otherwise it might become a standard hack for some bizarre
file-based locking paradigm.
createFile(name, { mode: failIfExists });
Creates the file if it doesn't exist. Error it if it does.
openWrite(name, createIfNeeded);
Creates and opens the file if it doesn't exist. Opens it if it does.
openWrite(name, dontCreate);
Error if the file doesn't exist. Opens it if it does.
openWrite(name, failIfExists);
Creates and opens the file if it doesn't exist. Error it if it does.
Is this correct? While I agree it's consistent, the number of
combinations here is a bit mind boggling. It took me quite a while to
write all of these out and make sure they are correct. The
createFile(dontCreate) combination is especially confusing.
I'd skip that one; I think the other 5 make sense, and are better than
having two different paradigms to say almost the same thing depending
on which function you call [boolean vs. string argument]. If you
decide you don't want failIfExists, that makes the list even simpler.
I think the 4 modes that are important to support are:
Create new file and error if there's already an existing one
Create new file and overwrite if there's already an existing one
Open for writing and create if doesn't exist
Open for writing and error if doesn't exist
The behavior that needs to be defined for the
Re: Polished FileSystem API proposal
On Tue, Jul 16, 2013 at 8:53 AM, Eric U er...@google.com wrote:
On Tue, Jul 16, 2013 at 12:32 AM, Jonas Sicking jo...@sicking.cc wrote:
On Mon, Jul 15, 2013 at 11:02 PM, Kinuko Yasuda kin...@chromium.org wrote:
Glad to see this proposal has a new draft.
On Sat, Jul 13, 2013 at 9:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters
parameters);
};
interface Directory {
readonly attribute DOMString name;
So neither File nor Directory has 'path' attribute but only exposes 'name'.
It feels a bit inconvenient but is it intentional?
It's something that I intended to add but forgot.
Having the full path within the filesystem removes some of the
security benefit of not allowing .., doesn't it? Conversely, is the
path useful if you can't use it to manipulate anything above the
current dir in the tree?
It removes some of the privacy benefits, but little of the security benefits.
It would at least be useful to let something like:
rootdir.get(foo/myfile.html).then(function(file) {
assert(file.path + file.name === foo/myfile.html);
});
The tricky thing is what to do if someone calls subdir.get(). Does
that return a File with .path set to a path relative to the root, or a
path relative to the subdir? If relative to the root, it does lose
some of the privacy benefits, but I'm not sure how important that is
within the sandbox. If relative to the subdir that means that you can
get two File objects representing the same file but with two different
paths. That's somewhat confusing.
Also note that we'd definitely never expose a path that goes outside
of the sandbox. I.e. the drag'n'drop or input type=file APIs should
never produce paths that contain directory names outside of the
directories that the user has attached to the page.
Keeping .name as just containing the leafname is probably the right thing to
do.
So we could introduce a .path property which contains the path within
the filesystem. The full filename would be .path + .name.
And doing exactly the same for both Directory and File objects seems
like a good thing.
File object is meant to be a snapshot and becomes invalid after modification
is made.
Will Directory follow the same model?
Say, if a Directory is acquired then moved (renamed) to another name, does
the Directory object keep functioning or become invalid?
Good question. I think the most sane way to implement the Directory
object is to internally keep a full path. Whenever an operation is to
be performed, we check if that full path still exists. If it doesn't,
the operation fails.
Just to be clear: if you rename a Directory's parent, it goes stale
and you can't use it. But if you then create a new directory with the
same name as the parent used to have, with a child of the right name,
this object will again be valid. The Directory is explicitly defined
as referring to a specific path string.
Agreed.
/ Jonas
Re: Polished FileSystem API proposal
Having the full path within the filesystem removes some of the security benefit of not allowing .., doesn't it? There's no problem if it's just an implementation detail. Conversely, is the path useful if you can't use it to manipulate anything above the current dir in the tree? It's useful to isolate it as a sandbox. Maybe we are talking about mount points that expose the root of a user selected filesystem subtree (I hope answer is yes... :-) )? -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Polished FileSystem API proposal
Hi,
On 2013-07-13 02:31, Jonas Sicking wrote:
[...]
interface FileHandle
{
readonly attribute FileOpenMode mode;
readonly attribute boolean active;
attribute long long? location;
location seems to be a bad name. It might be confused with location
in the filesystem. Why not use offset instead? Is this value 0-based?
PromiseFile getFile();
AbortableProgressPromiseArrayBuffer read(unsigned long long size);
AbortableProgressPromiseDOMString readText(unsigned long long
size, optional DOMString encoding = utf-8);
void abort();
};
interface FileHandleWritable : FileHandle
{
AbortableProgressPromisevoid write((DOMString or ArrayBuffer or
ArrayBufferView or Blob) value);
Promisevoid setSize(optional unsigned long long size);
Promisevoid flush();
};
partial interface URL {
static DOMString? getPersistentURL(File file);
}
// WebIDL cruft that's largely transparent
enum PersistenceType { temporary, persistent };
dictionary FilesystemParameters {
PersistenceType storage = temporary;
};
dictionary MakeFileOptions {
boolean overwriteIfExists = false;
(DOMString or Blob or ArrayBuffer or ArrayBufferView) data;
};
enum CreateMode { createifneeded, dontcreate }
enum DeleteMode { recursive, nonrecursive }
dictionary DestinationDict {
Directory dir;
DOMString name;
};
enum FileOpenMode { read, write, append };
[...]
Do we really need the .openAppend() function? Or is it ok to ask
people to use .openWrite() and then go to the end before writing?
In the append mode, is the location in file changed to the end of
file before every write? If not, then I think openAppend should be
removed. In the example you gave at the bottom of your message, it seems
that it's actually possible to write in the middle of the file.
Another non-posix thing is that you can read a file that is in write mode.
I would therefore propose to:
* remove openAppend()
* change enum FileOpenMode to {read, readWrite}
In another email you wrote that this API is not meant to be used to
access the OS filesystem, but a snadboxed filesystem abstraction for a
webapp. If so, why do we even need the two access modes? Let's just have
an openFile() that returns a File object you can read and write to.
--
Janusz Majnert
Samsung RD Institute Poland
Samsung Electronics
Re: Polished FileSystem API proposal
On Mon, Jul 15, 2013 at 12:18 AM, Janusz Majnert j.majn...@samsung.com wrote:
Hi,
On 2013-07-13 02:31, Jonas Sicking wrote:
[...]
interface FileHandle
{
readonly attribute FileOpenMode mode;
readonly attribute boolean active;
attribute long long? location;
location seems to be a bad name. It might be confused with location in
the filesystem. Why not use offset instead? Is this value 0-based?
That seems better yeah.
Do we really need the .openAppend() function? Or is it ok to ask
people to use .openWrite() and then go to the end before writing?
In the append mode, is the location in file changed to the end of file
before every write?
No. It's just initially set to the end of the file. Though since every
write moves the offset to the end of the write, that means that the
offset will remain at the end of the file unless explicitly set.
If not, then I think openAppend should be removed. In
the example you gave at the bottom of your message, it seems that it's
actually possible to write in the middle of the file.
It is indeed.
Another non-posix thing is that you can read a file that is in write mode.
I would therefore propose to:
* remove openAppend()
* change enum FileOpenMode to {read, readWrite}
In another email you wrote that this API is not meant to be used to access
the OS filesystem, but a snadboxed filesystem abstraction for a webapp. If
so, why do we even need the two access modes? Let's just have an openFile()
that returns a File object you can read and write to.
I can think of four reasons, though none of them particularly great.
* While it's not a primary target for this API, it's nice if we can
reuse the same API if we ever end up introducing APIs that will allow
accessing real filesystems. Though you could argue that it's ok if you
in that case too always open the file in readwrite mode.
* Even in a sandboxed filesystem you could end up with multiple actors
accessing the filesystem at the same time. For example the user might
have the same website open in two tabs. Or you might have a worker and
a page accessing the file at the same time. With openRead you can
allow multiple simultaneous readers.
* We might want to support a read-only filesystem in some situations.
For example if the user uses drag'n'drop to expose a folder to a page,
we could expose that folder using a read-only version of the Directory
interface. Though in that case not exposing open* at all and instead
relying on get() would probably work.
* A main reason we're considering exposing a filesystem API at all is
it's a familiar concept to authors. And most filesystem APIs have
openRead and openWrite (and posix has openAppend).
/ Jonas
Re: Polished FileSystem API proposal
15 lip 2013 10:47, Jonas Sicking jo...@sicking.cc napisał(a):
On Mon, Jul 15, 2013 at 12:18 AM, Janusz Majnert j.majn...@samsung.com
wrote:
Hi,
On 2013-07-13 02:31, Jonas Sicking wrote:
[...]
interface FileHandle
{
readonly attribute FileOpenMode mode;
readonly attribute boolean active;
attribute long long? location;
location seems to be a bad name. It might be confused with location
in
the filesystem. Why not use offset instead? Is this value 0-based?
That seems better yeah.
Do we really need the .openAppend() function? Or is it ok to ask
people to use .openWrite() and then go to the end before writing?
In the append mode, is the location in file changed to the end of
file
before every write?
No. It's just initially set to the end of the file. Though since every
write moves the offset to the end of the write, that means that the
offset will remain at the end of the file unless explicitly set.
Yes, but if you change the offset to point to the beginnig of the file,
then a subsequent write happens there, unlike in posix, which would first
move the pointer to the eof and then write the data.
IMHO openAppend should be dropped. It will confuse people used to posix
style file access, and for others it's just a shorthand for
openWrite+location=null.
If not, then I think openAppend should be removed. In
the example you gave at the bottom of your message, it seems that it's
actually possible to write in the middle of the file.
It is indeed.
Another non-posix thing is that you can read a file that is in write
mode.
I would therefore propose to:
* remove openAppend()
* change enum FileOpenMode to {read, readWrite}
In another email you wrote that this API is not meant to be used to
access
the OS filesystem, but a snadboxed filesystem abstraction for a
webapp. If
so, why do we even need the two access modes? Let's just have an
openFile()
that returns a File object you can read and write to.
I can think of four reasons, though none of them particularly great.
* While it's not a primary target for this API, it's nice if we can
reuse the same API if we ever end up introducing APIs that will allow
accessing real filesystems. Though you could argue that it's ok if you
in that case too always open the file in readwrite mode.
* Even in a sandboxed filesystem you could end up with multiple actors
accessing the filesystem at the same time. For example the user might
have the same website open in two tabs. Or you might have a worker and
a page accessing the file at the same time. With openRead you can
allow multiple simultaneous readers.
* We might want to support a read-only filesystem in some situations.
For example if the user uses drag'n'drop to expose a folder to a page,
we could expose that folder using a read-only version of the Directory
interface. Though in that case not exposing open* at all and instead
relying on get() would probably work.
* A main reason we're considering exposing a filesystem API at all is
it's a familiar concept to authors. And most filesystem APIs have
openRead and openWrite (and posix has openAppend).
/ Jonas
Re: Polished FileSystem API proposal
On Mon, Jul 15, 2013 at 2:42 PM, Eric U er...@google.com wrote:
75% bikeshedding, 25% lessons learned:
It seems inconsistent to have the API for recursion done differently
in two different parts of the API: you've got two functions for
enumerate/enumerateDeep but remove takes a parameter to be recursive.
I don't have a strong preference between them, but I think you should
only use one style.
Good point. I'll change it to remove and removeDeep for now. That also
avoids the what is default question.
Out of curiosity, has it been a problem for you that the
DirectoryEntry API doesn't have a non-recursive remove option?
Granted, you do have the ability to grab a sub-Entry and call .remove
on it. We didn't have that option since we didn't want to create an
equivalent of FileEntry.
What's the purpose of DestinationDict? It seems redundant.
There's basically three potential ways to specify a copy/move destination:
Specify a new name within the current directory.
Specify a new directory but keep the current name.
Specify a new directory as well as a new name within that directory.
We obviously don't have to support them all, for example you could
imagine requiring a two-step process in some cases. But we decided to
try to support all three as a single operation for now.
Coming up with a syntax for this is somewhat tricky. In C++ you'd
simply have three overloads that take the different combinations, but
that doesn't feel very javascripty.
Another is to use optional arguments like:
move(..., Directory? destDir, optional DOMString destName);
would work. That means that the three options would look like:
dir.move(srcfile, null, newName);
dir.move(srcfile, destDir);
dir.move(srcfile, destDir, newName);
Unfortunately the first one there looks pretty iffy. So instead we
went with a union which results in syntax like:
dir.move(srcfile, newName);
dir.move(srcfile, destDir);
dir.move(srcfile, { dir: destDir. name: newName });
Another solution might be to only use dictionaries. Resulting in
dir.move(srcfile, { name: newName });
dir.move(srcfile, { dir: destDir });
dir.move(srcfile, { dir: destDir. name: newName });
I think I'd prefer to defer to TC39 about which option is best here.
I'll point this out explicitly when emailing public-script-coord.
You have MakeFileOptions.overwriteIfExists for createFile, but
CreateMode { createIfNeeded | dontcreate } for openWrite; it feels
like those should share a set of flags. The options should probably
be {createIfNeeded, dontCreate, failIfExists}, where failIfExists
implies createIfNeeded and is the standard POSIX paradigm
[O_CREAT|O_EXCL]. It's useful for coordination between multiple pages
in the same origin, or the same page loaded twice.
So you're basically proposing (modulo naming):
createFile(name, { mode: createIfNeeded });
Creates the file if it doesn't exist. Overwrites it if it does.
createFile(name, { mode: dontCreate });
Error if the file doesn't exist. Overwrites it if it does.
createFile(name, { mode: failIfExists });
Creates the file if it doesn't exist. Error it if it does.
openWrite(name, createIfNeeded);
Creates and opens the file if it doesn't exist. Opens it if it does.
openWrite(name, dontCreate);
Error if the file doesn't exist. Opens it if it does.
openWrite(name, failIfExists);
Creates and opens the file if it doesn't exist. Error it if it does.
Is this correct? While I agree it's consistent, the number of
combinations here is a bit mind boggling. It took me quite a while to
write all of these out and make sure they are correct. The
createFile(dontCreate) combination is especially confusing.
If you don't allow .. to traverse directories, please define using
it as a path segment as an error, to allow for future expansion and to
catch bugs for folks who didn't read the spec.
Definitely.
openForWrite or openForWriting, while longer, would be clearer than
openWrite. Heck, why not just write?
I tried out just write for a while, but it seemed strange that
myDir.write() actually started writing to a file.
We've had folks ask for progress events on long move and copy calls.
Think about network drive latencies, big SD cards being brought into
the browser world, etc. It's especially important if you think you'll
ever use this API outside the sandbox, and cheap to add now even if
you won't. If you don't have it, people hack around it by
reimplementing copy using write and createDirectory, and it's tedious
and error-prone. In your case, you'll probably want them for
createFile as well, since you can pass in a gigantic Blob or
ArrayBuffer there.
Hmm.. Good points. For directory copy/move operations we couldn't
provide a total size, but we could still provide progress in the form
of total number number of bytes copied so far, as well as currently
copying/moving file with name X.
/ Jonas
Re: Polished FileSystem API proposal
On Mon, Jul 15, 2013 at 4:47 PM, Eric U er...@google.com wrote:
What's the purpose of DestinationDict? It seems redundant.
There's basically three potential ways to specify a copy/move destination:
Specify a new name within the current directory.
Specify a new directory but keep the current name.
Specify a new directory as well as a new name within that directory.
We obviously don't have to support them all, for example you could
imagine requiring a two-step process in some cases. But we decided to
try to support all three as a single operation for now.
Coming up with a syntax for this is somewhat tricky. In C++ you'd
simply have three overloads that take the different combinations, but
that doesn't feel very javascripty.
Another is to use optional arguments like:
move(..., Directory? destDir, optional DOMString destName);
would work. That means that the three options would look like:
dir.move(srcfile, null, newName);
dir.move(srcfile, destDir);
dir.move(srcfile, destDir, newName);
dir.move(srcfile, dir/and/new/name);
Granted, that doesn't help you if you're holding a Directory object
and don't have a way to generate a relative path to it. But perhaps
that's not a big lack? You can always munge paths yourself.
Obviously we went with the way you'd see as iffy.
Unfortunately the first one there looks pretty iffy. So instead we
went with a union which results in syntax like:
dir.move(srcfile, newName);
dir.move(srcfile, destDir);
dir.move(srcfile, { dir: destDir. name: newName });
In this syntax, could you say it this way?
dir.move(srcfile, { dir: dirName. name: newName });
No, but you could say
dir.move(srcfile, dirName/newName);
Or you can use .get() to grab the directory object named dirName and use that.
You have MakeFileOptions.overwriteIfExists for createFile, but
CreateMode { createIfNeeded | dontcreate } for openWrite; it feels
like those should share a set of flags. The options should probably
be {createIfNeeded, dontCreate, failIfExists}, where failIfExists
implies createIfNeeded and is the standard POSIX paradigm
[O_CREAT|O_EXCL]. It's useful for coordination between multiple pages
in the same origin, or the same page loaded twice.
So you're basically proposing (modulo naming):
createFile(name, { mode: createIfNeeded });
Creates the file if it doesn't exist. Overwrites it if it does.
Is the idea that createFile will overwrite an existing file's
contents, even if you don't supply any, e.g. nuking a file down to 0
bytes?
Yes.
That might be a bit surprising. Would you require
MakeFileOptions.data?
In the current proposal the function is called createFile and if you
want it to overwrite what's there you have to explicitly opt in to
that using
createFile(name, { overwriteIfExists: true });
so I don't think the current proposal is that surprising. Especially
since creating files usually will either error or remove what's there.
I'd like to keep that clarity.
createFile(name, { mode: dontCreate });
Error if the file doesn't exist. Overwrites it if it does.
I'm not sure why you'd ever call it this way. Perhaps disallow it for
now? Otherwise it might become a standard hack for some bizarre
file-based locking paradigm.
createFile(name, { mode: failIfExists });
Creates the file if it doesn't exist. Error it if it does.
openWrite(name, createIfNeeded);
Creates and opens the file if it doesn't exist. Opens it if it does.
openWrite(name, dontCreate);
Error if the file doesn't exist. Opens it if it does.
openWrite(name, failIfExists);
Creates and opens the file if it doesn't exist. Error it if it does.
Is this correct? While I agree it's consistent, the number of
combinations here is a bit mind boggling. It took me quite a while to
write all of these out and make sure they are correct. The
createFile(dontCreate) combination is especially confusing.
I'd skip that one; I think the other 5 make sense, and are better than
having two different paradigms to say almost the same thing depending
on which function you call [boolean vs. string argument]. If you
decide you don't want failIfExists, that makes the list even simpler.
I think the 4 modes that are important to support are:
Create new file and error if there's already an existing one
Create new file and overwrite if there's already an existing one
Open for writing and create if doesn't exist
Open for writing and error if doesn't exist
The behavior that needs to be defined for the create function is what
to do if the file already exists, because generally you aren't
expecting a file to exist when you are requesting it to be created.
The behavior that needs to be defined for the open function is what to
do if the function does not exist.
So I don't know that trying to align them too much is going to work
very well. I do totally agree that the current setup is not good
enough though.
How about:
createFile(name, { ifExists: overwrite });
createFile(name,
Re: Polished FileSystem API proposal
Hi,
I am not a big fan of the Directory approach of this proposal. It puts
the API between a high level, object oriented API and a low level API.
It is unfortunately not really high level because you have to use the
Directory for most operations and the File objects can't be subject to
any action.
I designed an API that looks very similar to the one proposed here but
based around DOMString instead of File and Directory. This API is meant
to be low level and more basic/simple. It gives room for JS libraries to
come up with their own high level object oriented API design and adds
very little overhead for simple use case.
This API is built around a FileSystem interface that exposes all the
methods needed to manipulate the virtual filesystem. Getting a
FileSystem object would be done with:
navigator.getFileSystem({temporary, permanent}).
An alternative would be to have navigator.filesystem that would return a
FileSystem object and navigator.requestPermanentFileSystem() that could
be used to make the navigator.filesystem object different (ie. permanent
vs temporary). That approach would make using temporary storage more
straightforward but could make using permanent storage a bit more painful.
This API is a bit less polished than the initial proposal (there were
less eyes on it) and contrary to the initial proposal, it is trying to
be very simple regarding the behaviour. For example, .remove() would
remove the directory/file and all sub-directories. The consumer of the
API is expected to check if the directory is empty before calling the
method. The API is re-using FileHandle and FileHandleWritable.
There are a couple of functions there only to make developers' life
easier assuming they would make a common use case simpler. Removing them
is definitely an option.
The interface below describes this alternative proposal. The comments
should give a rough explanation of what each method is doing.
enum FileSystemType {
temporary,
persistent
};
partial interface Navigator {
FileSystem getFileSystem(optional FileSystemType type = temporary);
};
interface FileSystem {
// Those two methods will create the directory/file if it does not
// exist but will keep there content as is. The returned value will
// be whether something as been created.
Promiseboolean createDir(DOMString path);
Promiseboolean createFile(DOMString path);
// Can be used to rename or move. Whether for a file or directory.
Promisevoid move(DOMString source, DOMString destination);
// Mostly to make developer's life easier.
Promisevoid copy(DOMString source, DOMString destination);
// Removes the file if the path is a file or the sub-tree if the path
// is a directory.
Promiseboolean remove(DOMString path);
// The next three functions would reject if the path is a directory.
PromiseFileHandle read(DOMString path);
PromiseFileHandleWritable write(DOMString path);
// This is for convenience so we can easily write a Blob.
Promisevoid write(DOMString path, Blob data);
// TODO: we could add append().
// For convenience. Could be expressed as:
// .read(path).then(f) { return f.getFile(); }
PromiseFile getFile(DOMString path);
EventStreamDOMString enumerate(DOMString path, optional boolean deep
= false);
// Those two methods are trivial. They could be merged into one
// method that would return a tri-state enum: file, directory,
// notfound.
Promiseboolean isDirectory(DOMString path);
Promiseboolean exists(DOMString path);
// TODO: we could add something like .childrenCount that would return
// how many files/directory a directory contains. Would be 0 for a
// file. This could also be .isEmpty(). But that might be
// mis-interpreted for files.
};
Thanks,
--
Mounir
On 12/07/13 17:31, Jonas Sicking wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters parameters);
};
interface Directory {
readonly attribute DOMString name;
PromiseFile createFile(DOMString path, MakeFileOptions options);
PromiseDirectory createDirectory(DOMString path);
Promise(File or Directory) get(DOMString path);
Promisevoid move((DOMString or File or Directory) entry,
Re: Polished FileSystem API proposal
On Sat, Jul 13, 2013 at 1:27 AM, David Rajchenbach-Teller
dtel...@mozilla.com wrote:
Why both createFile, open{Read, Write, Append} and get? Is it to avoid a
signature with dependent types?
I think you are asking why have createFile when it can be implemented
using openWrite or openAppend instead?
We believe that one of the most common operations is simply store
data X as a new file with filename Y. Having a simple function,
createFile, to accomplish this seems beneficial.
Compare
// With having creatFile:
navigator.getFilesystem().then(function(root) {
root.createFile(myfile.txt, { data: xhr.response });
});
// Only using openWrite
navigator.getFilesystem().then(function(root) {
return root.openWrite(myfile.txt);
}).then(function(handle) {
return Promise.every(
handle.setSize(0),
handle.write(xhr.response));
});
Does that answer the question?
/ Jonas
Cheers,
David
On 7/13/13 2:31 AM, Jonas Sicking wrote:
PromiseFile createFile(DOMString path, MakeFileOptions options);
PromiseDirectory createDirectory(DOMString path);
Promise(File or Directory) get(DOMString path);
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
};
--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla
Re: Polished FileSystem API proposal
Hi there!
A new file system API with a lot of promise :-)
On Sat, Jul 13, 2013 at 2:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters parameters);
};
interface Directory {
readonly attribute DOMString name;
PromiseFile createFile(DOMString path, MakeFileOptions options);
Why not CreateFileOptions? the method is called createFile and not makeFile
PromiseDirectory createDirectory(DOMString path);
You have a data as part of the MakeFileOptions, would that be useful here?
Promise(File or Directory) get(DOMString path);
Then shouldn't we have a convenience to see if a path is a directly or
not, like python has os.path.isdir(fileordirectoryname)
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
I don't like all these weird arguments like DeleteMode etc... just
make a separate method
removeRecursively(
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
Can't the user not just handle the creation in the error case of the
promise. or create it before... It would so very easy to create a
openWriteCreateIfNeeded method on top of the existing API. So such an
argument should only be needed if there would be performance benefits
- otherwise keep the API simple.
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
It is not obvious for me what that method does.
};
interface FileHandle
{
readonly attribute FileOpenMode mode;
readonly attribute boolean active;
attribute long long? location;
PromiseFile getFile();
AbortableProgressPromiseArrayBuffer read(unsigned long long size);
AbortableProgressPromiseDOMString readText(unsigned long long
size, optional DOMString encoding = utf-8);
void abort();
};
interface FileHandleWritable : FileHandle
{
AbortableProgressPromisevoid write((DOMString or ArrayBuffer or
ArrayBufferView or Blob) value);
Promisevoid setSize(optional unsigned long long size);
Promisevoid flush();
};
partial interface URL {
static DOMString? getPersistentURL(File file);
}
// WebIDL cruft that's largely transparent
enum PersistenceType { temporary, persistent };
Is temporary a kind of persistence? (sorry not native speaker)
dictionary FilesystemParameters {
PersistenceType storage = temporary;
};
dictionary MakeFileOptions {
boolean overwriteIfExists = false;
(DOMString or Blob or ArrayBuffer or ArrayBufferView) data;
};
enum CreateMode { createifneeded, dontcreate }
enum DeleteMode { recursive, nonrecursive }
dictionary DestinationDict {
Directory dir;
DOMString name;
};
enum FileOpenMode { read, write, append };
So this API introduces 2 classes: Directory and FileHandle. Directory
allows manipulation of the files and directories stored inside that
directory. FileHandle represents an exclusively opened file and allows
manipulation of the file contents.
The behavior is hopefully mostly obvious. A few general comments:
The functions on Directory that accept DOMString arguments for
filenames allow names like path/to/file. If the function creates a
file, then it creates the intermediate directories. Such paths are
always interpreted as relative to the directory itself, never relative
to the root.
We were thinking of *not* allowing paths that walk up the directory
tree. So paths like ../foo, .., /foo/bar or foo/../bar are not
allowed. This to keep things simple and avoid security issues for the
page.
Likewise, passing a File object to an operation of Directory where the
File object isn't contained in that
Re: Polished FileSystem API proposal
On Sat, Jul 13, 2013 at 2:37 AM, Kenneth Rohde Christiansen
kenneth.christian...@gmail.com wrote:
Hi there!
A new file system API with a lot of promise :-)
On Sat, Jul 13, 2013 at 2:31 AM, Jonas Sicking jo...@sicking.cc wrote:
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters parameters);
};
interface Directory {
readonly attribute DOMString name;
PromiseFile createFile(DOMString path, MakeFileOptions options);
Why not CreateFileOptions? the method is called createFile and not makeFile
Yup. I'll fix that.
PromiseDirectory createDirectory(DOMString path);
You have a data as part of the MakeFileOptions, would that be useful here?
What would it do?
Promise(File or Directory) get(DOMString path);
Then shouldn't we have a convenience to see if a path is a directly or
not, like python has os.path.isdir(fileordirectoryname)
The get method combines the check if exists, check if directory,
stat to get metadata functions into one. We could certainly add all
three, but it doesn't seem to really make the API easier to use, just
bigger.
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
I don't like all these weird arguments like DeleteMode etc... just
make a separate method
removeRecursively(
That's certainly an interesting idea. I'd personally prefer to just
have the recursive variant, but if we really need both then that's
probably better.
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
Can't the user not just handle the creation in the error case of the
promise. or create it before... It would so very easy to create a
openWriteCreateIfNeeded method on top of the existing API. So such an
argument should only be needed if there would be performance benefits
- otherwise keep the API simple.
Another question is if we need openWrite with support for not creating
the file if it's not there. You could likewise easily also create a
openWriteFailIfDoesntExist on top of existing API by using the get()
function.
I don't feel particularly strongly. Create-and-open-for-writing is
probably a decently common operation. Though createFile probably
covers the most common create-and-open-for-writing scenarios.
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
It is not obvious for me what that method does.
It recursively enumerates all files in this directory and all
subdirectories. This is useful for things like figuring out the total
size of files under a subdirectory, or sending a full directory tree
to the server.
/ Jonas
Re: Polished FileSystem API proposal
Why both createFile, open{Read, Write, Append} and get? Is it to avoid a
signature with dependent types?
Cheers,
David
On 7/13/13 2:31 AM, Jonas Sicking wrote:
PromiseFile createFile(DOMString path, MakeFileOptions options);
PromiseDirectory createDirectory(DOMString path);
Promise(File or Directory) get(DOMString path);
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
};
--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla
Polished FileSystem API proposal
Hi All,
Yesterday a few of us at mozilla went through the FileSystem API
proposal we previously sent [1] and tightened it up.
Executive Summary (aka TL;DR):
Below is the mozilla proposal for a simplified filesystem API. It
contains two new abstractions, a Directory object which allows
manipulating files and directories within it, and a FileHandle object
which allows holding an exclusive lock on a file while performing
multiple read/write operations on it.
It's largely modeled after posix, but because we've tried to keep it
author friendly despite it's asynchronous nature, it differs in a few
cases.
There are opportunities for further simplifications by straying
further from posix. It's unclear if this is desired or not.
Detailed proposal:
partial interface Navigator {
PromiseDirectory getFilesystem(optional FilesystemParameters parameters);
};
interface Directory {
readonly attribute DOMString name;
PromiseFile createFile(DOMString path, MakeFileOptions options);
PromiseDirectory createDirectory(DOMString path);
Promise(File or Directory) get(DOMString path);
Promisevoid move((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promisevoid copy((DOMString or File or Directory) entry,
(DOMString or Directory or DestinationDict) dest);
Promiseboolean remove((DOMString or File or Directory) path,
optional DeleteMode recursive = nonrecursive);
PromiseFileHandle openRead((DOMString or File) file);
PromiseFileHandleWritable openWrite((DOMString or File) file,
optional CreateMode createMode = createifneeded);
PromiseFileHandleWritable openAppend((DOMString or File) file,
optional CreateMode createMode = createifneeded);
EventStream(File or Directory) enumerate();
EventStreamFile enumerateDeep();
};
interface FileHandle
{
readonly attribute FileOpenMode mode;
readonly attribute boolean active;
attribute long long? location;
PromiseFile getFile();
AbortableProgressPromiseArrayBuffer read(unsigned long long size);
AbortableProgressPromiseDOMString readText(unsigned long long
size, optional DOMString encoding = utf-8);
void abort();
};
interface FileHandleWritable : FileHandle
{
AbortableProgressPromisevoid write((DOMString or ArrayBuffer or
ArrayBufferView or Blob) value);
Promisevoid setSize(optional unsigned long long size);
Promisevoid flush();
};
partial interface URL {
static DOMString? getPersistentURL(File file);
}
// WebIDL cruft that's largely transparent
enum PersistenceType { temporary, persistent };
dictionary FilesystemParameters {
PersistenceType storage = temporary;
};
dictionary MakeFileOptions {
boolean overwriteIfExists = false;
(DOMString or Blob or ArrayBuffer or ArrayBufferView) data;
};
enum CreateMode { createifneeded, dontcreate }
enum DeleteMode { recursive, nonrecursive }
dictionary DestinationDict {
Directory dir;
DOMString name;
};
enum FileOpenMode { read, write, append };
So this API introduces 2 classes: Directory and FileHandle. Directory
allows manipulation of the files and directories stored inside that
directory. FileHandle represents an exclusively opened file and allows
manipulation of the file contents.
The behavior is hopefully mostly obvious. A few general comments:
The functions on Directory that accept DOMString arguments for
filenames allow names like path/to/file. If the function creates a
file, then it creates the intermediate directories. Such paths are
always interpreted as relative to the directory itself, never relative
to the root.
We were thinking of *not* allowing paths that walk up the directory
tree. So paths like ../foo, .., /foo/bar or foo/../bar are not
allowed. This to keep things simple and avoid security issues for the
page.
Likewise, passing a File object to an operation of Directory where the
File object isn't contained in that directory or its descendents also
results in an error.
One thing that is probably not obvious is how the FileHandle.location
attribute works. This attribute is used by the read/readText/write
functions to select where the read or write operation starts. When
.read is called, it uses the current value of .location to determine
where the reading starts. It then fires off an asynchronous read
operation. It finally synchronously increases .location by the amount
of the 'size' argument before returning. Same thing for .write() and
.readText().
This means that the caller can simply set .location and then fire off
multiple read or write operations which automatically will happen
staggered in the file. It also means that the caller can set the
location for next operation by simply setting .location, or can check
the current location by simply getting .location.
Setting .location to null means go to the end.
Note that getting or setting .location does not need to synchronously
call seek, or do any IO operations, in the
