Re: Moving File API: Directories and System API to Note track?
Glenn Maynard wrote: I'm interested in the same from Mozilla side: what are the real issues that you think are unsolvable, or do you just think the underlying use cases aren't compelling enough for the work required? Speaking for myself, not for all Mozillans here, I find the use-cases Eric U listed two messages ahead of yours in the thread compelling. I like the interface Maciej synthesized from prior designs. Filesystems are not databases and they have their uses. This is not a difficult proposition! Of course, when all you have is a database then everything looks like your thumb... :-| /be
RE: Moving File API: Directories and System API to Note track?
-Original Message- From: Olli Pettay [mailto:olli.pet...@helsinki.fi] Sent: Thursday, October 04, 2012 3:48 AM To: SULLIVAN, BRYAN L Cc: Maciej Stachowiak; Glenn Maynard; Eric U; public-webapps@w3.org Subject: Re: Moving File API: Directories and System API to Note track? On 09/26/2012 01:32 AM, SULLIVAN, BRYAN L wrote: *From:*Maciej Stachowiak [mailto:m...@apple.com] *Sent:* Tuesday, September 25, 2012 2:59 PM *To:* Glenn Maynard *Cc:* Eric U; o...@pettay.fi; public-webapps@w3.org *Subject:* Re: Moving File API: Directories and System API to Note track? Hi Glenn, I read over your points. But I don't think they would change Apple's calculation about exposing an API to the real user filesystem in Safari, particularly as specified. I do think that my more minimal API might also be a better fit for the real filesystem use case, as it removes a bunch of unnecessary levels of indirection and abstraction that exist in the other proposals. But I think it is unlikely we would expose that aspect. We are still open to solving the sandboxed local storage area use case, with a more minimal API. But first we'd like to understand why those use cases can't be solved with targeted additions to IndexedDB (see forked thread). Can IndexedDB support the use case of managing a media or document library on the local device, in which the size of library can run into hundreds or files and gigabytes of storage? I don't see any technical problems with that. Other than hard limits on storage per origin and how the virtual filesystem would be organized and managed, neither do I, but I am not an expert in browser design, that's why I asked the question. If there are no technical problems, anyone have thoughts on how soon we can expect to see browsers that provide a virtual filesystem implementation based upon Indexed DB? Local storage of media is one of the key capabilities needed to enable HTML5-based offline media players as a realistic option. If the browser can effectively provide a virtual filesystem with performance characteristics equivalent to the underlying OS, then perhaps IndexedDB may suffice for filesystem access. But which browsers intend to support that level of functionality via IndexexDB? Isn't that part of IndexedDB, to be able to store files efficiently in it. So I would assume all the implementations will support that. I don't think specific efficiency objectives are a normative requirement of Indexed DB, but the question was more whether browser vendors would claim that their Indexed DB implementation will provide performance equivalent to the underlying OS. Apart from the technical challenges, it's unrealistic to use Indexed DB for this purpose if the performance cannot match the underlying OS.
Re: Sandboxed Filesystem use cases? (was Re: Moving File API: Directories and System API to Note track?)
Asking about use cases that can be served by a filesystem API, but not by IDB, is reasonable [and I'll respond to it below], but it misses a lot of the point. The users I've talked to like the FS API because it's a simple interface that everyone already understands, that's powerful enough to handle a huge variety of use cases. Sure, the async API makes it a bit more complicated. Every API that handles large data is stuck with the same overhead there. But underneath that, people know what to expect from it and can figure it out very quickly. You just need to store 100KB? 1) Request a filesystem. 2) Open a file. 3) Write your data. Need a URL for that? Sure, it's just a file, so obviously that works. Want it organized in directories just like your server or dev environment? Go ahead. You don't have to write SQL queries, learn how to organize data into noSQL tables, or deal with version change transactions. If you want to see what's in your data store, you don't need to write a viewer to dump your tables; you just go to the URL of any directory in your store and browse around. Our URLs have a natural structure that matches the directory tree. If you add URLs to IDB, with its free-form key/value arrangement, I don't forsee an immediate natural mapping that doesn't involve lots of escaping, ugly URLs, and/or limitations. On to the use cases: Things that work well in a sandboxed filesystem that don't work well in IDB [or any of the other current storage APIs] are those that involve nontransactional modifications of large blobs of data. For example, video/photo/audio editing, which involve data that's too big to store lots of extra copies of for rollback of failed transactions, and which you don't necessarily want to try to fit into memory. Overwriting just the ID3 tag of an MP3, or just the comment section of the EXIF in a JPEG, would be much more efficient via a filesystem interface. Larger series of modifications to those files, which you don't want to hold in memory, would be similar. I know Jonas wants to bolt nontransactional data onto the side of IDB via FileHandle, but I think that the cure there is far worse than the disease, and I don't think anyone at Google likes that proposal. I haven't polled everyone, but that's the impression I get. Beyond individual use cases: When looking at use cases for a filesystem API, people often want to separate the sandboxed cases and the non-sandboxed cases [My Photos, etc.]. It's also worthwhile to look at the added value of having a single API that works for both cases. You have a photo organizer that works in the sandbox with downloaded files? If your browser supports external filesystems, you can adapt your code to run in either place with a very small change [mainly dealing with paths that aren't legal on the local system]. If you're using IDB in the sandbox, and have a different API to expose media directories, you've got to start over, and then you have to maintain both systems. One added API? It's pretty clear that people see the value of an API that lets one access My Photos from the web. That API is necessarily going to cope with files and directories on some platforms, even if others don't expose directories as such. If we're going to need to add a filesystem API of some kind to deal with that, also using the same API to manage a sandboxed storage area seems like a very small addition to the web platform, unlike the other storage APIs we've added in the past. Regarding your final note: I'm not sure what you're talking about with BlobBuilder; is that the EXIF overwrite case you're trying to handle? If so, File[Handle|Writer] with BlobBuilder and seek seems to handle it better than anything else. Eric On Tue, Sep 25, 2012 at 11:57 AM, Maciej Stachowiak m...@apple.com wrote: On Sep 25, 2012, at 10:20 AM, James Graham jgra...@opera.com wrote: In addition, this would be the fourth storage API that we have tried to introduce to the platform in 5 years (localStorage, WebSQL, IndexedDB being the other three), and the fifth in total. Of the four APIs excluding this one, one has failed over interoperability concerns (WebSQL), one has significant performance issues and is discouraged from production use (localStorage) and one suffers from a significant problems due to its legacy design (cookies). The remaining API (IndexedDB) has not yet achieved widespread use. It seems to me that we don't have a great track record in this area, and rushing to add yet another API probably isn't wise. I would rather see JS-level implementations of a filesystem-like API on top of IndexedDB in order to work out the kinks without creating a legacy that has to be maintained for back-compat than native implementations at this time. I share your concerns about adding yet-another-storage API. (Although I believe there are major websites that have adopted or are in the process of adopting IndexedDB). I like my
Re: Moving File API: Directories and System API to Note track?
Maciej Stachowiak wrote: On Sep 22, 2012, at 9:35 PM, Maciej Stachowiakm...@apple.com wrote: On Sep 22, 2012, at 8:18 PM, Brendan Eichbren...@mozilla.com wrote: And two of the interfaces are generic and reusable in other contexts. Nice, and DOMRequest predates yours -- should it be done separately since (I believe) it is being used by other proposals unrelated to FileSystem-like ones? Sorry if I missed it and it's already being split out. Yes, I borrowed DOMRequest. I think DOMRequest and DOMMultiRequest could be a separate spec, if that sort of asynchronous response pattern is generally useful. And it seems like it might be. That would leave only two interfaces specific to the Minimal File System proposal, Directory and FileHandle. Here's an alternate version where I renamed some things to match Filesystem API and FileWriter, and added the missing key feature of getting a persistent URL for a file in a local filesystem (extending the URL interface). It's still a much simpler API that provides most of the same functionality. https://trac.webkit.org/wiki/MinimalFileStorageAlternate Even better. What's the next step? I would hope we can make device storage be this, but that's just old Unix-hacker me (yes, no pathnames, you have to namei by calling get for each component -- that's a feature). /be
Re: Moving File API: Directories and System API to Note track?
On Tue, 25 Sep 2012, Brendan Eich wrote: Maciej Stachowiak wrote: On Sep 22, 2012, at 9:35 PM, Maciej Stachowiakm...@apple.com wrote: On Sep 22, 2012, at 8:18 PM, Brendan Eichbren...@mozilla.com wrote: And two of the interfaces are generic and reusable in other contexts. Nice, and DOMRequest predates yours -- should it be done separately since (I believe) it is being used by other proposals unrelated to FileSystem-like ones? Sorry if I missed it and it's already being split out. Yes, I borrowed DOMRequest. I think DOMRequest and DOMMultiRequest could be a separate spec, if that sort of asynchronous response pattern is generally useful. And it seems like it might be. That would leave only two interfaces specific to the Minimal File System proposal, Directory and FileHandle. Here's an alternate version where I renamed some things to match Filesystem API and FileWriter, and added the missing key feature of getting a persistent URL for a file in a local filesystem (extending the URL interface). It's still a much simpler API that provides most of the same functionality. https://trac.webkit.org/wiki/MinimalFileStorageAlternate Even better. What's the next step? I would hope we can make device storage be this, but that's just old Unix-hacker me (yes, no pathnames, you have to namei by calling get for each component -- that's a feature). *Personally* I like this proposal more than the original Google one, but I'm still have concerns. Every operation requiring a callback seems likely to lead to a pyramid of doom and unmaintainable code. Since this seems to be an increasing problem with certain classes of JS APIs (the SQL API had similar issues and I wouldn't be surprised if IndexedDB has too), it seems like we should work out some solution, either at the langauge level or at the DOM level, for managing this complexity rather than adding new features that require it. Obviously workers and a sync API are one option, but having to keep a worker around just to do storage operations also seems rather burdensome. In addition, this would be the fourth storage API that we have tried to introduce to the platform in 5 years (localStorage, WebSQL, IndexedDB being the other three), and the fifth in total. Of the four APIs excluding this one, one has failed over interoperability concerns (WebSQL), one has significant performance issues and is discouraged from production use (localStorage) and one suffers from a significant problems due to its legacy design (cookies). The remaining API (IndexedDB) has not yet achieved widespread use. It seems to me that we don't have a great track record in this area, and rushing to add yet another API probably isn't wise. I would rather see JS-level implementations of a filesystem-like API on top of IndexedDB in order to work out the kinks without creating a legacy that has to be maintained for back-compat than native implementations at this time.
Sandboxed Filesystem use cases? (was Re: Moving File API: Directories and System API to Note track?)
On Sep 25, 2012, at 10:20 AM, James Graham jgra...@opera.com wrote: In addition, this would be the fourth storage API that we have tried to introduce to the platform in 5 years (localStorage, WebSQL, IndexedDB being the other three), and the fifth in total. Of the four APIs excluding this one, one has failed over interoperability concerns (WebSQL), one has significant performance issues and is discouraged from production use (localStorage) and one suffers from a significant problems due to its legacy design (cookies). The remaining API (IndexedDB) has not yet achieved widespread use. It seems to me that we don't have a great track record in this area, and rushing to add yet another API probably isn't wise. I would rather see JS-level implementations of a filesystem-like API on top of IndexedDB in order to work out the kinks without creating a legacy that has to be maintained for back-compat than native implementations at this time. I share your concerns about adding yet-another-storage API. (Although I believe there are major websites that have adopted or are in the process of adopting IndexedDB). I like my version better than the Google one, too, but I also worry about whether we should be adding another storage API at all. I think we need to go back to the use case for sandboxed filesystem storage and understand which use cases cannot be served with IndexedDB. Here are some use cases I have heard: (1) A webapp (possibly working on offline mode) wants to stage files for later upload (e.g. via XHR). Requirements: - Must be able to store distinct named items containing arbitrary binary data. - Must be able to read the data back for later upload. - Must be able to delete items. (2) A web-based mail client wants to download the user's attachments locally, then reference them by URL from the email and allow them to be extracted into the user's filesystem space. Requirements: - Must be able to store distinct named items containing arbitrary binary data. - Must be able to reference items by persistent URL from constructs in a webpage that use URLs. - Must be able to delete items. (3) A web-based web developer tool downloads copies of all the resources of a webpage, lets the user edit the webpage live potentially adding new resources, and then uploads it all again to one or more servers. Requirements: - Must be able to store distinct named items containing arbitrary binary data. - Must be able to replace items. - Must be able to reference items by persistent URL from constructs in a webpage that use URLs. - Must be able to delete items. - Must be able to enumerate items. Highly desirable: - Hierarchical namespace. (4) A game wants to download game resources locally for efficient operation, and later update them Requirements: - Must be able to store distinct named items containing arbitrary binary data. - Must be able to replace items. - Must be able to reference items by persistent URL from constructs in a webpage that use URLs. - Must be able to delete items. Highly desirable: - Hierarchical namespace. I believe the only requirement here that is not met by IndexedDB is: - The ability to reference an item by persistent URL. IndexedDB has enumeration, hierarchical namespace, ability to add, replace, remove, get, etc. Are there other use cases? In particular, are there use cases that justify a whole new storage API instead of adding this one feature to IndexedDB? Note: one aspect of the MinimalFileSystem proposal that is not obviously required by any of these use cases is the ability to incrementally update a file (beyond what you could already do with slice() and BlobBuilder). Basically the whole FileHandle interface. Is there truly a use case that you can't satisfy by using BlobBuilder to make your update and then atomically replacing? Regards, Maciej
Re: Moving File API: Directories and System API to Note track?
Hi Glenn, I read over your points. But I don't think they would change Apple's calculation about exposing an API to the real user filesystem in Safari, particularly as specified. I do think that my more minimal API might also be a better fit for the real filesystem use case, as it removes a bunch of unnecessary levels of indirection and abstraction that exist in the other proposals. But I think it is unlikely we would expose that aspect. We are still open to solving the sandboxed local storage area use case, with a more minimal API. But first we'd like to understand why those use cases can't be solved with targeted additions to IndexedDB (see forked thread). Regards, Maciej On Sep 25, 2012, at 12:35 PM, Glenn Maynard gl...@zewt.org wrote: On Wed, Sep 19, 2012 at 3:46 PM, James Graham jgra...@opera.com wrote: Indeed. We are not enthusiastic about implementing an API that has to traverse directory trees as this has significant technical challenges, or may expose user's path names, as this has security implications. Also AIUI this API is not a good fit for all platforms. I don't think there are any unsolvable problems for traversing native directory trees. (For example, hard links can be dealt with if they do turn up--keep track of inodes in parents--and pathname limitations put an upper limit, preventing infinite recursion. I think they're very unlikely in practice--infinite directory loops would break tons of native apps, too. No filesystem I've tried in Linux allows it.) I don't think any proposals expose user pathnames (eg. above the item that was dropped/opened). The main issue that needs to be explored is how to prevent users from accidentally giving more access than they mean to, eg. pages saying please open C:\ and drag in Users, and users not realizing what they're doing. This is critically important, but I think it's too early to conclude that this is unsolvable. (That's a UI issue more than a technical issue. There are other clearly technical issues, but I haven't seen any raised that look unsolvable. The two tricky ones I've seen are maximum filename/path length limitations, and valid characters varying between platforms, which have been discussed a lot and would need to be revisited--it's been too long and I forget where that left off.) On Fri, Sep 21, 2012 at 7:37 PM, Maciej Stachowiak m...@apple.com wrote: - I'm not keen on exposing portions of the user's filesystem. In particular: - This seems like a potential security/social-engineering risk. - This use case seems more relevant to system apps based on Web technology than Web apps as such; I fear that system-app-motivated complexity is bloating the api. These are the *primary* use cases of a filesystem-like API. There's lot of value in being able to let a media player app play and (optionally) manipulate my local media directories, for example--obviously I won't move my music into IndexedDB. (I don't find the sandboxed case very compelling, and while I consider IndexedDB to-be-proven for things like storing gigabytes of data for a game, it should definitely be given that chance before introducing another API for that set of use cases.) - Many of Apple's key target platforms don't even *have* a user-visible filesystem. It's not inconsistent to have APIs that aren't relevant on every single platform. When you don't have a user filesystem in the first place, the use cases themselves go away and you don't care about any of this. (For the sandboxed case, this isn't relevant. It would be useful to separate the sandboxed and native discussions, since except for the API style question, the issues and objections are almost completely distinct, and it's sometimes hard to keep track of which one we're talking about.) - We already have way too many storage APIs. To add another, the use cases would have to be overwhelmingly compelling, and highly impractical to serve by extending any of the other storage APIs (e.g. IndexedDB). I do find being able to allow users to work with their native data, just as native apps do, to be overwhelmingly compelling. I have about 4 TB of data, and the only way I can use any of it with web apps is by dragging in individual, flat lists of files, and even that's a strictly one-way street (FileSaver is only a very tiny help here). -- Glenn Maynard
Re: Moving File API: Directories and System API to Note track?
On Sat, Sep 22, 2012 at 8:32 AM, Eric U er...@google.com wrote: While I don't see any other browsers showing interest in implementing the FileSystem API as currently specced, Just for the record, Blackberry, Tizen/EFL and Netfront seem to have some support of FileSystem API. EFL has also uploaded a new FileSystem API patch just a few days ago. http://html5test.com/compare/feature/files-fileSystem.html PhoneGap's File API is mostly compatible to the current FileSystem API. http://docs.phonegap.com/en/1.5.0/phonegap_file_file.md.html#File I do see Firefox coming around to the belief that a filesystem-style API is a good thing, hence their DeviceStorage API. Rather than scrap the API that we've put 2 years of discussion and work into, why not work with us to evolve it to something you'd like more? If you have objections to specific attributes of the API, wouldn't it be more efficient to change just those things than to start over from scratch? Or worse, to have the Chrome filesystem API, the Firefox filesystem API, etc.? If I understand correctly, folks at Mozilla think having a directory abstraction is too heavy-weight, and would prefer users to slice and dice paths by hand. OK, that's a small change, and the functionality's roughly equivalent. We could probably even make migration fairly easy with a small polyfill. Jonas suggests FileHandle to replace FileWriter. That's clearly not a move to greater simplicity, and no polyfill is possible, but it does open up the potential for higher perfomance, especially in a multi-process browser. As i said when you proposed it, I'm interested, and we'd also like to solve the locking use cases. Let's talk about it, rather than throw the baby out with the bathwater. Eric On Tue, Sep 18, 2012 at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Moving File API: Directories and System API to Note track?
On Sep 22, 2012, at 9:35 PM, Maciej Stachowiak m...@apple.com wrote: On Sep 22, 2012, at 8:18 PM, Brendan Eich bren...@mozilla.com wrote: And two of the interfaces are generic and reusable in other contexts. Nice, and DOMRequest predates yours -- should it be done separately since (I believe) it is being used by other proposals unrelated to FileSystem-like ones? Sorry if I missed it and it's already being split out. Yes, I borrowed DOMRequest. I think DOMRequest and DOMMultiRequest could be a separate spec, if that sort of asynchronous response pattern is generally useful. And it seems like it might be. That would leave only two interfaces specific to the Minimal File System proposal, Directory and FileHandle. Here's an alternate version where I renamed some things to match Filesystem API and FileWriter, and added the missing key feature of getting a persistent URL for a file in a local filesystem (extending the URL interface). It's still a much simpler API that provides most of the same functionality. https://trac.webkit.org/wiki/MinimalFileStorageAlternate Regards, Maciej
Re: Moving File API: Directories and System API to Note track?
What does getMetadata a synchronously return? I think this API as written is still a fair bit more complex than needed for the sandboxed storage use case. It does seem simpler than Filesystem API. Regards, Maciej On Sep 21, 2012, at 10:10 PM, Jonas Sicking jo...@sicking.cc wrote: On Fri, Sep 21, 2012 at 5:37 PM, Maciej Stachowiak m...@apple.com wrote: My personal objections (ones that I think are shared by at least some other Safari folks): - It's way too complicated. (As one crude metric, I count 22 interfaces; and yes, I know many of those are callback interfaces or sync versions of interfaces; it still seems overengineered). - I see value in the use case of a private sandboxed storage area, as staging for uploads, to hold downloaded resources, etc. But: - It's not totally clear to me why a filesystemlike API is the best way to serve these use cases, particularly if in practice it is actually backed by a database. - The API as designed is way too complex for just that use case. - I'm not keen on exposing portions of the user's filesystem. In particular: - This seems like a potential security/social-engineering risk. - This use case seems more relevant to system apps based on Web technology than Web apps as such; I fear that system-app-motivated complexity is bloating the api. - Many of Apple's key target platforms don't even *have* a user-visible filesystem. - We'd like to keep Web APIs consistent between our various platforms consistent as much as possible. - I think trying to serve the real filesystem use cases adds a lot of complexity over the private storage area use case. - We already have way too many storage APIs. To add another, the use cases would have to be overwhelmingly compelling, and highly impractical to serve by extending any of the other storage APIs (e.g. IndexedDB). - In particular, I have heard an explanation of why IndexedDB as it currently exists can't handle all the use cases for file-like storage, but I have heard no explanation of why it can't be extended in that direction. For these reasons, I think it is unlikely that Safari would ever support Filesystem API in its current form. I could imagine considering a *much* narrower API, scoped only to the use case of private storage areas for Web apps, but only if a compelling case is made that there's no other way to serve that use case. For what it's worth, even the DeviceStorage API proposal is too complex for my tastes, in its current iteration. I think keeping the Filseystem API on the REC track in its current form is actively bad, because it leads outside observers to be misled about where the Web platform is going. For example, sites like http://html5test.com give out points for it even though it seems unlikely to advance on the standards track as it stands today. For what it's worth, I put together a draft for what an API would look like that has basically the same feature set as the current FileSystem API, but based on DeviceStorage. It's a much smaller API that the current FileSystem drafts, but supports things like shallow as well as deep directory iteration. https://wiki.mozilla.org/WebAPI/DeviceStorageAPI2 I think that if we at mozilla were to implement a sandboxed filesystem, it'd be something more like this. The FileHandle part of the API is definitely more complex to implement than the FileWriter API, but I'd argue that it's actually easier to use. For example you can start multiple write operations without having to wait for each individual write to finish. I also added an example of what a read-only DeviceStorage would look like if we wanted something like that for input type=file, drag'n'drop or a zip-reader. This shouldn't be considered as part of the remaining draft though since exposing a filesystem in those scenarios might very well be overkill. / Jonas
Re: Moving File API: Directories and System API to Note track?
On Sep 21, 2012, at 10:10 PM, Jonas Sicking jo...@sicking.cc wrote: For what it's worth, I put together a draft for what an API would look like that has basically the same feature set as the current FileSystem API, but based on DeviceStorage. It's a much smaller API that the current FileSystem drafts, but supports things like shallow as well as deep directory iteration. https://wiki.mozilla.org/WebAPI/DeviceStorageAPI2 I think that if we at mozilla were to implement a sandboxed filesystem, it'd be something more like this. I took a crack at a pruned and cleaned up version: https://trac.webkit.org/wiki/MinimalFileStorage Only 4 interfaces (excluding the Navigator addition), 16 methods, 9 attributes. And two of the interfaces are generic and reusable in other contexts. It removes cruft but adds features relative to the Mozilla version: - Atomic create/open operation - Renaming files and directories - A Directory interface for less weirdness in directory handling - Ability to open files in append mode (this makes more sense than an append() operation on the handle, given the way underlying filesystems work) Features omitted include: - Multiple named filesystems (just use directories) - Separate enumeration for writing (just use open) - Ability to fetch matadata (does not appear to be needed for the use case) It could be simplified a little bit, but really not that much, by removing hierarchy. Regards, Maciej
Re: Moving File API: Directories and System API to Note track?
Maciej Stachowiak wrote: On Sep 21, 2012, at 10:10 PM, Jonas Sickingjo...@sicking.cc wrote: For what it's worth, I put together a draft for what an API would look like that has basically the same feature set as the current FileSystem API, but based on DeviceStorage. It's a much smaller API that the current FileSystem drafts, but supports things like shallow as well as deep directory iteration. https://wiki.mozilla.org/WebAPI/DeviceStorageAPI2 I think that if we at mozilla were to implement a sandboxed filesystem, it'd be something more like this. I took a crack at a pruned and cleaned up version: https://trac.webkit.org/wiki/MinimalFileStorage Only 4 interfaces (excluding the Navigator addition), 16 methods, 9 attributes. Nice! And two of the interfaces are generic and reusable in other contexts. Nice, and DOMRequest predates yours -- should it be done separately since (I believe) it is being used by other proposals unrelated to FileSystem-like ones? Sorry if I missed it and it's already being split out. It removes cruft but adds features relative to the Mozilla version: - Atomic create/open operation As an old Unix hacker, this warms my heart. (I ignore the camelCaps method multiplication to avoid O_ flags, and the spelled-out creatE ;-) - Renaming files and directories Atomic rename, must have. - A Directory interface for less weirdness in directory handling +1 - Ability to open files in append mode (this makes more sense than an append() operation on the handle, given the way underlying filesystems work) Yup. Features omitted include: - Multiple named filesystems (just use directories) Exactly. Do we still call them mount points? - Separate enumeration for writing (just use open) Yup. - Ability to fetch matadata (does not appear to be needed for the use case) Whatsa mata with matadata? :-P Ok, defer. Fine by me unless Sicking has a pressing use-case to articulate. It could be simplified a little bit, but really not that much, by removing hierarchy. Hierarchy is much of the point here. I like the way this is going. I'm curious how things look in hindsight to Google folks. Was the whole of FileSystem truly needed? Even if whole properly excludes the synchronous APIs? /be
Re: Moving File API: Directories and System API to Note track?
On Sep 22, 2012, at 8:18 PM, Brendan Eich bren...@mozilla.com wrote: And two of the interfaces are generic and reusable in other contexts. Nice, and DOMRequest predates yours -- should it be done separately since (I believe) it is being used by other proposals unrelated to FileSystem-like ones? Sorry if I missed it and it's already being split out. Yes, I borrowed DOMRequest. I think DOMRequest and DOMMultiRequest could be a separate spec, if that sort of asynchronous response pattern is generally useful. And it seems like it might be. That would leave only two interfaces specific to the Minimal File System proposal, Directory and FileHandle. Regards, Maciej
Paths exposed in input type=file (was: Re: Moving File API: Directories and System API to Note track?)
On 09/20/2012 11:45 PM, Darin Fisher wrote: File path information is already exposed via input type=file multiple. File names may contain partial paths. http://www.whatwg.org/specs/web-apps/current-work/multipage/states-of-the-type-attribute.html#concept-input-type-file-selected I couldn't get any actual browser to expose paths in this way. Did I miss something? It also seems to be encouraging wilful violation of the FileAPI spec which clearly states that File objects don't return path info [1]. Having a may-level conformance criterion that supposes to override a must-level criterion in another spec, but no strong compat. requirement, doesn't seem like the optimum path to interoperability or sanity of people trying to understand and implement the platform. Is there any reason this paragraph shouldn't be considered a bug in HTML? [1] http://dev.w3.org/2006/webapi/FileAPI/#dfn-file
Re: Moving File API: Directories and System API to Note track?
No comment on the value of DirectoryEntry for enabling asynchronous listing of files in input type=file multiple? -Darin On Thu, Sep 20, 2012 at 4:48 PM, Maciej Stachowiak m...@apple.com wrote: +1 I don't see an indication of any major browser but Chrome planning to implement this and expose it to the Web. - Maciej On Sep 18, 2012, at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Paths exposed in input type=file (was: Re: Moving File API: Directories and System API to Note track?)
On Fri, Sep 21, 2012 at 1:14 AM, James Graham jgra...@opera.com wrote: On 09/20/2012 11:45 PM, Darin Fisher wrote: File path information is already exposed via input type=file multiple. File names may contain partial paths. http://www.whatwg.org/specs/**web-apps/current-work/** multipage/states-of-the-type-**attribute.html#concept-input-** type-file-selectedhttp://www.whatwg.org/specs/web-apps/current-work/multipage/states-of-the-type-attribute.html#concept-input-type-file-selected I couldn't get any actual browser to expose paths in this way. Did I miss something? It also seems to be encouraging wilful violation of the FileAPI spec which clearly states that File objects don't return path info [1]. Having a may-level conformance criterion that supposes to override a must-level criterion in another spec, but no strong compat. requirement, doesn't seem like the optimum path to interoperability or sanity of people trying to understand and implement the platform. Is there any reason this paragraph shouldn't be considered a bug in HTML? I'm not sure. Note: Chrome exposes relative file path information via File.webkitRelativePath. I haven't reviewed the history of this property. Exposing partial path information like this is useful in conjunction with input type=file directory [*]. It enables people to upload directories of files to a web server. The web server can re-create the exact directory structure, including empty directories. Regards, -Darin [*] http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2010-April/025764.html [1] http://dev.w3.org/2006/webapi/**FileAPI/#dfn-filehttp://dev.w3.org/2006/webapi/FileAPI/#dfn-file
Re: Moving File API: Directories and System API to Note track?
Or.. drag-n-drop receipt of a folder or set of folders. On Fri, Sep 21, 2012 at 3:50 PM, Darin Fisher da...@chromium.org wrote: No comment on the value of DirectoryEntry for enabling asynchronous listing of files in input type=file multiple? -Darin On Thu, Sep 20, 2012 at 4:48 PM, Maciej Stachowiak m...@apple.com wrote: +1 I don't see an indication of any major browser but Chrome planning to implement this and expose it to the Web. - Maciej On Sep 18, 2012, at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Moving File API: Directories and System API to Note track?
While I don't see any other browsers showing interest in implementing the FileSystem API as currently specced, I do see Firefox coming around to the belief that a filesystem-style API is a good thing, hence their DeviceStorage API. Rather than scrap the API that we've put 2 years of discussion and work into, why not work with us to evolve it to something you'd like more? If you have objections to specific attributes of the API, wouldn't it be more efficient to change just those things than to start over from scratch? Or worse, to have the Chrome filesystem API, the Firefox filesystem API, etc.? If I understand correctly, folks at Mozilla think having a directory abstraction is too heavy-weight, and would prefer users to slice and dice paths by hand. OK, that's a small change, and the functionality's roughly equivalent. We could probably even make migration fairly easy with a small polyfill. Jonas suggests FileHandle to replace FileWriter. That's clearly not a move to greater simplicity, and no polyfill is possible, but it does open up the potential for higher perfomance, especially in a multi-process browser. As i said when you proposed it, I'm interested, and we'd also like to solve the locking use cases. Let's talk about it, rather than throw the baby out with the bathwater. Eric On Tue, Sep 18, 2012 at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Moving File API: Directories and System API to Note track?
I like the idea of offering asynchronous listing of files in input type=file multiple. But I think Filesystem API is overkill for this use case. Regards, Maciej On Sep 21, 2012, at 3:50 PM, Darin Fisher da...@chromium.org wrote: No comment on the value of DirectoryEntry for enabling asynchronous listing of files in input type=file multiple? -Darin On Thu, Sep 20, 2012 at 4:48 PM, Maciej Stachowiak m...@apple.com wrote: +1 I don't see an indication of any major browser but Chrome planning to implement this and expose it to the Web. - Maciej On Sep 18, 2012, at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Moving File API: Directories and System API to Note track?
My personal objections (ones that I think are shared by at least some other Safari folks): - It's way too complicated. (As one crude metric, I count 22 interfaces; and yes, I know many of those are callback interfaces or sync versions of interfaces; it still seems overengineered). - I see value in the use case of a private sandboxed storage area, as staging for uploads, to hold downloaded resources, etc. But: - It's not totally clear to me why a filesystemlike API is the best way to serve these use cases, particularly if in practice it is actually backed by a database. - The API as designed is way too complex for just that use case. - I'm not keen on exposing portions of the user's filesystem. In particular: - This seems like a potential security/social-engineering risk. - This use case seems more relevant to system apps based on Web technology than Web apps as such; I fear that system-app-motivated complexity is bloating the api. - Many of Apple's key target platforms don't even *have* a user-visible filesystem. - We'd like to keep Web APIs consistent between our various platforms consistent as much as possible. - I think trying to serve the real filesystem use cases adds a lot of complexity over the private storage area use case. - We already have way too many storage APIs. To add another, the use cases would have to be overwhelmingly compelling, and highly impractical to serve by extending any of the other storage APIs (e.g. IndexedDB). - In particular, I have heard an explanation of why IndexedDB as it currently exists can't handle all the use cases for file-like storage, but I have heard no explanation of why it can't be extended in that direction. For these reasons, I think it is unlikely that Safari would ever support Filesystem API in its current form. I could imagine considering a *much* narrower API, scoped only to the use case of private storage areas for Web apps, but only if a compelling case is made that there's no other way to serve that use case. For what it's worth, even the DeviceStorage API proposal is too complex for my tastes, in its current iteration. I think keeping the Filseystem API on the REC track in its current form is actively bad, because it leads outside observers to be misled about where the Web platform is going. For example, sites like http://html5test.com give out points for it even though it seems unlikely to advance on the standards track as it stands today. Regards, Maciej On Sep 21, 2012, at 4:32 PM, Eric U er...@google.com wrote: While I don't see any other browsers showing interest in implementing the FileSystem API as currently specced, I do see Firefox coming around to the belief that a filesystem-style API is a good thing, hence their DeviceStorage API. Rather than scrap the API that we've put 2 years of discussion and work into, why not work with us to evolve it to something you'd like more? If you have objections to specific attributes of the API, wouldn't it be more efficient to change just those things than to start over from scratch? Or worse, to have the Chrome filesystem API, the Firefox filesystem API, etc.? If I understand correctly, folks at Mozilla think having a directory abstraction is too heavy-weight, and would prefer users to slice and dice paths by hand. OK, that's a small change, and the functionality's roughly equivalent. We could probably even make migration fairly easy with a small polyfill. Jonas suggests FileHandle to replace FileWriter. That's clearly not a move to greater simplicity, and no polyfill is possible, but it does open up the potential for higher perfomance, especially in a multi-process browser. As i said when you proposed it, I'm interested, and we'd also like to solve the locking use cases. Let's talk about it, rather than throw the baby out with the bathwater. Eric On Tue, Sep 18, 2012 at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Moving File API: Directories and System API to Note track?
On Fri, Sep 21, 2012 at 5:37 PM, Maciej Stachowiak m...@apple.com wrote: My personal objections (ones that I think are shared by at least some other Safari folks): - It's way too complicated. (As one crude metric, I count 22 interfaces; and yes, I know many of those are callback interfaces or sync versions of interfaces; it still seems overengineered). - I see value in the use case of a private sandboxed storage area, as staging for uploads, to hold downloaded resources, etc. But: - It's not totally clear to me why a filesystemlike API is the best way to serve these use cases, particularly if in practice it is actually backed by a database. - The API as designed is way too complex for just that use case. - I'm not keen on exposing portions of the user's filesystem. In particular: - This seems like a potential security/social-engineering risk. - This use case seems more relevant to system apps based on Web technology than Web apps as such; I fear that system-app-motivated complexity is bloating the api. - Many of Apple's key target platforms don't even *have* a user-visible filesystem. - We'd like to keep Web APIs consistent between our various platforms consistent as much as possible. - I think trying to serve the real filesystem use cases adds a lot of complexity over the private storage area use case. - We already have way too many storage APIs. To add another, the use cases would have to be overwhelmingly compelling, and highly impractical to serve by extending any of the other storage APIs (e.g. IndexedDB). - In particular, I have heard an explanation of why IndexedDB as it currently exists can't handle all the use cases for file-like storage, but I have heard no explanation of why it can't be extended in that direction. For these reasons, I think it is unlikely that Safari would ever support Filesystem API in its current form. I could imagine considering a *much* narrower API, scoped only to the use case of private storage areas for Web apps, but only if a compelling case is made that there's no other way to serve that use case. For what it's worth, even the DeviceStorage API proposal is too complex for my tastes, in its current iteration. I think keeping the Filseystem API on the REC track in its current form is actively bad, because it leads outside observers to be misled about where the Web platform is going. For example, sites like http://html5test.com give out points for it even though it seems unlikely to advance on the standards track as it stands today. For what it's worth, I put together a draft for what an API would look like that has basically the same feature set as the current FileSystem API, but based on DeviceStorage. It's a much smaller API that the current FileSystem drafts, but supports things like shallow as well as deep directory iteration. https://wiki.mozilla.org/WebAPI/DeviceStorageAPI2 I think that if we at mozilla were to implement a sandboxed filesystem, it'd be something more like this. The FileHandle part of the API is definitely more complex to implement than the FileWriter API, but I'd argue that it's actually easier to use. For example you can start multiple write operations without having to wait for each individual write to finish. I also added an example of what a read-only DeviceStorage would look like if we wanted something like that for input type=file, drag'n'drop or a zip-reader. This shouldn't be considered as part of the remaining draft though since exposing a filesystem in those scenarios might very well be overkill. / Jonas
Re: Moving File API: Directories and System API to Note track?
On Wed, 19 Sep 2012, Adam Barth wrote: On Wed, Sep 19, 2012 at 1:46 PM, James Graham jgra...@opera.com wrote: On Wed, 19 Sep 2012, Edward O'Connor wrote: Olli wrote: I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Sounds good to me. Indeed. We are not enthusiastic about implementing an API that has to traverse directory trees as this has significant technical challenges, or may expose user's path names, as this has security implications. Also AIUI this API is not a good fit for all platforms. There's nothing in the spec that exposes user paths. That's just FUD. I was thinking specifically of the combination of this and Drag and Drop and this API. I assumed that at some level one would end up with a bunch on Entry objects which seem to expose a path. It then seems that then a user who is tricked into dragging their root drive onto a webapp would expose all their paths. It is quite possible that this is a horrible misunderstanding of the spec, and if so I apologise. Nevertheless I think it's poor form to immediately characterise an error as a deliberate attempt to spread lies. In any case my central point remains which is that would support this spec moving off the Rec. track at this time.
Re: Moving File API: Directories and System API to Note track?
On Wed, Sep 19, 2012 at 11:50 PM, James Graham jgra...@opera.com wrote: On Wed, 19 Sep 2012, Adam Barth wrote: On Wed, Sep 19, 2012 at 1:46 PM, James Graham jgra...@opera.com wrote: On Wed, 19 Sep 2012, Edward O'Connor wrote: Olli wrote: I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Sounds good to me. Indeed. We are not enthusiastic about implementing an API that has to traverse directory trees as this has significant technical challenges, or may expose user's path names, as this has security implications. Also AIUI this API is not a good fit for all platforms. There's nothing in the spec that exposes user paths. That's just FUD. I was thinking specifically of the combination of this and Drag and Drop and this API. I assumed that at some level one would end up with a bunch on Entry objects which seem to expose a path. It then seems that then a user who is tricked into dragging their root drive onto a webapp would expose all their paths. It is quite possible that this is a horrible misunderstanding of the spec, and if so I apologise. Nevertheless I think it's poor form to immediately characterise an error as a deliberate attempt to spread lies. It just has nothing to do with the spec. It's like complaining that DOMString might leak user paths because if you use a DOMString with drag and drop, you might leak user paths. Adam
Re: Moving File API: Directories and System API to Note track?
On Wed, Sep 19, 2012 at 11:50 PM, James Graham jgra...@opera.com wrote: On Wed, 19 Sep 2012, Adam Barth wrote: On Wed, Sep 19, 2012 at 1:46 PM, James Graham jgra...@opera.com wrote: On Wed, 19 Sep 2012, Edward O'Connor wrote: Olli wrote: I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Sounds good to me. Indeed. We are not enthusiastic about implementing an API that has to traverse directory trees as this has significant technical challenges, or may expose user's path names, as this has security implications. Also AIUI this API is not a good fit for all platforms. There's nothing in the spec that exposes user paths. That's just FUD. I was thinking specifically of the combination of this and Drag and Drop and this API. I assumed that at some level one would end up with a bunch on Entry objects which seem to expose a path. It then seems that then a user who is tricked into dragging their root drive onto a webapp would expose all their paths. It is quite possible that this is a horrible misunderstanding of the spec, and if so I apologise. Nevertheless I think it's poor form to immediately characterise an error as a deliberate attempt to spread lies. In any case my central point remains which is that would support this spec moving off the Rec. track at this time. File path information is already exposed via input type=file multiple. File names may contain partial paths. http://www.whatwg.org/specs/web-apps/current-work/multipage/states-of-the-type-attribute.html#concept-input-type-file-selected -Darin
Re: Moving File API: Directories and System API to Note track?
+1 I don't see an indication of any major browser but Chrome planning to implement this and expose it to the Web. - Maciej On Sep 18, 2012, at 4:04 AM, Olli Pettay olli.pet...@helsinki.fi wrote: Hi all, I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Mainly because the API hasn't been widely accepted nor implemented and also because there are other proposals which handle the same use cases. The problem with keeping the API in recommendation track is that people outside standardization world think that the API is the one which all the browsers will implement and as of now that doesn't seem likely. -Olli
Re: Moving File API: Directories and System API to Note track?
Note: The {File,Directory}Entry types are also separately useful for
multi-file input and drag-n-drop applications:
http://wiki.whatwg.org/wiki/DragAndDropEntries
To summarize, it solves several problems with FileList:
1- A DirectoryEntry can be lazily resolved by the developer. With
FileList, the browser has to fully traverse a potentially deep directory
structure before it can provide the application with the FileList. That
can be very slow. DirectoryEntry enables applications to provide progress
UI.
2- FileList is not good at representing empty directories. Empty
directories can be important for cases where a user wishes to upload a
directory structure to a server.
(This has been implemented in Chrome for a little while now.)
My understand of the sysapps working group was that it was for APIs that
are not generally suitable for use in the conventional web sandbox. They
are for use in a different (system apps) sandbox. What I've described
above is useful in both, and so standardizing in webapps seems to make
sense.
Regards,
-Darin
On Tue, Sep 18, 2012 at 4:11 AM, Robin Berjon ro...@w3.org wrote:
On 18/09/2012 13:04 , Olli Pettay wrote:
I think we should discuss about moving File API: Directories and
System API from Recommendation track to Note. Mainly because the API
hasn't been widely accepted nor implemented and also because there
are other proposals which handle the same use cases. The problem with
keeping the API in recommendation track is that people outside
standardization world think that the API is the one which all the
browsers will implement and as of now that doesn't seem likely.
I wonder if a good case couldn't be made to move it to the upcoming
SysApps WG
(http://www.w3.org/2012/05/**sysapps-wg-charter.htmlhttp://www.w3.org/2012/05/sysapps-wg-charter.html
)?
It's very late to sneak anything into the charter, but I reckon it could
fall under their Media Storage API deliverable.
--
Robin Berjon - http://berjon.com/ - @robinberjon
Re: Moving File API: Directories and System API to Note track?
On Wed, 19 Sep 2012, Edward O'Connor wrote: Hi, Olli wrote: I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Sounds good to me. Indeed. We are not enthusiastic about implementing an API that has to traverse directory trees as this has significant technical challenges, or may expose user's path names, as this has security implications. Also AIUI this API is not a good fit for all platforms.
Re: Moving File API: Directories and System API to Note track?
On Wed, Sep 19, 2012 at 1:46 PM, James Graham jgra...@opera.com wrote: On Wed, 19 Sep 2012, Edward O'Connor wrote: Olli wrote: I think we should discuss about moving File API: Directories and System API from Recommendation track to Note. Sounds good to me. Indeed. We are not enthusiastic about implementing an API that has to traverse directory trees as this has significant technical challenges, or may expose user's path names, as this has security implications. Also AIUI this API is not a good fit for all platforms. There's nothing in the spec that exposes user paths. That's just FUD. Adam
