Re: File API - where are the missing parts?
On Tue, Feb 23, 2016 at 10:06 AM, Joshua Bell wrote: > I'm also very interested in hearing from other browser implementers; Chrome > is in the odd position of having made investments in related areas > (FileSystem API and FileWriter API) that did not see adoption in other > browsers, which is a disincentive to proceed here. I think we should really separate the type of APIs that Chrome implemented, which grants access to sandboxed filesystems, from APIs that lets a page interact with the user's filesystem. The two are generally much more different than they are similar. I think for mozilla implementing APIs which provide access to an origin-sandboxed filesystem is relatively small right now. I think we're much more interested in APIs which allow reading and writing to the user's filesystem. I couldn't give any timelines for when we'd implement any proposals, but I'd personally be happy to provide feedback to drafts. And I'd be even more happy to see other browsers experiment with the various UX challenges involved. / Jonas
Re: File API - where are the missing parts?
> > * Should permissions persist? If you're working in an editor and reload the > > tab, being hit with a flurry of permission prompts is less than ideal. But > > if you visit it again in a day or a year? And, similar to the "template" > > case above, what if you use a web-based editor to modify a file, then > > revisit the site a year later. > I don't think long persistence on file-location is a feasible idea. But the > option to choose persistence within a session seems a viable compromise. > You'll still need to click the dialog away once, but then you can work > uninterrupted. There is already a case similar to this with built in popup blockers in most browsers where you can allow once or always allow a similar approach could be taken with this. On Tue, Feb 23, 2016 at 4:50 PM, Florian Bösch wrote: > On Tue, Feb 23, 2016 at 7:06 PM, Joshua Bell wrote: >> >> On Tue, Feb 23, 2016 at 7:12 AM, Florian Bösch wrote: >>> >>> On Tue, Feb 23, 2016 at 2:48 AM, Jonas Sicking wrote: Is the last bullet here really accurate? How can you use existing APIs to listen to file modifications? >>> >>> I have not tested this on all UAs, but in Google Chrome what you can do >>> is to set an interval to check a files.lastModified date, and if a >>> modification is detected, read it in again with a FileReader and that works >>> fine. >> >> >> Huh... we should probably specify and/or fix that. > > Specify rather than fix, please. > There are also APIs implemented in several browsers for opening a whole directory of files from a webpage. This has been possible for some time in Chrome, and support was also recently added to Firefox and Edge. I'm not sure how interoperable these APIs are across browsers though :( >> >> >> IIRC, Edge's API[1] mimics Chrome's, and you can polyfill Firefox's API >> [2] on top of Chrome/Edge's[3]. So in theory if Firefox's pleases developers >> they can adopt the polyfill, and other browsers can transition to native >> support. >> >> [1] https://lists.w3.org/Archives/Public/public-wicg/2015Sep/.html >> [2] https://wicg.github.io/directory-upload/proposal.html >> [3] https://github.com/WICG/directory-upload/blob/gh-pages/polyfill.js >> >> ... or just read Ali's excellent summary: >> >> https://lists.w3.org/Archives/Public/public-webapps/2015AprJun/0245.html >> >> (But that's all a tangent to Florian's main use cases...) > > It's good to know this on a standards track. > >> True, but if we determine that permissions must be granted then the API >> needs to be designed to handle it, e.g. entry points to the API surface are >> through a requestPermission() API, everything is async, etc. > > > Ack > >> >> One concern is: what capabilities are granted by this action? Can the >> web-app re-save the file? Can it re-read the file? Does that permission >> persist across sessions? For example, if I save a document template from a >> site I would not expect the site to be able to read the file after I've >> edited with an unrelated file editor. >>> >>> Save many files to a user pickable folder: same as above >>> Working directory: this is more something that would go on in the >>> background of a UA, it would have to establish a "working directory" per tab >>> rather than UA-wide. No UX issues with that. >> >> Agreed. Likely doesn't even need to be specified - it'd just be a "least >> surprise" behavior by the UA. > > The save to directory case is less easy to handle because it impinges on > overwrite. After some thought, I'd move that to the more difficult UX cases. > >> >> * Since "File > Open" is supported today (via ) we must >> be careful about exposing functionality that has similar UX (i.e. a native >> file open dialog) but that implicitly grants extra permissions (e.g. being >> able to modify the file). This points to either additional UX during the >> action, UX when the app wants to write, or a more general permission granted >> to the origin for some scope (file? directory?). > > I'd think this to be a non formative note on implementation for UAs. The > mechanism of denying an action by the API should be fairly straightforward. > >> >> * Should permissions persist? If you're working in an editor and reload >> the tab, being hit with a flurry of permission prompts is less than ideal. >> But if you visit it again in a day or a year? And, similar to the "template" >> case above, what if you use a web-based editor to modify a file, then >> revisit the site a year later. > > I don't think long persistence on file-location is a feasible idea. But the > option to choose persistence within a session seems a viable compromise. > You'll still need to click the dialog away once, but then you can work > uninterrupted.
Re: File API - where are the missing parts?
On Tue, Feb 23, 2016 at 7:06 PM, Joshua Bell wrote: > On Tue, Feb 23, 2016 at 7:12 AM, Florian Bösch wrote: > >> On Tue, Feb 23, 2016 at 2:48 AM, Jonas Sicking wrote: >> >>> Is the last bullet here really accurate? How can you use existing APIs >>> to listen to file modifications? >>> >> I have not tested this on all UAs, but in Google Chrome what you can do >> is to set an interval to check a files.lastModified date, and if a >> modification is detected, read it in again with a FileReader and that works >> fine. >> > > Huh... we should probably specify and/or fix that. > Specify rather than fix, please. > There are also APIs implemented in several browsers for opening a whole >>> directory of files from a webpage. This has been possible for some time in >>> Chrome, and support was also recently added to Firefox and Edge. I'm not >>> sure how interoperable these APIs are across browsers though :( >>> >> > IIRC, Edge's API[1] mimics Chrome's, and you can polyfill Firefox's API > [2] on top of Chrome/Edge's[3]. So in theory if Firefox's pleases > developers they can adopt the polyfill, and other browsers can transition > to native support. > > [1] https://lists.w3.org/Archives/Public/public-wicg/2015Sep/.html > [2] https://wicg.github.io/directory-upload/proposal.html > [3] https://github.com/WICG/directory-upload/blob/gh-pages/polyfill.js > > ... or just read Ali's excellent summary: > > https://lists.w3.org/Archives/Public/public-webapps/2015AprJun/0245.html > > (But that's all a tangent to Florian's main use cases...) > It's good to know this on a standards track. True, but if we determine that permissions must be granted then the API > needs to be designed to handle it, e.g. entry points to the API surface are > through a requestPermission() API, everything is async, etc. > Ack > One concern is: what capabilities are granted by this action? Can the > web-app re-save the file? Can it re-read the file? Does that permission > persist across sessions? For example, if I save a document template from a > site I would not expect the site to be able to read the file after I've > edited with an unrelated file editor. > >> >>- Save many files to a user pickable folder: same as above >>- Working directory: this is more something that would go on in the >>background of a UA, it would have to establish a "working directory" per >>tab rather than UA-wide. No UX issues with that. >> >> Agreed. Likely doesn't even need to be specified - it'd just be a "least > surprise" behavior by the UA. > The save to directory case is less easy to handle because it impinges on overwrite. After some thought, I'd move that to the more difficult UX cases. > * Since "File > Open" is supported today (via ) we must > be careful about exposing functionality that has similar UX (i.e. a native > file open dialog) but that implicitly grants extra permissions (e.g. being > able to modify the file). This points to either additional UX during the > action, UX when the app wants to write, or a more general permission > granted to the origin for some scope (file? directory?). > I'd think this to be a non formative note on implementation for UAs. The mechanism of denying an action by the API should be fairly straightforward. > * Should permissions persist? If you're working in an editor and reload > the tab, being hit with a flurry of permission prompts is less than ideal. > But if you visit it again in a day or a year? And, similar to the > "template" case above, what if you use a web-based editor to modify a file, > then revisit the site a year later. > I don't think long persistence on file-location is a feasible idea. But the option to choose persistence within a session seems a viable compromise. You'll still need to click the dialog away once, but then you can work uninterrupted.
Re: File API - where are the missing parts?
On Tue, Feb 23, 2016 at 7:12 AM, Florian Bösch wrote: > On Tue, Feb 23, 2016 at 2:48 AM, Jonas Sicking wrote: >> >> Is the last bullet here really accurate? How can you use existing APIs to >> listen to file modifications? > > I have not tested this on all UAs, but in Google Chrome what you can do is > to set an interval to check a files.lastModified date, and if a modification > is detected, read it in again with a FileReader and that works fine. That is in violation of the spec. file.lastModified should be constant for the lifetime of the file object. Sadly the File object is grossly missnamed. It doesn't represent a OS filesystem file at all. It's more like large chunk of data with some metadata. Really we likely should have just had Blob and NamedBlob :( >> There are also APIs implemented in several browsers for opening a whole >> directory of files from a webpage. This has been possible for some time in >> Chrome, and support was also recently added to Firefox and Edge. I'm not >> sure how interoperable these APIs are across browsers though :( > > There does not seem to be a standard about this, or is there? It's an > essential functionality to be able to import OBJ and Collada files because > they are composites of the main file and other files (such as material > definitions or textures). There is no standard for this no. But there's clearly appetite for one given that all browsers now implement this functionality. >> However, before getting into such details, it is very important when >> discussing read/writing is to be precise about which files can be >> read/written. >> >> For example IndexedDB supports storing File (and Blob) objects inside >> IndexedDB. You can even get something very similar to incremental >> reads/writes by reading/writing slices. >> >> Here's a couple of libraries which implement filesystem APIs, which >> support incremental reading and writing, on top of IndexedDB: >> >> https://github.com/filerjs/filer >> https://github.com/ebidel/idb.filesystem.js >> >> However, IndexedDB, and thus any libraries built on top of it, only >> supports reading and writing files inside a origin-specific >> browser-sandboxed directory. >> >> This is also true for the the Filesystem API implemented in Google Chrome >> APIs that you are linking to. And it applies to the Filesystem API proposal >> at [1]. >> >> Writing files outside of any sandboxes requires not just an API >> specification, but also some sane, user understandable UX. >> >> So, to answer your questions, I would say: >> >> The APIs that you are linking to does not in fact meet the use cases that >> you are pointing to in your examples. Neither does [1], which is the closest >> thing that we have to a successor. >> >> The reason that no work has been done to meet the use cases that you are >> referring to, is that so far no credible solutions have been proposed for >> the UX problem. I.e. how do we make it clear to the user that they are >> granting access to the webpage to overwrite the contents of a file. >> >> [1] http://w3c.github.io/filesystem-api/ > > To be clear, I'm referring specifically to the ability of a user to pick any > destination on his mass-storage device to manage his data. I suspected you were. But you should then know that the Google Chrome proposals that you were referring to does not relate to those use cases at all. This is a very common misconception sadly. > I'm aware that there's thorny questions regarding UX (although UX itself is > rarely if ever specified in a W3C standard is it?). But that does not impact > all missing pieces. Notably not these: > > Save a file incrementally (and with the ability to abort): not a UX problem > because the mechanism to save files exists, it's just insufficiently > specified to allow for streaming writes. Indeed. This would be a solvable problem without introducing new UX concepts. The main missing piece here is likely the lack of Stream objects, which is what you'd want for incremental writing. Fortunately that part is much closer to a solved problem these days. So likely the main thing holding this use case up is simply having a proposal. > Save as pickable destination: also not a UX problem, the standard solution > here is to present the user with a standard operating system specific file > save dialog. I'm not sure I understand how this is different from what does? Is the problem that some browsers does not let the user choose directory to save in? It would be interesting to know why browsers don't let users choose directory for , and if they'd have concern supporting something like > Save many files to a user pickable folder: same as above > Working directory: this is more something that would go on in the background > of a UA, it would have to establish a "working directory" per tab rather > than UA-wide. No UX issues with that. I think there's a couple of UX concerns here. For normal save-as, we can ensure that the user is in full control over
Re: File API - where are the missing parts?
Thanks for starting this thread, Florian. I'm in broad agreement. On Tue, Feb 23, 2016 at 7:12 AM, Florian Bösch wrote: > On Tue, Feb 23, 2016 at 2:48 AM, Jonas Sicking wrote: > >> Is the last bullet here really accurate? How can you use existing APIs to >> listen to file modifications? >> > I have not tested this on all UAs, but in Google Chrome what you can do is > to set an interval to check a files.lastModified date, and if a > modification is detected, read it in again with a FileReader and that works > fine. > Huh... we should probably specify and/or fix that. > > >> There are also APIs implemented in several browsers for opening a whole >> directory of files from a webpage. This has been possible for some time in >> Chrome, and support was also recently added to Firefox and Edge. I'm not >> sure how interoperable these APIs are across browsers though :( >> > IIRC, Edge's API[1] mimics Chrome's, and you can polyfill Firefox's API [2] on top of Chrome/Edge's[3]. So in theory if Firefox's pleases developers they can adopt the polyfill, and other browsers can transition to native support. [1] https://lists.w3.org/Archives/Public/public-wicg/2015Sep/.html [2] https://wicg.github.io/directory-upload/proposal.html [3] https://github.com/WICG/directory-upload/blob/gh-pages/polyfill.js ... or just read Ali's excellent summary: https://lists.w3.org/Archives/Public/public-webapps/2015AprJun/0245.html (But that's all a tangent to Florian's main use cases...) > There does not seem to be a standard about this, or is there? It's an > essential functionality to be able to import OBJ and Collada files because > they are composites of the main file and other files (such as material > definitions or textures). > > >> Another important missing capability is the ability to modify an existing >> file. I.e. write 10 bytes in the middle of a 1GB file, without having to >> re-write the whole 1GB to disk. >> > Good point > > >> However, before getting into such details, it is very important when >> discussing read/writing is to be precise about which files can be >> read/written. >> >> For example IndexedDB supports storing File (and Blob) objects inside >> IndexedDB. You can even get something very similar to incremental >> reads/writes by reading/writing slices. >> >> Here's a couple of libraries which implement filesystem APIs, which >> support incremental reading and writing, on top of IndexedDB: >> >> https://github.com/filerjs/filer >> https://github.com/ebidel/idb.filesystem.js >> >> However, IndexedDB, and thus any libraries built on top of it, only >> supports reading and writing files inside a origin-specific >> browser-sandboxed directory. >> >> This is also true for the the Filesystem API implemented in Google Chrome >> APIs that you are linking to. And it applies to the Filesystem API proposal >> at [1]. >> >> Writing files outside of any sandboxes requires not just an API >> specification, but also some sane, user understandable UX. >> >> So, to answer your questions, I would say: >> >> The APIs that you are linking to does not in fact meet the use cases that >> you are pointing to in your examples. Neither does [1], which is the >> closest thing that we have to a successor. >> >> The reason that no work has been done to meet the use cases that you are >> referring to, is that so far no credible solutions have been proposed for >> the UX problem. I.e. how do we make it clear to the user that they are >> granting access to the webpage to overwrite the contents of a file. >> >> [1] http://w3c.github.io/filesystem-api/ >> > To be clear, I'm referring specifically to the ability of a user to pick > any destination on his mass-storage device to manage his data. This might > not be as sexy and easy as IndexDB & Co. but it's an essential > functionality for users to be able to organize their files to where they > want to have them, with the minimum of fuss. > > Yep, use cases acknowledged. I summarize them as: * "build a file editor" - you can't even build a non-terrible Notepad today, since "File > (Re)Save" and "File > Save As..." aren't supported, let alone performant (random access writes) * "build an IDE" - the above plus directory enumeration, file/directory watching, non-intrusive open/save. I agree these use cases are important, and I would like the platform to eventually support them both for native and sandboxed filesystems. I'm aware that there's thorny questions regarding UX (although UX itself is > rarely if ever specified in a W3C standard is it?). > True, but if we determine that permissions must be granted then the API needs to be designed to handle it, e.g. entry points to the API surface are through a requestPermission() API, everything is async, etc. > But that does not impact all missing pieces. Notably not these: > >- Save a file incrementally (and with the ability to abort): not a UX >problem because the mechanism to save files exists, it's just >insufficientl
Re: File API - where are the missing parts?
On Tue, Feb 23, 2016 at 2:48 AM, Jonas Sicking wrote: > Is the last bullet here really accurate? How can you use existing APIs to > listen to file modifications? > I have not tested this on all UAs, but in Google Chrome what you can do is to set an interval to check a files.lastModified date, and if a modification is detected, read it in again with a FileReader and that works fine. > There are also APIs implemented in several browsers for opening a whole > directory of files from a webpage. This has been possible for some time in > Chrome, and support was also recently added to Firefox and Edge. I'm not > sure how interoperable these APIs are across browsers though :( > There does not seem to be a standard about this, or is there? It's an essential functionality to be able to import OBJ and Collada files because they are composites of the main file and other files (such as material definitions or textures). > Another important missing capability is the ability to modify an existing > file. I.e. write 10 bytes in the middle of a 1GB file, without having to > re-write the whole 1GB to disk. > Good point > However, before getting into such details, it is very important when > discussing read/writing is to be precise about which files can be > read/written. > > For example IndexedDB supports storing File (and Blob) objects inside > IndexedDB. You can even get something very similar to incremental > reads/writes by reading/writing slices. > > Here's a couple of libraries which implement filesystem APIs, which > support incremental reading and writing, on top of IndexedDB: > > https://github.com/filerjs/filer > https://github.com/ebidel/idb.filesystem.js > > However, IndexedDB, and thus any libraries built on top of it, only > supports reading and writing files inside a origin-specific > browser-sandboxed directory. > > This is also true for the the Filesystem API implemented in Google Chrome > APIs that you are linking to. And it applies to the Filesystem API proposal > at [1]. > > Writing files outside of any sandboxes requires not just an API > specification, but also some sane, user understandable UX. > > So, to answer your questions, I would say: > > The APIs that you are linking to does not in fact meet the use cases that > you are pointing to in your examples. Neither does [1], which is the > closest thing that we have to a successor. > > The reason that no work has been done to meet the use cases that you are > referring to, is that so far no credible solutions have been proposed for > the UX problem. I.e. how do we make it clear to the user that they are > granting access to the webpage to overwrite the contents of a file. > > [1] http://w3c.github.io/filesystem-api/ > To be clear, I'm referring specifically to the ability of a user to pick any destination on his mass-storage device to manage his data. This might not be as sexy and easy as IndexDB & Co. but it's an essential functionality for users to be able to organize their files to where they want to have them, with the minimum of fuss. I'm aware that there's thorny questions regarding UX (although UX itself is rarely if ever specified in a W3C standard is it?). But that does not impact all missing pieces. Notably not these: - Save a file incrementally (and with the ability to abort): not a UX problem because the mechanism to save files exists, it's just insufficiently specified to allow for streaming writes. - Save as pickable destination: also not a UX problem, the standard solution here is to present the user with a standard operating system specific file save dialog. - Save many files to a user pickable folder: same as above - Working directory: this is more something that would go on in the background of a UA, it would have to establish a "working directory" per tab rather than UA-wide. No UX issues with that. Additionally this should be minimally UX controversial: - Overwrite a file (either previously saved or opened): I think it'd be a legitimate implementation of the UX to show an appropriate dialog at the time of overwrite that indicates what is overwritten, it's just a fast-track to save as pick file (and the UX can be improved by persistence of choice if that is deemed an acceptable risk). So it doesn't strike me that these missing features would create massive UX problems, indeed, most of them create no UX problem at all.
Re: File API - where are the missing parts?
On Sun, Feb 21, 2016 at 3:32 AM, Florian Bösch wrote:
>
> *What this covers*
>
>- Read one or many files in their entirety in one go (in python that
>would be open('foobar').read())
>- Save a completed binary string in its entirety in one go to the
>download folder (no overwrite)
>- Listen to file modifications
>
> Is the last bullet here really accurate? How can you use existing APIs to
listen to file modifications?
There are also APIs implemented in several browsers for opening a whole
directory of files from a webpage. This has been possible for some time in
Chrome, and support was also recently added to Firefox and Edge. I'm not
sure how interoperable these APIs are across browsers though :(
I would also add that existing APIs does allow partial reads. You can slice
a blob to get a smaller blob, and then read that. Slicing is a cheap
operation and should not copy any data.
Using this you can actually emulate incremental reads, though it is not as
easy as full streaming reads, and it might have some slight performance
differences.
> *What is missing*
>
>- Read a file incrementally (and with the ability to abort)
>- Save a file incrementally (and with the ability to abort)
>- Overwrite a file (either previously saved or opened)
>- Save as pickable destination
>- Save many files to a user pickable folder
>- Working directory
>
> Another important missing capability is the ability to modify an existing
file. I.e. write 10 bytes in the middle of a 1GB file, without having to
re-write the whole 1GB to disk.
However, before getting into such details, it is very important when
discussing read/writing is to be precise about which files can be
read/written.
For example IndexedDB supports storing File (and Blob) objects inside
IndexedDB. You can even get something very similar to incremental
reads/writes by reading/writing slices.
Here's a couple of libraries which implement filesystem APIs, which support
incremental reading and writing, on top of IndexedDB:
https://github.com/filerjs/filer
https://github.com/ebidel/idb.filesystem.js
However, IndexedDB, and thus any libraries built on top of it, only
supports reading and writing files inside a origin-specific
browser-sandboxed directory.
This is also true for the the Filesystem API implemented in Google Chrome
APIs that you are linking to. And it applies to the Filesystem API proposal
at [1].
Writing files outside of any sandboxes requires not just an API
specification, but also some sane, user understandable UX.
So, to answer your questions, I would say:
The APIs that you are linking to does not in fact meet the use cases that
you are pointing to in your examples. Neither does [1], which is the
closest thing that we have to a successor.
The reason that no work has been done to meet the use cases that you are
referring to, is that so far no credible solutions have been proposed for
the UX problem. I.e. how do we make it clear to the user that they are
granting access to the webpage to overwrite the contents of a file.
[1] http://w3c.github.io/filesystem-api/
/ Jonas
