Re: ZIP archive API?
On Mon, May 6, 2013 at 6:36 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 6, 2013 at 8:01 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 6, 2013 at 5:15 PM, Glenn Maynard gl...@zewt.org wrote: I'm not aware of any optimized inflate implementation in JS to compare against, and it's a complex algorithm, so nobody is likely to jump forward to spend a lot of time implementing and heavily optimizing it just to show how slow it is. I've seen an implementation around somewhere, but it didn't use typed arrays so it would need a lot of reworking to have any meaning. Likewise, I don't see any browser vendor jumping ahead and doing both the work to implement a library *and* and API to compare the two. Sorry, this didn't make sense. What library *and* API are you talking about? To compare what? A library for zip compression/decompression and an API for zip compression/decompression. So that we can compare the performance of the two. Every browser already has native inflate, though. This is unfortunately not a terribly strong argument. Exposing that implementation through a DOM API requires a fairly large amount of work. Not to add maintaining that over the years. You're arguing for allowing accessing files inside ZIPs by URL, which means you're going to have to do the work anyway, since you'd be able to create a blob URL, reference a file inside it using XHR, and get a Blob as a result. This is a small subset of that. No, the work to write and maintain an API for ZIP compress/decompression is pretty orthogonal, implementation-wise, to a protocol handler for ZIP decompression. Look, the fact that we need use cases should not be surprising to people on this mailing list. And I know that it's not surprising to you. One of the heavy arguments that has been brought up for a ZIP archive API has been that it provides better performance. That's a claim that needs to be backed up by numbers in order to carry any weight. The argument that you're bringing up is it's easy to implement (because you already have an zip library) is also a very weak argument. Until we get stronger arguments for an archive API, I'd be happy to see a proposal for a protocol handler. / Jonas
Re: ZIP archive API?
On Tue, May 7, 2013 at 8:09 AM, Jonas Sicking jo...@sicking.cc wrote: You're arguing for allowing accessing files inside ZIPs by URL, which means you're going to have to do the work anyway, since you'd be able to create a blob URL, reference a file inside it using XHR, and get a Blob as a result. This is a small subset of that. No, the work to write and maintain an API for ZIP compress/decompression is pretty orthogonal, implementation-wise, to a protocol handler for ZIP decompression. In order to implement zip you would use one of the ready made libraries supporting it, such as libzip (http://www.nih.at/libzip/index.html), minizip (http://www.winimage.com/zLibDll/minizip.html), Both libraries (and any other library you might encounter) already defines an API and an implementation. In order to support ZIP URLs you would use the API. In order to support a JS zip API, you would expose the API. These are not orthogonal, they are correlated.
Re: ZIP archive API?
* Florian Bösch wrote: It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Convenience is the first reason, since browsers by far and large already have complete bindings to compression algorithms and archive formats, letting the browser simply expose the software it already ships makes good sense rather than requiring every JS user to supply his own version. Speed may not matter to much on some platforms, but it matters a great deal on underpowered devices such as mobiles. Integration is where the support for archives goes beyond being an API, where URLs (to link.href, script.src, img.src, iframe.src, audio.src, video.src, css url(), etc.) could point into an archive. This cannot be done in JS. If we all agreed that such functionality should be provided by libraries rather than browser cores, and browser vendors could be expected to make things convenient, fast, and well-integrated, then using libraries would be convenient, libraries would be fast, and they would integrate well. -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
Re: ZIP archive API?
On 03/05/2013 21:05 , Florian Bösch wrote: It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Also, one of the reasons we compress things is because they're big.* Unpacking in JS is likely to mean unpacking to memory (unless the blobs are smarter than that), whereas the browser has access to strategies to mitigate this, e.g. using temporary files. Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. * Captain Obvious to the rescue! -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: ZIP archive API?
The main reason to use an archive (other than the space-savings) for me is to be able to transfer tens of thousands of small items that go into producing WebGL applications of non trivial scope. On Mon, May 6, 2013 at 1:27 PM, Robin Berjon ro...@w3.org wrote: On 03/05/2013 21:05 , Florian Bösch wrote: It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Also, one of the reasons we compress things is because they're big.* Unpacking in JS is likely to mean unpacking to memory (unless the blobs are smarter than that), whereas the browser has access to strategies to mitigate this, e.g. using temporary files. Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. * Captain Obvious to the rescue! -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: ZIP archive API?
On Mon, May 6, 2013 at 6:27 AM, Robin Berjon ro...@w3.org wrote: Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. Zips are streamable. That's what the local file headers are for. http://www.pkware.com/documents/casestudies/APPNOTE.TXT -- Glenn Maynard
Re: ZIP archive API?
I second that. Thanks Florian.
On 05/03/2013 02:52 PM, Florian Bösch wrote:
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in
filesystem API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to
selectively choose how they want to handle the data, I'd like to see
synchronous and asynchronous versions of each. I'd make synchronicity
an argument/flag or something to avoid API clutter like packSync,
packAsync, writeSync, writeAsync, and rather like write(data,
callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
over the top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to
use: http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of
types (Dataurl, Blob, URL object, File or Arraybuffer) and for
asynchronous operation needs to rely on a worker, which is bothersome
to setup: http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and
works synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nl
mailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com
mailto:pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to
transfer sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
smime.p7s
Description: S/MIME Cryptographic Signature
Re: ZIP archive API?
On Mon, May 6, 2013 at 5:03 AM, Glenn Maynard gl...@zewt.org wrote: On Mon, May 6, 2013 at 6:27 AM, Robin Berjon ro...@w3.org wrote: Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. Zips are streamable. That's what the local file headers are for. http://www.pkware.com/documents/casestudies/APPNOTE.TXT This came up a few years ago; Gregg Tavares explained in [1] that only /some/ zipfiles are streamable, and you don't know whether yours are or not until you've seen the whole file. Eric [1] http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0362.html
Re: ZIP archive API?
On Mon, May 6, 2013 at 4:27 AM, Robin Berjon ro...@w3.org wrote: On 03/05/2013 21:05 , Florian Bösch wrote: It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Also, one of the reasons we compress things is because they're big.* Unpacking in JS is likely to mean unpacking to memory (unless the blobs are smarter than that), whereas the browser has access to strategies to mitigate this, e.g. using temporary files. There's nothing here that implementations can do that JS can't. Both can read arbitrary parts of a file into memory and decompress only that data. And both would have to load data into memory in order to decompress it. The only things that implementations can do that JS can't is: * Implement new protocols. I definitely agree that we should specify a jar: or archive: protocol, but that's orthogonal to whether we need an API. * Implement Blob backends. In our prototype implementation we were able to return a Blob which represented data that hadn't been compressed yet. When the data was read it was incrementally decompressed. But a JS implementation could incrementally decompress and write the data to disk using something like FileHandle or FileWriter. So the same capabilities are there. Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. Indeed. This is IMO an argument for relying on libraries. Implementations are going to be a lot more conservative about adding new archive formats, since they have to be supported for eternity, than library authors will be. I'm still hoping to see some performance numbers from the people arguing that we should add this to the platform. Without that I see little hope of getting enough browser vendors behind this. / Jonas
Re: ZIP archive API?
On Mon, May 6, 2013 at 7:42 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 6, 2013 at 4:27 AM, Robin Berjon ro...@w3.org wrote: On 03/05/2013 21:05 , Florian Bösch wrote: It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Also, one of the reasons we compress things is because they're big.* Unpacking in JS is likely to mean unpacking to memory (unless the blobs are smarter than that), whereas the browser has access to strategies to mitigate this, e.g. using temporary files. There's nothing here that implementations can do that JS can't. Both can read arbitrary parts of a file into memory and decompress only that data. And both would have to load data into memory in order to decompress it. The only things that implementations can do that JS can't is: * Implement new protocols. I definitely agree that we should specify a jar: or archive: protocol, but that's orthogonal to whether we need an API. Is ZIP a protocol or simply a media type? Either way, browsers will need to implement operations on the archive format to understand its use in URL attributes. The question then becomes: if browsers have an implementation of the format for reference purposes, why do web apps need to ship their own implementations of the format for archive purposes? If web apps want to use some other archive/compression format, of course they will have to distribute an implementation. Thanks, David * Implement Blob backends. In our prototype implementation we were able to return a Blob which represented data that hadn't been compressed yet. When the data was read it was incrementally decompressed. But a JS implementation could incrementally decompress and write the data to disk using something like FileHandle or FileWriter. So the same capabilities are there. Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. Indeed. This is IMO an argument for relying on libraries. Implementations are going to be a lot more conservative about adding new archive formats, since they have to be supported for eternity, than library authors will be. I'm still hoping to see some performance numbers from the people arguing that we should add this to the platform. Without that I see little hope of getting enough browser vendors behind this. / Jonas
Re: ZIP archive API?
On 03.05.13 15:18, Jonas Sicking jo...@sicking.cc wrote: On Fri, May 3, 2013 at 12:12 PM, Paul Bakaus pbak...@zynga.com wrote: From: Florian Bösch pya...@gmail.com Date: Fri, 3 May 2013 21:05:17 +0200 To: Jonas Sicking jo...@sicking.cc Cc: Paul Bakaus pbak...@zynga.com, Anne van Kesteren ann...@annevk.nl, Webapps WG public-webapps@w3.org, Charles McCathie Nevile cha...@yandex-team.ru, Andrea Marchesini amarches...@mozilla.com Subject: Re: ZIP archive API? It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Convenience is the first reason, since browsers by far and large already have complete bindings to compression algorithms and archive formats, letting the browser simply expose the software it already ships makes good sense rather than requiring every JS user to supply his own version. Speed may not matter to much on some platforms, but it matters a great deal on underpowered devices such as mobiles. Show me some numbers to back this up and you'll have me convinced :) Remember that on underpowered devices native code is proportionally slower too. Integration is where the support for archives goes beyond being an API, where URLs (to link.href, script.src, img.src, iframe.src, audio.src, video.src, css url(), etc.) could point into an archive. This cannot be done in JS. I was going to say exactly that. I want to be able to have a virtual URL that I can point to. In my CSS, I want to do something like archive://assets/foo.png after I loaded and decompressed the ZIP file in JS. How does the assets part in the example above work? What does it mean? Is there some registry here or something? Actually assets was just referring to the package name. So what you're saying below makes total sense, and is what we want to have. Great stuff! Jonas, I'm intrigued do you see a way this could be done in JS? If so, maybe we should build a sample. I'm still thinking the performance won't be good enough, particular on mobile devices, but let's find out. You can actually do this in Gecko already. Any archive that you can refer to through a URL, you can also reach into. So if you have a .zip file in a Blob, and you generate a blob: URL like blob:123-abc then you can read the foo.html file out of that archive by using the URL jar:blob:123-abc!/foo.html. So far this doesn't work with ArrayBuffers since there is no way to have a URL that refers to an ArrayBuffer. You can even load something from inside a zip file from a server by doing img src=jar:http://example.com/foo/archive.zip!/image.jpg; Something like that I definitely agree that we should standardize. / Jonas On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote: The big question we kept running up against at Mozilla is why couldn't this simply be implemented as a JS library? If performance is the argument we need to back that up with data. / Jonas On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote: Hi Anne, Florian, I think the first baby step, or MVP, is the unpacking that Florian mentions below. I would definitely like to have the API available on both workers and normal context. Thanks, Paul From: Florian Bösch pya...@gmail.com Date: Fri, 3 May 2013 14:52:36 +0200 To: Anne van Kesteren ann...@annevk.nl Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org, Andrea Marchesini amarches...@mozilla.com Subject: Re: ZIP archive API? I'm interested a JS API that does the following: Unpacking: - Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem API) or Arraybuffer - List its content and metadata - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer Packing: - Create an archive - Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer To avoid the whole worker/proxy thing and to allow authors to selectively choose how they want to handle the data, I'd like to see synchronous and asynchronous versions of each. I'd make synchronicity an argument/flag or something to avoid API clutter like packSync, packAsync, writeSync, writeAsync, and rather like write(data, callback|boolean). - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the top: http://docs.python.org/3/library/zipfile.html - Pythons tarfile API is less clutered and easier to use: http://docs.python.org/3/library/tarfile.html - zip.js isn't really usable as it doesn't support the full range of types (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation needs to rely on a worker, which is bothersome to setup: http://stuk.github.io/jszip/ My own implementation of the tar format only targets array buffers and works synchronously, as in. var archive = new TarFile
Re: ZIP archive API?
On Mon, May 6, 2013 at 1:11 PM, Eric U er...@google.com wrote: This came up a few years ago; Gregg Tavares explained in [1] that only /some/ zipfiles are streamable, and you don't know whether yours are or not until you've seen the whole file. Eric [1] http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0362.html The file format is streamable. You can create files that follow the spec that will fail when streaming, but you can also create files that follow the spec that will fail when not streaming. (The end of central directory record sometimes has data after it, so you have to do a search; there's no spec defining how far you have to search, so if you put too much data there it'll start to fail.) Those are both problems with the spec that would have to be addressed. I don't think there's any reason to support tar (and it would significantly complicate the API, since tar *only* supports streaming). The bigger point here is that the ZIP appnote isn't enough. It doesn't define parsers or error handling. This means that defining an API to expose ZIPs isn't only a matter of defining an API, somebody will need to spec the file format itself. Also, the appnote isn't free, so this would probably need to be a clean-room spec. (However, it wouldn't need to specify all of the features of the format, a huge number of which are never used, only how to parse past them and ignore them.) On Mon, May 6, 2013 at 1:42 PM, Jonas Sicking jo...@sicking.cc wrote: Another question to take into account here is whether this should only be about zip. One of the limitations of zip archives is that they aren't streamable. Without boiling the ocean, adding support for a streamable format (which I don't think needs be more complex than tar) would be a big plus. Indeed. This is IMO an argument for relying on libraries. It's not. ZIP has been around longer than PNG and JPEG; its only real competors are tar.gz (which isn't useful here) and RAR (proprietary). It's not going away and there's no indication of a sudden influx of competing file formats, any more than image formats. That said, I don't know if a ZIP API is worthwhile. I'd start lower level here, and think about supporting inflating blobs. That's the same functionality any ZIP API will want, and it's the main part of the ZIP format that you really don't want to have to do in script. The surface area is also far simpler: new InflatedBlob(compressedBlob) I'm still hoping to see some performance numbers from the people arguing that we should add this to the platform. Without that I see little hope of getting enough browser vendors behind this. I'm not aware of any optimized inflate implementation in JS to compare against, and it's a complex algorithm, so nobody is likely to jump forward to spend a lot of time implementing and heavily optimizing it just to show how slow it is. I've seen an implementation around somewhere, but it didn't use typed arrays so it would need a lot of reworking to have any meaning. Every browser already has native inflate, though. -- Glenn Maynard
Re: ZIP archive API?
On Mon, May 6, 2013 at 5:15 PM, Glenn Maynard gl...@zewt.org wrote: I'm not aware of any optimized inflate implementation in JS to compare against, and it's a complex algorithm, so nobody is likely to jump forward to spend a lot of time implementing and heavily optimizing it just to show how slow it is. I've seen an implementation around somewhere, but it didn't use typed arrays so it would need a lot of reworking to have any meaning. Likewise, I don't see any browser vendor jumping ahead and doing both the work to implement a library *and* and API to compare the two. Every browser already has native inflate, though. This is unfortunately not a terribly strong argument. Exposing that implementation through a DOM API requires a fairly large amount of work. Not to add maintaining that over the years. / Jonas
Re: ZIP archive API?
On Mon, May 6, 2013 at 8:01 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 6, 2013 at 5:15 PM, Glenn Maynard gl...@zewt.org wrote: I'm not aware of any optimized inflate implementation in JS to compare against, and it's a complex algorithm, so nobody is likely to jump forward to spend a lot of time implementing and heavily optimizing it just to show how slow it is. I've seen an implementation around somewhere, but it didn't use typed arrays so it would need a lot of reworking to have any meaning. Likewise, I don't see any browser vendor jumping ahead and doing both the work to implement a library *and* and API to compare the two. Sorry, this didn't make sense. What library *and* API are you talking about? To compare what? Every browser already has native inflate, though. This is unfortunately not a terribly strong argument. Exposing that implementation through a DOM API requires a fairly large amount of work. Not to add maintaining that over the years. You're arguing for allowing accessing files inside ZIPs by URL, which means you're going to have to do the work anyway, since you'd be able to create a blob URL, reference a file inside it using XHR, and get a Blob as a result. This is a small subset of that. -- Glenn Maynard
Re: ZIP archive API?
On 03/05/2013 13:40 , Arthur Barstow wrote: Other than Mozilla, is there any other implementor interest? I'm wondering if steering this work to a CG would be `best` for now, especially if no one steps up to be Editor. I think that this is useful and I would rather it were inside a group than pushed off to a CG. It's not experimental in nature. -- Robin Berjon - http://berjon.com/ - @robinberjon
Re: ZIP archive API?
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote: Still waiting for it as well. I think it'd be very useful to transfer sets of assets etc. Do you have anything in particular you'd like to see happen first? It's pretty clear we should expose more here, but as with all things we should do it in baby steps. -- http://annevankesteren.nl/
Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in
filesystem API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync,
writeAsync, and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over
the top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
operation needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and
works synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer
sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian mentions
below. I would definitely like to have the API available on both workers and
normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Charles McCathie
Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem
API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync, writeAsync,
and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the
top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation
needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and works
synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
The big question we kept running up against at Mozilla is why couldn't
this simply be implemented as a JS library?
If performance is the argument we need to back that up with data.
/ Jonas
On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian
mentions below. I would definitely like to have the API available on both
workers and normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile
cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org, Andrea
Marchesini amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in
filesystem API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to
selectively choose how they want to handle the data, I'd like to see
synchronous and asynchronous versions of each. I'd make synchronicity an
argument/flag or something to avoid API clutter like packSync, packAsync,
writeSync, writeAsync, and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
over the top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
operation needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and
works synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nlwrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer
sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
It can be implemented by a JS library, but the three reasons to let the
browser provide it are Convenience, speed and integration.
Convenience is the first reason, since browsers by far and large already
have complete bindings to compression algorithms and archive formats,
letting the browser simply expose the software it already ships makes good
sense rather than requiring every JS user to supply his own version.
Speed may not matter to much on some platforms, but it matters a great deal
on underpowered devices such as mobiles.
Integration is where the support for archives goes beyond being an API,
where URLs (to link.href, script.src, img.src, iframe.src, audio.src,
video.src, css url(), etc.) could point into an archive. This cannot be
done in JS.
On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote:
The big question we kept running up against at Mozilla is why couldn't
this simply be implemented as a JS library?
If performance is the argument we need to back that up with data.
/ Jonas
On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian
mentions below. I would definitely like to have the API available on both
workers and normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile
cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org,
Andrea Marchesini amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in
filesystem API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to
selectively choose how they want to handle the data, I'd like to see
synchronous and asynchronous versions of each. I'd make synchronicity an
argument/flag or something to avoid API clutter like packSync, packAsync,
writeSync, writeAsync, and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
over the top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of
types (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
operation needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and
works synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nlwrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer
sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 21:05:17 +0200
To: Jonas Sicking jo...@sicking.ccmailto:jo...@sicking.cc
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Anne van
Kesteren ann...@annevk.nlmailto:ann...@annevk.nl, Webapps WG
public-webapps@w3.orgmailto:public-webapps@w3.org, Charles McCathie Nevile
cha...@yandex-team.rumailto:cha...@yandex-team.ru, Andrea Marchesini
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?
It can be implemented by a JS library, but the three reasons to let the browser
provide it are Convenience, speed and integration.
Convenience is the first reason, since browsers by far and large already have
complete bindings to compression algorithms and archive formats, letting the
browser simply expose the software it already ships makes good sense rather
than requiring every JS user to supply his own version.
Speed may not matter to much on some platforms, but it matters a great deal on
underpowered devices such as mobiles.
Integration is where the support for archives goes beyond being an API, where
URLs (to link.href, script.src, img.src, iframe.src, audio.src, video.src, css
url(), etc.) could point into an archive. This cannot be done in JS.
I was going to say exactly that. I want to be able to have a virtual URL that
I can point to. In my CSS, I want to do something like
archive://assets/foo.png after I loaded and decompressed the ZIP file in JS.
Jonas, I'm intrigued – do you see a way this could be done in JS? If so, maybe
we should build a sample. I'm still thinking the performance won't be good
enough, particular on mobile devices, but let's find out.
On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking
jo...@sicking.ccmailto:jo...@sicking.cc wrote:
The big question we kept running up against at Mozilla is why couldn't this
simply be implemented as a JS library?
If performance is the argument we need to back that up with data.
/ Jonas
On May 3, 2013 10:51 AM, Paul Bakaus
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian mentions
below. I would definitely like to have the API available on both workers and
normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Charles McCathie
Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem
API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync, writeAsync,
and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the
top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation
needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and works
synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
On Fri, May 3, 2013 at 12:12 PM, Paul Bakaus pbak...@zynga.com wrote:
From: Florian Bösch pya...@gmail.com
Date: Fri, 3 May 2013 21:05:17 +0200
To: Jonas Sicking jo...@sicking.cc
Cc: Paul Bakaus pbak...@zynga.com, Anne van Kesteren ann...@annevk.nl,
Webapps WG public-webapps@w3.org, Charles McCathie Nevile
cha...@yandex-team.ru, Andrea Marchesini amarches...@mozilla.com
Subject: Re: ZIP archive API?
It can be implemented by a JS library, but the three reasons to let the
browser provide it are Convenience, speed and integration.
Convenience is the first reason, since browsers by far and large already
have complete bindings to compression algorithms and archive formats,
letting the browser simply expose the software it already ships makes good
sense rather than requiring every JS user to supply his own version.
Speed may not matter to much on some platforms, but it matters a great deal
on underpowered devices such as mobiles.
Show me some numbers to back this up and you'll have me convinced :)
Remember that on underpowered devices native code is proportionally slower too.
Integration is where the support for archives goes beyond being an API,
where URLs (to link.href, script.src, img.src, iframe.src, audio.src,
video.src, css url(), etc.) could point into an archive. This cannot be
done in JS.
I was going to say exactly that. I want to be able to have a virtual URL
that I can point to. In my CSS, I want to do something like
archive://assets/foo.png after I loaded and decompressed the ZIP file in
JS.
How does the assets part in the example above work? What does it
mean? Is there some registry here or something?
Jonas, I'm intrigued – do you see a way this could be done in JS? If so,
maybe we should build a sample. I'm still thinking the performance won't be
good enough, particular on mobile devices, but let's find out.
You can actually do this in Gecko already. Any archive that you can
refer to through a URL, you can also reach into.
So if you have a .zip file in a Blob, and you generate a blob: URL
like blob:123-abc then you can read the foo.html file out of that
archive by using the URL jar:blob:123-abc!/foo.html. So far this
doesn't work with ArrayBuffers since there is no way to have a URL
that refers to an ArrayBuffer.
You can even load something from inside a zip file from a server by doing
img src=jar:http://example.com/foo/archive.zip!/image.jpg;
Something like that I definitely agree that we should standardize.
/ Jonas
On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote:
The big question we kept running up against at Mozilla is why couldn't
this simply be implemented as a JS library?
If performance is the argument we need to back that up with data.
/ Jonas
On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian
mentions below. I would definitely like to have the API available on both
workers and normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile
cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org, Andrea
Marchesini amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in
filesystem API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync,
writeAsync, and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
over the top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of
types (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
operation needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and
works synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nl
wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote
Re: ZIP archive API?
On May 3, 2013, at 3:18 PM, Jonas Sicking jo...@sicking.cc wrote: platforms, but it matters a great deal on underpowered devices such as mobiles. Show me some numbers to back this up and you'll have me convinced :) Remember that on underpowered devices native code is proportionally slower too. The word around town is that asm.js takes about 2x longer than native compiled code. So if we're looking at inflate and deflate, you've got that metric, and you have the memory and bandwidth overhead of including a js-based inflate deflate library and maintaining whatever licensing it comes with. Basic zip construction without compression is trivial to support in JS, and in some cases, it's desirable if you're targeting specific cases where the file format uses some magic numbers. -Charles
Re: ZIP archive API?
On May 3, 2013 4:20 PM, Charles Pritchard ch...@jumis.com wrote: On May 3, 2013, at 3:18 PM, Jonas Sicking jo...@sicking.cc wrote: platforms, but it matters a great deal on underpowered devices such as mobiles. Show me some numbers to back this up and you'll have me convinced :) Remember that on underpowered devices native code is proportionally slower too. The word around town is that asm.js takes about 2x longer than native compiled code. So how long does the actual compression/decompression take for the size of archives web developers usually deal with? IO aside given that that is the same in JS and native. / Jonas
Re: ZIP archive API?
Still waiting for it as well. I think it'd be very useful to transfer sets of assets etc. From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com Date: Tue, 30 Apr 2013 15:58:22 +0200 To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl Cc: Charles McCathie Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini amarches...@mozilla.commailto:amarches...@mozilla.com, Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com Subject: Re: ZIP archive API? I am very interested in working with archives. I'm currently using it as a delivery from server (like quake packs), import and export format for WebGL apps. On Tue, Apr 30, 2013 at 3:18 PM, Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl wrote: On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru wrote: Hi all, at the last TPAC there was discussion of a ZIP archive proposal. This has come and gone in various guises. Are there people currently interested in being able to work with ZIP in a web app? Are there implementors and is there an editor? We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is implemented as well (cc'd Andrea, who implemented it). There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about a somewhat-related proposal by Paul. -- http://annevankesteren.nl/
Re: ZIP archive API?
On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile cha...@yandex-team.ru wrote: Hi all, at the last TPAC there was discussion of a ZIP archive proposal. This has come and gone in various guises. Are there people currently interested in being able to work with ZIP in a web app? Are there implementors and is there an editor? We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is implemented as well (cc'd Andrea, who implemented it). There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about a somewhat-related proposal by Paul. -- http://annevankesteren.nl/
Re: ZIP archive API?
I am very interested in working with archives. I'm currently using it as a delivery from server (like quake packs), import and export format for WebGL apps. On Tue, Apr 30, 2013 at 3:18 PM, Anne van Kesteren ann...@annevk.nl wrote: On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile cha...@yandex-team.ru wrote: Hi all, at the last TPAC there was discussion of a ZIP archive proposal. This has come and gone in various guises. Are there people currently interested in being able to work with ZIP in a web app? Are there implementors and is there an editor? We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is implemented as well (cc'd Andrea, who implemented it). There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about a somewhat-related proposal by Paul. -- http://annevankesteren.nl/
