Re: jar protocol (was: ZIP archive API?)

2013-05-10 Thread Stian Soiland-Reyes
This seems very related to how prefixes/terms are expanded to IRIs in
JSON-LD - see http://www.w3.org/TR/json-ld/#iris

The JSON-LD approach is more like registering new local protocols,
as they look like URIs.

If we tried that out, then:
 link rel=bundle href=/bundle.zip anchor=b2 /
would mean that
   a href=b2:fred/hello.txt would resolve to fred/hello.txt within
bundle.zip.



The difference with Robin's proposal defines a new relative prefix -
almost like UNIX/Linux can let you mount /home/fred to a different
partition than /home - and therefore has this nice HTTP fall-back.
You won't have to worry about someone else defining the b2 protocol,
as you operate within your own URI namespace.

One downside with not having a URI scheme is that you need to
propagate the link bindings in any document that needs it - which is
probably OK, not very different from how RDF Turtle uses @prefix and
XML uses xmlns:fred =.


On 7 May 2013 21:31, David Sheets kosmo...@gmail.com wrote:
 On Tue, May 7, 2013 at 3:29 PM, Robin Berjon ro...@w3.org wrote:
 On 06/05/2013 20:42 , Jonas Sicking wrote:

 The only things that implementations can do that JS can't is:
 * Implement new protocols. I definitely agree that we should specify a
 jar: or archive: protocol, but that's orthogonal to whether we need an
 API.


 Have you looked at just reusing JAR for this (given that you support it in
 some form already)? I wonder how well it works. Off the top of my head I see
 at least two issues:

 • Its manifest format has lots of useless stuff, and is missing some things
 we would likely want (like MIME type mapping).

 • It requires its own URI scheme, which means that there is essentially no
 transition strategy for content: you can only start using it when everyone
 is (or you have to do UA detection).

 I wonder if we couldn't have a mechanism that would not require a separate
 URI scheme. Just throwing this against the wall, might be daft:

 We add a new link relationship: bundle (archive is taken, bikeshed later).
 The href points to the archive, and there can be as many as needed. The
 resolved absolute URL for this is added to a list of bundles (there is no
 requirement on when this gets fetched, UAs can do so immediately or on first
 use depending on what they wish to optimise for).

 After that, whenever there is a fetch for a resource the URL of which is a
 prefix match for this bundle the content is obtained from the bundle.

 This isn't very different from JAR but it does have the property of more
 easily enabling a transition. To give an example, say that the page at
 http://berjon.com/ contains:

 link rel=bundle href=bundle.wrap

 and

 img src=bundle.wrap/img/dahut.png alt=a dahut

 A UA supporting this would grab the bundle, then extract the image from it.
 A UA not supporting this would do nothing with the link, but would issue a
 request for /bundle.wrap/img/dahut.png. It is then fairly easy on the server
 side to be able to detect that it's a wrapped resource and serve it from
 inside the bundle (or whatever local convention it wants to adopt that
 allows it to cater to both — in any case it's trivial).

 This means no URL scheme to be supported by everyone, no nested URL scheme
 the way JAR does it (which is quite distasteful), no messing with escaping !
 in paths, etc.

 WDYT?

 This is really cool!

 Most servers already contain support for this in the form of index files.

 If you do

 link rel=bundle href=bundle.wrap/ /

 and set your server's file directory resolver to match index.zip, you
 don't need any special server-side extraction or handling: just
 extract the archive root as sibling to index.zip when you deploy!

 Additionally, this piggybacks application resource caching on top of
 HTTP caching.

 One quirk of this scheme (ha) is its notion of root path. With this
 path pattern match, the subresources in the archive exist in the
 domain's single top-level path structure. This means that for archives
 to be fully self-contained they must only use relative references that
 do not escape the archive root. Of course, this is also a feature when
 the containment of the archive is not a concern.

 How does directory resolution inside a bundle work? i.e. resolve
 bundle.wrap/dir/ ? It seems like this (listing) is a key feature of
 the API that was being discussed. I support a JSON object without a
 well-known name, personally.

 Can we use

 Link: bundle.wrap/; REL=bundle

 for generic resources?

 Does

 a href=bundle.wrap/page.htmlGo!/a

 make a server request or load from the bundle?

 Do bundle requests Accept archive media types?

 Do generic requests (e.g. address bar) Accept archive media types?

 What if I do

 link rel=bundle href= /

 ?

 Will this page be re-requested Accept-ing archive media types?

 Could bundles be entirely prefixed based?

 What does

 link rel=bundle href=bundle.wrap# /

 with

 img src=bundle.wrap#images/dahut.png / !-- or is it
 

Re: ZIP archive API?

2013-05-07 Thread Jonas Sicking
On Mon, May 6, 2013 at 6:36 PM, Glenn Maynard gl...@zewt.org wrote:
 On Mon, May 6, 2013 at 8:01 PM, Jonas Sicking jo...@sicking.cc wrote:

 On Mon, May 6, 2013 at 5:15 PM, Glenn Maynard gl...@zewt.org wrote:
  I'm not aware of any optimized inflate implementation in JS to compare
  against, and it's a complex algorithm, so nobody is likely to jump
  forward
  to spend a lot of time implementing and heavily optimizing it just to
  show
  how slow it is.  I've seen an implementation around somewhere, but it
  didn't
  use typed arrays so it would need a lot of reworking to have any
  meaning.

 Likewise, I don't see any browser vendor jumping ahead and doing both
 the work to implement a library *and* and API to compare the two.

 Sorry, this didn't make sense.  What library *and* API are you talking
 about?  To compare what?

A library for zip compression/decompression and an API for zip
compression/decompression. So that we can compare the performance of
the two.

  Every browser already has native inflate, though.

 This is unfortunately not a terribly strong argument. Exposing that
 implementation through a DOM API requires a fairly large amount of
 work. Not to add maintaining that over the years.


 You're arguing for allowing accessing files inside ZIPs by URL, which means
 you're going to have to do the work anyway, since you'd be able to create a
 blob URL, reference a file inside it using XHR, and get a Blob as a result.
 This is a small subset of that.

No, the work to write and maintain an API for ZIP
compress/decompression is pretty orthogonal, implementation-wise, to a
protocol handler for ZIP decompression.

Look, the fact that we need use cases should not be surprising to
people on this mailing list. And I know that it's not surprising to
you. One of the heavy arguments that has been brought up for a ZIP
archive API has been that it provides better performance. That's a
claim that needs to be backed up by numbers in order to carry any
weight.

The argument that you're bringing up is it's easy to implement
(because you already have an zip library) is also a very weak
argument.

Until we get stronger arguments for an archive API, I'd be happy to
see a proposal for a protocol handler.

/ Jonas



Re: ZIP archive API?

2013-05-07 Thread Florian Bösch
On Tue, May 7, 2013 at 8:09 AM, Jonas Sicking jo...@sicking.cc wrote:

  You're arguing for allowing accessing files inside ZIPs by URL, which
 means
  you're going to have to do the work anyway, since you'd be able to
 create a
  blob URL, reference a file inside it using XHR, and get a Blob as a
 result.
  This is a small subset of that.

 No, the work to write and maintain an API for ZIP
 compress/decompression is pretty orthogonal, implementation-wise, to a
 protocol handler for ZIP decompression.


In order to implement zip you would use one of the ready made libraries
supporting it, such as libzip (http://www.nih.at/libzip/index.html),
minizip (http://www.winimage.com/zLibDll/minizip.html),

Both libraries (and any other library you might encounter) already defines
an API and an implementation. In order to support ZIP URLs you would use
the API. In order to support a JS zip API, you would expose the API. These
are not orthogonal, they are correlated.


jar protocol (was: ZIP archive API?)

2013-05-07 Thread Robin Berjon

On 06/05/2013 20:42 , Jonas Sicking wrote:

The only things that implementations can do that JS can't is:
* Implement new protocols. I definitely agree that we should specify a
jar: or archive: protocol, but that's orthogonal to whether we need an
API.


Have you looked at just reusing JAR for this (given that you support it 
in some form already)? I wonder how well it works. Off the top of my 
head I see at least two issues:


• Its manifest format has lots of useless stuff, and is missing some 
things we would likely want (like MIME type mapping).


• It requires its own URI scheme, which means that there is essentially 
no transition strategy for content: you can only start using it when 
everyone is (or you have to do UA detection).


I wonder if we couldn't have a mechanism that would not require a 
separate URI scheme. Just throwing this against the wall, might be daft:


We add a new link relationship: bundle (archive is taken, bikeshed 
later). The href points to the archive, and there can be as many as 
needed. The resolved absolute URL for this is added to a list of bundles 
(there is no requirement on when this gets fetched, UAs can do so 
immediately or on first use depending on what they wish to optimise for).


After that, whenever there is a fetch for a resource the URL of which is 
a prefix match for this bundle the content is obtained from the bundle.


This isn't very different from JAR but it does have the property of more 
easily enabling a transition. To give an example, say that the page at 
http://berjon.com/ contains:


link rel=bundle href=bundle.wrap

and

img src=bundle.wrap/img/dahut.png alt=a dahut

A UA supporting this would grab the bundle, then extract the image from 
it. A UA not supporting this would do nothing with the link, but would 
issue a request for /bundle.wrap/img/dahut.png. It is then fairly easy 
on the server side to be able to detect that it's a wrapped resource and 
serve it from inside the bundle (or whatever local convention it wants 
to adopt that allows it to cater to both — in any case it's trivial).


This means no URL scheme to be supported by everyone, no nested URL 
scheme the way JAR does it (which is quite distasteful), no messing with 
escaping ! in paths, etc.


WDYT?

--
Robin Berjon - http://berjon.com/ - @robinberjon



Re: jar protocol (was: ZIP archive API?)

2013-05-07 Thread Anne van Kesteren
On Tue, May 7, 2013 at 7:29 AM, Robin Berjon ro...@w3.org wrote:
 This isn't very different from JAR but it does have the property of more
 easily enabling a transition. To give an example, say that the page at
 http://berjon.com/ contains:

 link rel=bundle href=bundle.wrap

 and

 img src=bundle.wrap/img/dahut.png alt=a dahut

You need a new URL scheme here. Otherwise the URL will be parsed
relative to the node's base URL.


--
http://annevankesteren.nl/



Re: jar protocol (was: ZIP archive API?)

2013-05-07 Thread Bjoern Hoehrmann
* Anne van Kesteren wrote:
On Tue, May 7, 2013 at 7:29 AM, Robin Berjon ro...@w3.org wrote:
 This isn't very different from JAR but it does have the property of more
 easily enabling a transition. To give an example, say that the page at
 http://berjon.com/ contains:

 link rel=bundle href=bundle.wrap

 and

 img src=bundle.wrap/img/dahut.png alt=a dahut

You need a new URL scheme here. Otherwise the URL will be parsed
relative to the node's base URL.

Robin seems to address that in the parts of his mail you didn't quote.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 



Re: jar protocol (was: ZIP archive API?)

2013-05-07 Thread Anne van Kesteren
On Tue, May 7, 2013 at 11:34 AM, Bjoern Hoehrmann derhoe...@gmx.net wrote:
 Robin seems to address that in the parts of his mail you didn't quote.

My bad :-(

Have to say it does seem quite elegant. And has great fallback (if
implemented on the server).


--
http://annevankesteren.nl/



Re: jar protocol (was: ZIP archive API?)

2013-05-07 Thread Jonas Sicking
On Tue, May 7, 2013 at 7:29 AM, Robin Berjon ro...@w3.org wrote:
 On 06/05/2013 20:42 , Jonas Sicking wrote:

 The only things that implementations can do that JS can't is:
 * Implement new protocols. I definitely agree that we should specify a
 jar: or archive: protocol, but that's orthogonal to whether we need an
 API.


 Have you looked at just reusing JAR for this (given that you support it in
 some form already)? I wonder how well it works. Off the top of my head I see
 at least two issues:

 • Its manifest format has lots of useless stuff, and is missing some things
 we would likely want (like MIME type mapping).

 • It requires its own URI scheme, which means that there is essentially no
 transition strategy for content: you can only start using it when everyone
 is (or you have to do UA detection).

 I wonder if we couldn't have a mechanism that would not require a separate
 URI scheme. Just throwing this against the wall, might be daft:

 We add a new link relationship: bundle (archive is taken, bikeshed later).
 The href points to the archive, and there can be as many as needed. The
 resolved absolute URL for this is added to a list of bundles (there is no
 requirement on when this gets fetched, UAs can do so immediately or on first
 use depending on what they wish to optimise for).

 After that, whenever there is a fetch for a resource the URL of which is a
 prefix match for this bundle the content is obtained from the bundle.

 This isn't very different from JAR but it does have the property of more
 easily enabling a transition. To give an example, say that the page at
 http://berjon.com/ contains:

 link rel=bundle href=bundle.wrap

 and

 img src=bundle.wrap/img/dahut.png alt=a dahut

 A UA supporting this would grab the bundle, then extract the image from it.
 A UA not supporting this would do nothing with the link, but would issue a
 request for /bundle.wrap/img/dahut.png. It is then fairly easy on the server
 side to be able to detect that it's a wrapped resource and serve it from
 inside the bundle (or whatever local convention it wants to adopt that
 allows it to cater to both — in any case it's trivial).

 This means no URL scheme to be supported by everyone, no nested URL scheme
 the way JAR does it (which is quite distasteful), no messing with escaping !
 in paths, etc.

 WDYT?

Will this let us support reading things from blob: URLs where the Blob
contains a zip file? I.e. what Gecko would support as
jar:blob:abc-123!/img/foo.jpg.

Also note that while we're using jar as scheme name, it's simply
just zip support. None of the other pieces of the jar spec is used.

/ Jonas



Re: jar protocol (was: ZIP archive API?)

2013-05-07 Thread Bjoern Hoehrmann
* Robin Berjon wrote:
I wonder if we couldn't have a mechanism that would not require a 
separate URI scheme. Just throwing this against the wall, might be daft:

We add a new link relationship: bundle (archive is taken, bikeshed 
later). The href points to the archive, and there can be as many as 
needed. The resolved absolute URL for this is added to a list of bundles 
(there is no requirement on when this gets fetched, UAs can do so 
immediately or on first use depending on what they wish to optimise for).

After that, whenever there is a fetch for a resource the URL of which is 
a prefix match for this bundle the content is obtained from the bundle.

There have been many proposals over the years that would allow for some-
thing like this, http://www.w3.org/TR/DataCache/ for instance, allows to
intercept certain requests to aid in supporting offline applications,
and `registerProtocolHandler` combined with `web+`-schemes go into a si-
milar direction. Those seem more worthwhile to explore to me than your
one-trick-strawman.

Also, it is not clear to me that avoiding a special scheme is a useful
design constraint (not to mention that bundling is something the com-
puter is supposed to do for me, so I would want to get that out of my
face). But I can see value in a more generic feature that allows me to
implement and reference IO objects as I see fit, which would provide for
bundling features.

This means no URL scheme to be supported by everyone, [...]

Well, `rel='bundle'` would have to be supported by everyone, because
past critical mass there would be too many nobody noticed the fallback
is not working until now cases, so that seems rather uninteresting in
the longer term.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 



Re: ZIP archive API?

2013-05-07 Thread Bjoern Hoehrmann
* Florian Bösch wrote:
It can be implemented by a JS library, but the three reasons to let the
browser provide it are Convenience, speed and integration.

Convenience is the first reason, since browsers by far and large already
have complete bindings to compression algorithms and archive formats,
letting the browser simply expose the software it already ships makes good
sense rather than requiring every JS user to supply his own version.

Speed may not matter to much on some platforms, but it matters a great deal
on underpowered devices such as mobiles.

Integration is where the support for archives goes beyond being an API,
where URLs (to link.href, script.src, img.src, iframe.src, audio.src,
video.src, css url(), etc.) could point into an archive. This cannot be
done in JS.

If we all agreed that such functionality should be provided by libraries
rather than browser cores, and browser vendors could be expected to make
things convenient, fast, and well-integrated, then using libraries would
be convenient, libraries would be fast, and they would integrate well.
-- 
Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 



Re: jar protocol (was: ZIP archive API?)

2013-05-07 Thread Glenn Maynard
On Tue, May 7, 2013 at 9:29 AM, Robin Berjon ro...@w3.org wrote:

 Have you looked at just reusing JAR for this (given that you support it in
 some form already)? I wonder how well it works. Off the top of my head I
 see at least two issues:


JARs are just ZIPs with Java metadata.  We don't need metadata, so plain
ZIPs are enough.

link rel=bundle href=bundle.wrap

 and

 img src=bundle.wrap/img/dahut.**png alt=a dahut


This depends on a document, so it wouldn't work in workers unless we add a
second API to register them in script.

--
Glenn Maynard


Re: ZIP archive API?

2013-05-06 Thread Robin Berjon

On 03/05/2013 21:05 , Florian Bösch wrote:

It can be implemented by a JS library, but the three reasons to let the
browser provide it are Convenience, speed and integration.


Also, one of the reasons we compress things is because they're big.* 
Unpacking in JS is likely to mean unpacking to memory (unless the blobs 
are smarter than that), whereas the browser has access to strategies to 
mitigate this, e.g. using temporary files.


Another question to take into account here is whether this should only 
be about zip. One of the limitations of zip archives is that they aren't 
streamable. Without boiling the ocean, adding support for a streamable 
format (which I don't think needs be more complex than tar) would be a 
big plus.




* Captain Obvious to the rescue!

--
Robin Berjon - http://berjon.com/ - @robinberjon



Re: ZIP archive API?

2013-05-06 Thread Florian Bösch
The main reason to use an archive (other than the space-savings) for me is
to be able to transfer tens of thousands of small items that go into
producing WebGL applications of non trivial scope.


On Mon, May 6, 2013 at 1:27 PM, Robin Berjon ro...@w3.org wrote:

 On 03/05/2013 21:05 , Florian Bösch wrote:

 It can be implemented by a JS library, but the three reasons to let the
 browser provide it are Convenience, speed and integration.


 Also, one of the reasons we compress things is because they're big.*
 Unpacking in JS is likely to mean unpacking to memory (unless the blobs are
 smarter than that), whereas the browser has access to strategies to
 mitigate this, e.g. using temporary files.

 Another question to take into account here is whether this should only be
 about zip. One of the limitations of zip archives is that they aren't
 streamable. Without boiling the ocean, adding support for a streamable
 format (which I don't think needs be more complex than tar) would be a big
 plus.



 * Captain Obvious to the rescue!


 --
 Robin Berjon - http://berjon.com/ - @robinberjon



Re: ZIP archive API?

2013-05-06 Thread Glenn Maynard
On Mon, May 6, 2013 at 6:27 AM, Robin Berjon ro...@w3.org wrote:

 Another question to take into account here is whether this should only be
 about zip. One of the limitations of zip archives is that they aren't
 streamable. Without boiling the ocean, adding support for a streamable
 format (which I don't think needs be more complex than tar) would be a big
 plus.


Zips are streamable.  That's what the local file headers are for.
http://www.pkware.com/documents/casestudies/APPNOTE.TXT

-- 
Glenn Maynard


Re: ZIP archive API?

2013-05-06 Thread Michaela Merz
I second that. Thanks Florian.




On 05/03/2013 02:52 PM, Florian Bösch wrote:
 I'm interested a JS API that does the following:

 Unpacking:
 - Receive an archive from a Dataurl, Blob, URL object, File (as in
 filesystem API) or Arraybuffer
 - List its content and metadata
 - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

 Packing:
 - Create an archive
 - Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
 - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

 To avoid the whole worker/proxy thing and to allow authors to
 selectively choose how they want to handle the data, I'd like to see
 synchronous and asynchronous versions of each. I'd make synchronicity
 an argument/flag or something to avoid API clutter like packSync,
 packAsync, writeSync, writeAsync, and rather like write(data,
 callback|boolean).

 - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
 over the top: http://docs.python.org/3/library/zipfile.html
 - Pythons tarfile API is less clutered and easier to
 use: http://docs.python.org/3/library/tarfile.html
 - zip.js isn't really usable as it doesn't support the full range of
 types (Dataurl, Blob, URL object, File or Arraybuffer) and for
 asynchronous operation needs to rely on a worker, which is bothersome
 to setup: http://stuk.github.io/jszip/

 My own implementation of the tar format only targets array buffers and
 works synchronously, as in.

 var archive = new TarFile(arraybuffer);
 var memberArrayBuffer = archive.get('filename');



 On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nl
 mailto:ann...@annevk.nl wrote:

 On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com
 mailto:pbak...@zynga.com wrote:
  Still waiting for it as well. I think it'd be very useful to
 transfer sets
  of assets etc.

 Do you have anything in particular you'd like to see happen first?
 It's pretty clear we should expose more here, but as with all things
 we should do it in baby steps.


 --
 http://annevankesteren.nl/





smime.p7s
Description: S/MIME Cryptographic Signature


Re: ZIP archive API?

2013-05-06 Thread Eric U
On Mon, May 6, 2013 at 5:03 AM, Glenn Maynard gl...@zewt.org wrote:
 On Mon, May 6, 2013 at 6:27 AM, Robin Berjon ro...@w3.org wrote:

 Another question to take into account here is whether this should only be
 about zip. One of the limitations of zip archives is that they aren't
 streamable. Without boiling the ocean, adding support for a streamable
 format (which I don't think needs be more complex than tar) would be a big
 plus.


 Zips are streamable.  That's what the local file headers are for.
 http://www.pkware.com/documents/casestudies/APPNOTE.TXT

This came up a few years ago; Gregg Tavares explained in [1] that only
/some/ zipfiles are streamable, and you don't know whether yours are
or not until you've seen the whole file.

 Eric

[1] http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0362.html



Re: ZIP archive API?

2013-05-06 Thread Jonas Sicking
On Mon, May 6, 2013 at 4:27 AM, Robin Berjon ro...@w3.org wrote:
 On 03/05/2013 21:05 , Florian Bösch wrote:

 It can be implemented by a JS library, but the three reasons to let the
 browser provide it are Convenience, speed and integration.

 Also, one of the reasons we compress things is because they're big.*
 Unpacking in JS is likely to mean unpacking to memory (unless the blobs are
 smarter than that), whereas the browser has access to strategies to mitigate
 this, e.g. using temporary files.

There's nothing here that implementations can do that JS can't. Both
can read arbitrary parts of a file into memory and decompress only
that data. And both would have to load data into memory in order to
decompress it.

The only things that implementations can do that JS can't is:
* Implement new protocols. I definitely agree that we should specify a
jar: or archive: protocol, but that's orthogonal to whether we need an
API.
* Implement Blob backends. In our prototype implementation we were
able to return a Blob which represented data that hadn't been
compressed yet. When the data was read it was incrementally
decompressed.

But a JS implementation could incrementally decompress and write the
data to disk using something like FileHandle or FileWriter. So the
same capabilities are there.

 Another question to take into account here is whether this should only be
 about zip. One of the limitations of zip archives is that they aren't
 streamable. Without boiling the ocean, adding support for a streamable
 format (which I don't think needs be more complex than tar) would be a big
 plus.

Indeed. This is IMO an argument for relying on libraries.
Implementations are going to be a lot more conservative about adding
new archive formats, since they have to be supported for eternity,
than library authors will be.

I'm still hoping to see some performance numbers from the people
arguing that we should add this to the platform. Without that I see
little hope of getting enough browser vendors behind this.

/ Jonas



Re: ZIP archive API?

2013-05-06 Thread David Sheets
On Mon, May 6, 2013 at 7:42 PM, Jonas Sicking jo...@sicking.cc wrote:
 On Mon, May 6, 2013 at 4:27 AM, Robin Berjon ro...@w3.org wrote:
 On 03/05/2013 21:05 , Florian Bösch wrote:

 It can be implemented by a JS library, but the three reasons to let the
 browser provide it are Convenience, speed and integration.

 Also, one of the reasons we compress things is because they're big.*
 Unpacking in JS is likely to mean unpacking to memory (unless the blobs are
 smarter than that), whereas the browser has access to strategies to mitigate
 this, e.g. using temporary files.

 There's nothing here that implementations can do that JS can't. Both
 can read arbitrary parts of a file into memory and decompress only
 that data. And both would have to load data into memory in order to
 decompress it.

 The only things that implementations can do that JS can't is:
 * Implement new protocols. I definitely agree that we should specify a
 jar: or archive: protocol, but that's orthogonal to whether we need an
 API.

Is ZIP a protocol or simply a media type?

Either way, browsers will need to implement operations on the archive
format to understand its use in URL attributes.

The question then becomes: if browsers have an implementation of the
format for reference purposes, why do web apps need to ship their own
implementations of the format for archive purposes?

If web apps want to use some other archive/compression format, of
course they will have to distribute an implementation.

Thanks,

David

 * Implement Blob backends. In our prototype implementation we were
 able to return a Blob which represented data that hadn't been
 compressed yet. When the data was read it was incrementally
 decompressed.

 But a JS implementation could incrementally decompress and write the
 data to disk using something like FileHandle or FileWriter. So the
 same capabilities are there.

 Another question to take into account here is whether this should only be
 about zip. One of the limitations of zip archives is that they aren't
 streamable. Without boiling the ocean, adding support for a streamable
 format (which I don't think needs be more complex than tar) would be a big
 plus.

 Indeed. This is IMO an argument for relying on libraries.
 Implementations are going to be a lot more conservative about adding
 new archive formats, since they have to be supported for eternity,
 than library authors will be.

 I'm still hoping to see some performance numbers from the people
 arguing that we should add this to the platform. Without that I see
 little hope of getting enough browser vendors behind this.

 / Jonas





Re: ZIP archive API?

2013-05-06 Thread Paul Bakaus


On 03.05.13 15:18, Jonas Sicking jo...@sicking.cc wrote:

On Fri, May 3, 2013 at 12:12 PM, Paul Bakaus pbak...@zynga.com wrote:


 From: Florian Bösch pya...@gmail.com
 Date: Fri, 3 May 2013 21:05:17 +0200
 To: Jonas Sicking jo...@sicking.cc
 Cc: Paul Bakaus pbak...@zynga.com, Anne van Kesteren
ann...@annevk.nl,
 Webapps WG public-webapps@w3.org, Charles McCathie Nevile
 cha...@yandex-team.ru, Andrea Marchesini amarches...@mozilla.com

 Subject: Re: ZIP archive API?

 It can be implemented by a JS library, but the three reasons to let the
 browser provide it are Convenience, speed and integration.

 Convenience is the first reason, since browsers by far and large already
 have complete bindings to compression algorithms and archive formats,
 letting the browser simply expose the software it already ships makes
good
 sense rather than requiring every JS user to supply his own version.

 Speed may not matter to much on some platforms, but it matters a great
deal
 on underpowered devices such as mobiles.

Show me some numbers to back this up and you'll have me convinced :)

Remember that on underpowered devices native code is proportionally
slower too.

 Integration is where the support for archives goes beyond being an API,
 where URLs (to link.href, script.src, img.src, iframe.src, audio.src,
 video.src, css url(), etc.) could point into an archive. This cannot
be
 done in JS.


 I was going to say exactly  that. I want to be able to have a virtual
URL
 that I can point to. In my CSS, I want to do something like
 archive://assets/foo.png after I loaded and decompressed the ZIP file
in
 JS.

How does the assets part in the example above work? What does it
mean? Is there some registry here or something?

Actually assets was just referring to the package name. So what you're
saying below makes total sense, and is what we want to have. Great stuff!


 Jonas, I'm intrigued ­ do you see a way this could be done in JS? If so,
 maybe we should build a sample. I'm still thinking the performance
won't be
 good enough, particular on mobile devices, but let's find out.

You can actually do this in Gecko already. Any archive that you can
refer to through a URL, you can also reach into.

So if you have a .zip file in a Blob, and you generate a blob: URL
like blob:123-abc then you can read the foo.html file out of that
archive by using the URL jar:blob:123-abc!/foo.html. So far this
doesn't work with ArrayBuffers since there is no way to have a URL
that refers to an ArrayBuffer.

You can even load something from inside a zip file from a server by doing
img src=jar:http://example.com/foo/archive.zip!/image.jpg;

Something like that I definitely agree that we should standardize.

/ Jonas

 On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote:

 The big question we kept running up against at Mozilla is why couldn't
 this simply be implemented as a JS library?

 If performance is the argument we need to back that up with data.

 / Jonas

 On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:

 Hi Anne, Florian,

 I think the first baby step, or MVP, is the unpacking that Florian
 mentions below. I would definitely like to have the API available on
both
 workers and normal context.

 Thanks,
 Paul

 From: Florian Bösch pya...@gmail.com
 Date: Fri, 3 May 2013 14:52:36 +0200
 To: Anne van Kesteren ann...@annevk.nl
 Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile
 cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org,
Andrea
 Marchesini amarches...@mozilla.com
 Subject: Re: ZIP archive API?

 I'm interested a JS API that does the following:

 Unpacking:
 - Receive an archive from a Dataurl, Blob, URL object, File (as in
 filesystem API) or Arraybuffer
 - List its content and metadata
 - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

 Packing:
 - Create an archive
 - Put in members passing a Dataurl, Blob, URL object, File or
Arraybuffer
 - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

 To avoid the whole worker/proxy thing and to allow authors to
selectively
 choose how they want to handle the data, I'd like to see synchronous
and
 asynchronous versions of each. I'd make synchronicity an
argument/flag or
 something to avoid API clutter like packSync, packAsync, writeSync,
 writeAsync, and rather like write(data, callback|boolean).

 - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
 over the top: http://docs.python.org/3/library/zipfile.html
 - Pythons tarfile API is less clutered and easier to use:
 http://docs.python.org/3/library/tarfile.html
 - zip.js isn't really usable as it doesn't support the full range of
 types (Dataurl, Blob, URL object, File or Arraybuffer) and for
asynchronous
 operation needs to rely on a worker, which is bothersome to setup:
 http://stuk.github.io/jszip/

 My own implementation of the tar format only targets array buffers and
 works synchronously, as in.

 var archive = new TarFile

Re: ZIP archive API?

2013-05-06 Thread Glenn Maynard
On Mon, May 6, 2013 at 1:11 PM, Eric U er...@google.com wrote:

 This came up a few years ago; Gregg Tavares explained in [1] that only
 /some/ zipfiles are streamable, and you don't know whether yours are
 or not until you've seen the whole file.

  Eric

 [1]
 http://lists.w3.org/Archives/Public/public-webapps/2010AprJun/0362.html


The file format is streamable.  You can create files that follow the spec
that will fail when streaming, but you can also create files that follow
the spec that will fail when not streaming.  (The end of central directory
record sometimes has data after it, so you have to do a search; there's no
spec defining how far you have to search, so if you put too much data there
it'll start to fail.)  Those are both problems with the spec that would
have to be addressed.  I don't think there's any reason to support tar (and
it would significantly complicate the API, since tar *only* supports
streaming).

The bigger point here is that the ZIP appnote isn't enough.  It doesn't
define parsers or error handling.  This means that defining an API to
expose ZIPs isn't only a matter of defining an API, somebody will need to
spec the file format itself.  Also, the appnote isn't free, so this would
probably need to be a clean-room spec.  (However, it wouldn't need to
specify all of the features of the format, a huge number of which are never
used, only how to parse past them and ignore them.)


On Mon, May 6, 2013 at 1:42 PM, Jonas Sicking jo...@sicking.cc wrote:

  Another question to take into account here is whether this should only be
  about zip. One of the limitations of zip archives is that they aren't
  streamable. Without boiling the ocean, adding support for a streamable
  format (which I don't think needs be more complex than tar) would be a
 big
  plus.

 Indeed. This is IMO an argument for relying on libraries.


It's not.  ZIP has been around longer than PNG and JPEG; its only real
competors are tar.gz (which isn't useful here) and RAR (proprietary).  It's
not going away and there's no indication of a sudden influx of competing
file formats, any more than image formats.

That said, I don't know if a ZIP API is worthwhile.  I'd start lower level
here, and think about supporting inflating blobs.  That's the same
functionality any ZIP API will want, and it's the main part of the ZIP
format that you really don't want to have to do in script.  The surface
area is also far simpler: new InflatedBlob(compressedBlob)

I'm still hoping to see some performance numbers from the people
 arguing that we should add this to the platform. Without that I see
 little hope of getting enough browser vendors behind this.


I'm not aware of any optimized inflate implementation in JS to compare
against, and it's a complex algorithm, so nobody is likely to jump forward
to spend a lot of time implementing and heavily optimizing it just to show
how slow it is.  I've seen an implementation around somewhere, but it
didn't use typed arrays so it would need a lot of reworking to have any
meaning.

Every browser already has native inflate, though.

-- 
Glenn Maynard


Re: ZIP archive API?

2013-05-06 Thread Jonas Sicking
On Mon, May 6, 2013 at 5:15 PM, Glenn Maynard gl...@zewt.org wrote:
 I'm not aware of any optimized inflate implementation in JS to compare
 against, and it's a complex algorithm, so nobody is likely to jump forward
 to spend a lot of time implementing and heavily optimizing it just to show
 how slow it is.  I've seen an implementation around somewhere, but it didn't
 use typed arrays so it would need a lot of reworking to have any meaning.

Likewise, I don't see any browser vendor jumping ahead and doing both
the work to implement a library *and* and API to compare the two.

 Every browser already has native inflate, though.

This is unfortunately not a terribly strong argument. Exposing that
implementation through a DOM API requires a fairly large amount of
work. Not to add maintaining that over the years.

/ Jonas



Re: ZIP archive API?

2013-05-06 Thread Glenn Maynard
On Mon, May 6, 2013 at 8:01 PM, Jonas Sicking jo...@sicking.cc wrote:

 On Mon, May 6, 2013 at 5:15 PM, Glenn Maynard gl...@zewt.org wrote:
  I'm not aware of any optimized inflate implementation in JS to compare
  against, and it's a complex algorithm, so nobody is likely to jump
 forward
  to spend a lot of time implementing and heavily optimizing it just to
 show
  how slow it is.  I've seen an implementation around somewhere, but it
 didn't
  use typed arrays so it would need a lot of reworking to have any meaning.

 Likewise, I don't see any browser vendor jumping ahead and doing both
 the work to implement a library *and* and API to compare the two.


Sorry, this didn't make sense.  What library *and* API are you talking
about?  To compare what?

  Every browser already has native inflate, though.

 This is unfortunately not a terribly strong argument. Exposing that
 implementation through a DOM API requires a fairly large amount of
 work. Not to add maintaining that over the years.


You're arguing for allowing accessing files inside ZIPs by URL, which means
you're going to have to do the work anyway, since you'd be able to create a
blob URL, reference a file inside it using XHR, and get a Blob as a
result.  This is a small subset of that.

-- 
Glenn Maynard


Re: ZIP archive API?

2013-05-03 Thread Robin Berjon

On 03/05/2013 13:40 , Arthur Barstow wrote:

Other than Mozilla, is there any other implementor interest?

I'm wondering if steering this work to a CG would be `best` for now,
especially if no one steps up to be Editor.


I think that this is useful and I would rather it were inside a group 
than pushed off to a CG. It's not experimental in nature.


--
Robin Berjon - http://berjon.com/ - @robinberjon



Re: ZIP archive API?

2013-05-03 Thread Anne van Kesteren
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
 Still waiting for it as well. I think it'd be very useful to transfer sets
 of assets etc.

Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.


--
http://annevankesteren.nl/



Re: ZIP archive API?

2013-05-03 Thread Florian Bösch
I'm interested a JS API that does the following:

Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in
filesystem API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync,
writeAsync, and rather like write(data, callback|boolean).

- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over
the top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
operation needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/

My own implementation of the tar format only targets array buffers and
works synchronously, as in.

var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');



On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
  Still waiting for it as well. I think it'd be very useful to transfer
 sets
  of assets etc.

 Do you have anything in particular you'd like to see happen first?
 It's pretty clear we should expose more here, but as with all things
 we should do it in baby steps.


 --
 http://annevankesteren.nl/



Re: ZIP archive API?

2013-05-03 Thread Paul Bakaus
Hi Anne, Florian,

I think the first baby step, or MVP, is the unpacking that Florian mentions 
below. I would definitely like to have the API available on both workers and 
normal context.

Thanks,
Paul

From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Charles McCathie 
Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG 
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini 
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?

I'm interested a JS API that does the following:

Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem 
API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

To avoid the whole worker/proxy thing and to allow authors to selectively 
choose how they want to handle the data, I'd like to see synchronous and 
asynchronous versions of each. I'd make synchronicity an argument/flag or 
something to avoid API clutter like packSync, packAsync, writeSync, writeAsync, 
and rather like write(data, callback|boolean).

- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the 
top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use: 
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types 
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation 
needs to rely on a worker, which is bothersome to setup: 
http://stuk.github.io/jszip/

My own implementation of the tar format only targets array buffers and works 
synchronously, as in.

var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');



On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren 
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus 
pbak...@zynga.commailto:pbak...@zynga.com wrote:
 Still waiting for it as well. I think it'd be very useful to transfer sets
 of assets etc.

Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.


--
http://annevankesteren.nl/



Re: ZIP archive API?

2013-05-03 Thread Jonas Sicking
The big question we kept running up against at Mozilla is why couldn't
this simply be implemented as a JS library?

If performance is the argument we need to back that up with data.

/ Jonas
On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:

  Hi Anne, Florian,

  I think the first baby step, or MVP, is the unpacking that Florian
 mentions below. I would definitely like to have the API available on both
 workers and normal context.

  Thanks,
 Paul

   From: Florian Bösch pya...@gmail.com
 Date: Fri, 3 May 2013 14:52:36 +0200
 To: Anne van Kesteren ann...@annevk.nl
 Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile 
 cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org, Andrea
 Marchesini amarches...@mozilla.com
 Subject: Re: ZIP archive API?

  I'm interested a JS API that does the following:

  Unpacking:
 - Receive an archive from a Dataurl, Blob, URL object, File (as in
 filesystem API) or Arraybuffer
 - List its content and metadata
 - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

  Packing:
 - Create an archive
 - Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
 - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

  To avoid the whole worker/proxy thing and to allow authors to
 selectively choose how they want to handle the data, I'd like to see
 synchronous and asynchronous versions of each. I'd make synchronicity an
 argument/flag or something to avoid API clutter like packSync, packAsync,
 writeSync, writeAsync, and rather like write(data, callback|boolean).

  - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
 over the top: http://docs.python.org/3/library/zipfile.html
 - Pythons tarfile API is less clutered and easier to use:
 http://docs.python.org/3/library/tarfile.html
 - zip.js isn't really usable as it doesn't support the full range of types
 (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
 operation needs to rely on a worker, which is bothersome to setup:
 http://stuk.github.io/jszip/

  My own implementation of the tar format only targets array buffers and
 works synchronously, as in.

  var archive = new TarFile(arraybuffer);
 var memberArrayBuffer = archive.get('filename');



 On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nlwrote:

 On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
  Still waiting for it as well. I think it'd be very useful to transfer
 sets
  of assets etc.

  Do you have anything in particular you'd like to see happen first?
 It's pretty clear we should expose more here, but as with all things
 we should do it in baby steps.


 --
 http://annevankesteren.nl/





Re: ZIP archive API?

2013-05-03 Thread Florian Bösch
It can be implemented by a JS library, but the three reasons to let the
browser provide it are Convenience, speed and integration.

Convenience is the first reason, since browsers by far and large already
have complete bindings to compression algorithms and archive formats,
letting the browser simply expose the software it already ships makes good
sense rather than requiring every JS user to supply his own version.

Speed may not matter to much on some platforms, but it matters a great deal
on underpowered devices such as mobiles.

Integration is where the support for archives goes beyond being an API,
where URLs (to link.href, script.src, img.src, iframe.src, audio.src,
video.src, css url(), etc.) could point into an archive. This cannot be
done in JS.



On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote:

 The big question we kept running up against at Mozilla is why couldn't
 this simply be implemented as a JS library?

 If performance is the argument we need to back that up with data.

 / Jonas
 On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:

  Hi Anne, Florian,

  I think the first baby step, or MVP, is the unpacking that Florian
 mentions below. I would definitely like to have the API available on both
 workers and normal context.

  Thanks,
 Paul

   From: Florian Bösch pya...@gmail.com
 Date: Fri, 3 May 2013 14:52:36 +0200
 To: Anne van Kesteren ann...@annevk.nl
 Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile 
 cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org,
 Andrea Marchesini amarches...@mozilla.com
 Subject: Re: ZIP archive API?

  I'm interested a JS API that does the following:

  Unpacking:
 - Receive an archive from a Dataurl, Blob, URL object, File (as in
 filesystem API) or Arraybuffer
 - List its content and metadata
 - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

  Packing:
 - Create an archive
 - Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
 - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

  To avoid the whole worker/proxy thing and to allow authors to
 selectively choose how they want to handle the data, I'd like to see
 synchronous and asynchronous versions of each. I'd make synchronicity an
 argument/flag or something to avoid API clutter like packSync, packAsync,
 writeSync, writeAsync, and rather like write(data, callback|boolean).

  - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
 over the top: http://docs.python.org/3/library/zipfile.html
 - Pythons tarfile API is less clutered and easier to use:
 http://docs.python.org/3/library/tarfile.html
 - zip.js isn't really usable as it doesn't support the full range of
 types (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
 operation needs to rely on a worker, which is bothersome to setup:
 http://stuk.github.io/jszip/

  My own implementation of the tar format only targets array buffers and
 works synchronously, as in.

  var archive = new TarFile(arraybuffer);
 var memberArrayBuffer = archive.get('filename');



 On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nlwrote:

 On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote:
  Still waiting for it as well. I think it'd be very useful to transfer
 sets
  of assets etc.

  Do you have anything in particular you'd like to see happen first?
 It's pretty clear we should expose more here, but as with all things
 we should do it in baby steps.


 --
 http://annevankesteren.nl/





Re: ZIP archive API?

2013-05-03 Thread Paul Bakaus


From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 21:05:17 +0200
To: Jonas Sicking jo...@sicking.ccmailto:jo...@sicking.cc
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Anne van 
Kesteren ann...@annevk.nlmailto:ann...@annevk.nl, Webapps WG 
public-webapps@w3.orgmailto:public-webapps@w3.org, Charles McCathie Nevile 
cha...@yandex-team.rumailto:cha...@yandex-team.ru, Andrea Marchesini 
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?

It can be implemented by a JS library, but the three reasons to let the browser 
provide it are Convenience, speed and integration.

Convenience is the first reason, since browsers by far and large already have 
complete bindings to compression algorithms and archive formats, letting the 
browser simply expose the software it already ships makes good sense rather 
than requiring every JS user to supply his own version.

Speed may not matter to much on some platforms, but it matters a great deal on 
underpowered devices such as mobiles.

Integration is where the support for archives goes beyond being an API, where 
URLs (to link.href, script.src, img.src, iframe.src, audio.src, video.src, css 
url(), etc.) could point into an archive. This cannot be done in JS.

I was going to say exactly  that. I want to be able to have a virtual URL that 
I can point to. In my CSS, I want to do something like 
archive://assets/foo.png after I loaded and decompressed the ZIP file in JS.

Jonas, I'm intrigued – do you see a way this could be done in JS? If so, maybe 
we should build a sample. I'm still thinking the performance won't be good 
enough, particular on mobile devices, but let's find out.




On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking 
jo...@sicking.ccmailto:jo...@sicking.cc wrote:

The big question we kept running up against at Mozilla is why couldn't this 
simply be implemented as a JS library?

If performance is the argument we need to back that up with data.

/ Jonas

On May 3, 2013 10:51 AM, Paul Bakaus 
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Hi Anne, Florian,

I think the first baby step, or MVP, is the unpacking that Florian mentions 
below. I would definitely like to have the API available on both workers and 
normal context.

Thanks,
Paul

From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Charles McCathie 
Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG 
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini 
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?

I'm interested a JS API that does the following:

Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem 
API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

To avoid the whole worker/proxy thing and to allow authors to selectively 
choose how they want to handle the data, I'd like to see synchronous and 
asynchronous versions of each. I'd make synchronicity an argument/flag or 
something to avoid API clutter like packSync, packAsync, writeSync, writeAsync, 
and rather like write(data, callback|boolean).

- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the 
top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use: 
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types 
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation 
needs to rely on a worker, which is bothersome to setup: 
http://stuk.github.io/jszip/

My own implementation of the tar format only targets array buffers and works 
synchronously, as in.

var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');



On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren 
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus 
pbak...@zynga.commailto:pbak...@zynga.com wrote:
 Still waiting for it as well. I think it'd be very useful to transfer sets
 of assets etc.

Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.


--
http://annevankesteren.nl/




Re: ZIP archive API?

2013-05-03 Thread Jonas Sicking
On Fri, May 3, 2013 at 12:12 PM, Paul Bakaus pbak...@zynga.com wrote:


 From: Florian Bösch pya...@gmail.com
 Date: Fri, 3 May 2013 21:05:17 +0200
 To: Jonas Sicking jo...@sicking.cc
 Cc: Paul Bakaus pbak...@zynga.com, Anne van Kesteren ann...@annevk.nl,
 Webapps WG public-webapps@w3.org, Charles McCathie Nevile
 cha...@yandex-team.ru, Andrea Marchesini amarches...@mozilla.com

 Subject: Re: ZIP archive API?

 It can be implemented by a JS library, but the three reasons to let the
 browser provide it are Convenience, speed and integration.

 Convenience is the first reason, since browsers by far and large already
 have complete bindings to compression algorithms and archive formats,
 letting the browser simply expose the software it already ships makes good
 sense rather than requiring every JS user to supply his own version.

 Speed may not matter to much on some platforms, but it matters a great deal
 on underpowered devices such as mobiles.

Show me some numbers to back this up and you'll have me convinced :)

Remember that on underpowered devices native code is proportionally slower too.

 Integration is where the support for archives goes beyond being an API,
 where URLs (to link.href, script.src, img.src, iframe.src, audio.src,
 video.src, css url(), etc.) could point into an archive. This cannot be
 done in JS.


 I was going to say exactly  that. I want to be able to have a virtual URL
 that I can point to. In my CSS, I want to do something like
 archive://assets/foo.png after I loaded and decompressed the ZIP file in
 JS.

How does the assets part in the example above work? What does it
mean? Is there some registry here or something?

 Jonas, I'm intrigued – do you see a way this could be done in JS? If so,
 maybe we should build a sample. I'm still thinking the performance won't be
 good enough, particular on mobile devices, but let's find out.

You can actually do this in Gecko already. Any archive that you can
refer to through a URL, you can also reach into.

So if you have a .zip file in a Blob, and you generate a blob: URL
like blob:123-abc then you can read the foo.html file out of that
archive by using the URL jar:blob:123-abc!/foo.html. So far this
doesn't work with ArrayBuffers since there is no way to have a URL
that refers to an ArrayBuffer.

You can even load something from inside a zip file from a server by doing
img src=jar:http://example.com/foo/archive.zip!/image.jpg;

Something like that I definitely agree that we should standardize.

/ Jonas

 On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote:

 The big question we kept running up against at Mozilla is why couldn't
 this simply be implemented as a JS library?

 If performance is the argument we need to back that up with data.

 / Jonas

 On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote:

 Hi Anne, Florian,

 I think the first baby step, or MVP, is the unpacking that Florian
 mentions below. I would definitely like to have the API available on both
 workers and normal context.

 Thanks,
 Paul

 From: Florian Bösch pya...@gmail.com
 Date: Fri, 3 May 2013 14:52:36 +0200
 To: Anne van Kesteren ann...@annevk.nl
 Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile
 cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org, Andrea
 Marchesini amarches...@mozilla.com
 Subject: Re: ZIP archive API?

 I'm interested a JS API that does the following:

 Unpacking:
 - Receive an archive from a Dataurl, Blob, URL object, File (as in
 filesystem API) or Arraybuffer
 - List its content and metadata
 - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer

 Packing:
 - Create an archive
 - Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
 - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer

 To avoid the whole worker/proxy thing and to allow authors to selectively
 choose how they want to handle the data, I'd like to see synchronous and
 asynchronous versions of each. I'd make synchronicity an argument/flag or
 something to avoid API clutter like packSync, packAsync, writeSync,
 writeAsync, and rather like write(data, callback|boolean).

 - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit
 over the top: http://docs.python.org/3/library/zipfile.html
 - Pythons tarfile API is less clutered and easier to use:
 http://docs.python.org/3/library/tarfile.html
 - zip.js isn't really usable as it doesn't support the full range of
 types (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous
 operation needs to rely on a worker, which is bothersome to setup:
 http://stuk.github.io/jszip/

 My own implementation of the tar format only targets array buffers and
 works synchronously, as in.

 var archive = new TarFile(arraybuffer);
 var memberArrayBuffer = archive.get('filename');



 On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren ann...@annevk.nl
 wrote:

 On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus pbak...@zynga.com wrote

Re: ZIP archive API?

2013-05-03 Thread Charles Pritchard
On May 3, 2013, at 3:18 PM, Jonas Sicking jo...@sicking.cc wrote:

 platforms, but it matters a great deal
 on underpowered devices such as mobiles.
 
 Show me some numbers to back this up and you'll have me convinced :)
 
 Remember that on underpowered devices native code is proportionally slower 
 too.


The word around town is that asm.js takes about 2x longer than native compiled 
code.

So if we're looking at inflate and deflate, you've got that metric, and you 
have the memory and bandwidth overhead of including a js-based inflate deflate 
library and maintaining whatever licensing it comes with.

Basic zip construction without compression is trivial to support in JS, and in 
some cases, it's desirable if you're targeting specific cases where the file 
format uses some magic numbers.

-Charles


Re: ZIP archive API?

2013-05-03 Thread Jonas Sicking
On May 3, 2013 4:20 PM, Charles Pritchard ch...@jumis.com wrote:

 On May 3, 2013, at 3:18 PM, Jonas Sicking jo...@sicking.cc wrote:

  platforms, but it matters a great deal
  on underpowered devices such as mobiles.
 
  Show me some numbers to back this up and you'll have me convinced :)
 
  Remember that on underpowered devices native code is proportionally
slower too.


 The word around town is that asm.js takes about 2x longer than native
compiled code.

So how long does the actual compression/decompression take for the size of
archives web developers usually deal with? IO aside given that that is the
same in JS and native.

/ Jonas


Re: ZIP archive API?

2013-05-01 Thread Paul Bakaus
Still waiting for it as well. I think it'd be very useful to transfer sets of 
assets etc.

From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Tue, 30 Apr 2013 15:58:22 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Charles McCathie Nevile 
cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG 
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini 
amarches...@mozilla.commailto:amarches...@mozilla.com, Paul Bakaus 
pbak...@zynga.commailto:pbak...@zynga.com
Subject: Re: ZIP archive API?

I am very interested in working with archives. I'm currently using it as a 
delivery from server (like quake packs), import and export format for WebGL 
apps.


On Tue, Apr 30, 2013 at 3:18 PM, Anne van Kesteren 
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile
cha...@yandex-team.rumailto:cha...@yandex-team.ru wrote:
 Hi all, at the last TPAC there was discussion of a ZIP archive proposal.
 This has come and gone in various guises.

 Are there people currently interested in being able to work with ZIP in a
 web app? Are there implementors and is there an editor?

We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is
implemented as well (cc'd Andrea, who implemented it).

There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about
a somewhat-related proposal by Paul.


--
http://annevankesteren.nl/




ZIP archive API?

2013-04-30 Thread Charles McCathie Nevile
Hi all, at the last TPAC there was discussion of a ZIP archive proposal.  
This has come and gone in various guises.


Are there people currently interested in being able to work with ZIP in a  
web app? Are there implementors and is there an editor?


cheers

Chaals

--
Charles McCathie Nevile - Consultant (web standards) CTO Office, Yandex
  cha...@yandex-team.ru Find more at http://yandex.com



Re: ZIP archive API?

2013-04-30 Thread Anne van Kesteren
On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile
cha...@yandex-team.ru wrote:
 Hi all, at the last TPAC there was discussion of a ZIP archive proposal.
 This has come and gone in various guises.

 Are there people currently interested in being able to work with ZIP in a
 web app? Are there implementors and is there an editor?

We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is
implemented as well (cc'd Andrea, who implemented it).

There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about
a somewhat-related proposal by Paul.


--
http://annevankesteren.nl/



Re: ZIP archive API?

2013-04-30 Thread Florian Bösch
I am very interested in working with archives. I'm currently using it as a
delivery from server (like quake packs), import and export format for WebGL
apps.


On Tue, Apr 30, 2013 at 3:18 PM, Anne van Kesteren ann...@annevk.nl wrote:

 On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile
 cha...@yandex-team.ru wrote:
  Hi all, at the last TPAC there was discussion of a ZIP archive proposal.
  This has come and gone in various guises.
 
  Are there people currently interested in being able to work with ZIP in a
  web app? Are there implementors and is there an editor?

 We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is
 implemented as well (cc'd Andrea, who implemented it).

 There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about
 a somewhat-related proposal by Paul.


 --
 http://annevankesteren.nl/