Re: ZIP archive API?
On 03.05.13 15:18, Jonas Sicking jo...@sicking.cc wrote: On Fri, May 3, 2013 at 12:12 PM, Paul Bakaus pbak...@zynga.com wrote: From: Florian Bösch pya...@gmail.com Date: Fri, 3 May 2013 21:05:17 +0200 To: Jonas Sicking jo...@sicking.cc Cc: Paul Bakaus pbak...@zynga.com, Anne van Kesteren ann...@annevk.nl, Webapps WG public-webapps@w3.org, Charles McCathie Nevile cha...@yandex-team.ru, Andrea Marchesini amarches...@mozilla.com Subject: Re: ZIP archive API? It can be implemented by a JS library, but the three reasons to let the browser provide it are Convenience, speed and integration. Convenience is the first reason, since browsers by far and large already have complete bindings to compression algorithms and archive formats, letting the browser simply expose the software it already ships makes good sense rather than requiring every JS user to supply his own version. Speed may not matter to much on some platforms, but it matters a great deal on underpowered devices such as mobiles. Show me some numbers to back this up and you'll have me convinced :) Remember that on underpowered devices native code is proportionally slower too. Integration is where the support for archives goes beyond being an API, where URLs (to link.href, script.src, img.src, iframe.src, audio.src, video.src, css url(), etc.) could point into an archive. This cannot be done in JS. I was going to say exactly that. I want to be able to have a virtual URL that I can point to. In my CSS, I want to do something like archive://assets/foo.png after I loaded and decompressed the ZIP file in JS. How does the assets part in the example above work? What does it mean? Is there some registry here or something? Actually assets was just referring to the package name. So what you're saying below makes total sense, and is what we want to have. Great stuff! Jonas, I'm intrigued do you see a way this could be done in JS? If so, maybe we should build a sample. I'm still thinking the performance won't be good enough, particular on mobile devices, but let's find out. You can actually do this in Gecko already. Any archive that you can refer to through a URL, you can also reach into. So if you have a .zip file in a Blob, and you generate a blob: URL like blob:123-abc then you can read the foo.html file out of that archive by using the URL jar:blob:123-abc!/foo.html. So far this doesn't work with ArrayBuffers since there is no way to have a URL that refers to an ArrayBuffer. You can even load something from inside a zip file from a server by doing img src=jar:http://example.com/foo/archive.zip!/image.jpg; Something like that I definitely agree that we should standardize. / Jonas On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking jo...@sicking.cc wrote: The big question we kept running up against at Mozilla is why couldn't this simply be implemented as a JS library? If performance is the argument we need to back that up with data. / Jonas On May 3, 2013 10:51 AM, Paul Bakaus pbak...@zynga.com wrote: Hi Anne, Florian, I think the first baby step, or MVP, is the unpacking that Florian mentions below. I would definitely like to have the API available on both workers and normal context. Thanks, Paul From: Florian Bösch pya...@gmail.com Date: Fri, 3 May 2013 14:52:36 +0200 To: Anne van Kesteren ann...@annevk.nl Cc: Paul Bakaus pbak...@zynga.com, Charles McCathie Nevile cha...@yandex-team.ru, public-webapps WG public-webapps@w3.org, Andrea Marchesini amarches...@mozilla.com Subject: Re: ZIP archive API? I'm interested a JS API that does the following: Unpacking: - Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem API) or Arraybuffer - List its content and metadata - Unpack members to Dataurl, Blob, URL object, File or Arraybuffer Packing: - Create an archive - Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer - Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer To avoid the whole worker/proxy thing and to allow authors to selectively choose how they want to handle the data, I'd like to see synchronous and asynchronous versions of each. I'd make synchronicity an argument/flag or something to avoid API clutter like packSync, packAsync, writeSync, writeAsync, and rather like write(data, callback|boolean). - Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the top: http://docs.python.org/3/library/zipfile.html - Pythons tarfile API is less clutered and easier to use: http://docs.python.org/3/library/tarfile.html - zip.js isn't really usable as it doesn't support the full range of types (Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation needs to rely on a worker, which is bothersome to setup: http://stuk.github.io/jszip/ My own implementation of the tar format only targets array buffers and works synchronously, as in. var archive = new TarFile
Re: Collecting real world use cases (Was: Fixing appcache: a proposal to get us started)
On 02.05.13 06:58, Charles McCathie Nevile cha...@yandex-team.ru wrote: On Thu, 18 Apr 2013 19:19:17 +0300, Paul Bakaus pbak...@zynga.com wrote: Hi Jonas, Thanks for this I feel this is heading somewhere, finally! I still need to work on submitting my full feedback, but I'd like to mention this: Why did nobody so far in this thread include real world use cases? For a highly complex topic like this in particular, I would think that collecting a large number of user use cases, not only requirements, and furthermore finding the lowest common denominator based on them, would prove very helpful, even if it's just about validation and making people understand your lengthy proposal. I.e. a news reader that needs to sync content, but has an offline UI. Do you have a list collected somewhere? I started to collect them: http://www.w3.org/wiki/Webapps/AppCacheUseCases Excellent, thanks! So far I added the stuff from this thread in very rough form. I think it is helpful in the Wiki to name people who are actually trying to implement this stuff (the more the merrier, obviously) so we know who are the minimum set of people to talk to about reviewing solutions etc. I will add Zynga to the list. cheers Chaals Thanks! Paul On 26.03.13 16:02, Jonas Sicking jo...@sicking.cc wrote: Hi WebApps! Apologies in advance for a long email. This is a complex subject and I wanted to present a coherent proposal. Please don't be shy about starting separate threads when providing feedback. There has been a lot of debating about fixing appcache. Last year mozilla got a few people together mostly with the goal of understanding what the actual problems were. The notes from that meeting are available at [1]. Those discussions, and a few followup ones, has made it clear that there were a few big ticket items that we needed to fix: * The fact that master entries are automatically added to the cache works very poorly for a lot of developers. * Once a website is cached the user will only see the new version on second load, even if the user is online. This is good for performance but is a behavior many websites aren't willing to live with. * You have to tweak a comment of the manifest in order to trigger an update-check of the cached resources. * We need an escape hatch for people running into missing features in the appcache. I.e. a way for websites to use script to complement the set of behaviors supported by the appcache spec. * The fact that FALLBACK combined the hit network first, fall back to a cached resource and allow a cached resource to handle requests to a whole URL space behaviors is problematic since many times you want one and not the other. * People want to use appcache not just to make offline apps possible, but also make online apps fast. * There isn't enough ability to control the appcache through javascript. There are certainly other things that people have mentioned, but the above have been a reoccurring theme. Feel free to comment here if you have other issues with the current appcache, but it might be worth doing that as separate threads. I believe that some of these problems stem from a relatively small set of design problems: The appcache appears to be aimed at too simple applications. It works fine if the website you want to cache consists of a small set of static resources and otherwise only use features like IndexedDB or localStorage to manage dynamic data. But once an application uses server-side processing to dynamically generate resources based on query parameters or other parts of the URL, then it requires that you change the way that your application works. Another design aspect that appears to be causing problem is that appcache is optimized too heavily for minimizing the amount of typing that the author had to do. It attempts to help the author too much, for example by automatically adding master entries that link to an appcache but aren't enumerated in it. Or automatically adding the handler URLs from the FALLBACK section to the set of URLs to be automatically downloaded. The result is that the appcache contains too much magic. In theory an author can just type very little and the appcache will automatically do the right thing. However this magic is making it too hard for authors to understand what's going on. The result is that people don't use the appcache even if they might have needed to type very little to get it working. Implementations certainly hasn't helped here either, by not exposing the behind-the-scenes logic through debugging and developer tools. The fact that the appcache is aimed at simple applications is generally a good thing given that it's the first version. However the desire to make applications available offline, as well as make them faster when the user is online, has been so great that people have wanted to use the appcache to solve a larger set of types of applications. So to some
Re: ZIP archive API?
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian mentions
below. I would definitely like to have the API available on both workers and
normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Charles McCathie
Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem
API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync, writeAsync,
and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the
top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation
needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and works
synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 21:05:17 +0200
To: Jonas Sicking jo...@sicking.ccmailto:jo...@sicking.cc
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Anne van
Kesteren ann...@annevk.nlmailto:ann...@annevk.nl, Webapps WG
public-webapps@w3.orgmailto:public-webapps@w3.org, Charles McCathie Nevile
cha...@yandex-team.rumailto:cha...@yandex-team.ru, Andrea Marchesini
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?
It can be implemented by a JS library, but the three reasons to let the browser
provide it are Convenience, speed and integration.
Convenience is the first reason, since browsers by far and large already have
complete bindings to compression algorithms and archive formats, letting the
browser simply expose the software it already ships makes good sense rather
than requiring every JS user to supply his own version.
Speed may not matter to much on some platforms, but it matters a great deal on
underpowered devices such as mobiles.
Integration is where the support for archives goes beyond being an API, where
URLs (to link.href, script.src, img.src, iframe.src, audio.src, video.src, css
url(), etc.) could point into an archive. This cannot be done in JS.
I was going to say exactly that. I want to be able to have a virtual URL that
I can point to. In my CSS, I want to do something like
archive://assets/foo.png after I loaded and decompressed the ZIP file in JS.
Jonas, I'm intrigued – do you see a way this could be done in JS? If so, maybe
we should build a sample. I'm still thinking the performance won't be good
enough, particular on mobile devices, but let's find out.
On Fri, May 3, 2013 at 8:04 PM, Jonas Sicking
jo...@sicking.ccmailto:jo...@sicking.cc wrote:
The big question we kept running up against at Mozilla is why couldn't this
simply be implemented as a JS library?
If performance is the argument we need to back that up with data.
/ Jonas
On May 3, 2013 10:51 AM, Paul Bakaus
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Hi Anne, Florian,
I think the first baby step, or MVP, is the unpacking that Florian mentions
below. I would definitely like to have the API available on both workers and
normal context.
Thanks,
Paul
From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com
Date: Fri, 3 May 2013 14:52:36 +0200
To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl
Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Charles McCathie
Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG
public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini
amarches...@mozilla.commailto:amarches...@mozilla.com
Subject: Re: ZIP archive API?
I'm interested a JS API that does the following:
Unpacking:
- Receive an archive from a Dataurl, Blob, URL object, File (as in filesystem
API) or Arraybuffer
- List its content and metadata
- Unpack members to Dataurl, Blob, URL object, File or Arraybuffer
Packing:
- Create an archive
- Put in members passing a Dataurl, Blob, URL object, File or Arraybuffer
- Serialize archive to Dataurl, Blob, URL object, File or Arraybuffer
To avoid the whole worker/proxy thing and to allow authors to selectively
choose how they want to handle the data, I'd like to see synchronous and
asynchronous versions of each. I'd make synchronicity an argument/flag or
something to avoid API clutter like packSync, packAsync, writeSync, writeAsync,
and rather like write(data, callback|boolean).
- Pythons zipfile API is ok, except the getinfo/setinfo stuff is a bit over the
top: http://docs.python.org/3/library/zipfile.html
- Pythons tarfile API is less clutered and easier to use:
http://docs.python.org/3/library/tarfile.html
- zip.js isn't really usable as it doesn't support the full range of types
(Dataurl, Blob, URL object, File or Arraybuffer) and for asynchronous operation
needs to rely on a worker, which is bothersome to setup:
http://stuk.github.io/jszip/
My own implementation of the tar format only targets array buffers and works
synchronously, as in.
var archive = new TarFile(arraybuffer);
var memberArrayBuffer = archive.get('filename');
On Fri, May 3, 2013 at 2:37 PM, Anne van Kesteren
ann...@annevk.nlmailto:ann...@annevk.nl wrote:
On Thu, May 2, 2013 at 1:15 AM, Paul Bakaus
pbak...@zynga.commailto:pbak...@zynga.com wrote:
Still waiting for it as well. I think it'd be very useful to transfer sets
of assets etc.
Do you have anything in particular you'd like to see happen first?
It's pretty clear we should expose more here, but as with all things
we should do it in baby steps.
--
http://annevankesteren.nl/
Re: ZIP archive API?
Still waiting for it as well. I think it'd be very useful to transfer sets of assets etc. From: Florian Bösch pya...@gmail.commailto:pya...@gmail.com Date: Tue, 30 Apr 2013 15:58:22 +0200 To: Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl Cc: Charles McCathie Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru, public-webapps WG public-webapps@w3.orgmailto:public-webapps@w3.org, Andrea Marchesini amarches...@mozilla.commailto:amarches...@mozilla.com, Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com Subject: Re: ZIP archive API? I am very interested in working with archives. I'm currently using it as a delivery from server (like quake packs), import and export format for WebGL apps. On Tue, Apr 30, 2013 at 3:18 PM, Anne van Kesteren ann...@annevk.nlmailto:ann...@annevk.nl wrote: On Tue, Apr 30, 2013 at 1:07 PM, Charles McCathie Nevile cha...@yandex-team.rumailto:cha...@yandex-team.ru wrote: Hi all, at the last TPAC there was discussion of a ZIP archive proposal. This has come and gone in various guises. Are there people currently interested in being able to work with ZIP in a web app? Are there implementors and is there an editor? We have https://wiki.mozilla.org/WebAPI/ArchiveAPI which is implemented as well (cc'd Andrea, who implemented it). There's also https://bugzilla.mozilla.org/show_bug.cgi?id=681967 about a somewhat-related proposal by Paul. -- http://annevankesteren.nl/
Re: Collecting real world use cases (Was: Fixing appcache: a proposal to get us started)
Hi Jonas, hi Alec, I think all of this is great stuff that helps a ton – thanks! Let's definitely put them onto a wiki somewhere. Charles: Where would be the right place to put that list? I'm imagining a big table with columns for the proposal and then each of the AppCache 2.0 proposals validated against it in the others. Thanks, Paul From: Alec Flett alecfl...@chromium.orgmailto:alecfl...@chromium.org Date: Wed, 1 May 2013 08:50:33 -0700 To: Jonas Sicking jo...@sicking.ccmailto:jo...@sicking.cc Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, Webapps WG public-webapps@w3.orgmailto:public-webapps@w3.org Subject: Re: Collecting real world use cases (Was: Fixing appcache: a proposal to get us started) I think there are some good use cases for not-quite-offline as well. Sort of a combination of your twitter and wikipedia use cases: Community-content site: Logged-out users have content cached aggressively offline - meaning every page visited should be cached until told otherwise. Intermediate caches / proxies should be able to cache the latest version of a URL. As soon as a user logs in, the same urls they just used should now have editing controls. (note that actual page contents *may* not have not changed, just the UI) Pages now need to be fresh meaning that users should never edit stale content. In an ideal world, once a logged in user has edited a page, that page is pushed to users or proxies who have previously cached that page and will likely visit it again soon. I know this example in particular seems like it could be accomplished with a series of If-Modified-Since / 304's, but connection latency is the killer here, especially for mobile - the fact that you have a white screen while you wait to see if the page has changed. The idea that you could visit a cached page, (i.e. avoid hitting the network) and then a few seconds later be told there is a newer version of this page available after the fact, (or even just silently update the page so the next visit delivers a fresh but network-free page) would be pretty huge. Especially if you could then proactively fetch a select set of pages - i.e. imagine an in-browser process that says for each link on this page, if I have a stale copy of the url, go fetch it in the background so it is ready in the cache (On this note it would probably be worth reaching out to the wiki foundation to learn about the hassle they've gone through over the years trying to distribute the load of wikipedia traffic given the constraints of HTTP caching, broken proxies, CDNs, ISPs, etc) Alec On Tue, Apr 30, 2013 at 9:06 PM, Jonas Sicking jo...@sicking.ccmailto:jo...@sicking.cc wrote: On Apr 18, 2013 6:19 PM, Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com wrote: Hi Jonas, Thanks for this I feel this is heading somewhere, finally! I still need to work on submitting my full feedback, but I'd like to mention this: Why did nobody so far in this thread include real world use cases? For a highly complex topic like this in particular, I would think that collecting a large number of user use cases, not only requirements, and furthermore finding the lowest common denominator based on them, would prove very helpful, even if it's just about validation and making people understand your lengthy proposal. I.e. a news reader that needs to sync content, but has an offline UI. Do you have a list collected somewhere? Sorry for not including the list in the initial email. It was long enough as it was so I decided to stop. Some of the use cases we discussed were: Small simple game The game consists of a set of static resources. A few HTML pages, like high score page, start page, in-game page, etc. A larger number of media resources. A few data resources which contain level metadata. Small amount of dynamic data being generated, such as progress on each level, high score, user info. In-game performance is critical, all resources must be guaranteed to be available locally once the game starts. Little need for network connectivity other than to update game resources whenever an update is available. Advanced game Same as simple game, but also downloads additional levels dynamically. Also wants to store game progress on servers so that it can be synced across devices. Wikipedia Top level page and its resources are made available offline. Application logic can enable additional pages to be made available offline. When such a page is made available offline both the page and any media resources that it uses needs to be cached. Doesn't need to be updated very aggressively, maybe only upon user request. Twitter A set of HTML templates that are used to create a UI for a database of tweets. The same data is visualized in several different ways, for example in the user's default tweet stream, in the page for an individual tweet, and in the conversation thread view. Downloading the actual tweet contents and metadata shouldn't need to happen
Re: Collecting real world use cases (Was: Fixing appcache: a proposal to get us started)
On 19.04.13 00:01, Charles McCathie Nevile cha...@yandex-team.ru wrote: On Thu, 18 Apr 2013 19:19:17 +0300, Paul Bakaus pbak...@zynga.com wrote: Hi Jonas, Thanks for this I feel this is heading somewhere, finally! I still need to work on submitting my full feedback, but I'd like to mention this: Why did nobody so far in this thread include real world use cases? We submitted a few (they are not completely fleshed out but they are actual things we do) in the related thread http://lists.w3.org/Archives/Public/public-webapps/2013JanMar/0949.html Thanks both! I will dig through the links. Might be worth to collect them all into a list that's easily accessible so we can base any proposal or update to proposals on the UC's and validate against them. For a highly complex topic like this in particular, I would think that collecting a large number of user use cases, not only requirements, and furthermore finding the lowest common denominator based on them, would prove very helpful, even if it's just about validation and making people understand your lengthy proposal. I.e. a news reader that needs to sync content, but has an offline UI. Do you have a list collected somewhere? There are also use cases in the discussions of hosted apps in the SysApps WG - they are essentially working on the same problem. cheers Chaals Thanks! Paul On 26.03.13 16:02, Jonas Sicking jo...@sicking.cc wrote: Hi WebApps! Apologies in advance for a long email. This is a complex subject and I wanted to present a coherent proposal. Please don't be shy about starting separate threads when providing feedback. There has been a lot of debating about fixing appcache. Last year mozilla got a few people together mostly with the goal of understanding what the actual problems were. The notes from that meeting are available at [1]. Those discussions, and a few followup ones, has made it clear that there were a few big ticket items that we needed to fix: * The fact that master entries are automatically added to the cache works very poorly for a lot of developers. * Once a website is cached the user will only see the new version on second load, even if the user is online. This is good for performance but is a behavior many websites aren't willing to live with. * You have to tweak a comment of the manifest in order to trigger an update-check of the cached resources. * We need an escape hatch for people running into missing features in the appcache. I.e. a way for websites to use script to complement the set of behaviors supported by the appcache spec. * The fact that FALLBACK combined the hit network first, fall back to a cached resource and allow a cached resource to handle requests to a whole URL space behaviors is problematic since many times you want one and not the other. * People want to use appcache not just to make offline apps possible, but also make online apps fast. * There isn't enough ability to control the appcache through javascript. There are certainly other things that people have mentioned, but the above have been a reoccurring theme. Feel free to comment here if you have other issues with the current appcache, but it might be worth doing that as separate threads. I believe that some of these problems stem from a relatively small set of design problems: The appcache appears to be aimed at too simple applications. It works fine if the website you want to cache consists of a small set of static resources and otherwise only use features like IndexedDB or localStorage to manage dynamic data. But once an application uses server-side processing to dynamically generate resources based on query parameters or other parts of the URL, then it requires that you change the way that your application works. Another design aspect that appears to be causing problem is that appcache is optimized too heavily for minimizing the amount of typing that the author had to do. It attempts to help the author too much, for example by automatically adding master entries that link to an appcache but aren't enumerated in it. Or automatically adding the handler URLs from the FALLBACK section to the set of URLs to be automatically downloaded. The result is that the appcache contains too much magic. In theory an author can just type very little and the appcache will automatically do the right thing. However this magic is making it too hard for authors to understand what's going on. The result is that people don't use the appcache even if they might have needed to type very little to get it working. Implementations certainly hasn't helped here either, by not exposing the behind-the-scenes logic through debugging and developer tools. The fact that the appcache is aimed at simple applications is generally a good thing given that it's the first version. However the desire to make applications available offline, as well as make them faster when the user is online, has been so
Collecting real world use cases (Was: Fixing appcache: a proposal to get us started)
Hi Jonas, Thanks for this I feel this is heading somewhere, finally! I still need to work on submitting my full feedback, but I'd like to mention this: Why did nobody so far in this thread include real world use cases? For a highly complex topic like this in particular, I would think that collecting a large number of user use cases, not only requirements, and furthermore finding the lowest common denominator based on them, would prove very helpful, even if it's just about validation and making people understand your lengthy proposal. I.e. a news reader that needs to sync content, but has an offline UI. Do you have a list collected somewhere? Thanks! Paul On 26.03.13 16:02, Jonas Sicking jo...@sicking.cc wrote: Hi WebApps! Apologies in advance for a long email. This is a complex subject and I wanted to present a coherent proposal. Please don't be shy about starting separate threads when providing feedback. There has been a lot of debating about fixing appcache. Last year mozilla got a few people together mostly with the goal of understanding what the actual problems were. The notes from that meeting are available at [1]. Those discussions, and a few followup ones, has made it clear that there were a few big ticket items that we needed to fix: * The fact that master entries are automatically added to the cache works very poorly for a lot of developers. * Once a website is cached the user will only see the new version on second load, even if the user is online. This is good for performance but is a behavior many websites aren't willing to live with. * You have to tweak a comment of the manifest in order to trigger an update-check of the cached resources. * We need an escape hatch for people running into missing features in the appcache. I.e. a way for websites to use script to complement the set of behaviors supported by the appcache spec. * The fact that FALLBACK combined the hit network first, fall back to a cached resource and allow a cached resource to handle requests to a whole URL space behaviors is problematic since many times you want one and not the other. * People want to use appcache not just to make offline apps possible, but also make online apps fast. * There isn't enough ability to control the appcache through javascript. There are certainly other things that people have mentioned, but the above have been a reoccurring theme. Feel free to comment here if you have other issues with the current appcache, but it might be worth doing that as separate threads. I believe that some of these problems stem from a relatively small set of design problems: The appcache appears to be aimed at too simple applications. It works fine if the website you want to cache consists of a small set of static resources and otherwise only use features like IndexedDB or localStorage to manage dynamic data. But once an application uses server-side processing to dynamically generate resources based on query parameters or other parts of the URL, then it requires that you change the way that your application works. Another design aspect that appears to be causing problem is that appcache is optimized too heavily for minimizing the amount of typing that the author had to do. It attempts to help the author too much, for example by automatically adding master entries that link to an appcache but aren't enumerated in it. Or automatically adding the handler URLs from the FALLBACK section to the set of URLs to be automatically downloaded. The result is that the appcache contains too much magic. In theory an author can just type very little and the appcache will automatically do the right thing. However this magic is making it too hard for authors to understand what's going on. The result is that people don't use the appcache even if they might have needed to type very little to get it working. Implementations certainly hasn't helped here either, by not exposing the behind-the-scenes logic through debugging and developer tools. The fact that the appcache is aimed at simple applications is generally a good thing given that it's the first version. However the desire to make applications available offline, as well as make them faster when the user is online, has been so great that people have wanted to use the appcache to solve a larger set of types of applications. So to some extent the appcache has been a victim of its own success. The other week a few of us at mozilla got together to discuss how to fix appcache. I.e. how to come up with a solution for the above mentioned problems. We came to a few conclusions. We still want to try to keep a declarative solution. While the current appcache appears to only work really well for very simple applications, we hope that it is possible to find a declarative format which is simple enough to be understandable and practical, while still supporting a large number of applications. However, we do think that there needs to be a script-driven fallback. A declarative solution
Re: Shrinking existing libraries as a goal
I wholeheartedly support this proposal. +1000. From: Yehuda Katz wyc...@gmail.commailto:wyc...@gmail.com Date: Wed, 16 May 2012 00:32:51 -0400 To: public-webapps@w3.orgmailto:public-webapps@w3.org Subject: Shrinking existing libraries as a goal Resent-From: public-webapps@w3.orgmailto:public-webapps@w3.org Resent-Date: Wed, 16 May 2012 04:33:57 + In the past year or so, I've participated in a number of threads that were implicitly about adding features to browsers that would shrink the size of existing libraries. Inevitably, those discussions end up litigating whether making it easier for jQuery (or some other library) to do the task is a good idea in the first place. While those discussions are extremely useful, I feel it would be useful for a group to focus on proposals that would shrink the size of existing libraries with the implicit assumption that it was a good idea. From some basic experimentation I've personally done with the jQuery codebase, I feel that such a group could rather quickly identify enough areas to make a much smaller version of jQuery that ran on modern browsers plausible. I also think that having data to support or refute that assertion would be useful, as it's often made casually in meta-discussions. If there is a strong reason that people feel that a focused effort to identify ways to shrink existing popular libraries in new browsers would be a bad idea, I'd be very interested to hear it. Thanks so much for your consideration, Yehuda Katz jQuery Foundation (ph) 718.877.1325
Re: CfC Re: Charter addition proposal: screen orientation lock
Great! Am 30.03.12 11:28 schrieb Mounir Lamouri unter mou...@lamouri.fr: On 02/15/2012 10:09 AM, Vincent Scheib wrote: Mounir, I ran into the same confusion regarding how does the API expose locking?. May I suggest that you explicitly state in the abstract that the API is pending? I just updated the draft so the locking part is now part of it. Cheers, -- Mounir
Re: CfC Re: Charter addition proposal: screen orientation lock
Hi Arthur, In the Screen Orientation API draft, I don't see any references to locking. Is this by design? Thanks, Paul Am 06.02.12 13:19 schrieb Arthur Barstow unter art.bars...@nokia.com: Given the positive responses to this CfC, the Screen Orientation API has now been added to the Additions Agreed section of charter changes wiki http://www.w3.org/2008/webapps/wiki/CharterChanges. On 1/30/12 8:26 AM, ext Charles McCathieNevile wrote: OK, since I was planning to have the charter up today, let's have a quick call for consensus on this. Please reply by end of business Wednesday if you support or object to this - silence will be taken as not explicitly supporting it, and without support it isn't going to get into the draft charter. If it does go there, there will still be opportunities to object but it will be harder to squeeze in. cheers Chaals On Mon, 30 Jan 2012 12:22:30 +0100, Robin Berjon ro...@berjon.com wrote: Hi all! Sorry for bringing this to the group this late, but it's a topic that's been discussed in other places and that I believe is both useful and mature enough to be ready for standardisation. Some applications are designed in such a way that they only make sense in one device orientation. The archetypical example would be a game that only works in landscape mode, but there are other examples. Right now native apps can support this rather easily, but web apps have been stuck with silly hacks such as detecting that the orientation is wrong and asking the user to rotate. This further leads to trouble when the device itself is used as a controller (e.g. in racing games) as this can sometimes trigger an undesired orientation change mid-game ‹ hardly a user-friendly experience. Note that this is not about system-level orientation lock (which would be fodder for another group) but application-level orientation. Options to address this have been discussed (amongst other places) here: http://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/f 38bb05e66c01a77# There is discussion as to whether this ought to be only an API or if it should use a meta element (which would also give it an API since it could be changed dynamically), with an overall leaning towards the latter. I am rather confident that we should be able to agree on the best approach relatively quickly. I will let implementers speak for themselves, but my understanding is that there is interest in this feature. It is certainly a regular request from developers. In previous discussions we haven't hashed out who would stand up as editor and test facilitator, but I'm confident that we can find people. If no one else steps up, I'll take the testing hat. WDYT?
Re: CfC: Proposal to add web packaging / asset compression
Hi guys, I'm not particularly set on one direction to solve this problem. I just want to get it solved, and if SPDY, along with a much improved programmatically controllable appCache is the preferred solution, let's go for it. I, along with probably plenty of other web developers, am still inexperienced with SPDY, and thus, maybe we should take this conversation into a different direction, trying to come up with a backward-compatible SPDY demo that solves every issue of asset delivering and caching. Here's a rough list of things it needs to do: * transfer an initial set of assets to the client to display the page (game) * later on, pull subsequent packages of assets when needed * Cache sets of assets locally * Request delta updates for cached asset packages If these are all manageable with SPDY, I'm in. Last question – where is the standards work of SPDY happening? All I could find was this page http://www.chromium.org/spdy, that points to three expired documents submitted to IETF. Thanks, Paul Von: Yehuda Katz wyc...@gmail.commailto:wyc...@gmail.com Datum: Tue, 14 Feb 2012 11:54:47 -0800 An: Tab Atkins Jr. jackalm...@gmail.commailto:jackalm...@gmail.com Cc: Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com, public-webapps@w3.orgmailto:public-webapps@w3.org public-webapps@w3.orgmailto:public-webapps@w3.org Betreff: Re: CfC: Proposal to add web packaging / asset compression I would agree with this. My initial thought when reading the proposal was SPDY as well. That said, there is ongoing discussion about improving the app-cache that is also relevant[1]. I am also planning on opening a discussion about programmatic control of a cache (probably not piggy-backed onto app-cache, which has important atomicity guarantees and no programmatic control, but possibly piggy-backed off of the File API). Between SPDY, improving app cache semantics, and a clean way to programmatically store remote assets that can be loaded via script, link and img tags, I think we have a solution that does not require creating a new packaging format. [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=14702 Yehuda Katz (ph) 718.877.1325 On Tue, Feb 14, 2012 at 10:26 AM, Tab Atkins Jr. jackalm...@gmail.commailto:jackalm...@gmail.com wrote: On Tue, Feb 14, 2012 at 1:24 AM, Paul Bakaus pbak...@zynga.commailto:pbak...@zynga.com wrote: Hi everybody, This is a proposal to add a packaging format transparent to browsers to the charter. At Zynga, we have identified this as one of our most pressuring issues. Developers want to be able to send a collection of assets to the browser through a single request, instead of hundreds. Today, we misuse image and audio sprites, slicing them again as base64 only to put them into weird caches. These are workarounds, and ugly ones, as well. None of the workarounds is satisfying, either in terms of robustness, performance or simply, a sane API. Coincidentally, this is also one of the most pressuring issues of WebGL. Since you are dealing with a lot of assets with WebGL games, proper solutions must be found. I was once a believer in an approach like this, and supported previous attempts at it like Mozilla's use of a zip + virtual paths. Now, though, SPDY seems to be moving along nicely enough that we don't really need to worry about this. It's already supported in Chrome and Firefox, and it lets you pull multiple assets in a single connection, push assets that haven't yet been requested, and prioritize asset retrieval. I don't feel there's any real need to worry about asset packaging formats anymore. ~TJ
Re: CfC Re: Charter addition proposal: screen orientation lock
Cool, thanks for the update! Am 15.02.12 13:21 schrieb Mounir Lamouri unter mou...@lamouri.fr: On 02/15/2012 04:29 AM, Tobie Langel wrote: In the Screen Orientation API draft, I don't see any references to locking. Is this by design? It's in the abstract: The Screen Orientation API's goal is to provide an interface for web applications to be able to read the screen orientation state, to be informed when this state changes and to be able to lock the screen orientation to a specific state. --http://dvcs.w3.org/hg/screen-orientation/raw-file/tip/Overview.html I only wrote the reading part for the moment because there are still some discussions in Mozilla's WebAPI mailing list [1] about the locking part. When the specification draft will be complete, I will send a message here. [1] https://www.mozilla.org/about/forums/#dev-webapi -- Mounir
CfC: Proposal to add web packaging / asset compression
Hi everybody,
This is a proposal to add a packaging format transparent to browsers to the
charter. At Zynga, we have identified this as one of our most pressuring
issues. Developers want to be able to send a collection of assets to the
browser through a single request, instead of hundreds.
Today, we misuse image and audio sprites, slicing them again as base64 only to
put them into weird caches. These are workarounds, and ugly ones, as well. None
of the workarounds is satisfying, either in terms of robustness, performance or
simply, a sane API. Coincidentally, this is also one of the most pressuring
issues of WebGL. Since you are dealing with a lot of assets with WebGL games,
proper solutions must be found.
A ticket at Mozilla, describing the issue further, has been opened by us here:
https://bugzilla.mozilla.org/show_bug.cgi?id=681967
Here's an actual code draft I attached to the ticket:
window.loadPackage('package.webpf', function() {
var img = new Image();
img.src = package.webpf/myImage.png;
})
Or alternatively, with a local storage system (I prefer option one):
window.loadPackage('package.webpf', function(files) {
files[0].saveTo('myImage.png');
var img = new Image();
img.src = local://absolute path of url of site/myImage.png;
})
No big deal if the whole API looks entirely different when it's done. The
format needs to be able to handle delta updates well, and must be cacheable. It
needs to be transparent to the browser, and assets of uncompressed web packages
need to be able to be included from CSS. I am aware this is a more inconvenient
addition to work on, but there is immediate need.
This is also a call for implementors, testers and editors. I am unfortunately
not experienced enough to handle any of those jobs.
Thanks, I'm looking forward to feedback!
Paul Bakaus
W3C Rep and Studio CTO, Zynga
Re: CfC Re: Charter addition proposal: screen orientation lock
Zynga wholeheartedly supports screen orientation locking! Am 30.01.12 14:26 schrieb Charles McCathieNevile unter cha...@opera.com: OK, since I was planning to have the charter up today, let's have a quick call for consensus on this. Please reply by end of business Wednesday if you support or object to this - silence will be taken as not explicitly supporting it, and without support it isn't going to get into the draft charter. If it does go there, there will still be opportunities to object but it will be harder to squeeze in. cheers Chaals On Mon, 30 Jan 2012 12:22:30 +0100, Robin Berjon ro...@berjon.com wrote: Hi all! Sorry for bringing this to the group this late, but it's a topic that's been discussed in other places and that I believe is both useful and mature enough to be ready for standardisation. Some applications are designed in such a way that they only make sense in one device orientation. The archetypical example would be a game that only works in landscape mode, but there are other examples. Right now native apps can support this rather easily, but web apps have been stuck with silly hacks such as detecting that the orientation is wrong and asking the user to rotate. This further leads to trouble when the device itself is used as a controller (e.g. in racing games) as this can sometimes trigger an undesired orientation change mid-game ‹ hardly a user-friendly experience. Note that this is not about system-level orientation lock (which would be fodder for another group) but application-level orientation. Options to address this have been discussed (amongst other places) here: http://groups.google.com/group/mozilla.dev.webapi/browse_thread/thread/f3 8bb05e66c01a77# There is discussion as to whether this ought to be only an API or if it should use a meta element (which would also give it an API since it could be changed dynamically), with an overall leaning towards the latter. I am rather confident that we should be able to agree on the best approach relatively quickly. I will let implementers speak for themselves, but my understanding is that there is interest in this feature. It is certainly a regular request from developers. In previous discussions we haven't hashed out who would stand up as editor and test facilitator, but I'm confident that we can find people. If no one else steps up, I'll take the testing hat. WDYT? -- Charles 'chaals' McCathieNevile Opera Software, Standards Group je parle français -- hablo español -- jeg kan litt norsk http://my.opera.com/chaals Try Opera: http://www.opera.com
Re: Enable compression of a blob to .zip file
Fyi - it's been a long time request from Zynga to have a web packaging format to serve and store assets in large chunks. A relevant ticket with a discussion can be found here: https://bugzilla.mozilla.org/show_bug.cgi?id=681967 I'm all for it, of course. Am 31.10.11 12:20 schrieb Charles Pritchard unter ch...@jumis.com: On 10/31/11 11:34 AM, Boris Zbarsky wrote: On 10/31/11 1:42 PM, Charles Pritchard wrote: I'm almost certain that somewhere, it is specified that the browser should sniff the first few bytes of the file to see if it is compressed. I don't believe it is. In fact, doing that would contradict the specs as they stand, to my knowledge. I could be wrong, of course, but in that case citation needed... -Boris Well I've failed to find a citation. I will post one if I find it. In the meantime: Blob and Data uris can and should have sniffing for image/svg+xml. It's trivial, and it makes a whole lot more sense than extending both the Blob and data uri specs to include transfer encoding semantics. file: and filesystem: and widget urls are items that -might- be supported on an OS level, and thus out of scope here. Back to deflate use cases: PDF.js I'm sure implements deflate (for PDF FlateEncode), I've recently done some docx work which required deflate. Many servers do not host .svgz files well, and I use XHR with deflate to deal with that (though I would have just used blob urls if they worked). localStorage, indexedDB and WebSQL all require DOMString, as do most WebSocket implementations -- in practical use, this means base64 encoding data. It's another area where deflate could squeeze a few more bytes out of existing constraints. Especially with localStorage. As we continue to support more-and-more document formats, deflate support on the client side becomes more important. Apple and MS, two very large vendors, have pushed ahead with using deflate in various file formats they use. Adobe has been doing it for some time. Though HTTP works quite well for negotiated compression there are more and more cases that are not http bound. I would very much like to see deflate and inflate exposed to the scripting environment. From there I can easily interact with several file formats. Zip included. -Charles -Charles
Re: Joystick support
Hi Scott, We should definitely open it up for a broader discussion. I think the right place for the discussion might be the new device apis WG, which I've cc'ed. Thanks, Paul Am 24.08.11 21:23 schrieb Scott Graham unter scot...@chromium.org: Hello, I noticed that Mozilla has started to prototype support for Joystick events. There's some documentation on this effort https://wiki.mozilla.org/JoystickAPI as well as a prototype https://bugzilla.mozilla.org/show_bug.cgi?id=604039 I'm also interested in adding support for joysticks to Chromium and so would like to open the discussion up to a broader audience. Do others see this as a valuable API? Or have comments on the design? scott
Re: API for matrix manipulation
Just wanted to throw in Sylvester, which is a fairly cool Math JS lib for generic Matrix and Vector computations (http://sylvester.jcoglan.com/). I especially like it as the API feels like it is designed for human beings. Having Matrix/Vector as generic interfaces sounds like a big win. Am 18.03.11 21:50 schrieb Chris Marrin unter cmar...@apple.com: On Mar 15, 2011, at 5:08 PM, Tab Atkins Jr. wrote: On Tue, Mar 15, 2011 at 5:00 PM, Chris Marrin cmar...@apple.com wrote: I think it would be nice to unify the classes somehow. But that might be difficult since SVG and CSS are (necessarily) separate specs. But maybe one of the API gurus has a solution? We just discussed this on Monday at the FXTF telcon. Sounds like people are, in general, okay with just using a 4x4 matrix, though there are some possible implementation issues with devices that can't do 3d at all. (It was suggested that they can simply do a 2d projection, which is simple.) I don't think there are implementation issues other than performance related ones. As you say, you can always flatten a 3D matrix for use in a purely 2D renderer. We do this in the WebKit implementation in some cases. But doing 4x4 matrix math can be expensive, especially on less capable hardware,. So it would probably be valuable to have a set of 2D affine calls on any future 4x4 class, so an implementation can easily optimize by knowing they can get away with doing a subset of the math if all the operands are 2D affine matrices. But that's just a bit of extra API. - ~Chris cmar...@apple.com
