Re: Fixing appcache: a proposal to get us started
On Thu, 28 Nov 2013 16:38:55 +1000, pira...@gmail.com pira...@gmail.com
wrote:
Json manifest seems a nice solution to me :-)
Blergh! :-)
More to the point, it seems that the approach of ServiceWorker discussed
at the TPAC meeting, and based on Alex Russell's proposal which is
unfortunately lacking an actual spec, is getting considerable traction as
the path to follow for fixing appcache.
It would be useful to get a sense of whether people think we should do
something else.
cheers
Chaals
Send from my Samsung Galaxy Note II
El 28/11/2013 07:21, eli elib...@gmail.com escribió:
The web is server + client sides. Trying to fix issues you have
with
client technologies only (appcache, JavaScript, ...) will always be
a
bad
choice.
I disagree, Javascript and web browsers are becoming powerful enough
to delegate servers to their barebones, just offering storage or
databases or specific web services, being able to delegate all the
operatibility to the client-side code. In the new web, web servers
are
just plain ol' API
It's not that much a question of available power, it's just operations
that needs to be done before any file hit the device.
To be available offline, the device has to hit a server first, then the
appcache magic happens.
No reason the server couldn't prepare / select what to send to the
device: iOS won't support WebM anytime soon, there is no reason to
constantly ask iOS device the same info again again. That just makes
no
sense, and force devs to produce device/os specific files (manifest)
anyway.
And it's not AppCache job to do so. Its job is just make a web document
available offline + make updates simple easy.
Example : Not being able to update one single file keeping the others
cached is a structural mistake. Sub-manifests sounds like an
over-engineered fix to me, just making things more complicated for
developers, browser vendors for future evolution of this
specification.
Could the problems of not being able to update one single file in the
cache, and not sending WebM files to iOS devices, both be solved by
adding
additional file info to the cache manifest?
For example, if the manifest were in JSON:
{'CACHE': [
{'file':'index.html','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02
00:00:00','type':'text/html'},
{'file':'video.webm','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02
00:00:00','type':'video/webm'},
{'file':'video.mp4','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02
00:00:00','type':'video/mp4'}
],
'NETWORK':'*',
'FALLBACK':[['online.jpg','offline.jpg'],['online.htm','offline.htm']],
'SETTINGS':'prefer-online'}
This way, a browser can compare a file's timestamp in the newly
downloaded
manifest to the one in its stored manifest to determine whether or not
to
download a new version. And an iOS device could ignore 'video/webm'
file-types.
-Eli
--
Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix.
– Linus Tordvals, creador del sistema operativo Linux
--
Charles McCathie Nevile - Consultant (web standards) CTO Office, Yandex
cha...@yandex-team.ru Find more at http://yandex.com
Re: Fixing appcache: a proposal to get us started
The only problem I see now with that approach is that a browser that
supports
both webm and mp4 videos would download both videos. It seems to me there
should be a way for the browser to determine duplicate content in the
manifest
file in the same way that it determines which video file to download
based on
the file-type.
Something like this might work better:
{'CACHE': [
{'file':'index.html','timestamp':'2013-11-27 00:00:00','type':'text/html'},
{'video': {'sources': [
{'file':'video.webm','timestamp':'2013-11-27
00:00:00','type':'video/webm'},
{'file':'video.mp4','timestamp':'2013-11-27
00:00:00','type':'video/mp4'}
]}
}
],
'NETWORK':'*',
'FALLBACK':[['online.jpg','offline.jpg'],['online.htm','offline.htm']],
'SETTINGS':'prefer-online'}
At least this way the appcache inherits the problem of video in HTML,
and the browser vendors can apply the present solution to the appcache.
-Eli
On 11/27/2013 10:38 PM, pira...@gmail.com wrote:
Json manifest seems a nice solution to me :-)
Send from my Samsung Galaxy Note II
El 28/11/2013 07:21, eli elib...@gmail.com
mailto:elib...@gmail.com escribió:
The web is server + client sides. Trying to fix issues
you have with
client technologies only (appcache, JavaScript, ...) will
always be a bad
choice.
I disagree, Javascript and web browsers are becoming
powerful enough
to delegate servers to their barebones, just offering storage or
databases or specific web services, being able to delegate
all the
operatibility to the client-side code. In the new web, web
servers are
just plain ol' API
It's not that much a question of available power, it's just
operations that needs to be done before any file hit the device.
To be available offline, the device has to hit a server first,
then the appcache magic happens.
No reason the server couldn't prepare / select what to send to
the device: iOS won't support WebM anytime soon, there is no
reason to constantly ask iOS device the same info again
again. That just makes no sense, and force devs to produce
device/os specific files (manifest) anyway.
And it's not AppCache job to do so. Its job is just make a web
document available offline + make updates simple easy.
Example : Not being able to update one single file keeping the
others cached is a structural mistake. Sub-manifests sounds
like an over-engineered fix to me, just making things more
complicated for developers, browser vendors for future
evolution of this specification.
Could the problems of not being able to update one single file in
the cache, and not sending WebM files to iOS devices, both be
solved by adding additional file info to the cache manifest?
For example, if the manifest were in JSON:
{'CACHE': [
{'file':'index.html','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02 00:00:00','type':'text/html'},
{'file':'video.webm','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02 00:00:00','type':'video/webm'},
{'file':'video.mp4','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02 00:00:00','type':'video/mp4'}
],
'NETWORK':'*',
'FALLBACK':[['online.jpg','offline.jpg'],['online.htm','offline.htm']],
'SETTINGS':'prefer-online'}
This way, a browser can compare a file's timestamp in the newly
downloaded manifest to the one in its stored manifest to determine
whether or not to download a new version. And an iOS device could
ignore 'video/webm' file-types.
-Eli
--
Si quieres viajar alrededor del mundo y ser invitado a
hablar en un
monton de sitios diferentes, simplemente escribe un sistema
operativo
Unix.
– Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
The web is server + client sides. Trying to fix issues you have with
client technologies only (appcache, JavaScript, ...) will always be a bad
choice.
I disagree, Javascript and web browsers are becoming powerful enough
to delegate servers to their barebones, just offering storage or
databases or specific web services, being able to delegate all the
operatibility to the client-side code. In the new web, web servers are
just plain ol' API
It's not that much a question of available power, it's just operations that
needs to be done before any file hit the device.
To be available offline, the device has to hit a server first, then the appcache
magic happens.
No reason the server couldn't prepare / select what to send to the device: iOS
won't support WebM anytime soon, there is no reason to constantly ask iOS device
the same info again again. That just makes no sense, and force devs to
produce device/os specific files (manifest) anyway.
And it's not AppCache job to do so. Its job is just make a web document available
offline + make updates simple easy.
Example : Not being able to update one single file keeping the others cached is a
structural mistake. Sub-manifests sounds like an over-engineered fix to me, just
making things more complicated for developers, browser vendors for future
evolution of this specification.
Could the problems of not being able to update one single file in the
cache, and not sending WebM files to iOS devices, both be solved by
adding additional file info to the cache manifest?
For example, if the manifest were in JSON:
{'CACHE': [
{'file':'index.html','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02 00:00:00','type':'text/html'},
{'file':'video.webm','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02 00:00:00','type':'video/webm'},
{'file':'video.mp4','timestamp':'2013-11-27
00:00:00','expires':'2013-12-02 00:00:00','type':'video/mp4'}
],
'NETWORK':'*',
'FALLBACK':[['online.jpg','offline.jpg'],['online.htm','offline.htm']],
'SETTINGS':'prefer-online'}
This way, a browser can compare a file's timestamp in the newly
downloaded manifest to the one in its stored manifest to determine
whether or not to download a new version. And an iOS device could ignore
'video/webm' file-types.
-Eli
--
Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix.
– Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
Json manifest seems a nice solution to me :-)
Send from my Samsung Galaxy Note II
El 28/11/2013 07:21, eli elib...@gmail.com escribió:
The web is server + client sides. Trying to fix issues you have with
client technologies only (appcache, JavaScript, ...) will always be a
bad
choice.
I disagree, Javascript and web browsers are becoming powerful enough
to delegate servers to their barebones, just offering storage or
databases or specific web services, being able to delegate all the
operatibility to the client-side code. In the new web, web servers are
just plain ol' API
It's not that much a question of available power, it's just operations
that needs to be done before any file hit the device.
To be available offline, the device has to hit a server first, then the
appcache magic happens.
No reason the server couldn't prepare / select what to send to the
device: iOS won't support WebM anytime soon, there is no reason to
constantly ask iOS device the same info again again. That just makes no
sense, and force devs to produce device/os specific files (manifest) anyway.
And it's not AppCache job to do so. Its job is just make a web document
available offline + make updates simple easy.
Example : Not being able to update one single file keeping the others
cached is a structural mistake. Sub-manifests sounds like an
over-engineered fix to me, just making things more complicated for
developers, browser vendors for future evolution of this specification.
Could the problems of not being able to update one single file in the
cache, and not sending WebM files to iOS devices, both be solved by adding
additional file info to the cache manifest?
For example, if the manifest were in JSON:
{'CACHE': [
{'file':'index.html','timestamp':'2013-11-27 00:00:00','expires':'2013-12-02
00:00:00','type':'text/html'},
{'file':'video.webm','timestamp':'2013-11-27 00:00:00','expires':'2013-12-02
00:00:00','type':'video/webm'},
{'file':'video.mp4','timestamp':'2013-11-27 00:00:00','expires':'2013-12-02
00:00:00','type':'video/mp4'}
],
'NETWORK':'*',
'FALLBACK':[['online.jpg','offline.jpg'],['online.htm','offline.htm']],
'SETTINGS':'prefer-online'}
This way, a browser can compare a file's timestamp in the newly downloaded
manifest to the one in its stored manifest to determine whether or not to
download a new version. And an iOS device could ignore 'video/webm'
file-types.
-Eli
--
Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix.
– Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
The web is server + client sides. Trying to fix issues you have with client technologies only (appcache, JavaScript, ...) will always be a bad choice. I disagree, Javascript and web browsers are becoming powerful enough to delegate servers to their barebones, just offering storage or databases or specific web services, being able to delegate all the operatibility to the client-side code. In the new web, web servers are just plain ol' APIs. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
The web is server + client sides. Trying to fix issues you have with client technologies only (appcache, JavaScript, ...) will always be a bad choice. I disagree, Javascript and web browsers are becoming powerful enough to delegate servers to their barebones, just offering storage or databases or specific web services, being able to delegate all the operatibility to the client-side code. In the new web, web servers are just plain ol' API It's not that much a question of available power, it's just operations that needs to be done before any file hit the device. To be available offline, the device has to hit a server first, then the appcache magic happens. No reason the server couldn't prepare / select what to send to the device: iOS won't support WebM anytime soon, there is no reason to constantly ask iOS device the same info again again. That just makes no sense, and force devs to produce device/os specific files (manifest) anyway. And it's not AppCache job to do so. Its job is just make a web document available offline + make updates simple easy. Example : Not being able to update one single file keeping the others cached is a structural mistake. Sub-manifests sounds like an over-engineered fix to me, just making things more complicated for developers, browser vendors for future evolution of this specification. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
To be available offline, the device has to hit a server first, then the appcache magic happens. Obviously. No reason the server couldn't prepare / select what to send to the device: iOS won't support WebM anytime soon, there is no reason to constantly ask iOS device the same info again again. That just makes no sense, and force devs to produce device/os specific files (manifest) anyway. And it's not AppCache job to do so. Its job is just make a web document available offline + make updates simple easy. I'm not talking about remove the servers functionality, I only disagree with your statement about using only client-side technologies is bad, giving the impression that servers are necesary. Not today, they are necesary for bootstraping and access content on a reference place, but we should remove the idea of a centralized web as far as possible. AppCache here is a good tool for this, but here the server is only neccesary to generate the custom manifest file, no more, and also probably this manifest could be generated directly on client-side (don't know if it could be done with today technologies, but at least with ServiceWorkers it will be possible, and using them probably the AppCache manifest becames useless in this case of use). -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
Your manifest file should be dynamically generated by your server, based on what you know about the user's browser. Now you have one single manifest file which is easier for updates, + server-side language comments so documentation is easy. The web is server + client sides. Trying to fix issues you have with client technologies only (appcache, JavaScript, ...) will always be a bad choice. I would contest this point as now you're suggesting that the right thing to do is to force everyone to maintain custom browser detection libraries on their servers again in order to guess what the browser's capabilities are. As we know from much experience with this technique already, it is brittle and must be constantly monitored and updated as new browser versions are released. Why not attempt to give the browser-side manifest functionality the ability to feature test for file support instead? Then the browsers can be the trusted source instead of everyone having to create new divergent browser file support inference hacks. Sincerely, James Greene On Mon, Nov 25, 2013 at 9:30 AM, pira...@gmail.com pira...@gmail.comwrote: To be available offline, the device has to hit a server first, then the appcache magic happens. Obviously. No reason the server couldn't prepare / select what to send to the device: iOS won't support WebM anytime soon, there is no reason to constantly ask iOS device the same info again again. That just makes no sense, and force devs to produce device/os specific files (manifest) anyway. And it's not AppCache job to do so. Its job is just make a web document available offline + make updates simple easy. I'm not talking about remove the servers functionality, I only disagree with your statement about using only client-side technologies is bad, giving the impression that servers are necesary. Not today, they are necesary for bootstraping and access content on a reference place, but we should remove the idea of a centralized web as far as possible. AppCache here is a good tool for this, but here the server is only neccesary to generate the custom manifest file, no more, and also probably this manifest could be generated directly on client-side (don't know if it could be done with today technologies, but at least with ServiceWorkers it will be possible, and using them probably the AppCache manifest becames useless in this case of use). -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
Why not attempt to give the browser-side manifest functionality the ability to feature test for file support instead? Then the browsers can be the trusted source instead of everyone having to create new divergent browser file support inference hacks. This seems to me that this is some kind of add scripting habilities to the AppCache manifest, while we already have Javascript, and allowing it to do that will lead us to something fairly similar (in fact, a sub-set) of what ServiceWorkers can do. Why duplicate efforts then? Manifest files are good for static content, but for other cases there are better tools. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
Le 25 nov. 2013 à 17:27, James Greene james.m.gre...@gmail.com a écrit : Your manifest file should be dynamically generated by your server, based on what you know about the user's browser. Now you have one single manifest file which is easier for updates, + server-side language comments so documentation is easy. The web is server + client sides. Trying to fix issues you have with client technologies only (appcache, JavaScript, ...) will always be a bad choice. I would contest this point as now you're suggesting that the right thing to do is to force everyone to maintain custom browser detection libraries on their servers again in order to guess what the browser's capabilities are. As we know from much experience with this technique already, it is brittle and must be constantly monitored and updated as new browser versions are released. Why not attempt to give the browser-side manifest functionality the ability to feature test for file support instead? Then the browsers can be the trusted source instead of everyone having to create new divergent browser file support inference hacks. Sincerely, James Greene I'm not rejecting anything or trying to state something is good or bad. Just like i'm not rejecting the server side as a devil to kill. Current AppCache state is not THAT bad for static document cache. It's pretty much 100% client-side, and with a little bit of extra work to fix how it handles files updates, it might be ok, still easy to deal with. (file by file update without complete manifest swip, grouped files versionning, end of validity datetime, …) Then for further / more complicate / specific use cases, some server-side scripts might be needed. And trying to do everything only from the client-side is a bad choice, just like previous things we did thinking server-side only was a bad choice too. Remi
Re: Fixing appcache: a proposal to get us started
This seems to me that this is some kind of add scripting habilities to the AppCache manifest, while we already have Javascript, and allowing it to do that will lead us to something fairly similar (in fact, a sub-set) of what ServiceWorkers can do. Why duplicate efforts then? Manifest files are good for static content, but for other cases there are better tools. Yes and no. Primary differences: 1. I'm not advocating for full scriptability, just basic support detection, e.g.: `if accepts(audio/ogg) ...` The main problem I'd see there is if the browser also needs to know what plugins (or even JS libs, I suppose) are capable of consuming. 2. AFAIK, using ServiceWorkers requires that the resources are programmatically fetched *by* the ServiceWorker. This is different than a manifest which is capable of figuring the cache out on the fly as resources are loaded. I'm not opposed to just punted and saying that ServiceWorker is the way to deal with this, though. I have no personal use cases at the moment for this functionality... just playing devil's advocate for the OP. :) Sincerely, James Greene On Mon, Nov 25, 2013 at 11:05 AM, pira...@gmail.com pira...@gmail.comwrote: Why not attempt to give the browser-side manifest functionality the ability to feature test for file support instead? Then the browsers can be the trusted source instead of everyone having to create new divergent browser file support inference hacks. This seems to me that this is some kind of add scripting habilities to the AppCache manifest, while we already have Javascript, and allowing it to do that will lead us to something fairly similar (in fact, a sub-set) of what ServiceWorkers can do. Why duplicate efforts then? Manifest files are good for static content, but for other cases there are better tools. -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
1. I'm not advocating for full scriptability, just basic support detection, e.g.: `if accepts(audio/ogg) ...` Some kind of basic scriptability like the one on CSS, isn't it? Ok, it's good. The main problem I'd see there is if the browser also needs to know what plugins (or even JS libs, I suppose) are capable of consuming. More or less why I was going down the full Javascript path :-) 2. AFAIK, using ServiceWorkers requires that the resources are programmatically fetched by the ServiceWorker. This is different than a manifest which is capable of figuring the cache out on the fly as resources are loaded. Not really, ServiceWorker doesn't mandate that all resources go though it, but it's being showed as the main use case (not mine, I want to combine it with ProtocolHandler to create my own protocols and manage them on client-side...). I'm not opposed to just punted and saying that ServiceWorker is the way to deal with this, though. I have no personal use cases at the moment for this functionality... just playing devil's advocate for the OP. :) Lol! Ok :-) -- Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton de sitios diferentes, simplemente escribe un sistema operativo Unix. – Linus Tordvals, creador del sistema operativo Linux
Re: Fixing appcache: a proposal to get us started
Come on!? Deal with flash players seems to me like a regression.
On Wed, May 1, 2013 at 1:14 AM, Jonas Sicking jo...@sicking.cc wrote:
On Wed, Apr 3, 2013 at 5:50 AM, Robin Berjon ro...@w3.org wrote:
On 29/03/2013 21:08 , Jonas Sicking wrote:
* Cache both files (poor bandwidth)
* We could enable some way of flagging which context different URLs
are expected to be used in. That way the UA can send the normal
content negotiation headers for images vs media files. I'm not sure
that this is worth it though given how few websites use content
negotiation headers.
* Use script to detect which formats are supported by the UA and then
use cacheURL to add the appropriate URL to the cache.
* Use the NavigationController feature.
* Use UA-string detection. You can either send different manifests
that point to different URLs for the media, or use a single manifest
but do the UA detection and serve different media files from the same
media URL. This is a pretty crappy solution though.
Another option: in your list of URLs to cache, instead of just strings
you
can also have objects of the form { video/webm: kittens.webm,
video/evil: dead-kittens.mv4 } that would operate in a manner
modelled
on source, caching only what's needed.
It's a bit verbose, but it's a lot less verbose than loading the content
twice.
Yes. The proposal already suggests using objects to express individual
resources. Something like the above seems like a natural extension.
Audio and video in general will be tricky until there's a set of
codecs that can be universally depended on. We'd likely have to deal
with video players written in flash too :(
/ Jonas
Re: Fixing appcache: a proposal to get us started
On Mon, Apr 1, 2013 at 4:03 AM, Glenn Jones
glennjones...@googlemail.com wrote:
I think this outline for improving AppCache is a really good start, but I
can see one major problem with it. As you point out AppCache was initially
designed for building simple single page web apps.
The appcache appears to be aimed at too simple applications. It works
fine if the website you want to cache consists of a small set of
static resources and otherwise only use features like IndexedDB or
localStorage to manage dynamic data. But once an application uses
server-side processing to dynamically generate resources based on
query parameters or other parts of the URL, then it requires that you
change the way that your application works.
I completely agree with the above point, but I don’t think it can be done
with prefix URLs, which are all but useless in dealing with “dynamically
generated resource based” sites. The example code you give for urlmaps only
shows how prefixs can be used to handle querystrings.
What is required is a way to cover all the common URL design patterns found
in the real world i.e. paving cow paths. Most importantly we need url
matching that works against section namespace design patterns i.e.
https://github.com/glennjones/microformat-node/issues
http://lanyrd.com/2012/jsconf-us/speakers/
http://www.quora.com/URLs/recent
To deal with theses URL structures you need to use something like the route
matching built into JavaScript MVC frameworks
urlmap: [{
url: https://github/:*/:*/issues;,
page: https://github/profile/project/issues;
}]
In the example above the :* wildcards match against the section of the URL
which needs to be ignored and allows you to match at a deeper level than
prefix urls.
Unless the new version of AppCache supports the current patterns of URL
design used on the web it will be forever assigned to build single page
apps. There will be no chance of progressively enhancing a large complex
site with AppCache, if it cannot deal with pre-existing URL design.
I like this a lot! I definitely want to make sure that we don't end up
with full regexp complexity in the type of patterns that we support,
but I agree that only supporting prefixes does seem like too small of
a subset.
I wouldn't be surprised if we'll end up having to expand to support
things like query parameters as well, but starting with prefixes and
patterns that can do wild-cards for individual path segments sounds
great.
/ Jonas
Re: Fixing appcache: a proposal to get us started
On Wed, Apr 3, 2013 at 5:50 AM, Robin Berjon ro...@w3.org wrote:
On 29/03/2013 21:08 , Jonas Sicking wrote:
* Cache both files (poor bandwidth)
* We could enable some way of flagging which context different URLs
are expected to be used in. That way the UA can send the normal
content negotiation headers for images vs media files. I'm not sure
that this is worth it though given how few websites use content
negotiation headers.
* Use script to detect which formats are supported by the UA and then
use cacheURL to add the appropriate URL to the cache.
* Use the NavigationController feature.
* Use UA-string detection. You can either send different manifests
that point to different URLs for the media, or use a single manifest
but do the UA detection and serve different media files from the same
media URL. This is a pretty crappy solution though.
Another option: in your list of URLs to cache, instead of just strings you
can also have objects of the form { video/webm: kittens.webm,
video/evil: dead-kittens.mv4 } that would operate in a manner modelled
on source, caching only what's needed.
It's a bit verbose, but it's a lot less verbose than loading the content
twice.
Yes. The proposal already suggests using objects to express individual
resources. Something like the above seems like a natural extension.
Audio and video in general will be tricky until there's a set of
codecs that can be universally depended on. We'd likely have to deal
with video players written in flash too :(
/ Jonas
Re: Fixing appcache: a proposal to get us started
On 01/05/2013, at 2:20 PM, Jonas Sicking jo...@sicking.cc wrote: The current AppCache spec suffers from this too, but only once users go offline. I.e. I can use FALLBACK to take over http://users.example.edu/~alice/ using a resource from from http://users.example.edu/~bob/newalice.html But that only works when the user is offline, which limits the damage a little. Yeah, I noticed that too; like you say, it's pretty limited. I did some quick testing and couldn't get current implementations to do it (I think because either they haven't completely implemented fallback, or I wasn't properly triggering it). The only solution that I can see to this problem is requiring that manifests, or navigationcontroller-scripts are only allowed to take over URLs that are below them. I.e. http://users.example.edu/~bob/manifest.json could only control navigations to URLs with the prefix http://users.example.edu/~bob/;. You could still redirect resource loading in more flexible ways, but maybe top-level page loads needs to have this restriction. Possibly. I'm still a bit wary of that; there are some *weird* CMSs out that that hide lots of things behind opaque, unstructured URLs. Cheers, -- Mark Nottingham http://www.mnot.net/
Re: Fixing appcache: a proposal to get us started
On 2013-03-27 01:20, Jonas Sicking wrote:
On Tue, Mar 26, 2013 at 1:48 AM, James Graham jgra...@opera.com wrote:
On 03/26/2013 08:02 AM, Jonas Sicking wrote:
Another feature that we are proposing is to drop the current
manifest format and instead use a JSON based one. The most simple
reason for this is that we noticed that the information we need to
express quickly became complex enough that using a format with simple
parsing rules was beneficial.
A format based on extending the current appcache format would be no
problem for a UA to parse. However the complexity that we need to
express resulted in something that's too hard for a human to manually
write, or for a human to understand when looking at somebody else's
manifest in order to learn.
The simple parsing rules for JSON seemed like a better fit. It also
provides more of an opportunity to extend the format in the future.
JSON also has advantages when it comes to creating APIs exposed to
webpages for interacting with appcaches. More about this below.
Some slightly trivial feedback: I am worried about using a format with no
support for comments. I agree that some hypothetical JSON+comments format
would be a good fit, but without the ability to document complex rulesets,
it seems like we are going to create a maintenance nightmare.
I completely agree. I feel like we're stuck between a rock and a hard
place here.
On one hand we need something that supports comments. JSON only
supports comments using ugly hacks like:
{
//: Remember to update this as needed,
version: 2,
//: Need nav.css for sidebar,
cache: [index.html, index.css, nav.css]
}
On the other hand, using something other than JSON means that we loose
the ability to use existing tool chains for processing the manifest.
I'm not so worried about the UA having to write new code in order to
implement the AppCache feature itself. I'm more concerned that
serverside code that wants to process the manifest can't use a normal
JSON parser, and that client-side code that wants to load a manifest
can't use things like xhr.responseType=json.
I really don't know which downside is worse.
I definitely wish that JSON supported comments, but that doesn't seem
likely to ever happen :(
...
Well, there is a format with good server and client support that allows
comments. It's called XML.
Best regards, Julian
Re: Fixing appcache: a proposal to get us started
Hi Jonas, I don't get a good sense from this proposal (and NavigationController) about what the scope of an application is. E.g., if both http://example.com/fooApp and http://example.com/barApp say that they both grab the URL http://example.com/app who wins? More to the point, what's stopping http://users.example.edu/~bob/ from taking over responses from http://users.example.edu/~alice/ ? Also along these lines - I see an outstanding question that some people want to host multiple apps on the same URL. Is that really wise? If anything, I'd go the other way and say that one hostname = one app; then the scoping of and separation between apps is intuitive (e.g., fooApp.example.com, barApp.example.com). You could even define a well-known location for the manifest, so that people could just type the hostname in to install / view the app... Cheers, -- Mark Nottingham http://www.mnot.net/
Re: Fixing appcache: a proposal to get us started
On Wed, 03 Apr 2013 14:50:53 +0200, Robin Berjon ro...@w3.org wrote:
On 29/03/2013 21:08 , Jonas Sicking wrote:
* Cache both files (poor bandwidth)
* We could enable some way of flagging which context different URLs
are expected to be used in. That way the UA can send the normal
content negotiation headers for images vs media files. I'm not sure
that this is worth it though given how few websites use content
negotiation headers.
* Use script to detect which formats are supported by the UA and then
use cacheURL to add the appropriate URL to the cache.
* Use the NavigationController feature.
* Use UA-string detection. You can either send different manifests
that point to different URLs for the media, or use a single manifest
but do the UA detection and serve different media files from the same
media URL. This is a pretty crappy solution though.
Another option: in your list of URLs to cache, instead of just strings
you can also have objects of the form { video/webm: kittens.webm,
video/evil: dead-kittens.mv4 } that would operate in a manner
modelled on source, caching only what's needed.
Is this intended only for video resources, or arbitrary resources?
Non-media elements (and hence, non-media resources) don't have the
source mechanism, so maybe the syntax should make it clear that it's
media-specific.
e.g.:
MEDIA
kittens.webm video/webm; codecs=...
kittens.mp4 video/mp4; codecs=...
# one MEDIA block per resource?
This doesn't include the media= attribute of source, but I think that
should be dropped from the element anyway.
Something similar could be done for img srcset, maybe, which is a
different beast than source.
It's a bit verbose, but it's a lot less verbose than loading the content
twice.
--
Simon Pieters
Opera Software
Re: Fixing appcache: a proposal to get us started
On 04/04/2013 15:41 , Simon Pieters wrote:
On Wed, 03 Apr 2013 14:50:53 +0200, Robin Berjon ro...@w3.org wrote:
On 29/03/2013 21:08 , Jonas Sicking wrote:
* Cache both files (poor bandwidth)
* We could enable some way of flagging which context different URLs
are expected to be used in. That way the UA can send the normal
content negotiation headers for images vs media files. I'm not sure
that this is worth it though given how few websites use content
negotiation headers.
* Use script to detect which formats are supported by the UA and then
use cacheURL to add the appropriate URL to the cache.
* Use the NavigationController feature.
* Use UA-string detection. You can either send different manifests
that point to different URLs for the media, or use a single manifest
but do the UA detection and serve different media files from the same
media URL. This is a pretty crappy solution though.
Another option: in your list of URLs to cache, instead of just strings
you can also have objects of the form { video/webm: kittens.webm,
video/evil: dead-kittens.mv4 } that would operate in a manner
modelled on source, caching only what's needed.
Is this intended only for video resources, or arbitrary resources?
Non-media elements (and hence, non-media resources) don't have the
source mechanism, so maybe the syntax should make it clear that it's
media-specific.
I thought about that, but I don't think that's needed, really. Assume
someone did something like:
{
shiny/new-type: whatever.new
, crummy/old-type: whatever.old
}
Then the UA will pick one to cache. Worst case scenario, this is useless
but harmless. But it's conceivable that script could then look at what's
in the cache to know which to use.
Just to be clear: I don't think that using this for non-media cases will
be incredibly useful. But I don't see any reason to actively prevent it.
--
Robin Berjon - http://berjon.com/ - @robinberjon
Re: Fixing appcache: a proposal to get us started
On 29/03/2013 21:08 , Jonas Sicking wrote:
* Cache both files (poor bandwidth)
* We could enable some way of flagging which context different URLs
are expected to be used in. That way the UA can send the normal
content negotiation headers for images vs media files. I'm not sure
that this is worth it though given how few websites use content
negotiation headers.
* Use script to detect which formats are supported by the UA and then
use cacheURL to add the appropriate URL to the cache.
* Use the NavigationController feature.
* Use UA-string detection. You can either send different manifests
that point to different URLs for the media, or use a single manifest
but do the UA detection and serve different media files from the same
media URL. This is a pretty crappy solution though.
Another option: in your list of URLs to cache, instead of just strings
you can also have objects of the form { video/webm: kittens.webm,
video/evil: dead-kittens.mv4 } that would operate in a manner
modelled on source, caching only what's needed.
It's a bit verbose, but it's a lot less verbose than loading the content
twice.
--
Robin Berjon - http://berjon.com/ - @robinberjon
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 7:02 AM, Jonas Sicking jo...@sicking.cc wrote: The details of the API in this worker is something we haven't looked at yet. It's a very big task in and of itself. Fortunately, Alex Russell and a few others have worked on a proposal for exactly this at [2]. The intent is for these two proposals to be aligned such that they work well together. They are already very complementary in their feature sets, so this should not be a problem. However this is something that we've just started looking at, and since both proposals are still under heavy development, I didn't want to wait until they are both aligned before publishing what we have so far. [2] https://github.com/slightlyoff/NavigationController So I think working on both makes a lot of sense as the requirements of one will influence the other in positive ways. However, when it comes to deploying, I think if we deployed the controller first and see what kind of feedback we get from library developers down the line and use that to further influence the development of the declarative model we'll yield better results. There are many hard problems to solve, some of which come down to heuristics, and it seems like letting hundreds of developers experiment rather than a couple of browser implementers will lead to better results down the line. Having said, what we need to resolve for both, and I think is one of the trickier bits, is the life cycle. How is it added, when does it go into effect, when and how is it replaced, etc. -- http://annevankesteren.nl/
Re: Fixing appcache: a proposal to get us started
I love the idea! It is the best solution listed so far.
On Wed, Apr 3, 2013 at 9:50 AM, Robin Berjon ro...@w3.org wrote:
On 29/03/2013 21:08 , Jonas Sicking wrote:
* Cache both files (poor bandwidth)
* We could enable some way of flagging which context different URLs
are expected to be used in. That way the UA can send the normal
content negotiation headers for images vs media files. I'm not sure
that this is worth it though given how few websites use content
negotiation headers.
* Use script to detect which formats are supported by the UA and then
use cacheURL to add the appropriate URL to the cache.
* Use the NavigationController feature.
* Use UA-string detection. You can either send different manifests
that point to different URLs for the media, or use a single manifest
but do the UA detection and serve different media files from the same
media URL. This is a pretty crappy solution though.
Another option: in your list of URLs to cache, instead of just strings you
can also have objects of the form { video/webm: kittens.webm,
video/evil: dead-kittens.mv4 } that would operate in a manner modelled
on source, caching only what's needed.
It's a bit verbose, but it's a lot less verbose than loading the content
twice.
--
Robin Berjon - http://berjon.com/ - @robinberjon
Re: Fixing appcache: a proposal to get us started
I think this outline for improving AppCache is a really good start, but I
can see one major problem with it. As you point out AppCache was initially
designed for building simple single page web apps.
The appcache appears to be aimed at too simple applications. It works
fine if the website you want to cache consists of a small set of
static resources and otherwise only use features like IndexedDB or
localStorage to manage dynamic data. But once an application uses
server-side processing to dynamically generate resources based on
query parameters or other parts of the URL, then it requires that you
change the way that your application works.
I completely agree with the above point, but I don’t think it can be done
with prefix URLs, which are all but useless in dealing with “dynamically
generated resource based” sites. The example code you give for urlmaps only
shows how prefixs can be used to handle querystrings.
What is required is a way to cover all the common URL design patterns found
in the real world i.e. paving cow paths. Most importantly we need url
matching that works against section namespace design patterns i.e.
https://github.com/glennjones/microformat-node/issues
http://lanyrd.com/2012/jsconf-us/speakers/
http://www.quora.com/URLs/recent
To deal with theses URL structures you need to use something like the route
matching built into JavaScript MVC frameworks
urlmap: [{
url: https://github/:*/:*/issues;,
page: https://github/profile/project/issues;
}]
In the example above the :* wildcards match against the section of the URL
which needs to be ignored and allows you to match at a deeper level than
prefix urls.
Unless the new version of AppCache supports the current patterns of URL
design used on the web it will be forever assigned to build single page
apps. There will be no chance of progressively enhancing a large complex
site with AppCache, if it cannot deal with pre-existing URL design.
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 7:40 PM, Alec Flett alecfl...@chromium.org wrote:
This is a tricky problem indeed.
The current appcache actually has the behavior that you're advocating,
but that's something that a lot of developers has complained about. In
fact, that's the second biggest complaint that I've heard only
trailing the confusing master entries behavior.
I personally think the problem with this particular aspect of the existing
appcache is that its so incredibly hard to clear the cache and go online
during development - i.e. once you're offline you have to jump through hoops
to get back online. A secondary issue was that once you as a developer got
used to dealing with that, If your users somehow get stuck in an offline
state because of a bug, there is/was no real way to repair them other than
telling them to clear their cache.
On the other hand, it creates the load twice to get latest version
behavior that a lot of developers dislike. I.e. when a user opens a
website they end up getting the previous version of the website and
have to reload to get the new version.
I think that if there is a programmatic API that is available early-on, then
at least starting in offline gives the developer the option of going online
if they so choose - and it could be done even before the onload handler if
they want to avoid flashing the old/deprecated page in the browser. If you
require hitting the network first, then I can't think of how you'd write
programmatic hooks to bypass that.
You might be right and that off-line first behavior is the right
behavior in the majority of cases.
Though based on feedback from talking to people I'm fairly sure that
the on-line first behavior is something that at least *some*
developers want.
But maybe making offline-first be the default and allow people to opt
in to online-first?
Do note though that with the proposal as it stands today it is fairly
easy to get the offline first behavior. Simply set the expiration time
to a really long time and then use the explicit API for doing update
checking against the network as needed.
However I'm definitely willing to try changing things around such that
offline-first is more of a default.
How about this new proposal:
{
cache: [...],
}
This would use a default expiration time of one day. Whether the
cache has expired or not we here always use the offline resources
first. However if the cache has expired we also do an update check in
the background after the page has loaded.
{
expiration: 300,
cache: [...],
}
This still has the semantics of always using offline-first. The only
difference is that already after 5 minutes would we do a background
update check once the page has loaded.
{
dont-use-expired: true,
expiration: 3600,
cache: [...],
}
This manifest would hit the network if the user is online and if the
cache has expired, i.e. if it's been more than an hour since we last
checked for update.
I personally think that no matter how expressive the declarative syntax is,
developers are always going to need to work around it - expiration or
staleness is simply too complex to just give an absolute or relative date
or time - caching policy in apps can simply depend on things that extend
beyond your caching syntax - I mean imagine a caching policy that depends on
it being before or after sunset in your locale.
I definitely agree that having a programmatic API is needed! That's
the intent of the checkForUpdate() and download() functions.
If we don't have those functions I think very few websites would be
able to use the manifest at all and would be forced to use the
NavigationController instead.
If you have other ideas for how we can solve this then I'd love to
hear it. If we need to add more knobs to allow authors to choose which
policies they want to use then I have no problem with that. It would
be particularly interesting to hear what policies people are planning
on implementing using NavigationController to see if we can enable
those.
A more complex, real example: at my last company had a systemwide horizon
expiration policy that we implemented with a caching proxy. Imagine this: a
very large interconnected data set where individuals spent their time
editing data in small regions of the corpus. The goal was if you made an
edit, then everything YOU saw would be consistent with that edit. It was
perfectly reasonable to have other users see stale versions of any page -
a poor man's (i.e. startup with only a few application server's)
eventually-consistent solution.
The way this worked, if any individual user made changes to a particular
dataset that affected a page, they would get a cookie set on their client
saying you have made changes through time T and all future pages that they
visited had to be newer than time T. When the browser would hit the proxy
with an If-Modified-Since, the proxy would look at the cookie and say Hmm..
I have a stale version of this page
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 10:03 PM, Emerson Estrella emerson.estre...@gmail.com wrote: I'm writing a web application that uses the AppCache API for offline browsing. But I'm also using the Audio API to play back-ground music and a few audio effects. For audio support in different browsers I'm delivering each sound/music in two different file formats: OGG and MP3. The problem is regarding the cache manifest file. If we add all audio files (MP3 and OGG) in the cache manifest file, all browsers will cache all files. Including the unsupported ones. So, we end up with a huge storage requirement. Which is really bad if you are on a 3G connection for example. To prevent browser caching unsupported resources, the best approach I've found was to split the manifest file. This way we can tell browsers that sopport OGG files to cache only OGG files, and do tha same for other formats like MP3. But it is a ugly solution. Now I have two manifest cache files, one with all OGG files listed (ogg.appcache) and the other one with the MP3 files (mp3.appcache). I could see a few ways of handling this: * Cache both files (poor bandwidth) * We could enable some way of flagging which context different URLs are expected to be used in. That way the UA can send the normal content negotiation headers for images vs media files. I'm not sure that this is worth it though given how few websites use content negotiation headers. * Use script to detect which formats are supported by the UA and then use cacheURL to add the appropriate URL to the cache. * Use the NavigationController feature. * Use UA-string detection. You can either send different manifests that point to different URLs for the media, or use a single manifest but do the UA detection and serve different media files from the same media URL. This is a pretty crappy solution though. / Jonas
Re: Fixing appcache: a proposal to get us started
I'm writing a web application that uses the AppCache API for offline browsing. But I'm also using the Audio API to play back-ground music and a few audio effects. For audio support in different browsers I'm delivering each sound/music in two different file formats: OGG and MP3. The problem is regarding the cache manifest file. If we add all audio files (MP3 and OGG) in the cache manifest file, all browsers will cache all files. Including the unsupported ones. So, we end up with a huge storage requirement. Which is really bad if you are on a 3G connection for example. To prevent browser caching unsupported resources, the best approach I've found was to split the manifest file. This way we can tell browsers that sopport OGG files to cache only OGG files, and do tha same for other formats like MP3. But it is a ugly solution. Now I have two manifest cache files, one with all OGG files listed (ogg.appcache) and the other one with the MP3 files (mp3.appcache). Good luck guys! Best regards, Emerson Estrella
Re: Fixing appcache: a proposal to get us started
On 03/26/2013 08:02 AM, Jonas Sicking wrote: Another feature that we are proposing is to drop the current manifest format and instead use a JSON based one. The most simple reason for this is that we noticed that the information we need to express quickly became complex enough that using a format with simple parsing rules was beneficial. A format based on extending the current appcache format would be no problem for a UA to parse. However the complexity that we need to express resulted in something that's too hard for a human to manually write, or for a human to understand when looking at somebody else's manifest in order to learn. The simple parsing rules for JSON seemed like a better fit. It also provides more of an opportunity to extend the format in the future. JSON also has advantages when it comes to creating APIs exposed to webpages for interacting with appcaches. More about this below. Some slightly trivial feedback: I am worried about using a format with no support for comments. I agree that some hypothetical JSON+comments format would be a good fit, but without the ability to document complex rulesets, it seems like we are going to create a maintenance nightmare.
Re: Fixing appcache: a proposal to get us started
One feature I'd like to see is respect for compression headers. I've got an app which results in a 30Mb app cache, but it's only 8Mb over the wire due to GZIP compression. I'd much prefer the appcache to see that the content was served compressed, cache it compressed, and serve it to the browser offline compressed too. Whilst not strictly part of a JS API, it would probably be a worthwhile note to implementors that the preferred behaviour is to store and cache with the same compression headers with which the content was originally served. Another approach would be to modify the appcache manifest with a compress property per item to cache, or perhaps transparently compress the entire appcache for a given manifest within the browser (totally transparent to the user/developer/API). Either way, it'd be nice to prevent the scenario of downloading 8Mb content and resulting in a 30Mb cache. N On 26 March 2013 08:48, James Graham jgra...@opera.com wrote: On 03/26/2013 08:02 AM, Jonas Sicking wrote: Another feature that we are proposing is to drop the current manifest format and instead use a JSON based one. The most simple reason for this is that we noticed that the information we need to express quickly became complex enough that using a format with simple parsing rules was beneficial. A format based on extending the current appcache format would be no problem for a UA to parse. However the complexity that we need to express resulted in something that's too hard for a human to manually write, or for a human to understand when looking at somebody else's manifest in order to learn. The simple parsing rules for JSON seemed like a better fit. It also provides more of an opportunity to extend the format in the future. JSON also has advantages when it comes to creating APIs exposed to webpages for interacting with appcaches. More about this below. Some slightly trivial feedback: I am worried about using a format with no support for comments. I agree that some hypothetical JSON+comments format would be a good fit, but without the ability to document complex rulesets, it seems like we are going to create a maintenance nightmare.
Re: Fixing appcache: a proposal to get us started
On 26 March 2013 07:02, Jonas Sicking jo...@sicking.cc wrote:
{
expiration: 300,
cache: [index.html, index.js, index.css]
}
If the user navigates to index.html The following happens:
If the user is online and we haven't checked for update for the
appcache in the last 5 minutes (300 seconds) we simply ignore the
cache and load index.html and any resources it links to from the
network.
How is online defined here? What if we're online but the page 404s, what
if we're online but DNS to that domain fails etc etc.
Offline-first isn't intuitive, I agree. I did some hacking with FALLBACK to
get online-first behaviour in Lanyrd and quickly learned why it's a bad
idea, offline performance was dire as the device had to fail to connect
before it'd show me cached data. I'd rather stick with offline-first but
present it in a way that's expected.
If the user is offline, or if we checked for update for the appcache
within the last 5 minutes, we use the index.html from the appcache
without hitting the network first. If index.html uses index.js or
index.css, those will be immediately loaded from the cache.
Is the opposite true? If index.html is loaded from the network will
index.js ever come from the cache? (current appcache says no)
Whenever we check for updates for an appcache with the above manifest
we do an if-modified-since/if-none-match for the manifest. We then do
an update check for any resource requested by the manifest. I.e. even
if the manifest hasn't changed we still do an update check for each
resource linked to by the manifest. If any resources were added since
the previous manifest those are obviously simply downloaded. If any
resources were removed from the manifest those are discarded. As an
optimization the UA can start doing update checks on the same set of
URLs that the previous version of the manifest contained.
Don't get the last line, does the same set of URLs that the previous
version of the manifest contained mean URLs that are in the current
version, and also the previous version? Is this suggesting that obeying
HTTP cache headers is an optional optimisation?
{
version: 5.1,
expiration: 300,
cache: [index.html, index.js, index.css]
}
For this manifest, when we want check for update for the cache we
first do a if-none-match/if-modified-since check for the cache object
itself. If we get back a new resource, *and* that resource contains a
new value for the version property, then we do update checks for all
resources as well as download any new ones.
+1, having an explicit version property is a much better idea than the
string matching in the current manifest.
In order to further cut down on the number of network requests, we'd
also enable providing last-modified dates or etags directly in the
manifest:
{
expiration: 300,
cache: [{ url: index.html, etag: 84ba9f},
{ url: index.js, last-modified: Wed, 1 May 2013
04:58:08 GMT },
index.css]
}
Adding these into the manifest would require some kind of automation, if
there's a level of automation couldn't it just change the url to
index.84ba9f.js and have rewriting or static generation take care of the
rest?
{
expiration: 300,
cache: [index.html, index.js, index.css],
cookie-vary: uid
}
This would mean that even if the user is offline and navigates to
index.html, if the value of the uid cookie is different from when
the appcache was last updated, the appcache would not be returned. A
UA could even use the value of the uid cookie as an additional key
in its appcache registry and thus support keeping appcaches for
different users on the same device.
If I have an expiration of a day, and log out as one user log in with
another, then get on a plane, none of my content would be available to me?
The actual AppCache object has the following API:
I'm worried the API here loses the simplicity that this proposal is
supposed to have over the scripted solution, where you have to learn a
manifest format and how it interacts with items added via its scripting
API, whereas the navigation controller only has a scripting API.
Is there anything here that couldn't be done with the NavigationController
a library? I'm not suggesting that's reason for it not to exist, just
wondering if it's offering anything unique.
Jake.
Re: Fixing appcache: a proposal to get us started
* Jonas Sicking wrote: There has been a lot of debating about fixing appcache. Last year mozilla got a few people together mostly with the goal of understanding what the actual problems were. The notes from that meeting are available at [1]. (I take it the fixing-appcache mailing list has since been closed in http://www.w3.org/community/fixing-appcache/ favour of discussion here.) -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
Re: Fixing appcache: a proposal to get us started
On Tuesday, March 26, 2013 at 12:12 PM, Bjoern Hoehrmann wrote: (I take it the fixing-appcache mailing list has since been closed in http://www.w3.org/community/fixing-appcache/ favour of discussion here.) Yes, see: http://lists.w3.org/Archives/Public/public-fixing-appcache/2013Feb/0005.html --tobie
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 3:21 AM, Jake Archibald jaffathec...@gmail.com wrote:
On 26 March 2013 07:02, Jonas Sicking jo...@sicking.cc wrote:
{
expiration: 300,
cache: [index.html, index.js, index.css]
}
If the user navigates to index.html The following happens:
If the user is online and we haven't checked for update for the
appcache in the last 5 minutes (300 seconds) we simply ignore the
cache and load index.html and any resources it links to from the
network.
How is online defined here? What if we're online but the page 404s, what
if we're online but DNS to that domain fails etc etc.
Offline-first isn't intuitive, I agree. I did some hacking with FALLBACK to
get online-first behaviour in Lanyrd and quickly learned why it's a bad
idea, offline performance was dire as the device had to fail to connect
before it'd show me cached data. I'd rather stick with offline-first but
present it in a way that's expected.
This is a tricky problem indeed.
The current appcache actually has the behavior that you're advocating,
but that's something that a lot of developers has complained about. In
fact, that's the second biggest complaint that I've heard only
trailing the confusing master entries behavior.
On one hand I totally agree that we hitting the network means
potentially doing a terribly slow load over an 2g network connection
which ends up failing half-way through. There is simply no way to
ensure a good user experience when you decide to hit the network
first.
On the other hand, it creates the load twice to get latest version
behavior that a lot of developers dislike. I.e. when a user opens a
website they end up getting the previous version of the website and
have to reload to get the new version.
Two strong concerns that developers have expressed with this is that
it means that when website X announces that they've launched a new
version of the website, a user which goes to X will still see the old
version. Another issue is that if a website rolls out a security fix,
they don't want people running a cached copy of the exploitable
version.
I don't know how to fully reconcile these two conflicting
requirements. I.e. don't hit a potentially slow network if there's a
cached copy and if the user is online, don't run an old version.
The expiration property is the best solution I've been able to come
up with so far. A website that considers it important to have users
always run the latest version can set the number low which prevents
running an old version if possible. A website that prefers to optimize
for performance can set a high number.
This doesn't seem like a problem that's specific to this appcache
proposal. A website using the NavigationController API will have to
make exactly the same types of decisions as far as I can tell. It's
just that it'll get to make them using explicit JS code rather than by
adjusting properties in a manifest.
If you have other ideas for how we can solve this then I'd love to
hear it. If we need to add more knobs to allow authors to choose which
policies they want to use then I have no problem with that. It would
be particularly interesting to hear what policies people are planning
on implementing using NavigationController to see if we can enable
those.
If the user is offline, or if we checked for update for the appcache
within the last 5 minutes, we use the index.html from the appcache
without hitting the network first. If index.html uses index.js or
index.css, those will be immediately loaded from the cache.
Is the opposite true? If index.html is loaded from the network will index.js
ever come from the cache? (current appcache says no)
To start simple I'm so far proposing that the answer is no. What's
the plan on the NavigationController side?
Whenever we check for updates for an appcache with the above manifest
we do an if-modified-since/if-none-match for the manifest. We then do
an update check for any resource requested by the manifest. I.e. even
if the manifest hasn't changed we still do an update check for each
resource linked to by the manifest. If any resources were added since
the previous manifest those are obviously simply downloaded. If any
resources were removed from the manifest those are discarded. As an
optimization the UA can start doing update checks on the same set of
URLs that the previous version of the manifest contained.
Don't get the last line, does the same set of URLs that the previous
version of the manifest contained mean URLs that are in the current
version, and also the previous version? Is this suggesting that obeying
HTTP cache headers is an optional optimisation?
Sorry, I might have derailed this by mentioning the performance optimization.
The basic idea is that even if the manifest hasn't changed, we still
do an update check on all resources. I suspect that both for the
update check of the manifest itself as well as the resources we still
honor the normal http caching semantics. Except
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 1:48 AM, James Graham jgra...@opera.com wrote:
On 03/26/2013 08:02 AM, Jonas Sicking wrote:
Another feature that we are proposing is to drop the current
manifest format and instead use a JSON based one. The most simple
reason for this is that we noticed that the information we need to
express quickly became complex enough that using a format with simple
parsing rules was beneficial.
A format based on extending the current appcache format would be no
problem for a UA to parse. However the complexity that we need to
express resulted in something that's too hard for a human to manually
write, or for a human to understand when looking at somebody else's
manifest in order to learn.
The simple parsing rules for JSON seemed like a better fit. It also
provides more of an opportunity to extend the format in the future.
JSON also has advantages when it comes to creating APIs exposed to
webpages for interacting with appcaches. More about this below.
Some slightly trivial feedback: I am worried about using a format with no
support for comments. I agree that some hypothetical JSON+comments format
would be a good fit, but without the ability to document complex rulesets,
it seems like we are going to create a maintenance nightmare.
I completely agree. I feel like we're stuck between a rock and a hard
place here.
On one hand we need something that supports comments. JSON only
supports comments using ugly hacks like:
{
//: Remember to update this as needed,
version: 2,
//: Need nav.css for sidebar,
cache: [index.html, index.css, nav.css]
}
On the other hand, using something other than JSON means that we loose
the ability to use existing tool chains for processing the manifest.
I'm not so worried about the UA having to write new code in order to
implement the AppCache feature itself. I'm more concerned that
serverside code that wants to process the manifest can't use a normal
JSON parser, and that client-side code that wants to load a manifest
can't use things like xhr.responseType=json.
I really don't know which downside is worse.
I definitely wish that JSON supported comments, but that doesn't seem
likely to ever happen :(
/ Jonas
Re: Fixing appcache: a proposal to get us started
This is a lot to digest, but I know the developer community will greatly
appreciate the work that has gone into this—thank you.
On Tue, Mar 26, 2013 at 3:02 AM, Jonas Sicking jo...@sicking.cc wrote:
(snip)
First we need a way to get at AppCache objects:
No mention of installAppCache, removeAppCache or getAppCacheList anywhere
else in the email—not even to say they are in progress?
partial interface Navigator {
FutureAppCache installAppCache(url);
FutureAppCache getAppCache(url);
Futureboolean removeAppCache(url);
FutureDOMString[] getAppCacheList();
}
There is no way to create an alias binding for shortening these calls—the
kids love shorthanding ;)
This isn't really a navigator capability is it? Perhaps instead of adding
surface to navigator, a new global called platform, for platform related
APIs that aren't quite navigator, document or window (?) capabilities...
partial interface Window {
Object platform
}
partial interface platform {
Object appCache
}
interface appCache {
FutureAppCache install(url);
FutureAppCache get(url);
Futureboolean remove(url);
FutureDOMString[] list();
}
If that's not desirable (which I can easily understand), then dump it on
the window. Yes, it stinks to put more things on the window object, but
developers understand recognize the pattern (it's also minifier friendly)
partial interface Window {
Object appCache
}
interface appCache {
FutureAppCache install(url);
FutureAppCache get(url);
Futureboolean remove(url);
FutureDOMString[] list();
}
With this API surface, the code I might ship:
(function(window) {
var appCache = window.appCache;
appCache.get(url a).then(function(cache) {
// do stuff and things...
});
appCache.remove(url b).then(function(cache) {
// do stuff and things...
});
}(this));
Minifies to:
(function(a){var b=a.appCache;b.get(url
a).then(function(){}),b.remove(url b).then(function(){})})(this);
vs. the proposed API:
(function(window) {
var navigator = window.navigator;
navigator.getAppCache(url a).then(function(cache) {
// do stuff and things...
});
navigator.removeAppCache(url b).then(function(cache) {
// do stuff and things...
});
}(this));
(function(a){var b=a.navigator;b.getAppCache(url
a).then(function(){}),b.removeAppCache(url b).then(function(){})})(this);
Specifically, the interesting parts are:
b.get(url a).then(function(){}),b.remove(url b).then(function(){})
b.getAppCache(url a).then(function(){}),b.removeAppCache(url
b).then(function(){})
partial interface Document {
AppCache appCache;
readonly attribute boolean appCacheUpdateAvailable;
attribute EventHandler onappcacheupdateavailable;
}
As an author, I would already know that I'm working with an appCache
object, therefore the word appCache prefix on the boolean property is
just junk that's not minifiable. So...
appCacheUpdateAvailable =
- updateAvailable
- hasUpdate
- isAvailable
- ?
Anyway, great work from all involved. I'm excited to see such momentum and
dedication to getting app cache back on track :)
Rick
snip
/ Jonas
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 6:28 PM, Rick Waldron waldron.r...@gmail.com wrote:
This is a lot to digest, but I know the developer community will greatly
appreciate the work that has gone into this—thank you.
Yeah, I hope this is possible to consume despite its length. I'll
create a shorter writeup with less background for the next iteration.
On Tue, Mar 26, 2013 at 3:02 AM, Jonas Sicking jo...@sicking.cc wrote:
(snip)
First we need a way to get at AppCache objects:
No mention of installAppCache, removeAppCache or getAppCacheList anywhere
else in the email—not even to say they are in progress?
Sorry, i missed this. I hope they are generally pretty obvious what
they do on a high level though?
partial interface Navigator {
FutureAppCache installAppCache(url);
FutureAppCache getAppCache(url);
Futureboolean removeAppCache(url);
FutureDOMString[] getAppCacheList();
}
There is no way to create an alias binding for shortening these calls—the
kids love shorthanding ;)
This isn't really a navigator capability is it? Perhaps instead of adding
surface to navigator, a new global called platform, for platform related
APIs that aren't quite navigator, document or window (?) capabilities...
Yeah, I wasn't sure what to do about this. I'm not really exited about
adding a property to the global object which has a name that is as
generic as platform.
partial interface Window {
Object platform
}
partial interface platform {
Object appCache
}
interface appCache {
FutureAppCache install(url);
FutureAppCache get(url);
Futureboolean remove(url);
FutureDOMString[] list();
}
I like this general approach other than the issue mentioned above.
If that's not desirable (which I can easily understand), then dump it on the
window. Yes, it stinks to put more things on the window object, but
developers understand recognize the pattern (it's also minifier friendly)
partial interface Window {
Object appCache
}
interface appCache {
FutureAppCache install(url);
FutureAppCache get(url);
Futureboolean remove(url);
FutureDOMString[] list();
}
This might be more ok. As would sticking the appCache property on the
navigator object be.
partial interface Document {
AppCache appCache;
readonly attribute boolean appCacheUpdateAvailable;
attribute EventHandler onappcacheupdateavailable;
}
As an author, I would already know that I'm working with an appCache
object, therefore the word appCache prefix on the boolean property is just
junk that's not minifiable.
The updateAvailable isn't really a property of the AppCache object
though. An appcache is updated as soon as it has been downloaded. It's
the view that the user is currently looking at that is out-of-date.
Possibly if we add a appCache property on window or navigator then
putting the property there might work.
/ Jonas
Re: Fixing appcache: a proposal to get us started
On Tue, Mar 26, 2013 at 9:58 PM, Jonas Sicking jo...@sicking.cc wrote:
On Tue, Mar 26, 2013 at 6:28 PM, Rick Waldron waldron.r...@gmail.com
wrote:
This is a lot to digest, but I know the developer community will greatly
appreciate the work that has gone into this—thank you.
Yeah, I hope this is possible to consume despite its length. I'll
create a shorter writeup with less background for the next iteration.
On Tue, Mar 26, 2013 at 3:02 AM, Jonas Sicking jo...@sicking.cc wrote:
(snip)
First we need a way to get at AppCache objects:
No mention of installAppCache, removeAppCache or getAppCacheList anywhere
else in the email—not even to say they are in progress?
Sorry, i missed this. I hope they are generally pretty obvious what
they do on a high level though?
Yes, they are—you were so thorough that their absence was alarming ;)
partial interface Navigator {
FutureAppCache installAppCache(url);
FutureAppCache getAppCache(url);
Futureboolean removeAppCache(url);
FutureDOMString[] getAppCacheList();
}
There is no way to create an alias binding for shortening these calls—the
kids love shorthanding ;)
This isn't really a navigator capability is it? Perhaps instead of adding
surface to navigator, a new global called platform, for platform
related
APIs that aren't quite navigator, document or window (?) capabilities...
Yeah, I wasn't sure what to do about this. I'm not really exited about
adding a property to the global object which has a name that is as
generic as platform.
partial interface Window {
Object platform
}
partial interface platform {
Object appCache
}
interface appCache {
FutureAppCache install(url);
FutureAppCache get(url);
Futureboolean remove(url);
FutureDOMString[] list();
}
I like this general approach other than the issue mentioned above.
If that's not desirable (which I can easily understand), then dump it on
the
window. Yes, it stinks to put more things on the window object, but
developers understand recognize the pattern (it's also minifier friendly)
partial interface Window {
Object appCache
}
interface appCache {
FutureAppCache install(url);
FutureAppCache get(url);
Futureboolean remove(url);
FutureDOMString[] list();
}
This might be more ok. As would sticking the appCache property on the
navigator object be.
partial interface Document {
AppCache appCache;
readonly attribute boolean appCacheUpdateAvailable;
attribute EventHandler onappcacheupdateavailable;
}
As an author, I would already know that I'm working with an appCache
object, therefore the word appCache prefix on the boolean property is
just
junk that's not minifiable.
The updateAvailable isn't really a property of the AppCache object
though. An appcache is updated as soon as it has been downloaded. It's
the view that the user is currently looking at that is out-of-date.
Yep, I misread that—thanks for clarifying.
Possibly if we add a appCache property on window or navigator then
putting the property there might work.
Even better :)
/ Jonas
Re: Fixing appcache: a proposal to get us started
This is a tricky problem indeed. The current appcache actually has the behavior that you're advocating, but that's something that a lot of developers has complained about. In fact, that's the second biggest complaint that I've heard only trailing the confusing master entries behavior. I personally think the problem with this particular aspect of the existing appcache is that its so incredibly hard to clear the cache and go online during development - i.e. once you're offline you have to jump through hoops to get back online. A secondary issue was that once you as a developer got used to dealing with that, If your users somehow get stuck in an offline state because of a bug, there is/was no real way to repair them other than telling them to clear their cache. On the other hand, it creates the load twice to get latest version behavior that a lot of developers dislike. I.e. when a user opens a website they end up getting the previous version of the website and have to reload to get the new version. I think that if there is a programmatic API that is available early-on, then at least starting in offline gives the developer the option of going online if they so choose - and it could be done even before the onload handler if they want to avoid flashing the old/deprecated page in the browser. If you require hitting the network first, then I can't think of how you'd write programmatic hooks to bypass that. I personally think that no matter how expressive the declarative syntax is, developers are always going to need to work around it - expiration or staleness is simply too complex to just give an absolute or relative date or time - caching policy in apps can simply depend on things that extend beyond your caching syntax - I mean imagine a caching policy that depends on it being before or after sunset in your locale. If you have other ideas for how we can solve this then I'd love to hear it. If we need to add more knobs to allow authors to choose which policies they want to use then I have no problem with that. It would be particularly interesting to hear what policies people are planning on implementing using NavigationController to see if we can enable those. A more complex, real example: at my last company had a systemwide horizon expiration policy that we implemented with a caching proxy. Imagine this: a very large interconnected data set where individuals spent their time editing data in small regions of the corpus. The goal was if you made an edit, then everything YOU saw would be consistent with that edit. It was perfectly reasonable to have other users see stale versions of any page - a poor man's (i.e. startup with only a few application server's) eventually-consistent solution. The way this worked, if any individual user made changes to a particular dataset that affected a page, they would get a cookie set on their client saying you have made changes through time T and all future pages that they visited had to be newer than time T. When the browser would hit the proxy with an If-Modified-Since, the proxy would look at the cookie and say Hmm.. I have a stale version of this page at time T-6, I'd better regenerate it or I have a version of the page at time T+2, so I can give this to you - To make this work we had to set max-age=0, essentially bypassing the entire user's browser cache for every page, even if the server mostly responded with a 304. (so the proxy server sitting in our colo in Santa Clara functioned as your browser's cache because that was the place we could programatically write a policy) That really sucked for performance though, so we increased max-age to maybe 30 seconds, and put a generated script in the head that included the time the page was generated, and then compared the cookie to the embedded time. If the cookie was higher, then we know the page was served stale (by our definition of stale) from the browser cache so we forced a refresh. Since this was all in the head, the page didn't even flicker. Something like head scriptvar lastWriteTime=1292830; // generated in-page by the template engine if (lastWriteTime extractLWT(document.cookie)) reload(); // boilerplate cache policy /script But of course the problem there is that ONLY works on HTML - all other resources had to have a different policy. With a NavigationController model (or some other programmatic model) you can write arbitrary logic. to deal with these kinds of cases. I'm not saying the declarative model doesn't fix 80% of the issues, but more that you still need a programmatic model in addition. If the user is offline, or if we checked for update for the appcache within the last 5 minutes, we use the index.html from the appcache without hitting the network first. If index.html uses index.js or index.css, those will be immediately loaded from the cache. Is the opposite true? If index.html is loaded from the network will index.js ever come from the cache?
