Re: [whatwg] Unlimited pageStorage for App Cached web pages
On 5/31/11, Felix Halim wrote: > On Mon, May 30, 2011 at 10:39 PM, Bjartur Thorlacius > wrote: > > The dynamic resources only updated if the user visit the particular > app cached web-page. > Yeah, that's logical. Caches should still be allowed to refetch resources just before they're expected to be used. I might want my home computer to fetch the latest news in the morning and evening, so I can start reading when I wake up and when I get home from school. > Remember that the dynamic resources I'm talking about here is NOT > shared between other web-cached pages (even they are in the same > domain). > That's fine. I don't think caches need to know that, but I'll get back to you after some sleep. It may be hard to get quotas right; but multiple HTML documents *could* link to the same resources. I think quotas should only be enforced per resource and on the user agent, leaving the user agent to use the quota for small files only as effectively as it can, e.g. by keeping only frequently used resources. >> The former is easy to achieve, but user agents tend to throw away stale >> versions as to not present outdated information to the user and to save >> storage space. > > The user agent only need to keep the latest version. > It's fine to throw away the outdated one if you have the latest. > Sorry, I meant potentially stale. User agents should of course not keep obsolete versions of resources when they have fresh ones, but they may end up with versions of resources are not fresh. In this case they SHOULD validate or refetch the resource - except when "working offline". >> You want user agents to fetch the latest version whenever possible, but >> keep >> an old copy for when your servers are unreachable. > > The "whenever possible" is when the user revisit the cached page. ...and a cache has a fresh copy or an authoritative server for the resource is reachable and responsive. > The "old" here means the latest version that was cached.. > Yes. >> 5MB ought to be enough for anyone. > > You were joking, right? :D > Yeah, I'm kidding. :P I believe quota size should be decided on case-by-case basis (unlike localStorage where it's probably useful to make assumptions as to the available storage space). > 5MB for each App Cached web-page is probably OK. > However, 5MB localStorage quota for each domain is NOT OK. > The right amount to reserve for caching depends on the scarcity of space on the user's machine. >> If all you want to do is store mutable resources for offline use and >> validate them if possible, but returning the cached entry (or entity) if >> validation is impossible, simply serve the resources with an Expires >> header >> set to a date in the past. > > Here is an example of how I want the App Cache to behave: > > First, it always try to fetch the main page, and all its > static/dynamic resources and display it (just like normal web page). > Then some time later, if the user want's to visit the SAME page again > but the user is offline or the server is unavailable, then the latest > cache of the page is displayed. > HTTP user agents MAY implement the behaviour you describe; i.e. use potentially stale entries when validation (checking if it's fresh) is impossible - as long as that doesn't happen "normally". I don't consider "working offline" normal. Caches are free to serve potentially stale entries as long as they they disclose how old they are (so the user agent can determine if it's usable, or warn the user). My impression is that HTTP caching fulfills your needs and that the HTTP specification doesn't forbid the behaviour you prescribe. Thus you're free to implement your ideal behaviour without modifying any specifications. Are you unable to use HTTP caching? > In the sense, it's exactly like "working offline mode" whenever the > user is offline or the server is not responding. > Why can't you use "offline mode"? Serve the dynamic content with the expiration date set to the past. That way the UA can store it for offline use if it has enough storage space at it's disposal (making space for it using an implementation defined cache algorithm such as LFU). > The current App Cache design updates the cache to the latest version > in the background when the user visit the page for the second time and > then it needs to refresh the page to actually update the display. This > is annoying since the user will first see stale data, then a few > second later, it's updated with a giant refresh (including all the > static resources). This is because the App Cache is too COARSE > grained. It doesn't know what actually changes (which data are static, > which data are dynamic). That is another reason why we need > pageStorage: to separate the dynamic and the static resources. > Disclaimer: I'll have to do some reading on App Cache; I don't even understand why good ol' HTTP caching doesn't do the job. HTTP caches know which resources are static or dynamic, as the HTTP server tells them. A networked cache will
Re: [whatwg] Unlimited pageStorage for App Cached web pages
On Mon, May 30, 2011 at 10:39 PM, Bjartur Thorlacius wrote: > The following is how I understand your requirements; please correct me where > correction is due. > You've got two types of resources: > > 1. static resources, to be retrieved once and cached indefinitely > 2. dynamic resources, to be updated whenever possible while always > keeping the last version The dynamic resources only updated if the user visit the particular app cached web-page. Remember that the dynamic resources I'm talking about here is NOT shared between other web-cached pages (even they are in the same domain). > The former is easy to achieve, but user agents tend to throw away stale > versions as to not present outdated information to the user and to save > storage space. The user agent only need to keep the latest version. It's fine to throw away the outdated one if you have the latest. > You want user agents to fetch the latest version whenever possible, but keep > an old copy for when your servers are unreachable. The "whenever possible" is when the user revisit the cached page. The "old" here means the latest version that was cached.. > 5MB ought to be enough for anyone. You were joking, right? :D 5MB for each App Cached web-page is probably OK. However, 5MB localStorage quota for each domain is NOT OK. > If all you want to do is store mutable resources for offline use and > validate them if possible, but returning the cached entry (or entity) if > validation is impossible, simply serve the resources with an Expires header > set to a date in the past. Here is an example of how I want the App Cache to behave: First, it always try to fetch the main page, and all its static/dynamic resources and display it (just like normal web page). Then some time later, if the user want's to visit the SAME page again but the user is offline or the server is unavailable, then the latest cache of the page is displayed. In the sense, it's exactly like "working offline mode" whenever the user is offline or the server is not responding. The current App Cache design updates the cache to the latest version in the background when the user visit the page for the second time and then it needs to refresh the page to actually update the display. This is annoying since the user will first see stale data, then a few second later, it's updated with a giant refresh (including all the static resources). This is because the App Cache is too COARSE grained. It doesn't know what actually changes (which data are static, which data are dynamic). That is another reason why we need pageStorage: to separate the dynamic and the static resources. Felix Halim
Re: [whatwg] Unlimited pageStorage for App Cached web pages
Þann mán 30.maí 2011 03:42, skrifaði Felix Halim: Hmm.. yes, I think "unlimited" is a bad word (I just use it because currently App Cache quota is unlimited). Let me explain my need for pageStorage in a different way: Suppose I have a web page and want to store it in an App Cache. This web page requires a few resources (.ccs, .js, images, etc..). But all of them are "static" resources. I want to store "dynamic" resources as well for that particular page only (not shared). Think of the dynamic resources as "data" that changes from time to time for that particular page only. localStorage can be used to store the "dynamic" resources, but localStorage has very limited quota and it is shared to the entire domain. Different unrelated pages in the same domain will use the shared quota! The following is how I understand your requirements; please correct me where correction is due. You've got two types of resources: 1. static resources, to be retrieved once and cached indefinitely 2. dynamic resources, to be updated whenever possible while always keeping the last version The former is easy to achieve, but user agents tend to throw away stale versions as to not present outdated information to the user and to save storage space. You want user agents to fetch the latest version whenever possible, but keep an old copy for when your servers are unreachable. Currently I can "hack" the App Cache to simulate the pageStorage like this: We can turn one of the .js files "dynamic" by updating the .js file, then edit the MANIFEST file a bit, so that the browser re-download *ALL* the resources again. This way, the .js file quota gets in the App Cache quota which is currently *UNLIMITED*. But this "hack" is very costly, and inconvenient. 5MB ought to be enough for anyone. If all you want to do is store mutable resources for offline use and validate them if possible, but returning the cached entry (or entity) if validation is impossible, simply serve the resources with an Expires header set to a date in the past. That way caches SHOULD validate the response before reusing it, but may reuse it without validation under abnormal situations, such as when working offline. Of course, doing this for all resources doesn't make sense when storage space is scarce. In that case discard stylesheets, huge videos, heavy graphics other files with a high noise to content ratio.* If your (or your client's) cache doesn't do that already, you're free to modify it, or pay someone to do it for you. *Warning: The result of calculating the noise to content ratio of CSS files is undefined.
Re: [whatwg] Unlimited pageStorage for App Cached web pages
Hmm.. yes, I think "unlimited" is a bad word (I just use it because currently App Cache quota is unlimited). Let me explain my need for pageStorage in a different way: Suppose I have a web page and want to store it in an App Cache. This web page requires a few resources (.ccs, .js, images, etc..). But all of them are "static" resources. I want to store "dynamic" resources as well for that particular page only (not shared). Think of the dynamic resources as "data" that changes from time to time for that particular page only. localStorage can be used to store the "dynamic" resources, but localStorage has very limited quota and it is shared to the entire domain. Different unrelated pages in the same domain will use the shared quota! Currently I can "hack" the App Cache to simulate the pageStorage like this: We can turn one of the .js files "dynamic" by updating the .js file, then edit the MANIFEST file a bit, so that the browser re-download *ALL* the resources again. This way, the .js file quota gets in the App Cache quota which is currently *UNLIMITED*. But this "hack" is very costly, and inconvenient. Then I think, would it be better just to introduce a pageStorage which behaves like localStorage, but tied to the AppCache quota (not shared and tied to the main page only)? Felix Halim On Mon, May 30, 2011 at 5:05 AM, Bjartur Thorlacius wrote: > On 5/28/11, Felix Halim wrote: >> To summarize, the pageStorage offers "unlimited" storage for dynamic >> content for the App Cached web pages. > User agents may store expired pages for offline use. Internet Explorer > and Firefox have 'Work offline' modes automatically enabled on > complete disconnection from the network. Currently, only cached pages > and sites explicitly selected by the user are available offline, but > given enough disk space, user agents might keep all files of MIME type > "text" (e.g. text/html and text/plain) - or even all files. > The variation on constraints between systems is such that even looking > only at my desk there's a system with over 1.7GiB of free read-write > memory (0.5MiB magnetic, 1.3GiB volatile RAM) and another one with > under 300MiB (volatile RAM). I don't want authors to be able to use up > my memory by storing most or all content for offline use, nor to > unnecessarily loose access to content when storage space is plentiful. >
Re: [whatwg] Unlimited pageStorage for App Cached web pages
On 5/28/11, Felix Halim wrote: > To summarize, the pageStorage offers "unlimited" storage for dynamic > content for the App Cached web pages. User agents may store expired pages for offline use. Internet Explorer and Firefox have 'Work offline' modes automatically enabled on complete disconnection from the network. Currently, only cached pages and sites explicitly selected by the user are available offline, but given enough disk space, user agents might keep all files of MIME type "text" (e.g. text/html and text/plain) - or even all files. The variation on constraints between systems is such that even looking only at my desk there's a system with over 1.7GiB of free read-write memory (0.5MiB magnetic, 1.3GiB volatile RAM) and another one with under 300MiB (volatile RAM). I don't want authors to be able to use up my memory by storing most or all content for offline use, nor to unnecessarily loose access to content when storage space is plentiful.
[whatwg] Unlimited pageStorage for App Cached web pages
AFAIK, currently there is no storage limit for the App Cache. However, localStorage does has limit of 5 MB. It is silly to force the user to "install" a web page just to get unlimited localStorage. I'm thinking why not introduce an "unlimited pageStorage" that behaves like App Cache? The pageStorage is tied to a particular web page's URL just like what App Cache does. Any key/value that is inserted to the pageStorage is associated with the App Cache. What is the use case for the pageStorage? Suppose a web page has N users. Each user has different public page. These public pages contains static content and dynamic content. The App Cache can only captures the static content (if we use App Cache to store the dynamic content, then everytime the user visit the page, the user will has to "double" refresh the page to get the latest App Cache). We are missing a way to store the dynamic content! (here is the job of pageStorage). We can use localStorage for the dynamic content, but it is limited to 5 MB and it is shared between pages of the same domain (we don't need shared storage). What we need is an unlimited "pageStorage" that stores dynamic content for each App Cache. To summarize, the pageStorage offers "unlimited" storage for dynamic content for the App Cached web pages. Is this possible to do? Felix Halim
