Re: [IndexedDB] Granting storage quotas
Dumi,
I am not sure what the API expectations are for different levels of durability
of storage APIs. Is it:
1. Options passed to individual APIs selecting durability level
2. Separate API calls for different durability level
3. Allocations occurring through markup requiring user actions which acts as
ambient durability level for a site.
I surely would like to avoid 1 and 2. I felt like the discussion (based on your
proposal) was leaning towards 3. However, I can't read this email in that way.
Thanks for clarifying for my sake.
Nikunj
On May 4, 2010, at 6:08 PM, Dumitru Daniliuc wrote:
ian, it seems to me that nobody objects to adding a isPersistent optional
parameter to openDatabase{Sync}() in the WebSQLDatabases spec (default =
false). can you please add it to the spec? if isPersistent = true and the UA
doesn't support persistent storage, then i believe openDatabase{Sync}()
should throw a SECURITY_ERR just like it does when the DB cannot be opened.
thanks,
dumi
On Thu, Apr 29, 2010 at 1:20 PM, Shawn Wilsher sdwi...@mozilla.com wrote:
On 4/29/2010 1:08 PM, Tab Atkins Jr. wrote:
When you say per site do you mean per subdomain, or per domain? The
former is too permissive, the latter is too restrictive.
I believe he means per origin. At least that's what I took from our
discussion.
Cheers,
Shawn
Re: [IndexedDB] Granting storage quotas
On Thu, May 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Dumi, I am not sure what the API expectations are for different levels of durability of storage APIs. Is it: 1. Options passed to individual APIs selecting durability level 2. Separate API calls for different durability level 3. Allocations occurring through markup requiring user actions which acts as ambient durability level for a site. I surely would like to avoid 1 and 2. I felt like the discussion (based on your proposal) was leaning towards 3. However, I can't read this email in that way. Thanks for clarifying for my sake. I don't think you've accurately characterized the situation. 1 vs. 2 has been bike-shedding so far. In other words, I don't think there have been any super compelling arguments for or against them, but I think 1 has had a bit more support. Either 1 or 2 is required in order for a single origin to have multiple levels of durability. For example, I have 14GB of email in my GMail account. I'd like to have a guarantee that I have at least the last 7 days of email + any drafts + emails in my outbox. I'd also like it to cache all the rest, but if my hard drive fills up, I'm happy for it to drop the extra. Another example: I have a photo sharing site. I upload a bunch of photos while offline that I'd like it to upload once I get back online. I also have several gigs of data cached to make the photo browsing experience better, but I'm fine with it being deleted. These types of scenarios become even more compelling when we're talking about mobile devices and net-books. I know it's been argued that these devices keep getting more storage, so we won't need to worry about it soon, but I'd argue that storage usage keeps increasing as well. 3 is a separate issue and is related to HOW we grant permissions. I think several of us at Google are in favor of something like this, but I haven't seen anyone else express interest so we're going to continue on without the new piece of markup for now, but if there was support for it, I think we'd be happy to go that route. J
Re: [IndexedDB] Granting storage quotas
nikunj, i agree with what jeremy said. i think we need each storage API to be able to specify what kind of storage it needs (and i'm trying to add an optional flag for that to WebSQLDatabases, which is your option #1). in addition to that, i think we need an API that would allow an app to request persistent storage up front independent of the specific storage APIs the app will use (your option #3?), but it doesn't look like that is going anywhere. dumi On Thu, May 6, 2010 at 3:30 AM, Jeremy Orlow jor...@chromium.org wrote: On Thu, May 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Dumi, I am not sure what the API expectations are for different levels of durability of storage APIs. Is it: 1. Options passed to individual APIs selecting durability level 2. Separate API calls for different durability level 3. Allocations occurring through markup requiring user actions which acts as ambient durability level for a site. I surely would like to avoid 1 and 2. I felt like the discussion (based on your proposal) was leaning towards 3. However, I can't read this email in that way. Thanks for clarifying for my sake. I don't think you've accurately characterized the situation. 1 vs. 2 has been bike-shedding so far. In other words, I don't think there have been any super compelling arguments for or against them, but I think 1 has had a bit more support. Either 1 or 2 is required in order for a single origin to have multiple levels of durability. For example, I have 14GB of email in my GMail account. I'd like to have a guarantee that I have at least the last 7 days of email + any drafts + emails in my outbox. I'd also like it to cache all the rest, but if my hard drive fills up, I'm happy for it to drop the extra. Another example: I have a photo sharing site. I upload a bunch of photos while offline that I'd like it to upload once I get back online. I also have several gigs of data cached to make the photo browsing experience better, but I'm fine with it being deleted. These types of scenarios become even more compelling when we're talking about mobile devices and net-books. I know it's been argued that these devices keep getting more storage, so we won't need to worry about it soon, but I'd argue that storage usage keeps increasing as well. 3 is a separate issue and is related to HOW we grant permissions. I think several of us at Google are in favor of something like this, but I haven't seen anyone else express interest so we're going to continue on without the new piece of markup for now, but if there was support for it, I think we'd be happy to go that route. J
Re: [IndexedDB] Granting storage quotas
ian, it seems to me that nobody objects to adding a isPersistent optional
parameter to openDatabase{Sync}() in the WebSQLDatabases spec (default =
false). can you please add it to the spec? if isPersistent = true and the UA
doesn't support persistent storage, then i believe openDatabase{Sync}()
should throw a SECURITY_ERR just like it does when the DB cannot be opened.
thanks,
dumi
On Thu, Apr 29, 2010 at 1:20 PM, Shawn Wilsher sdwi...@mozilla.com wrote:
On 4/29/2010 1:08 PM, Tab Atkins Jr. wrote:
When you say per site do you mean per subdomain, or per domain? The
former is too permissive, the latter is too restrictive.
I believe he means per origin. At least that's what I took from our
discussion.
Cheers,
Shawn
Re: [IndexedDB] Granting storage quotas
On Thu, Apr 29, 2010 at 11:56 AM, Michael Nordman micha...@google.com wrote: Sounds like we agree on there being a distinction between two levels of persistence with one being more permanent than the other. Great, provided we have that agreement we can craft interfaces that allow callers to make the distinction! Well, I wouldn't say that we have committed one way or another. The main thing we want to do at mozilla is experimenting with different UIs and see if we can make something good. I'm fine with making changes to the specs, as long as we're fine with making further edits down the road should it be needed. / Jonas
Re: [IndexedDB] Granting storage quotas
On Thu, Apr 29, 2010 at 10:57 AM, Jonas Sicking jo...@sicking.cc wrote: I think we were operating under the assumption that we're going to avoid involving the user until neccesary. So for example letting the site store a few MB of data without the user getting involved, and only once enough storage is wanted by the site, ask the user if this is ok. When you say per site do you mean per subdomain, or per domain? The former is too permissive, the latter is too restrictive. On Thu, Apr 29, 2010 at 11:56 AM, Michael Nordman micha...@google.com wrote: Sounds like we agree on there being a distinction between two levels of persistence with one being more permanent than the other. Great, provided we have that agreement we can craft interfaces that allow callers to make the distinction! Not quite. If we agree that there are multiple levels, but one browser interprets that as being varying levels of temporary storage, but another interprets that as temporary and permanent storage, then authors are still unhappy. :/ ~TJ
Re: [IndexedDB] Granting storage quotas
On 4/29/2010 1:08 PM, Tab Atkins Jr. wrote: When you say per site do you mean per subdomain, or per domain? The former is too permissive, the latter is too restrictive. I believe he means per origin. At least that's what I took from our discussion. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Granting storage quotas
shawn, did you have a chance to give this some thought? how would mozilla like to handle cases like the ones jeremy and robin mentioned? how would you like to manage quotas? thanks, dumi On Fri, Apr 23, 2010 at 11:08 AM, Shawn Wilsher sdwi...@mozilla.com wrote: On 4/23/2010 7:39 AM, Nikunj Mehta wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I haven't talked to anyone at Mozilla that thinks that having permanent and non-permanent-but-possibly-long-lasting data to be a good idea. There does seem to be support for a session-only version of indexedDB. Cheers, Shawn
Re: [IndexedDB] Granting storage quotas
On 4/28/2010 2:54 PM, Dumitru Daniliuc wrote: shawn, did you have a chance to give this some thought? how would mozilla like to handle cases like the ones jeremy and robin mentioned? how would you like to manage quotas? We chatted yesterday, but I haven't had a chance to get it down into writing. Sorry about the delay, but I've had some other things come up. Should be able to get it down in writing to this list this week. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Granting storage quotas
We had some discussions about this at mozilla yesterday. I think the summary is something like this: * We'd like to expire data in IndexDB after some time. This will likely be based on heuristics, such as haven't visited the site for an extended period of time, though possibly keep the data a bit longer if it was put in the database during offline mode and thus likely contains data from the user. So in other words, we'd like to prevent data staying on the users system indefinitely for a site that the user no longer uses. * We'll expose UI to allow the user to remove data at any point. Similar to cookie UIs. * We'll expose APIs to extension developers to allow extensions to manage data lifetime as well, similar to cookie APIs. * We support the idea of temporary object stores used to for example implement cross joins. One solution is that these as never inserted into a database, but are only useable if you have a reference to a store object. Once the storage object is no longer used and is GCed, the store is nuked. * There was some support for non-permanent-but-possibly-long-lasting, useful for for example caching data that is also available on the server. The main concern here was that sites will always just opt for the most permanent storage option. So we need to ensure that there is incentive not to do this. We definitely realize that automatically expiring data stored in IndexDB needs to be done with much care. Whatever the spec says, developers will likely think of the storage area as basically permanent. So we'll likely need to experiment with the expiring a good deal. Ultimately it probably doesn't make sense to have MUST requirements around this as this can't be tested anyway. I.e. there is no way to test that data put in the storage won't be expired in half a year. It might make sense to have some informal language in the spec so that authors have some idea of how this works, but for now we have no language to recommend since we'd like to get implementation experience. Again, this is similar to how cookies work, where we (and I believe other browsers), have played around with different expiration policies throughout the years. / Jonas On Wed, Apr 28, 2010 at 2:54 PM, Dumitru Daniliuc d...@chromium.org wrote: shawn, did you have a chance to give this some thought? how would mozilla like to handle cases like the ones jeremy and robin mentioned? how would you like to manage quotas? thanks, dumi On Fri, Apr 23, 2010 at 11:08 AM, Shawn Wilsher sdwi...@mozilla.com wrote: On 4/23/2010 7:39 AM, Nikunj Mehta wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I haven't talked to anyone at Mozilla that thinks that having permanent and non-permanent-but-possibly-long-lasting data to be a good idea. There does seem to be support for a session-only version of indexedDB. Cheers, Shawn
Re: [IndexedDB] Granting storage quotas
This thinking resonates with what we've been thinking too (I think). On Wed, Apr 28, 2010 at 3:42 PM, Jonas Sicking jo...@sicking.cc wrote: We had some discussions about this at mozilla yesterday. I think the summary is something like this: * We'd like to expire data in IndexDB after some time. This will likely be based on heuristics, such as haven't visited the site for an extended period of time, though possibly keep the data a bit longer if it was put in the database during offline mode and thus likely contains data from the user. So in other words, we'd like to prevent data staying on the users system indefinitely for a site that the user no longer uses. * We'll expose UI to allow the user to remove data at any point. Similar to cookie UIs. * We'll expose APIs to extension developers to allow extensions to manage data lifetime as well, similar to cookie APIs. * We support the idea of temporary object stores used to for example implement cross joins. One solution is that these as never inserted into a database, but are only useable if you have a reference to a store object. Once the storage object is no longer used and is GCed, the store is nuked. * There was some support for non-permanent-but-possibly-long-lasting, useful for for example caching data that is also available on the server. The main concern here was that sites will always just opt for the most permanent storage option. So we need to ensure that there is incentive not to do this. We have in mind that the incentives for developers to not always utilize the most permanent storage option are... 1) Non-permanent storage is accessible w/o explicit user opt-in, so less annoying UI. 2) Less of a burden to 'manage' the data if the user-agent will clean it up for you. We definitely realize that automatically expiring data stored in IndexDB needs to be done with much care. Whatever the spec says, developers will likely think of the storage area as basically permanent. So we'll likely need to experiment with the expiring a good deal. Ultimately it probably doesn't make sense to have MUST requirements around this as this can't be tested anyway. I.e. there is no way to test that data put in the storage won't be expired in half a year. It might make sense to have some informal language in the spec so that authors have some idea of how this works, but for now we have no language to recommend since we'd like to get implementation experience. Again, this is similar to how cookies work, where we (and I believe other browsers), have played around with different expiration policies throughout the years. / Jonas On Wed, Apr 28, 2010 at 2:54 PM, Dumitru Daniliuc d...@chromium.org wrote: shawn, did you have a chance to give this some thought? how would mozilla like to handle cases like the ones jeremy and robin mentioned? how would you like to manage quotas? thanks, dumi On Fri, Apr 23, 2010 at 11:08 AM, Shawn Wilsher sdwi...@mozilla.com wrote: On 4/23/2010 7:39 AM, Nikunj Mehta wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I haven't talked to anyone at Mozilla that thinks that having permanent and non-permanent-but-possibly-long-lasting data to be a good idea. There does seem to be support for a session-only version of indexedDB. Cheers, Shawn
Re: [IndexedDB] Granting storage quotas
On Wed, Apr 28, 2010 at 4:03 PM, Michael Nordman micha...@google.com wrote: This thinking resonates with what we've been thinking too (I think). On Wed, Apr 28, 2010 at 3:42 PM, Jonas Sicking jo...@sicking.cc wrote: We had some discussions about this at mozilla yesterday. I think the summary is something like this: * We'd like to expire data in IndexDB after some time. This will likely be based on heuristics, such as haven't visited the site for an extended period of time, though possibly keep the data a bit longer if it was put in the database during offline mode and thus likely contains data from the user. So in other words, we'd like to prevent data staying on the users system indefinitely for a site that the user no longer uses. * We'll expose UI to allow the user to remove data at any point. Similar to cookie UIs. * We'll expose APIs to extension developers to allow extensions to manage data lifetime as well, similar to cookie APIs. * We support the idea of temporary object stores used to for example implement cross joins. One solution is that these as never inserted into a database, but are only useable if you have a reference to a store object. Once the storage object is no longer used and is GCed, the store is nuked. * There was some support for non-permanent-but-possibly-long-lasting, useful for for example caching data that is also available on the server. The main concern here was that sites will always just opt for the most permanent storage option. So we need to ensure that there is incentive not to do this. We have in mind that the incentives for developers to not always utilize the most permanent storage option are... 1) Non-permanent storage is accessible w/o explicit user opt-in, so less annoying UI. This is interesting. I think that we'll want to allow sites some amount of permanent storage without explicit user opt-in. Though we could allow much more for the caching storage. This sounds like a very good idea. 2) Less of a burden to 'manage' the data if the user-agent will clean it up for you. I'm not convinced this will be much of incentive. I think few sites are as interested in cleaning up the users hard drive as the user is. I can see many sites dropping data into the permanent storage and then caring very little when that is cleaned up. I'd imagine many ad networks would love it if it was never cleaned up. The incentive I had in mind was that if the UA decides it needs to purge data for a specific site, for example due to hitting some global quota, then we'd first clear out the data with the lowest level of permanence first. So it wouldn't make a difference if all data for a site is permanent, or if all of it is semi-permanent, in both cases it'd all get nuked. However if the site has some permanent and some semi-permanent, then we'd clear out the semi-permanent first and remeasure if we're up against the global quota still. But I also really like the idea of having less UI for semi-permanent. / Jonas
Re: [IndexedDB] Granting storage quotas
On Wed, Apr 28, 2010 at 4:17 PM, Eric Uhrhane er...@google.com wrote: On Wed, Apr 28, 2010 at 3:42 PM, Jonas Sicking jo...@sicking.cc wrote: We had some discussions about this at mozilla yesterday. I think the summary is something like this: * We'd like to expire data in IndexDB after some time. This will likely be based on heuristics, such as haven't visited the site for an extended period of time, though possibly keep the data a bit longer if it was put in the database during offline mode and thus likely contains data from the user. So in other words, we'd like to prevent data staying on the users system indefinitely for a site that the user no longer uses. Just to clarify: you're looking to expire storage that's marked as persistent [or something like that] or that's marked as temporary, or you're not planning to distinguish strongly between types of storage? We're definitely looking at expiring both. Though possibly with different levels of aggressiveness. See other emails in regards to having incentives for sites to use the temporary storage. / Jonas
Re: [IndexedDB] Granting storage quotas
We probably want to have different policies for different kinds of devices. For mobile, pruning unused storage is definitely important, but for modern desktops with 1TB drives most users probably won't ever need to free up disk space unless they're hit with some kind of denial-of-service attack, in which case policy should focus on preventing the latter. And this is going to get more true over time. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6]
Re: [IndexedDB] Granting storage quotas
On Wed, Apr 28, 2010 at 4:32 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Apr 28, 2010 at 4:03 PM, Michael Nordman micha...@google.com wrote: We have in mind that the incentives for developers to not always utilize the most permanent storage option are... 1) Non-permanent storage is accessible w/o explicit user opt-in, so less annoying UI. This is interesting. I think that we'll want to allow sites some amount of permanent storage without explicit user opt-in. Though we could allow much more for the caching storage. This sounds like a very good idea. The problems with allowing permanent no-user-ask storage have already been hashed out. It's impossible to do right - either you're too lenient (amount per subdomain) and malicious sites can fill up your permanent storage trivially by using iframes to subdomains, or you're too strict (amount per domain, or disallowing storage in nested browsing contexts) which kills a lot of legitimate and good use-cases. Tying permanence to user opt-in, and specifically to proactive user opt-in rather than a script-triggered dialog (modal or otherwise), appears to be the only way to chart the appropriate course between power and protection. 2) Less of a burden to 'manage' the data if the user-agent will clean it up for you. I'm not convinced this will be much of incentive. I think few sites are as interested in cleaning up the users hard drive as the user is. I can see many sites dropping data into the permanent storage and then caring very little when that is cleaned up. I'd imagine many ad networks would love it if it was never cleaned up. The incentive I had in mind was that if the UA decides it needs to purge data for a specific site, for example due to hitting some global quota, then we'd first clear out the data with the lowest level of permanence first. So it wouldn't make a difference if all data for a site is permanent, or if all of it is semi-permanent, in both cases it'd all get nuked. However if the site has some permanent and some semi-permanent, then we'd clear out the semi-permanent first and remeasure if we're up against the global quota still. But I also really like the idea of having less UI for semi-permanent. Requiring a user to explicitly hit an input or similar to allocate permanent storage should be enough to make sites use volatile storage unless they really need permanence. Expiring permanent storage automatically, though, means that I can never be sure that my mail archives are actually secure on my computer, even though I told Gmail to save it to my hard drive. I can never be sure that my draft emails are really saved, and won't disappear. I have no problem with a browser simply not *exposing* permanent storage, and not allowing authors to request it. But if any major browser exposes a permanent storage that isn't actually permanent, then we app developers no longer have a permanent storage to rely on. It's simply gone, unless we put up a Best viewed in anything but Firefox sticker. On mobile platforms where storage is at a premium and it's much more difficult for users to manipulate the filesystem, just don't allow permanent storage. But don't lie to the application and say you support something that you explicitly don't. Lying browsers cause horrific confusion and bugs. _ ~TJ
Re: [IndexedDB] Granting storage quotas
On Apr 23, 2010, at 20:08 , Shawn Wilsher wrote: On 4/23/2010 7:39 AM, Nikunj Mehta wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I haven't talked to anyone at Mozilla that thinks that having permanent and non-permanent-but-possibly-long-lasting data to be a good idea. There does seem to be support for a session-only version of indexedDB. How would you handle the case of a web site offering a video editing service? Would you require that all the footage be uploaded (and redownloaded every time the browser deletes the cache due to its massive size)? Why not just let the user upload the final product? That seems rather wasteful, and even though video editing is a slightly extreme example, there are plenty of other cases that can generate a lot of data that is only of local use. -- Robin Berjon - http://berjon.com/
Re: [IndexedDB] Granting storage quotas
On Apr 21, 2010, at 1:03 PM, Michael Nordman wrote: On Wed, Apr 21, 2010 at 12:10 PM, Mike Clement mi...@google.com wrote: FWIW, the transient vs. permanent storage support is exactly why I eagerly await an implementation of EricU's Filesystem API. Being able to guarantee that the UA will not discard potentially irreplaceable data is of paramount importance to web apps that want to work in an offline mode. I also find that the current arbitrary quota limit of 5MB per domain makes local storage APIs unusable for all but the most rudimentary apps (e.g., sticky note demo apps). There is an asymmetric distribution of local storage needs out there that no one is yet addressing (e.g., a photo- or video-related app might need GBs of local storage, an offline mail app might need tens or hundreds of MB, a TODO list app might only need kilobytes, etc.). I wholeheartedly support any effort to coordinate quota management among all of the various local storage APIs. The issue of quota limits is something that browser vendors will need to address soon enough, and it's probably best left up to them. The need for permanent storage across all local storage APIs, though, is something that in my opinion should come out of the standardization process. Here's a stab at defining programming interfaces that make a distinction between transient vs permanent for the storage mechanisms. If we make additions like this, we should use the same terminology across the board. // WebSqlDBs, also could work for IndexedDBs window.openDatabase(...); // temporary window.openPermanentDatabase(...); // AppCaches, embellish the first line of the manifest file CACHE MANIFEST CACHE MANIFEST PERMANENT // FileSystem, see the draft, i've change the terms a little here window.requestFilesystem(...);// evictable window.requestPermanentFilesystem(...) // LocalStorage window.localStorage;// purgeable window.permanentLocalStorage; Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea?
Re: [IndexedDB] Granting storage quotas
On Fri, Apr 23, 2010 at 3:39 PM, Nikunj Mehta nik...@o-micron.com wrote: On Apr 21, 2010, at 1:03 PM, Michael Nordman wrote: On Wed, Apr 21, 2010 at 12:10 PM, Mike Clement mi...@google.com wrote: FWIW, the transient vs. permanent storage support is exactly why I eagerly await an implementation of EricU's Filesystem API. Being able to guarantee that the UA will not discard potentially irreplaceable data is of paramount importance to web apps that want to work in an offline mode. I also find that the current arbitrary quota limit of 5MB per domain makes local storage APIs unusable for all but the most rudimentary apps (e.g., sticky note demo apps). There is an asymmetric distribution of local storage needs out there that no one is yet addressing (e.g., a photo- or video-related app might need GBs of local storage, an offline mail app might need tens or hundreds of MB, a TODO list app might only need kilobytes, etc.). I wholeheartedly support any effort to coordinate quota management among all of the various local storage APIs. The issue of quota limits is something that browser vendors will need to address soon enough, and it's probably best left up to them. The need for permanent storage across all local storage APIs, though, is something that in my opinion should come out of the standardization process. Here's a stab at defining programming interfaces that make a distinction between transient vs permanent for the storage mechanisms. If we make additions like this, we should use the same terminology across the board. // WebSqlDBs, also could work for IndexedDBs window.openDatabase(...); // temporary window.openPermanentDatabase(...); // AppCaches, embellish the first line of the manifest file CACHE MANIFEST CACHE MANIFEST PERMANENT // FileSystem, see the draft, i've change the terms a little here window.requestFilesystem(...);// evictable window.requestPermanentFilesystem(...) // LocalStorage window.localStorage;// purgeable window.permanentLocalStorage; Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I think this is what Dumi suggested. The only hitch is that both IndexedDB and WebSQLDatabase's open commands already take optional parameters, so it'd start to get kind of ugly. (Presumably, you'd have to add undefineds in for any optional param you want to accept the default for.) But I don't necessarily think that problem is a show stopper.
Re: [IndexedDB] Granting storage quotas
On Fri, Apr 23, 2010 at 7:39 AM, Nikunj Mehta nik...@o-micron.com wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I don't think we can expose the type of permanence to the user in any sort of sane way. The fact that we have to bug the user at all for permanent storage is already bad, but necessary. As long as everyone thinks it's fine to expose an identical user interface for all types of permanent storage, then I'm cool with it. (I don't see any problem with doing so, given the types of permanence listed earlier.) ~TJ
Re: [IndexedDB] Granting storage quotas
On 4/23/2010 7:39 AM, Nikunj Mehta wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I haven't talked to anyone at Mozilla that thinks that having permanent and non-permanent-but-possibly-long-lasting data to be a good idea. There does seem to be support for a session-only version of indexedDB. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Granting storage quotas
On Fri, Apr 23, 2010 at 11:08 AM, Shawn Wilsher sdwi...@mozilla.com wrote: On 4/23/2010 7:39 AM, Nikunj Mehta wrote: Could we create an additional optional parameter for an open request with the type of permanence required? Or is it not a good idea? I haven't talked to anyone at Mozilla that thinks that having permanent and non-permanent-but-possibly-long-lasting data to be a good idea. There does seem to be support for a session-only version of indexedDB. What then do people at Mozilla propose for use cases like flickr/picasa wanting to cache files on the hard disk iff there's plenty of space to do so, but making sure that said caches are not a burden if that users hard disk later fills up? J
Re: [IndexedDB] Granting storage quotas
On Thu, Apr 22, 2010 at 3:14 PM, Dumitru Daniliuc d...@chromium.org wrote:
fwiw, i agree with michael and tab that we should split all storage into
permanent/persistent and temporary/evictable/purgeable. however, i don't
think we need separate calls such as openDatabase() and
openPermanentDatabase(). i think we should just add an optional isPersistent
parameter to openDatabase() which would default to 'false'.
On Wed, Apr 21, 2010 at 2:27 AM, Mark Seaborn mseab...@chromium.orgwrote:
On Tue, Apr 20, 2010 at 7:17 PM, Shawn Wilsher sdwi...@mozilla.comwrote:
On 4/20/2010 4:11 AM, Mark Seaborn wrote:
1) It doesn't allow a web app to ask for a storage allocation up front,
before it starts to consume the storage.
Why does that matter?
It doesn't support the use cases that I suggested at the start of the
thread:
* An e-mail web app requests an amount of storage that is large enough to
store all your current e-mail, plus your e-mail for the next year at
projected rates. As this runs out, it can request more.
* A backup web app requests an amount that is large enough to store your
data.
Suppose I leave an e-mail web app running on my laptop while I'm not using
it, expecting it to sync my mail. Then I take the laptop somewhere where
there's no network connectivity. I don't want to find that the e-mail app
failed to fetch my e-mail because it ran out of storage and was stuck while
the browser displayed a prompt that I wasn't there to see.
The same goes for the backup example. I wouldn't want to find that the
backup app has failed to back up my data because it got stuck at a prompt.
This wouldn't happen if the backup app had the ability to request, up front,
the amount of storage that it knows it will need. This request might happen
at the point where the user specifies what files they want the app to copy.
how about adding a simple async API that would allow apps to request
(permanent) storage up front?
window.getPermSpace(desiredPermSpace, permSpaceGrantedCallback);
function permSpaceGrantedCallback(permSpaceQuota, permSpaceUsed) {
// store stuff
}
the apps could use this API to:
1. request enough space before doing anything -- no quota-related failures
in the middle of a storage operation (unless the hard drive unexpectedly
fills up, the user loses the network connection, etc.)
2. figure out what to store on the user's machine (number of emails to
sync, etc.) based on how much permanent space they got and how much of it
they're already using.
3. figure out the amount of space they are using/can still use at any point
in time (and potentially run a clean up job when they're getting close to
their quota, etc.).
the UAs could use this API to:
1. asynchronously get user's input when needed.
2. cleanly delete any permanent data stored by an app before the app starts
using that data, if the user decides not to give any permanent storage to
that app.
3. do better perm space book-keeping and prevent potential abuses before
they happen.
and future storage APIs (sync or async) wouldn't have to worry at all about
managing quotas.
There are a few details I might tweak, but overall I would be happy with an
approach like this. In Chromium, we would probably give 0 permanent storage
by default (all evictable/volatile/etc) and make apps request it like this.
In Chromium, there are other ways to request more storage (for example, in
the manifest of Chrome extensions) so this would even fit in seamlessly with
them (since they could request and we'd always say sure!).
I know that we've resisted baking explicit quotas into the web platform
before, but the feedback we've gotten from virtually every developer on this
is negative. I think this is a good compromise to say that platforms which
absolutely don't want to bake in the concept of quotas can only offer
evictable/volatile storage. And I like that requesting permission is
detached from opening any specific database/file. This means that we can
add other methods of requesting quota fairly easily as devices evolve.
J
Re: [IndexedDB] Granting storage quotas
On Wed, Apr 21, 2010 at 7:58 AM, Robin Berjon ro...@berjon.com wrote: On Tue, 20 Apr 2010 14:37:33 -0700 Michael Nordman micha...@google.com wrote: I think ericu is baking in a distinction in between 'permanent' and 'temporary' in the FileSystem API he's working on. Some harmony across all flavors of local storage could be good. He is: http://dev.w3.org/2009/dap/file-system/file-dir-sys.html#using-localfilesystemsync I haven't seen ericu pop up in this thread but I guess he's reading it. Either way, it's certainly an approach that I'd be happy to see fine-tuned to be more or less consistent across various local storage mechanisms. Thanks Robin--I'm actually just now catching up on most of my email, as I've been out of the office for a week. I agree completely that we need consistency across storage mechanisms. Using Nikunj's terminology, I'd love to see an evictable/persistent split on all storage types, and some kind of unified quota management for all client-side storage. I'm not sure that we need to spec out all the quota management stuff--we should leave a lot of room for UAs to experiment with how to interact with the user--but if we can add a simple mechanism for declaring/requesting quota across all storage types, I think that would be very helpful. Dumi has proposed a programmatic API, and Tab has suggested a new input type. I think either of those would be OK as long as interaction is completely asynchronous. Eric
Re: [IndexedDB] Granting storage quotas
On Tue, Apr 20, 2010 at 7:17 PM, Shawn Wilsher sdwi...@mozilla.com wrote: On 4/20/2010 4:11 AM, Mark Seaborn wrote: 1) It doesn't allow a web app to ask for a storage allocation up front, before it starts to consume the storage. Why does that matter? It doesn't support the use cases that I suggested at the start of the thread: * An e-mail web app requests an amount of storage that is large enough to store all your current e-mail, plus your e-mail for the next year at projected rates. As this runs out, it can request more. * A backup web app requests an amount that is large enough to store your data. Suppose I leave an e-mail web app running on my laptop while I'm not using it, expecting it to sync my mail. Then I take the laptop somewhere where there's no network connectivity. I don't want to find that the e-mail app failed to fetch my e-mail because it ran out of storage and was stuck while the browser displayed a prompt that I wasn't there to see. The same goes for the backup example. I wouldn't want to find that the backup app has failed to back up my data because it got stuck at a prompt. This wouldn't happen if the backup app had the ability to request, up front, the amount of storage that it knows it will need. This request might happen at the point where the user specifies what files they want the app to copy. 2) In Opera, the quota can only be increased in multiples of about 15, so it takes three prompts to get up into the range of gigabytes. But there is an unlimited option, yeah? True, but it would be better not to have to give the web app carte blanche. However, it is a minor point because it is easily fixable. 3) The web app can't choose when the question is put to the user. 4) The web app doesn't know how much storage has been allocated, so it doesn't know when a question will be asked. 5) In Opera, if the user chooses Reject, they don't get prompted again. This means that asking the user at an appropriate time is important for the continued functioning of the web app. Prompting the user at the wrong time will interrupt them with a page-modal dialog which they might want to get rid of with Reject, which would potentially break the web app by leaving it unable to get more storage. These all feel like user-agent specific worries on how the user agent wants to bring this to the attention of the user. It's not user agent specific if the fix involves adding or changing a programmatic interface, or involves defining behaviour that web apps are likely to rely on to function correctly. Regards, Mark
Re: [IndexedDB] Granting storage quotas
On Tue, Apr 13, 2010 at 4:53 PM, João Eiras jo...@opera.com wrote:
On Tue, 13 Apr 2010 12:09:14 +0200, Mark Seaborn mseab...@chromium.org
wrote:
Is there any plan for involving the user in storage allocation decisions
for
IndexedDB? [1]
For comparison, the WebStorage API [2] doesn't have any special support
for
the user to make allocation choices. My understanding is that browsers
have
a fixed storage limit per origin -- in Chromium, 5Mb per origin. The
problem with this model is that it is both too permissive and too
restrictive.
Not really. The user agent can ask for quota from the user when the limit
is being hit without the webpage even having to worry about it. Opera 10.50
does that.
I tried this out in Opera with a script that doubles the amount of storage
it uses on each reload [1]. I got a dialog saying:
Web Storage Quota
foo.com wants to store data on your computer, but needs more space.
Current limit: 2.7 MB
Requested limit: 5.4 MB
[this is a dropdown, also containing:
8.1 MB
10.7 MB
21.4 MB
42.7 MB
Unlimited]
[Allow] [Reject]
This is still too restrictive because:
1) It doesn't allow a web app to ask for a storage allocation up front,
before it starts to consume the storage.
2) In Opera, the quota can only be increased in multiples of about 15, so it
takes three prompts to get up into the range of gigabytes.
3) The web app can't choose when the question is put to the user.
4) The web app doesn't know how much storage has been allocated, so it
doesn't know when a question will be asked.
5) In Opera, if the user chooses Reject, they don't get prompted again.
This means that asking the user at an appropriate time is important for the
continued functioning of the web app. Prompting the user at the wrong time
will interrupt them with a page-modal dialog which they might want to get
rid of with Reject, which would potentially break the web app by leaving
it unable to get more storage.
The workaround for (1) is to write junk data into local storage in an
attempt to force the browser to produce a prompt. This is obviously
inefficient. Also, this strategy is dependent on knowing what the initial
quota is, but because of (4), a web app would have to guess based on
knowledge of particular browsers' policies.
(2) is easily addressed by providing an editable field and not just a
dropdown.
(1) and (3) would be addressed by adding a requestQuota() interface as I
suggested.
For (5), I couldn't find any options in the UI for resetting the
prompt/reject state so that Opera would prompt again. Delete Private Data
- Delete Web storage databases did not reset the prompt/reject state.
However, by searching I found that opera:webstorage provides a page for
granting larger quotas and resetting the prompt/reject state.
(5) could be addressed by using a non-modal info bar, or by addressing (3).
I don't mean to pick on Opera, by the way. I think it's positive that it
does allow the quota to be increased.
Regards,
Mark
[1] Here's the page I used to test this:
p
You have viewed this page
span id=countan untold number of/span
time(s).
/p
p
Space consumed: span id=bytes/span
/p
script
try {
if (!localStorage.pageLoadCount) {
localStorage.pageLoadCount = 0;
localStorage.data = x;
}
localStorage.pageLoadCount = parseInt(localStorage.pageLoadCount) + 1;
document.getElementById('count').textContent =
localStorage.pageLoadCount;
var data = localStorage.data;
data = data + data;
localStorage.data = data;
document.getElementById('bytes').textContent = data.length + chars;
}
catch (exn) {
document.getElementById('bytes').textContent = exn;
}
/script
Re: [IndexedDB] Granting storage quotas
On 4/20/2010 4:11 AM, Mark Seaborn wrote: 1) It doesn't allow a web app to ask for a storage allocation up front, before it starts to consume the storage. Why does that matter? 2) In Opera, the quota can only be increased in multiples of about 15, so it takes three prompts to get up into the range of gigabytes. But there is an unlimited option, yeah? 3) The web app can't choose when the question is put to the user. 4) The web app doesn't know how much storage has been allocated, so it doesn't know when a question will be asked. 5) In Opera, if the user chooses Reject, they don't get prompted again. This means that asking the user at an appropriate time is important for the continued functioning of the web app. Prompting the user at the wrong time will interrupt them with a page-modal dialog which they might want to get rid of with Reject, which would potentially break the web app by leaving it unable to get more storage. These all feel like user-agent specific worries on how the user agent wants to bring this to the attention of the user. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Granting storage quotas
This way of thinking is incompatible with offline web apps. If I'm offline and I send and email, it needs to stay queued up to send until I'm reconnected to the internet. Anyone wanting to debate whether or not the UA should be free to clean up persistent storage without asking the user should read http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-August/thread.html#22289 and the various other threads it spawned first and only re-start things if they have new information to add. J On Tue, Apr 20, 2010 at 3:10 PM, Nikunj Mehta nik...@o-micron.com wrote: As I see it, there's no such thing as permanent storage for Web browser managed data. Even if a site expresses preferences that it would like to keep its data resident for a long time, there cannot be a guarantee for the data to be there permanently. If applications are bound to have to deal with data disappearing while they are not running, we should not need to spec browser behavior around the notion of purgeable or permanent. It makes sense for an application, OTOH, to say that it does not need data to be stored on disk. IOW, create a database that is non-durable and, hence, kept only in memory. Such databases are required to be empty upon creation. They may be spilled over to disk, if implementations like to, but they will not be retained from session to session. Nikunj On Apr 20, 2010, at 2:37 PM, Michael Nordman wrote: I'd like to back up and challenge the notion of a per-site quota. In this discussion and others there is an underlying assumption that each site has some well defined limit that the user-agent has granted it. I doubt that's the best model. (Fyi: the chrome team's overly simplistic model whereby each site gets 5M was not chosen because its a good model, this was done just to proceed with building out the storage APIs independent of a real storage management strategy). I'd like to set aside the per-site quota assumption and explore some alternative models for web storage management. Some thoughts about the world we're designing for. There are an open ended number of sites, each of which *could* use web storage in some form. From that fact alone, it's impossible to come up with a quota that could be granted to each and every site. It seems likely that the number of sites that will actually require permanent storage is small compared to the number of sites that *could* make use of more volatile storage, to borrow jorlow's term, where the volatile data on disk can get scrapped by the user-agent as needed. Maybe a better term for that class of storage is purgeable? Maybe we should be designing for what seems to be the more common case, lots of sites that make use of volatile/purgeable storage. But also come up a means whereby the smaller number of sites that require stronger guarantees can express the need for more permanent storage. What if by default all local storage is purgeable. A lot of quota issues melt away in this case since the user agent is free to reclaim anything at anytime. I think it'd be reasonable if the user-agent never asked the user anything on a per-site basis. A user-agent could warn when system disk space crossed thresholds and give the user an option to set limits on system disk space usage for webstorage as a whole. What if when creating local storage repositories (WebDBs or IndexedDBs or WebFileSystems or AppCaches) there was an optional means for the webapp to express please consider this a permanent storage repository. The first time a site request permanent storage could be a reasonable time to interact with the user in some form, or to consult the user prefs about allowing permanent storage w/o being asked. I think ericu is baking in a distinction in between 'permanent' and 'temporary' in the FileSystem API he's working on. Some harmony across all flavors of local storage could be good. I actually think local storage management is an area where the webplatform has a chance to do a much better job then the desktop platforms have historically done. We don't need no stinking quotas ;) But we also don't need untold amounts of unused permanent storage littering disk drives needlessly around the globe (until the user gets a new system). A silly analogy. A computer is like a ship at sea. After years of usage, a whole bunch of barnacles build up on the hull and slow the vessel down. The webplatform to date is barnacle free in this area because there are no permanent local storage facilities... lets try to make these new features not so barnacle prone too. Cheers -Michael On Tue, Apr 20, 2010 at 11:17 AM, Shawn Wilsher sdwi...@mozilla.comwrote: On 4/20/2010 4:11 AM, Mark Seaborn wrote: 1) It doesn't allow a web app to ask for a storage allocation up front, before it starts to consume the storage. Why does that matter? 2) In Opera, the quota can only be increased in multiples of about 15, so it takes three prompts to get
Re: [IndexedDB] Granting storage quotas
On Tue, Apr 20, 2010 at 3:10 PM, Nikunj Mehta nik...@o-micron.com wrote: As I see it, there's no such thing as permanent storage for Web browser managed data. Even if a site expresses preferences that it would like to keep its data resident for a long time, there cannot be a guarantee for the data to be there permanently. If applications are bound to have to deal with data disappearing while they are not running, we should not need to spec browser behavior around the notion of purgeable or permanent. I see a difference between a cached version of a picture you've downloaded vs a new picture taken while on vacation by a camera built into the device and placed into a local repository managed by the user-agent. There is only one copy of that picture in the world. I'm looking for ways to make these storage APIs widely available w/o a lot of user-prompting, but also for ways for webapps to express stronger guarantees when needed. I think the notion of purgeable vs permanent may help reconcile these conflicting goals. It makes sense for an application, OTOH, to say that it does not need data to be stored on disk. IOW, create a database that is non-durable and, hence, kept only in memory. Such databases are required to be empty upon creation. They may be spilled over to disk, if implementations like to, but they will not be retained from session to session. Nikunj On Apr 20, 2010, at 2:37 PM, Michael Nordman wrote: I'd like to back up and challenge the notion of a per-site quota. In this discussion and others there is an underlying assumption that each site has some well defined limit that the user-agent has granted it. I doubt that's the best model. (Fyi: the chrome team's overly simplistic model whereby each site gets 5M was not chosen because its a good model, this was done just to proceed with building out the storage APIs independent of a real storage management strategy). I'd like to set aside the per-site quota assumption and explore some alternative models for web storage management. Some thoughts about the world we're designing for. There are an open ended number of sites, each of which *could* use web storage in some form. From that fact alone, it's impossible to come up with a quota that could be granted to each and every site. It seems likely that the number of sites that will actually require permanent storage is small compared to the number of sites that *could* make use of more volatile storage, to borrow jorlow's term, where the volatile data on disk can get scrapped by the user-agent as needed. Maybe a better term for that class of storage is purgeable? Maybe we should be designing for what seems to be the more common case, lots of sites that make use of volatile/purgeable storage. But also come up a means whereby the smaller number of sites that require stronger guarantees can express the need for more permanent storage. What if by default all local storage is purgeable. A lot of quota issues melt away in this case since the user agent is free to reclaim anything at anytime. I think it'd be reasonable if the user-agent never asked the user anything on a per-site basis. A user-agent could warn when system disk space crossed thresholds and give the user an option to set limits on system disk space usage for webstorage as a whole. What if when creating local storage repositories (WebDBs or IndexedDBs or WebFileSystems or AppCaches) there was an optional means for the webapp to express please consider this a permanent storage repository. The first time a site request permanent storage could be a reasonable time to interact with the user in some form, or to consult the user prefs about allowing permanent storage w/o being asked. I think ericu is baking in a distinction in between 'permanent' and 'temporary' in the FileSystem API he's working on. Some harmony across all flavors of local storage could be good. I actually think local storage management is an area where the webplatform has a chance to do a much better job then the desktop platforms have historically done. We don't need no stinking quotas ;) But we also don't need untold amounts of unused permanent storage littering disk drives needlessly around the globe (until the user gets a new system). A silly analogy. A computer is like a ship at sea. After years of usage, a whole bunch of barnacles build up on the hull and slow the vessel down. The webplatform to date is barnacle free in this area because there are no permanent local storage facilities... lets try to make these new features not so barnacle prone too. Cheers -Michael On Tue, Apr 20, 2010 at 11:17 AM, Shawn Wilsher sdwi...@mozilla.comwrote: On 4/20/2010 4:11 AM, Mark Seaborn wrote: 1) It doesn't allow a web app to ask for a storage allocation up front, before it starts to consume the storage. Why does that matter? 2) In Opera, the quota can only be increased in multiples of about 15, so
Re: [IndexedDB] Granting storage quotas
On 4/19/2010 10:08 PM, Jeremy Orlow wrote: On Tue, Apr 13, 2010 at 3:09 AM, Mark Seabornmseab...@chromium.org wrote: 2) It is too permissive because it enforces no limit on the amount of space a web app can use: A web app from example.com can create an unlimited number of puppet subdomains: aaa.example.com, bbb.example.com, etc. It can use aaa.example.com's 5Mb allocation by loading a script from aaa.example.com in an iframe and communicating with it using postMessage(). As far as I'm aware, no one (including Chromium) has a solution to at the moment. Which probably should be a cause for concern. :-) In the thread you linked to earlier, it was suggested that user agents could also factor in the top level domain into the cap (e.g. example.com only gets 50MB of space). But really, I don't see how any of this is testable in a deterministic matter, so I don't see the point in adding it to the spec. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Granting storage quotas
On Tue, Apr 20, 2010 at 5:59 PM, Jeremy Orlow jor...@chromium.org wrote: On Tue, Apr 20, 2010 at 5:42 PM, Nikunj Mehta nik...@o-micron.com wrote: On Apr 20, 2010, at 5:25 PM, Jeremy Orlow wrote: On Tue, Apr 20, 2010 at 5:07 PM, Shawn Wilsher sdwi...@mozilla.comwrote: On 4/20/2010 3:19 PM, Jeremy Orlow wrote: This way of thinking is incompatible with offline web apps. If I'm offline and I send and email, it needs to stay queued up to send until I'm reconnected to the internet. I think a smart browser would include am I offline in it's heuristic for granting storage space. Granting storage space is only part of the problem. Extrapolating from what you said, I guess I could see us keeping track of which origins have ever been accessed offline and making sure that data is never deleted without consulting the user. But, that doesn't solve this use case: something tike TiddlyWiki which lives totally on your system and is not supposed to be synced to the cloud. The problem is that if you used this on your desktop, which is never offline, then the browser would not know it's precious. We could use the existence of an appCache manifest as a hint. I guess that might be good enough. But then we still have the malware problem. Anyone wanting to debate whether or not the UA should be free to clean up persistent storage without asking the user should read http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-August/thread.html#22289 and the various other threads it spawned first and only re-start things if they have new information to add. I read it, but I don't see a consensus formed before it died off. Am I missing something? It was kind of difficult to track. The basic consensus was that persistent data can and should not be deleted without explicit user approval. On Tue, Apr 20, 2010 at 5:18 PM, Nikunj Mehta nik...@o-micron.com wrote: On Apr 20, 2010, at 3:19 PM, Jeremy Orlow wrote: This way of thinking is incompatible with offline web apps. If I'm offline and I send and email, it needs to stay queued up to send until I'm reconnected to the internet. I think the problem is that data loss could occur regardless of guarantees. Sure, and that came up in the original giant thread. Even if something's in the cloud, it could be lost. So what it really comes down to is best effort. Here are the classes of storage I hear you asking for: 1. temporary (no likelihood of data being being stored after session ends) I don't think this type is very interesting. The only use case is when you have so much data that the UA might need to spill to disk. And if you're doing that, I'd imagine you'd want to use one of the other storage types anyway. Oops, one thing I didn't consider is session as in per tab (like SessionStorage) rather than per document. I don't know if this is what you originally meant (based on your arguments, I assume it's not), in which case I'd consider this a 4th kind of storage. Now that I think about it, I'm pretty sure the enumeration of all of these types was done in some thread in the past. But I believe that's it: per document, per session/tab, caches/evictable/volatile/transient/purgable/etc, and persistient/non-evictable/etc.
Re: [IndexedDB] Granting storage quotas
On Tue, Apr 20, 2010 at 5:18 PM, Nikunj Mehta nik...@o-micron.com wrote: On Apr 20, 2010, at 3:19 PM, Jeremy Orlow wrote: This way of thinking is incompatible with offline web apps. If I'm offline and I send and email, it needs to stay queued up to send until I'm reconnected to the internet. I think the problem is that data loss could occur regardless of guarantees. Here are the classes of storage I hear you asking for: 1. temporary (no likelihood of data being being stored after session ends) 2. evictable (no limit per site, except global limits, eviction candidates chosen arbitrarily, including while application is running) 3. non-evictable (no eviction, but data loss possible, limited by user) Is this making sense? That categorization does make sense, including the temporary. I think we're (i know i am) most interested in the distinction between evictable and non-evictable right now because the quota management and ui implications. We've introduced these storage APIs that result in files on disk, and now we want to put in some real-world storage management features. The dumbed down 5MB per origin was a stop gap measure. This is reminding me of Mike Wilson's state handling post about different scopes and lifetimes... http://old.nabble.com/html5-state-handling:-overview-and-extensions-td24034773.html Anyone wanting to debate whether or not the UA should be free to clean up persistent storage without asking the user should read http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-August/thread.html#22289 and the various other threads it spawned first and only re-start things if they have new information to add. J On Tue, Apr 20, 2010 at 3:10 PM, Nikunj Mehta nik...@o-micron.com wrote: As I see it, there's no such thing as permanent storage for Web browser managed data. Even if a site expresses preferences that it would like to keep its data resident for a long time, there cannot be a guarantee for the data to be there permanently. If applications are bound to have to deal with data disappearing while they are not running, we should not need to spec browser behavior around the notion of purgeable or permanent. It makes sense for an application, OTOH, to say that it does not need data to be stored on disk. IOW, create a database that is non-durable and, hence, kept only in memory. Such databases are required to be empty upon creation. They may be spilled over to disk, if implementations like to, but they will not be retained from session to session. Nikunj On Apr 20, 2010, at 2:37 PM, Michael Nordman wrote: I'd like to back up and challenge the notion of a per-site quota. In this discussion and others there is an underlying assumption that each site has some well defined limit that the user-agent has granted it. I doubt that's the best model. (Fyi: the chrome team's overly simplistic model whereby each site gets 5M was not chosen because its a good model, this was done just to proceed with building out the storage APIs independent of a real storage management strategy). I'd like to set aside the per-site quota assumption and explore some alternative models for web storage management. Some thoughts about the world we're designing for. There are an open ended number of sites, each of which *could* use web storage in some form. From that fact alone, it's impossible to come up with a quota that could be granted to each and every site. It seems likely that the number of sites that will actually require permanent storage is small compared to the number of sites that *could* make use of more volatile storage, to borrow jorlow's term, where the volatile data on disk can get scrapped by the user-agent as needed. Maybe a better term for that class of storage is purgeable? Maybe we should be designing for what seems to be the more common case, lots of sites that make use of volatile/purgeable storage. But also come up a means whereby the smaller number of sites that require stronger guarantees can express the need for more permanent storage. What if by default all local storage is purgeable. A lot of quota issues melt away in this case since the user agent is free to reclaim anything at anytime. I think it'd be reasonable if the user-agent never asked the user anything on a per-site basis. A user-agent could warn when system disk space crossed thresholds and give the user an option to set limits on system disk space usage for webstorage as a whole. What if when creating local storage repositories (WebDBs or IndexedDBs or WebFileSystems or AppCaches) there was an optional means for the webapp to express please consider this a permanent storage repository. The first time a site request permanent storage could be a reasonable time to interact with the user in some form, or to consult the user prefs about allowing permanent storage w/o being asked. I think ericu is baking in a distinction in between 'permanent' and
Re: [IndexedDB] Granting storage quotas
On 4/13/2010 8:53 AM, João Eiras wrote: Not really. The user agent can ask for quota from the user when the limit is being hit without the webpage even having to worry about it. Opera 10.50 does that. I agree with this, and do see the benefit of adding more to the spec. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Granting storage quotas
On Mon, Apr 19, 2010 at 9:13 PM, Shawn Wilsher sdwi...@mozilla.com wrote: On 4/13/2010 8:53 AM, João Eiras wrote: Not really. The user agent can ask for quota from the user when the limit is being hit without the webpage even having to worry about it. Opera 10.50 does that. I agree with this, and do see the benefit of adding more to the spec. Sure. That addresses item 1 of Mark's original question, but what about this: On Tue, Apr 13, 2010 at 3:09 AM, Mark Seaborn mseab...@chromium.org wrote: 2) It is too permissive because it enforces no limit on the amount of space a web app can use: A web app from example.com can create an unlimited number of puppet subdomains: aaa.example.com, bbb.example.com, etc. It can use aaa.example.com's 5Mb allocation by loading a script from aaa.example.com in an iframe and communicating with it using postMessage(). As far as I'm aware, no one (including Chromium) has a solution to at the moment. Which probably should be a cause for concern. :-)
Re: [IndexedDB] Granting storage quotas
On Tue, 13 Apr 2010 12:09:14 +0200, Mark Seaborn mseab...@chromium.org wrote: Is there any plan for involving the user in storage allocation decisions for IndexedDB? [1] For comparison, the WebStorage API [2] doesn't have any special support for the user to make allocation choices. My understanding is that browsers have a fixed storage limit per origin -- in Chromium, 5Mb per origin. The problem with this model is that it is both too permissive and too restrictive. Not really. The user agent can ask for quota from the user when the limit is being hit without the webpage even having to worry about it. Opera 10.50 does that.
