Re: IndexedDB: Syntax for specifying persistent/temporary storage
Hi Kinuko and Eric, Thanks for providing feedback from a Google perspective! Definitely still feels like there's a lot of possibilities and options here. No combination of solutions seem obviously best to me, so please do take the below as me thinking out loud rather than having strong opinions. On Wed, Dec 11, 2013 at 10:53 PM, Kinuko Yasuda kin...@chromium.org wrote: Fyi, Chrome applies similar policy to (A) for installed apps, but only when a special permission is requested in its manifest file. For regular apps their 'temporary' storage is still under eviction control. Interesting! I had hoped that we would be able to align on storage policies for installed/bookmarked hosted apps (not packaged apps). Maybe that was too naive :-) Though sounds like we still are very close. Close enough that it we can probably align on API. The problem with this solution is that it doesn't give bookmarked apps the ability to create truly temporary data. Even data that a bookmarked app puts in the temporary storage is effectively treated as persistent and so not deleted if we start to run low on disk space. Temporary storage for apps is a feature that Android has, and that to some extent *nix OSs has had through use of /tmp. It definite is something that seems nice for constrained mobile devices. As a slightly conservative variation, would it work if we just 'unlock' persistent storage for webapps when they're 'bookmarked' or got an installed app-like permission? The app could check if it could use persistent storage without requesting (by issuing quota query API, or maybe via some event), and convert some data into persistent if they like, but UA doesn't do so automatically. The UA continues to evict 'temporary' data even for 'bookmarked' webapps (UA could prioritize temporary data for bookmarked ones if it wants). I think this would work, but it has downsides. See my answer below to Eric. B) We could create a temporary/default/persistent split. I.e. we create three different storage categories. The default is what's used if no storage category is explicitly specified a IDB database is created. For normal webpages default is treated like temporary. I.e. it is counted towards the same quotas as temporary storage and it's deleted automatically if we run low on space. However once a page transitions to being an installed app data in default is treated as persistent. temporary storage would always be treated as truly temporary. I.e. the implementation is always free to delete data there if the user is running low on storage. If data stored by bookmarked apps count towards the same quotas as temporary data stored by web pages is an implementation decision. persistent storage would behave as in A. I.e. webpages can't use it without there being a prompt involved at some point. Bookmarked webapps can use it without prompt. This sounds kinda reasonable to me, but I'd imagine some webapps (who are serious about their storage) would probably want to know if the default is either temporary or persistent in this case, and we may need a tiny API for that. Definitely, if we go with this solution we should provide a way to know what policy is used for default. D) Alex Russell proposed adding an event which asks a page I'm low on storage, please clear out any data that is not important. This could be combined with either A or C to enable installed apps to have really temporary data. It has the advantage that data that isn't critical can be stored in the same IDB database as critical data which is really nice. I.e. you no longer have to separate critical and non-critical data into separate databases which prevent transactions that touch both. However it has the disadvantage that it makes clearing out temporary data a *really* heavy operation. It means having to launch all apps in the background and have them run code. This could mean spawning lots of processes and doing lots of complicated queries and other IO. I'm actually a fan of this idea, as it gives most flexible option (but with some cost). I wouldn't say it doesn't make the eviction operation heavy, but UA could probably do smarter than launching all apps at once. Most apps may not be interested in such an event at all, and UA could only check the apps that have shown interest on adjusting their data on their own, probably via some registration mechanism. UA could also use some order like LRU to launch apps one by one until it can get enough available space. One of the situations when we need to clear out temporary storage is when we're in the middle of a IDB (or presumably WebSQL) transaction and run low on disk space. The page might already have been granted permanent storage so we don't want to simply abort the transaction and blame quota. Would a notification such as the one described here help in that situation? I.e. would you really want to pause the current transaction and start firing up
Re: IndexedDB: Syntax for specifying persistent/temporary storage
Option C) using numeric priority are good enough for most web application. It has advantage of easy to implement. Option C) can be combined with option B) by defining that 0 means temporary, 1 means persistent and undefined means default. Any other values should throw error. Since we can query allocated quota and knowing estimated app data size, persistent will be grantee. Kyaw
Re: IndexedDB: Syntax for specifying persistent/temporary storage
Good writeup, Jonas--I think you've hit the major points. I think numeric priorities are both overkill and underpowered, depending upon their specific implementation. Without the promise we're currently making for Persistent storage [this will never be cleared unless you do it or the user explicitly requests it], numeric priorities are ultimately weaker than apps want. Unless we say that the top priority is the same as persistent, in which case we've added complexity without taking any away. The idea of Default is kind of appealing, and easy to migrate over to, but I'm not sure it's necessary. As Kinuko says, we can just unlock Persistent storage for apps on install, and let them migrate over whichever data needs it. This would work better if we supplied a tool to do an atomic migration, though--using the current APIs, apps would have to use 2x their storage during the transition, and browser developers might be able to implement it internally with a simple flag change or directory rename. I don't have a strong opinion there, but I lean toward just the two types rather than three. As for Alex's please clear up space event--it's not clear to me how to do that cleanly for apps that aren't currently loaded, which may need to talk to servers that aren't currently running, which the user may never plan to run again, or which require credentials to access their stored data, etc. On Wed, Dec 11, 2013 at 7:39 PM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Thanks Jan for sending this. Now let me throw a giant wrench into this discussion :-) Unfortunately as we've been discussing webapps, manifests etc at mozilla I've slowly come to the realization that the temporary/persistent categorization isn't really fulfilling all the envisioned use cases. The background is that multiple platforms are now building the functionality to run normal websites outside of the browser. iOS was one of the first popular implementations of this. If you put a meta name=apple-mobile-web-app-capable content=yes in the markup of a page, and the user use bookmark to homescreen feature in iOS Safari, that almost turns the website into an app [1]. Google is currently working on implementing the same feature in Chrome for Android. At mozilla we created a proposal [2] for what is essentially a standardized version of the same idea. I think this approach is a really awesome use of the web and something that I'm very interested in supporting when designing these storage APIs. To support this use case, I think it needs to be possible for a website to first start as a website which the user only has a casual connection with, then gradually grow into something that the user essentially treats as a trusted app. Such a trusted app should have much more ability to store data without having to ask the user for permission, or without that data being suddenly deleted because we're low on disk space. In short, such an app should be treated more like a native app when it comes to storage. There are a few ways we can enable this use case. In the discussion below I'll use IndexedDB as an example of storage API, but it applies to all storage APIs equally. A) The temporary/persistent split almost enables this. We could say that when something that's a normal website stores data in temporary storage we count that data towards both per-origin and global quotas. If the global quota fills up, then we silently delete data from websites in an LRU fashion. If the user converts the website to an app by using bookmark to homescreen then we simply start treating the data stored in the temporary storage as persistent. I.e. we don't count it towards the global temporary-storage quota and we never delete it in order to make room for other websites. For persistent databases we would for normal websites put up a prompt (I'll leave out details like if this happens only when the quota API is used, or if can happen when the database is being written to). If persistent storage is used by a bookmarked app we simply would not prompt. In neither case would data stored in persistent storage ever be silently deleted in order to make room for other storage. The problem with this solution is that it doesn't give bookmarked apps the ability to create truly temporary data. Even data that a bookmarked app puts in the temporary storage is effectively treated as persistent and so not deleted if we start to run low on disk space. Temporary storage for apps is a feature that Android has, and that to some extent *nix OSs has had through use of /tmp. It definite is something that seems nice for constrained mobile devices. B) We could create a temporary/default/persistent split. I.e. we create three different storage categories. The default is what's used if no storage category is explicitly specified a IDB database is created. For normal webpages default is treated like temporary. I.e. it is counted towards
Re: IndexedDB: Syntax for specifying persistent/temporary storage
On Dec 11, 2013, at 10:53 PM, Kinuko Yasuda kin...@chromium.org wrote: persistent storage would behave as in A. I.e. webpages can't use it without there being a prompt involved at some point. Bookmarked webapps can use it without prompt. This sounds kinda reasonable to me, but I'd imagine some webapps (who are serious about their storage) would probably want to know if the default is either temporary or persistent in this case, and we may need a tiny API for that. In my experience, perm/persistent and temporary are misnomers. Temporary is closer to a cache. And persistent is all-or-nothing. That is, if you write two files to it, you're guaranteed that you'll always be able to fetch both files or neither. This is different than a pass-through to the host file system, something that Chrome supports retained pointers to in their packaged app API. I've lost persistent storage about four times with Chrome, following an update. Just the way of the world :-). In all of those cases, if the storage were pass-thru, I would not have lost that data. Though the consistency would be managed via OS, not UA. -Charles
Re: IndexedDB: Syntax for specifying persistent/temporary storage
Hi All, Thanks Jan for sending this. Now let me throw a giant wrench into this discussion :-) Unfortunately as we've been discussing webapps, manifests etc at mozilla I've slowly come to the realization that the temporary/persistent categorization isn't really fulfilling all the envisioned use cases. The background is that multiple platforms are now building the functionality to run normal websites outside of the browser. iOS was one of the first popular implementations of this. If you put a meta name=apple-mobile-web-app-capable content=yes in the markup of a page, and the user use bookmark to homescreen feature in iOS Safari, that almost turns the website into an app [1]. Google is currently working on implementing the same feature in Chrome for Android. At mozilla we created a proposal [2] for what is essentially a standardized version of the same idea. I think this approach is a really awesome use of the web and something that I'm very interested in supporting when designing these storage APIs. To support this use case, I think it needs to be possible for a website to first start as a website which the user only has a casual connection with, then gradually grow into something that the user essentially treats as a trusted app. Such a trusted app should have much more ability to store data without having to ask the user for permission, or without that data being suddenly deleted because we're low on disk space. In short, such an app should be treated more like a native app when it comes to storage. There are a few ways we can enable this use case. In the discussion below I'll use IndexedDB as an example of storage API, but it applies to all storage APIs equally. A) The temporary/persistent split almost enables this. We could say that when something that's a normal website stores data in temporary storage we count that data towards both per-origin and global quotas. If the global quota fills up, then we silently delete data from websites in an LRU fashion. If the user converts the website to an app by using bookmark to homescreen then we simply start treating the data stored in the temporary storage as persistent. I.e. we don't count it towards the global temporary-storage quota and we never delete it in order to make room for other websites. For persistent databases we would for normal websites put up a prompt (I'll leave out details like if this happens only when the quota API is used, or if can happen when the database is being written to). If persistent storage is used by a bookmarked app we simply would not prompt. In neither case would data stored in persistent storage ever be silently deleted in order to make room for other storage. The problem with this solution is that it doesn't give bookmarked apps the ability to create truly temporary data. Even data that a bookmarked app puts in the temporary storage is effectively treated as persistent and so not deleted if we start to run low on disk space. Temporary storage for apps is a feature that Android has, and that to some extent *nix OSs has had through use of /tmp. It definite is something that seems nice for constrained mobile devices. B) We could create a temporary/default/persistent split. I.e. we create three different storage categories. The default is what's used if no storage category is explicitly specified a IDB database is created. For normal webpages default is treated like temporary. I.e. it is counted towards the same quotas as temporary storage and it's deleted automatically if we run low on space. However once a page transitions to being an installed app data in default is treated as persistent. temporary storage would always be treated as truly temporary. I.e. the implementation is always free to delete data there if the user is running low on storage. If data stored by bookmarked apps count towards the same quotas as temporary data stored by web pages is an implementation decision. persistent storage would behave as in A. I.e. webpages can't use it without there being a prompt involved at some point. Bookmarked webapps can use it without prompt. C) Rather than having different storage categories, we could simply introduce something like priorities. I.e. the page could set a numeric priority when creating a database. It could additionally use the quota API to request a number of MB of storage that wouldn't be deleted when we run low on storage. For a webpage, when we start running low on storage we delete storage containers, starting with the lowest priority one, until that origin's total storage is below the amount of storage that it had requested through the quota API. For a bookmarked app, we simply automatically grant it unlimited storage. This approach has the advantage that it gets rid of the split between different storage categories, which certainly simplifies things. However it also seems quite fragile. First of all it means that rather than getting rid of none or all temporary databases, the page
Re: IndexedDB: Syntax for specifying persistent/temporary storage
Hi, I'm glad to hear that IDB's going to support storage types! On Thu, Dec 12, 2013 at 12:39 PM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Thanks Jan for sending this. Now let me throw a giant wrench into this discussion :-) Unfortunately as we've been discussing webapps, manifests etc at mozilla I've slowly come to the realization that the temporary/persistent categorization isn't really fulfilling all the envisioned use cases. The background is that multiple platforms are now building the functionality to run normal websites outside of the browser. iOS was one of the first popular implementations of this. If you put a meta name=apple-mobile-web-app-capable content=yes in the markup of a page, and the user use bookmark to homescreen feature in iOS Safari, that almost turns the website into an app [1]. Google is currently working on implementing the same feature in Chrome for Android. At mozilla we created a proposal [2] for what is essentially a standardized version of the same idea. I think this approach is a really awesome use of the web and something that I'm very interested in supporting when designing these storage APIs. To support this use case, I think it needs to be possible for a website to first start as a website which the user only has a casual connection with, then gradually grow into something that the user essentially treats as a trusted app. Such a trusted app should have much more ability to store data without having to ask the user for permission, or without that data being suddenly deleted because we're low on disk space. In short, such an app should be treated more like a native app when it comes to storage. There are a few ways we can enable this use case. In the discussion below I'll use IndexedDB as an example of storage API, but it applies to all storage APIs equally. A) The temporary/persistent split almost enables this. We could say that when something that's a normal website stores data in temporary storage we count that data towards both per-origin and global quotas. If the global quota fills up, then we silently delete data from websites in an LRU fashion. If the user converts the website to an app by using bookmark to homescreen then we simply start treating the data stored in the temporary storage as persistent. I.e. we don't count it towards the global temporary-storage quota and we never delete it in order to make room for other websites. For persistent databases we would for normal websites put up a prompt (I'll leave out details like if this happens only when the quota API is used, or if can happen when the database is being written to). If persistent storage is used by a bookmarked app we simply would not prompt. In neither case would data stored in persistent storage ever be silently deleted in order to make room for other storage. Fyi, Chrome applies similar policy to (A) for installed apps, but only when a special permission is requested in its manifest file. For regular apps their 'temporary' storage is still under eviction control. The problem with this solution is that it doesn't give bookmarked apps the ability to create truly temporary data. Even data that a bookmarked app puts in the temporary storage is effectively treated as persistent and so not deleted if we start to run low on disk space. Temporary storage for apps is a feature that Android has, and that to some extent *nix OSs has had through use of /tmp. It definite is something that seems nice for constrained mobile devices. As a slightly conservative variation, would it work if we just 'unlock' persistent storage for webapps when they're 'bookmarked' or got an installed app-like permission? The app could check if it could use persistent storage without requesting (by issuing quota query API, or maybe via some event), and convert some data into persistent if they like, but UA doesn't do so automatically. The UA continues to evict 'temporary' data even for 'bookmarked' webapps (UA could prioritize temporary data for bookmarked ones if it wants). B) We could create a temporary/default/persistent split. I.e. we create three different storage categories. The default is what's used if no storage category is explicitly specified a IDB database is created. For normal webpages default is treated like temporary. I.e. it is counted towards the same quotas as temporary storage and it's deleted automatically if we run low on space. However once a page transitions to being an installed app data in default is treated as persistent. temporary storage would always be treated as truly temporary. I.e. the implementation is always free to delete data there if the user is running low on storage. If data stored by bookmarked apps count towards the same quotas as temporary data stored by web pages is an implementation decision. persistent storage would behave as in A. I.e. webpages can't use it without there being a prompt
IndexedDB: Syntax for specifying persistent/temporary storage
IndexedDB implementation in Firefox 26 (the current beta) supports a
new storage type called temporary storage.
In short, it's a storage with LRU eviction policy, so the least
recently used data is automatically deleted when
a limit is reached. Chrome supports something similar [1].
Obviously, the IndexedDB spec needs to be updated to allow specifying
different storage types.
We might need to add other parameters in future so we propose creating
a dictionary with a version and
storage property for now.
Here is the current interface:
interface IDBFactory {IDBOpenDBRequest open (DOMString name,
[EnforceRange] optional unsigned long long version);
IDBOpenDBRequest
https://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#idl-def-IDBOpenDBRequest
deleteDatabase (DOMString name);shortcmp (any first,
any second);
};
and the interface with the dictionary:
interface IDBFactory {
IDBOpenDBRequest open (DOMString name, [EnforceRange] unsigned
long long version);
IDBOpenDBRequest open (DOMString name, optional IDBOpenDBOptions options);
IDBOpenDBRequest deleteDatabase (DOMString name, optional
IDBOpenDBOptions options);
short cmp (any first, any second);
};
enum StorageType { persistent, temporary };
dictionary IDBOpenDBOptions
{
[EnforceRange] unsigned long long version;
StorageType storage;
};
Some simple examples:
Example 1a - Opening a database in default storage:
var request = indexedDB.open('AddressBook', 15);
request.onsuccess = function(evt) {...};
request.onerror = function(evt) {...};
Example 1b - Deleting a database in default storage:
var request = indexedDB.deleteDatabase(AddressBook);request.onsuccess
= function(evt) {...};
request.onerror = function(evt) {...};
Example 2a - Opening a database in temporary storage:
var request = indexedDB.open(AddressBook, { version: 15, storage:
temporary });request.onsuccess = function(evt) {...};
request.onerror = function(evt) {...};
Example 2b - Deleting a database in temporary storage:
var request = indexedDB.deleteDatabase(AddressBook, { storage:
temporary });request.onsuccess = function(evt) {...};
request.onerror = function(evt) {...};
What do you think ?
[1] https://developers.google.com/chrome/whitepapers/storage
Jan
Re: IndexedDB: Syntax for specifying persistent/temporary storage
Thanks for sending this!
On Fri, Dec 6, 2013 at 7:19 AM, Jan Varga jan.va...@gmail.com wrote:
IndexedDB implementation in Firefox 26 (the current beta) supports a new
storage type called temporary storage.
In short, it's a storage with LRU eviction policy, so the least recently used
data is automatically deleted when
a limit is reached. Chrome supports something similar [1].
Obviously, the IndexedDB spec needs to be updated to allow specifying
different storage types.
Since the spec isn't one of those new-fangled Living Standards, this would
be in a different spec. Indexed DB Level 2 or something. There hasn't
been a discussion about that yet.
FYI we've been tracking specific items/proposals as RESOLVED/LATER bugs
[1]; there's a Wiki page that desperately needs updating [2] and I had a
doc capturing some discussion notes as well [3]. We should file a tracking
bug for this issue.
We might need to add other parameters in future so we propose creating
a dictionary with a version and
storage property for now.
Sounds reasonable, since name is always required, keeping it out of the
dict makes sense to me vs. what I captured in [3]. I like how this looks,
although...
Here is the current interface:
interface IDBFactory {IDBOpenDBRequest open (DOMString name,
[EnforceRange] optional unsigned long long version);IDBOpenDBRequest
https://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#idl-def-IDBOpenDBRequest
deleteDatabase (DOMString name);shortcmp (any first, any
second);
};
and the interface with the dictionary:
interface IDBFactory {
IDBOpenDBRequest open (DOMString name, [EnforceRange] unsigned long long
version);
IDBOpenDBRequest open (DOMString name, optional IDBOpenDBOptions options);
IDBOpenDBRequest deleteDatabase (DOMString name, optional
IDBOpenDBOptions options);
short cmp (any first, any second);
};
Issue: How can script detect that this updated API is available?
It seems like this could be done:
var r;
try {
r = indexedDB.open(db, {version: 1, storage: temporary});
// Throws TypeError on older implementations since Dictionary won't
coerce to Number (?)
} catch (e) {
// Fall back to default storage type
r = indexedDB.open(db, 1);
}
... which could be shimmed.
I don't think overloading has many proponents at the moment, though. The
other options are a different method name, or passing |undefined| as the
version, neither of which are great. Allowing null/undefined/0/falsy to
mean current version wouldn't be too terrible, though, and isn't a compat
concern since it explicitly throws today.
enum StorageType { persistent, temporary };
dictionary IDBOpenDBOptions
{
[EnforceRange] unsigned long long version;
StorageType storage;
};
[1]
https://www.w3.org/Bugs/Public/buglist.cgi?cmdtype=runnamedlist_id=25048namedcmd=IndexedDB%20Later
[2] http://www.w3.org/2008/webapps/wiki/IndexedDatabaseFeatures
[3]
https://docs.google.com/a/chromium.org/document/d/1vvC5tFZCZ9T8Cwd2DteUvw5WlU4YJa2NajdkHn6fu-I/edit
Re: IndexedDB: Syntax for specifying persistent/temporary storage
On 12/6/13 1:29 PM, Joshua Bell wrote: // Throws TypeError on older implementations since Dictionary won't coerce to Number (?) Sure it will. It'll do ToNumber() and probably end up NaN (which becomes 0 as an unsigned long long) unless your object has a valueOf method that returns something interesting. I don't think overloading has many proponents at the moment, though. Sure; in practice this would be done as a union type, not overloading. That still has the same how do I tell? issue, of course. other options are a different method name, or passing |undefined| as the version, neither of which are great. Allowing null/undefined/0/falsy to mean current version wouldn't be too terrible, though, and isn't a compat concern since it explicitly throws today. It sure doesn't. null and 0 are perfectly fine values for an unsigned long long (null becomes 0 after ToNumber()). undefined is treated as not passed. -Boris
Re: IndexedDB: Syntax for specifying persistent/temporary storage
On Fri, Dec 6, 2013 at 11:09 AM, Boris Zbarsky bzbar...@mit.edu wrote: On 12/6/13 1:29 PM, Joshua Bell wrote: // Throws TypeError on older implementations since Dictionary won't coerce to Number (?) Sure it will. It'll do ToNumber() and probably end up NaN (which becomes 0 as an unsigned long long) unless your object has a valueOf method that returns something interesting. Whoops - my brain was conflating distinguishable with coercable, and then I fell into the special case below when testing... I don't think overloading has many proponents at the moment, though. Sure; in practice this would be done as a union type, not overloading. That still has the same how do I tell? issue, of course. other options are a different method name, or passing |undefined| as the version, neither of which are great. Allowing null/undefined/0/falsy to mean current version wouldn't be too terrible, though, and isn't a compat concern since it explicitly throws today. It sure doesn't. null and 0 are perfectly fine values for an unsigned long long (null becomes 0 after ToNumber()). This behavior is specified in prose, not IDL: https://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#widl-IDBFactory-open-IDBOpenDBRequest-DOMString-name-unsigned-long-long-version If the value of version is 0 (zero), the implementation must throw a TypeError. undefined is treated as not passed. ... and we could default to 0 here, if we went down this path. Upshot is that falsy throws in implementations today due to unsigned long long coercion to 0, and the explicit case in the IDB spec. Conveniently, this would throw for objects as dictionaries per Boris' comments above (unless someone is trying to confuse themselves with a valueOf method on a dict...) -Boris
Re: IndexedDB: Syntax for specifying persistent/temporary storage
On 12/6/13 2:37 PM, Joshua Bell wrote: This behavior is specified in prose, not IDL: https://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#widl-IDBFactory-open-IDBOpenDBRequest-DOMString-name-unsigned-long-long-version If the value of version is 0 (zero), the implementation must throw a TypeError. Oh, huh. I was mislead here by Gecko not having indexeddb on webidl yet, so in fact seems to allow null. :( OK, great. undefined is treated as not passed. ... and we could default to 0 here, if we went down this path. Ah, make sense. -Boris
