RE: [IndexedDB] Lots of small nits and clarifying questions
Sorry for having disappeared for a while, odata was keeping me busy. I agree with all the clarifications listed in this thread that are required, so I won't redundantly mark each with same here, but I have a few comments on one or two of them below. On Mon, Mar 15, 2010 at 8:14 AM, Jeremy Orlow wrote: On Sat, Mar 13, 2010 at 9:02 AM, Nikunj Mehta nik...@o-micron.com wrote: Thanks for your patience. Most questions below don't seem to need new spec text. On Feb 18, 2010, at 9:08 AM, Jeremy Orlow wrote: 6) The specific ordering of elements should probably be specced including a mix of types. Can you propose spec text for this? What do you think about the text in http://www.w3.org/TR/IndexedDB/#key-construct? If we're only adding long long for v1, then I think language similar to what's there now is probably OK. But now that I think about it, I'm a bit concerned that we might be backing ourselves into a corner for the future. I also noticed that the sort order of JavaScript seems to order it numbers, strings, and then nulls (not strings, numbers, nulls). I wonder if there is some other spec on sort order we can cite rather than rolling our own. I really think that just doing long/strings won't do, even for v1. For non-primary-key indexes we'll need at least Date and number (not just integers) in addition to long/string. Without that there is no ordering by date sent for emails or list price for products or lots of other scenarios where you're caching data coming from a server. 2) What happens when data mutates while you're iterating via a cursor? This is covered by http://www.w3.org/TR/IndexedDB/#dfn-mode That applies to two separate transactions. As far as I can tell, it should be possible to have a cursor open and then delete an element that the cursor is currently traversing all within the same transaction. Am I missing something? I was assuming that within the same transaction you could change rows and those changes would be observable from open cursors. If it happens to be the current row then you won't be able to fetch it anymore but you can still move to the next one and continue scanning (and seeing any new changes that happened since you last moved). 1) Structured clone is going to change over time. And, realistically, UAs won't support every type right away anyway. What do we do when a value is inserted that we do not support? We will evolve the text as and when the same evolves in WebStorage. I don't know of any implementations which have moved away from only allowing strings within WebStorage. I suspect that not fully supporting the structured clone algorithm as specced is one of the reasons for this. As far as I can tell, you're essentially saying that fully supporting the the structured clone algorithm a pre-req for IndexedDB? I guess I can't argue too much with that, but I'm not sure how realistic it is. I know we only half support it at the moment in Chromium. I have the same worry about structured clones...it's right in principle but I can't see implementations converging and that will just hurt interoperability. Unfortunately there doesn't seem to be a well-known middle-ground. JSON is way too restrictive (e.g. no Date). Should we consider defining a subset of structured clones that work (maybe something like Javascript primitives plus Date plus whatever extra we feel we should include such as perhaps File objects)? Thanks -pablo
Re: [IndexedDB] Lots of small nits and clarifying questions
On Tue, Mar 30, 2010 at 9:10 AM, Pablo Castro pablo.cas...@microsoft.comwrote: Sorry for having disappeared for a while, odata was keeping me busy. I agree with all the clarifications listed in this thread that are required, so I won't redundantly mark each with same here, but I have a few comments on one or two of them below. On Mon, Mar 15, 2010 at 8:14 AM, Jeremy Orlow wrote: On Sat, Mar 13, 2010 at 9:02 AM, Nikunj Mehta nik...@o-micron.com wrote: Thanks for your patience. Most questions below don't seem to need new spec text. On Feb 18, 2010, at 9:08 AM, Jeremy Orlow wrote: 6) The specific ordering of elements should probably be specced including a mix of types. Can you propose spec text for this? What do you think about the text in http://www.w3.org/TR/IndexedDB/#key-construct? If we're only adding long long for v1, then I think language similar to what's there now is probably OK. But now that I think about it, I'm a bit concerned that we might be backing ourselves into a corner for the future. I also noticed that the sort order of JavaScript seems to order it numbers, strings, and then nulls (not strings, numbers, nulls). I wonder if there is some other spec on sort order we can cite rather than rolling our own. I really think that just doing long/strings won't do, even for v1. For non-primary-key indexes we'll need at least Date and number (not just integers) in addition to long/string. Without that there is no ordering by date sent for emails or list price for products or lots of other scenarios where you're caching data coming from a server. 2) What happens when data mutates while you're iterating via a cursor? This is covered by http://www.w3.org/TR/IndexedDB/#dfn-mode That applies to two separate transactions. As far as I can tell, it should be possible to have a cursor open and then delete an element that the cursor is currently traversing all within the same transaction. Am I missing something? I was assuming that within the same transaction you could change rows and those changes would be observable from open cursors. If it happens to be the current row then you won't be able to fetch it anymore but you can still move to the next one and continue scanning (and seeing any new changes that happened since you last moved). 1) Structured clone is going to change over time. And, realistically, UAs won't support every type right away anyway. What do we do when a value is inserted that we do not support? We will evolve the text as and when the same evolves in WebStorage. I don't know of any implementations which have moved away from only allowing strings within WebStorage. I suspect that not fully supporting the structured clone algorithm as specced is one of the reasons for this. As far as I can tell, you're essentially saying that fully supporting the the structured clone algorithm a pre-req for IndexedDB? I guess I can't argue too much with that, but I'm not sure how realistic it is. I know we only half support it at the moment in Chromium. I have the same worry about structured clones...it's right in principle but I can't see implementations converging and that will just hurt interoperability. Unfortunately there doesn't seem to be a well-known middle-ground. JSON is way too restrictive (e.g. no Date). Should we consider defining a subset of structured clones that work (maybe something like Javascript primitives plus Date plus whatever extra we feel we should include such as perhaps File objects)? There is some precedent for what you suggest: the spec for LocalStorage already specifies that storing ImageData isn't allowed. ( http://dev.w3.org/html5/webstorage/#the-storage-interface see setItem section.) On the other hand, I'm not sure I like the idea of each API supporting different subsets of the structured clone algorithm. Even if all UAs support the same subset for each API, it still seems fairly confusing to web developers. And I'm guessing that UAs won't be to keen on adding more complex control flow to their structured clone implementations to disallow different parts of the algorithm based on what it's using. Thus any specced subset of the algorithm will probably need to be a MAY not a MUST. I still think we should spec an error to be returned when the UA doesn't fully support the structured clone algorithm and thus can't handle the data provided. I agree it's sub-optimal, but I think it's the pragmatic choice. Especially if the structured clone algorithm ever changes (and thus implementations can fall out of compliance with the spec).
Re: [IndexedDB] Lots of small nits and clarifying questions
On Sat, Mar 13, 2010 at 9:02 AM, Nikunj Mehta nik...@o-micron.com wrote: Thanks for your patience. Most questions below don't seem to need new spec text. On Feb 18, 2010, at 9:08 AM, Jeremy Orlow wrote: I'm sorry that I let so much IndexedDB feedback get backlogged. In the future, I'll try to trickle things out slower. * * *Indexes:* 1) Creation of indexes really needs to be made more clear. Did you mean the algorithm for this process or the API documentation? For example, does creation of the index block everything until it's complete or does the database get created in the background? If the index is created using the synchronous call, then the calling worker will block until the index is created. If the async API is used, the rest of the database can continue to work and the worker should be able to continue to other calls without waiting for the index creation to get over. What if I have 1gb of my mail stored in IndexedDB and then a database migration adds an index? Depends on your choice of workers and sync/async API. Is my app completely unusable during that time? It doesn't have to be in a well-designed app. What if the browser is exited half way through building (you can't just delete it)? Regardless of whether the sync or the async API, if you choose to perform DDL (not just index creation, but also database creation) in a transaction, then it could be done atomically. What happens if you query the database while it's building in the background-building case (should it emulate it via entity-store-scans)? The index doesn't exist until it is created. If you use an index before it exists (even if it is in the process of being created), then the outcome depends on whether the index is created in a transaction. If yes, the call will fail because the index will not exist. If no, then the call may use partial index results. These are all very important questions whose answers should be standardized. This feels like good non-normative text to me. All of this seems fine for v1. 2) Why are Indexes in some database-global namespace rather than some entity-store-global namespace? This approach is fine by me. I know in SQL, most people use the table name as a prefix for their index names to make sure they're unique. Why inherit such silliness into IndexedDB? Why not connect every index to a particular entity-store? Indexes in IndexDB are already dependent on an entity store. Ok, then I'd suggest we move the create/open/delete index methods to the IDBObjectStoreSync/IDBObjectStoreRequest interfaces to reflect this. 6) The specific ordering of elements should probably be specced including a mix of types. Can you propose spec text for this? What do you think about the text in http://www.w3.org/TR/IndexedDB/#key-construct? If we're only adding long long for v1, then I think language similar to what's there now is probably OK. But now that I think about it, I'm a bit concerned that we might be backing ourselves into a corner for the future. I also noticed that the sort order of JavaScript seems to order it numbers, strings, and then nulls (not strings, numbers, nulls). I wonder if there is some other spec on sort order we can cite rather than rolling our own. *Key ranges / cursors:* 1) Is an open or closed key range the default? Not sure what you mean by default here? The parameters are optional. If you don't specify them, what should they be? 2) What happens when data mutates while you're iterating via a cursor? This is covered by http://www.w3.org/TR/IndexedDB/#dfn-mode That applies to two separate transactions. As far as I can tell, it should be possible to have a cursor open and then delete an element that the cursor is currently traversing all within the same transaction. Am I missing something? 3) In the spec, get and getObject seem to assume that only one element can be returned...but that's only true if unique is true. What do you do if there are multiple? ObjectStore doesn't allow storage of multiple values with a given key. See http://www.w3.org/TR/IndexedDB/#object-store-concept Multiples only make sense if you are retrieving from an Index. The following spec text could be added to 3.2.5 and 3.2.6 to return the first value in the sorted order of their keys. This text should work. 3.2.5 3. Let /value/ be the value of the record first in the sorted order of their keys corresponding to the key /key/ in /index/ 3.2.6 3. Return the value for the record first in the sorted order of their keys corresponding to the key /key/ in /index/ 4) Why can the cursor only travel in one direction? Many things are possible. This is a scope limitation. You can open multiple cursors each going in different directions. I was just trying to understand why. I agree there's no super compelling use case. 5) What if you modify a value that then implicitly (via the
Re: [IndexedDB] Lots of small nits and clarifying questions
On Mon, Mar 1, 2010 at 6:03 AM, Nikunj Mehta nik...@o-micron.com wrote: On Feb 28, 2010, at 3:24 PM, Jeremy Orlow wrote: Another nit: as far as I can tell, all of the common parts of the interfaces are named Foo, the synchronous API portion is FooSync, and the async API portion is FooRequest. This is true except for IndexedDatabase where the sync version is simply IndexedDatabase and the async version is IndexedDatabaseRequest. Can we please change IndexedDatabase to IndexedDatabaseSync for consistency, even though there is no common shared base class? I have no problems with renaming. However, before we go too much in to renaming, it is important to finalize the async API style. In general, I agree, but this was the one place where the naming seemed inconsistent, so I thought it was worth bringing up even with other large outstanding issues. You're right though that we do need to finalize the async API style. I've responded to [IndexedDB] Promises and will try to drive that to consensus soon. In the mean time, it'd be great if you (and others) could find the time to offer your opinions on the other issues I've brought up in both this and other threads (all with [IndexedDB] in the subject). J P.S. Would it be useful to accompany requests like this with a patch against Overview.html? That certainly helps. Attached. As far as I can tell, it's just a 2 liner, but maybe I missed something? On Thu, Feb 18, 2010 at 5:08 PM, Jeremy Orlow jor...@google.com wrote: I'm sorry that I let so much IndexedDB feedback get backlogged. In the future, I'll try to trickle things out slower. * * *Indexes:* 1) Creation of indexes really needs to be made more clear. For example, does creation of the index block everything until it's complete or does the database get created in the background? What if I have 1gb of my mail stored in IndexedDB and then a database migration adds an index? Is my app completely unusable during that time? What if the browser is exited half way through building (you can't just delete it)? What happens if you query the database while it's building in the background-building case (should it emulate it via entity-store-scans)? These are all very important questions whose answers should be standardized. 2) Why are Indexes in some database-global namespace rather than some entity-store-global namespace? I know in SQL, most people use the table name as a prefix for their index names to make sure they're unique. Why inherit such silliness into IndexedDB? Why not connect every index to a particular entity-store? 3) What happens when unique constraints are violated? 4) I don't remember anything explicitly stating that when a value changes that an index has a keypath referring to, that index should be updated. 5) It definitely would be nice to be able to index more than just longs and strings. 6) The specific ordering of elements should probably be specced including a mix of types. *Key ranges / cursors:* 1) Is an open or closed key range the default? 2) What happens when data mutates while you're iterating via a cursor? 3) In the spec, get and getObject seem to assume that only one element can be returned...but that's only true if unique is true. What do you do if there are multiple? 4) Why can the cursor only travel in one direction? 5) What if you modify a value that then implicitly (via the key-path) changes the index that your cursor is currently iterating over? *Transactions:* 1) We feel strongly that nested transactions should be allowed. Closed nested transactions should be simple to implement and will make it much easier for multiple layers of abstraction to use IndexedDB without knowledge of each other. 2) In the spec, dynamic transactions and the difference between static and dynamic are not very well explained. 3) I'm not sure that I like how the spec talks about commits being durable but then later says Applications must not assume that committing the transaction produces an instantaneously durable result. The user agent may delay flushing data to durable storage until an appropriate time. It seems like the language should be made more consistient. Also, it seems like there should be some way to ensure it is durable on disk for when it's absolutely necessary. (But maybe with a note that UAs are free to rate limit this.) *Misc:* 1) Structured clone is going to change over time. And, realistically, UAs won't support every type right away anyway. What do we do when a value is inserted that we do not support? 2) It seems that you can only be connected to one database at a time? If so, why? 3) Do we have enough distinct error codes? For example, there are multiple ways to get a NON_TRANSIENT_ERR when creating a transaction. Error strings can help with debugging, but they can differ between UAs. It seems as though all errors should be diagnosable via the error
Re: [IndexedDB] Lots of small nits and clarifying questions
Another nit: as far as I can tell, all of the common parts of the interfaces are named Foo, the synchronous API portion is FooSync, and the async API portion is FooRequest. This is true except for IndexedDatabase where the sync version is simply IndexedDatabase and the async version is IndexedDatabaseRequest. Can we please change IndexedDatabase to IndexedDatabaseSync for consistency, even though there is no common shared base class? J P.S. Would it be useful to accompany requests like this with a patch against Overview.html? On Thu, Feb 18, 2010 at 5:08 PM, Jeremy Orlow jor...@google.com wrote: I'm sorry that I let so much IndexedDB feedback get backlogged. In the future, I'll try to trickle things out slower. * * *Indexes:* 1) Creation of indexes really needs to be made more clear. For example, does creation of the index block everything until it's complete or does the database get created in the background? What if I have 1gb of my mail stored in IndexedDB and then a database migration adds an index? Is my app completely unusable during that time? What if the browser is exited half way through building (you can't just delete it)? What happens if you query the database while it's building in the background-building case (should it emulate it via entity-store-scans)? These are all very important questions whose answers should be standardized. 2) Why are Indexes in some database-global namespace rather than some entity-store-global namespace? I know in SQL, most people use the table name as a prefix for their index names to make sure they're unique. Why inherit such silliness into IndexedDB? Why not connect every index to a particular entity-store? 3) What happens when unique constraints are violated? 4) I don't remember anything explicitly stating that when a value changes that an index has a keypath referring to, that index should be updated. 5) It definitely would be nice to be able to index more than just longs and strings. 6) The specific ordering of elements should probably be specced including a mix of types. *Key ranges / cursors:* 1) Is an open or closed key range the default? 2) What happens when data mutates while you're iterating via a cursor? 3) In the spec, get and getObject seem to assume that only one element can be returned...but that's only true if unique is true. What do you do if there are multiple? 4) Why can the cursor only travel in one direction? 5) What if you modify a value that then implicitly (via the key-path) changes the index that your cursor is currently iterating over? *Transactions:* 1) We feel strongly that nested transactions should be allowed. Closed nested transactions should be simple to implement and will make it much easier for multiple layers of abstraction to use IndexedDB without knowledge of each other. 2) In the spec, dynamic transactions and the difference between static and dynamic are not very well explained. 3) I'm not sure that I like how the spec talks about commits being durable but then later says Applications must not assume that committing the transaction produces an instantaneously durable result. The user agent may delay flushing data to durable storage until an appropriate time. It seems like the language should be made more consistient. Also, it seems like there should be some way to ensure it is durable on disk for when it's absolutely necessary. (But maybe with a note that UAs are free to rate limit this.) *Misc:* 1) Structured clone is going to change over time. And, realistically, UAs won't support every type right away anyway. What do we do when a value is inserted that we do not support? 2) It seems that you can only be connected to one database at a time? If so, why? 3) Do we have enough distinct error codes? For example, there are multiple ways to get a NON_TRANSIENT_ERR when creating a transaction. Error strings can help with debugging, but they can differ between UAs. It seems as though all errors should be diagnosable via the error codes. 4) In 3.3.2, openCursor takes in an optional IDBKeyRange and then an optional direction. But what if you don't want a range but you do want a particular direction? Are implementations expected to handle this by looking at whether the first parameter is a IDBKeyRange or not? Same goes for IDBIndexSync. 5) Similarly, put takes 2 optionals. Depending on the object store it may or may not make sense for there to be a key param. I guess the javascript bindings will need to have knowledge of whether a key is expected and/or disallow boolean keys? It'd probably be better to avoid this from a bindings point of view. 3.2.2.4 - why would you skip the next step? 3.2.2.6 - should be preform one or the other, right? 3.2.2.6.1 - should be if it has a key generator right? 3.3.2 - if createObjectStore converts a null name to the empty string, why woudln't openObjectStore, create/open
Re: [IndexedDB] Lots of small nits and clarifying questions
On Feb 28, 2010, at 3:24 PM, Jeremy Orlow wrote: Another nit: as far as I can tell, all of the common parts of the interfaces are named Foo, the synchronous API portion is FooSync, and the async API portion is FooRequest. This is true except for IndexedDatabase where the sync version is simply IndexedDatabase and the async version is IndexedDatabaseRequest. Can we please change IndexedDatabase to IndexedDatabaseSync for consistency, even though there is no common shared base class? I have no problems with renaming. However, before we go too much in to renaming, it is important to finalize the async API style. J P.S. Would it be useful to accompany requests like this with a patch against Overview.html? That certainly helps. On Thu, Feb 18, 2010 at 5:08 PM, Jeremy Orlow jor...@google.com wrote: I'm sorry that I let so much IndexedDB feedback get backlogged. In the future, I'll try to trickle things out slower. Indexes: 1) Creation of indexes really needs to be made more clear. For example, does creation of the index block everything until it's complete or does the database get created in the background? What if I have 1gb of my mail stored in IndexedDB and then a database migration adds an index? Is my app completely unusable during that time? What if the browser is exited half way through building (you can't just delete it)? What happens if you query the database while it's building in the background-building case (should it emulate it via entity-store-scans)? These are all very important questions whose answers should be standardized. 2) Why are Indexes in some database-global namespace rather than some entity-store-global namespace? I know in SQL, most people use the table name as a prefix for their index names to make sure they're unique. Why inherit such silliness into IndexedDB? Why not connect every index to a particular entity-store? 3) What happens when unique constraints are violated? 4) I don't remember anything explicitly stating that when a value changes that an index has a keypath referring to, that index should be updated. 5) It definitely would be nice to be able to index more than just longs and strings. 6) The specific ordering of elements should probably be specced including a mix of types. Key ranges / cursors: 1) Is an open or closed key range the default? 2) What happens when data mutates while you're iterating via a cursor? 3) In the spec, get and getObject seem to assume that only one element can be returned...but that's only true if unique is true. What do you do if there are multiple? 4) Why can the cursor only travel in one direction? 5) What if you modify a value that then implicitly (via the key- path) changes the index that your cursor is currently iterating over? Transactions: 1) We feel strongly that nested transactions should be allowed. Closed nested transactions should be simple to implement and will make it much easier for multiple layers of abstraction to use IndexedDB without knowledge of each other. 2) In the spec, dynamic transactions and the difference between static and dynamic are not very well explained. 3) I'm not sure that I like how the spec talks about commits being durable but then later says Applications must not assume that committing the transaction produces an instantaneously durable result. The user agent may delay flushing data to durable storage until an appropriate time. It seems like the language should be made more consistient. Also, it seems like there should be some way to ensure it is durable on disk for when it's absolutely necessary. (But maybe with a note that UAs are free to rate limit this.) Misc: 1) Structured clone is going to change over time. And, realistically, UAs won't support every type right away anyway. What do we do when a value is inserted that we do not support? 2) It seems that you can only be connected to one database at a time? If so, why? 3) Do we have enough distinct error codes? For example, there are multiple ways to get a NON_TRANSIENT_ERR when creating a transaction. Error strings can help with debugging, but they can differ between UAs. It seems as though all errors should be diagnosable via the error codes. 4) In 3.3.2, openCursor takes in an optional IDBKeyRange and then an optional direction. But what if you don't want a range but you do want a particular direction? Are implementations expected to handle this by looking at whether the first parameter is a IDBKeyRange or not? Same goes for IDBIndexSync. 5) Similarly, put takes 2 optionals. Depending on the object store it may or may not make sense for there to be a key param. I guess the javascript bindings will need to have knowledge of whether a key is expected and/or disallow boolean keys? It'd probably be better to avoid this from a bindings point of view. 3.2.2.4 -
