Re: [IndexedDB] Current editor's draft
On Tue, Jul 27, 2010 at 12:22 AM, Jonas Sicking jo...@sicking.cc wrote: On Sat, Jul 24, 2010 at 8:29 AM, Jeremy Orlow jor...@chromium.org wrote: And is it only possible to lock existing rows, or can you prevent new records from being created? There's no way to lock yet to be created rows since until a transaction ends, its effects cannot be made visible to other transactions. So if you have an objectStore with auto-incrementing indexes, there is the possibility that two dynamic transactions both can add a row to said objectStore at the same time. Both transactions would then add a row with the same autogenerated id (one higher than the highest id in the table). Upon commit, how is this conflict resolved? What if the objectStore didn't use auto-incrementing indexes, but you still had two separate dynamic transactions which both insert a row with the same key. How is the conflict resolved? I believe a common trick to reconcile this is stipulating that if you add 1000 rows the id's may not necessarily be 1000 sequential numbers. This allows transactions to increment the id and leave it incremented even if the transaction fails. Which also means that other transactions can be grabbing an ID of their own as well. And if a transaction fails, well, we've wasted one possible ID. This does not answer the question what happens if two transactions add the same key value though? If you're using optimistic transactions, whichever commits first succeeds. Not sure with the pessimistic/lock-based implementations. And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? An implementation could use shared locks for read operations even though the object store might have been opened in READ_WRITE mode, and later upgrade the locks if the read data is being modified. However, I am not keen to push for this as a specced behavior. What do you mean by an implementation could? Is this left intentionally undefined and left up to the implementation? Doesn't that mean that there is significant risk that code could work very well in a conservative implementation, but often cause race conditions in a implementation that uses narrower locks? Wouldn't this result in a race to the bottom where implementations are forced to eventually use very wide locks in order to work well in websites? In general, there are a lot of details that are unclear in the dynamic transactions proposals. I'm also not sure if these things are unclear to me because they are intentionally left undefined, or if you guys just haven't had time yet to define the details? As the spec is now, as an implementor I'd have no idea of how to implement dynamic transactions. And as a user I'd have no idea what level of protection to expect from implementations, nor what strategies to use to avoid bugs. In all the development I've done deadlocks and race conditions are generally unacceptable, and instead strategies are developed that avoids them, such as always grab locks in the same order, and always grab locks when using shared data. I currently have no idea what strategy to recommend in IndexedDB documentation to developers to allow them to avoid race conditions and deadlocks. To get clarity in these questions, I'd *really* *really* like to see a more detailed proposal. I think a detailed proposal would be a good thing (maybe from Pablo or Nikunj since they're who are really pushing them at this point), but at the same time, I think you're getting really bogged down in the details, Jonas. What we should be concerned about and speccing is the behavior the user sees. For example, can any operation on data fail due to transient issues (like deadlocks, serialization issues) or will the implementation shield web developers from this? And will we guarantee 100% serializable semantics? (I strongly believe we should on both counts.) How things are implemented, granularity of locks, or even if an implementation uses locks at all for dynamic transactions should be explicitly out of scope for any spec. After all, it's only the behavior users care about. If we can guarantee no deadlocks and 100% serializable semantics, then I agree, it doesn't matter beyond that. However I don't think the current proposals for dynamic transactions guarantee that. In fact, a central point of the dynamic transactions proposal seems to be that the author can grow the lock space dynamically, in an author defined order. As long as that is the case you can't prevent deadlocks other than by forbidding multiple concurrent (dynamic) transactions. There has been a lot of talk on list about how you could implement dynamic transactions with locks and I agree that I don't see how what's been mentioned can be guaranteed to be serializable and/or not create deadlocks.
Re: [IndexedDB] Current editor's draft
On Tue, Jul 27, 2010 at 4:38 PM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 27, 2010 at 3:14 AM, Jeremy Orlow jor...@chromium.org wrote: On Tue, Jul 27, 2010 at 12:22 AM, Jonas Sicking jo...@sicking.cc wrote: On Sat, Jul 24, 2010 at 8:29 AM, Jeremy Orlow jor...@chromium.org wrote: And is it only possible to lock existing rows, or can you prevent new records from being created? There's no way to lock yet to be created rows since until a transaction ends, its effects cannot be made visible to other transactions. So if you have an objectStore with auto-incrementing indexes, there is the possibility that two dynamic transactions both can add a row to said objectStore at the same time. Both transactions would then add a row with the same autogenerated id (one higher than the highest id in the table). Upon commit, how is this conflict resolved? What if the objectStore didn't use auto-incrementing indexes, but you still had two separate dynamic transactions which both insert a row with the same key. How is the conflict resolved? I believe a common trick to reconcile this is stipulating that if you add 1000 rows the id's may not necessarily be 1000 sequential numbers. This allows transactions to increment the id and leave it incremented even if the transaction fails. Which also means that other transactions can be grabbing an ID of their own as well. And if a transaction fails, well, we've wasted one possible ID. This does not answer the question what happens if two transactions add the same key value though? If you're using optimistic transactions, whichever commits first succeeds. Not sure with the pessimistic/lock-based implementations. That doesn't count as serializable, does it? It's not a dead-lock hazard, but a race condition where whoever commits last experiences dataloss unless properly handling the commit error. If they don't handle commit errors, yes, you would have data loss. I didn't realize we were even considering trying to make dynamic transactions work without the possibility of the commit failing. And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? An implementation could use shared locks for read operations even though the object store might have been opened in READ_WRITE mode, and later upgrade the locks if the read data is being modified. However, I am not keen to push for this as a specced behavior. What do you mean by an implementation could? Is this left intentionally undefined and left up to the implementation? Doesn't that mean that there is significant risk that code could work very well in a conservative implementation, but often cause race conditions in a implementation that uses narrower locks? Wouldn't this result in a race to the bottom where implementations are forced to eventually use very wide locks in order to work well in websites? In general, there are a lot of details that are unclear in the dynamic transactions proposals. I'm also not sure if these things are unclear to me because they are intentionally left undefined, or if you guys just haven't had time yet to define the details? As the spec is now, as an implementor I'd have no idea of how to implement dynamic transactions. And as a user I'd have no idea what level of protection to expect from implementations, nor what strategies to use to avoid bugs. In all the development I've done deadlocks and race conditions are generally unacceptable, and instead strategies are developed that avoids them, such as always grab locks in the same order, and always grab locks when using shared data. I currently have no idea what strategy to recommend in IndexedDB documentation to developers to allow them to avoid race conditions and deadlocks. To get clarity in these questions, I'd *really* *really* like to see a more detailed proposal. I think a detailed proposal would be a good thing (maybe from Pablo or Nikunj since they're who are really pushing them at this point), but at the same time, I think you're getting really bogged down in the details, Jonas. What we should be concerned about and speccing is the behavior the user sees. For example, can any operation on data fail due to transient issues (like deadlocks, serialization issues) or will the implementation shield web developers from this? And will we guarantee 100% serializable semantics? (I strongly believe we should on both counts.) How things are implemented, granularity of locks, or even if an implementation uses locks at all for dynamic transactions should be explicitly out of scope for any spec. After all, it's only the behavior users care about. If we can guarantee no deadlocks and
Re: [IndexedDB] Current editor's draft
On Sat, Jul 24, 2010 at 8:29 AM, Jeremy Orlow jor...@chromium.org wrote: And is it only possible to lock existing rows, or can you prevent new records from being created? There's no way to lock yet to be created rows since until a transaction ends, its effects cannot be made visible to other transactions. So if you have an objectStore with auto-incrementing indexes, there is the possibility that two dynamic transactions both can add a row to said objectStore at the same time. Both transactions would then add a row with the same autogenerated id (one higher than the highest id in the table). Upon commit, how is this conflict resolved? What if the objectStore didn't use auto-incrementing indexes, but you still had two separate dynamic transactions which both insert a row with the same key. How is the conflict resolved? I believe a common trick to reconcile this is stipulating that if you add 1000 rows the id's may not necessarily be 1000 sequential numbers. This allows transactions to increment the id and leave it incremented even if the transaction fails. Which also means that other transactions can be grabbing an ID of their own as well. And if a transaction fails, well, we've wasted one possible ID. This does not answer the question what happens if two transactions add the same key value though? And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? An implementation could use shared locks for read operations even though the object store might have been opened in READ_WRITE mode, and later upgrade the locks if the read data is being modified. However, I am not keen to push for this as a specced behavior. What do you mean by an implementation could? Is this left intentionally undefined and left up to the implementation? Doesn't that mean that there is significant risk that code could work very well in a conservative implementation, but often cause race conditions in a implementation that uses narrower locks? Wouldn't this result in a race to the bottom where implementations are forced to eventually use very wide locks in order to work well in websites? In general, there are a lot of details that are unclear in the dynamic transactions proposals. I'm also not sure if these things are unclear to me because they are intentionally left undefined, or if you guys just haven't had time yet to define the details? As the spec is now, as an implementor I'd have no idea of how to implement dynamic transactions. And as a user I'd have no idea what level of protection to expect from implementations, nor what strategies to use to avoid bugs. In all the development I've done deadlocks and race conditions are generally unacceptable, and instead strategies are developed that avoids them, such as always grab locks in the same order, and always grab locks when using shared data. I currently have no idea what strategy to recommend in IndexedDB documentation to developers to allow them to avoid race conditions and deadlocks. To get clarity in these questions, I'd *really* *really* like to see a more detailed proposal. I think a detailed proposal would be a good thing (maybe from Pablo or Nikunj since they're who are really pushing them at this point), but at the same time, I think you're getting really bogged down in the details, Jonas. What we should be concerned about and speccing is the behavior the user sees. For example, can any operation on data fail due to transient issues (like deadlocks, serialization issues) or will the implementation shield web developers from this? And will we guarantee 100% serializable semantics? (I strongly believe we should on both counts.) How things are implemented, granularity of locks, or even if an implementation uses locks at all for dynamic transactions should be explicitly out of scope for any spec. After all, it's only the behavior users care about. If we can guarantee no deadlocks and 100% serializable semantics, then I agree, it doesn't matter beyond that. However I don't think the current proposals for dynamic transactions guarantee that. In fact, a central point of the dynamic transactions proposal seems to be that the author can grow the lock space dynamically, in an author defined order. As long as that is the case you can't prevent deadlocks other than by forbidding multiple concurrent (dynamic) transactions. / Jonas
Re: [IndexedDB] Current editor's draft
On Fri, Jul 23, 2010 at 4:21 PM, Jonas Sicking jo...@sicking.cc wrote: On Fri, Jul 23, 2010 at 8:09 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 22, 2010, at 11:27 AM, Jonas Sicking wrote: On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? That is not correct. I said that the original intention was to make dynamic transactions lock as little and as late as possible. However, the current spec does not state explicitly that dynamic transactions should not lock the entire objectStore, but it could. If so, how does the author specify which rows are locked? Again, the intention is to do this directly from the actions performed by the application and the affected keys The two above statements confuse me. The important question is: Pablo is clearly suggesting that dynamic transactions should not use whole-objectStore locks, but rather row-level locks, or possibly range locks. Is this what you are suggesting too? I'm not sure why this matters at all. The point is that we need to use some locking scheme that ensures serializability and ideally it'd be done in a way that allows a good deal of concurrency. I don't think the spec should force any particular locking scheme (or pessimistic transactions in general). And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? The open call is used to check if some static transaction has the entire store locked for READ_WRITE. If so, the open call will block. Nikunj, as has been explained
Re: [IndexedDB] Current editor's draft
On Jul 22, 2010, at 11:27 AM, Jonas Sicking wrote: On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? That is not correct. I said that the original intention was to make dynamic transactions lock as little and as late as possible. However, the current spec does not state explicitly that dynamic transactions should not lock the entire objectStore, but it could. If so, how does the author specify which rows are locked? Again, the intention is to do this directly from the actions performed by the application and the affected keys And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? The open call is used to check if some static transaction has the entire store locked for READ_WRITE. If so, the open call will block. And is it only possible to lock existing rows, or can you prevent new records from being created? There's no way to lock yet to be created rows since until a transaction ends, its effects cannot be made visible to other transactions. And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? An implementation could use shared locks for read operations even though the object store might have been opened in READ_WRITE mode, and later upgrade the locks if the read data is being modified. However, I am not keen to push for this as a specced behavior.
Re: [IndexedDB] Current editor's draft
On Fri, Jul 23, 2010 at 8:09 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 22, 2010, at 11:27 AM, Jonas Sicking wrote: On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? That is not correct. I said that the original intention was to make dynamic transactions lock as little and as late as possible. However, the current spec does not state explicitly that dynamic transactions should not lock the entire objectStore, but it could. If so, how does the author specify which rows are locked? Again, the intention is to do this directly from the actions performed by the application and the affected keys The two above statements confuse me. The important question is: Pablo is clearly suggesting that dynamic transactions should not use whole-objectStore locks, but rather row-level locks, or possibly range locks. Is this what you are suggesting too? And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? The open call is used to check if some static transaction has the entire store locked for READ_WRITE. If so, the open call will block. Given that synchronous vs. asynchronous is just an optimization, I'll defer this topic until I understand the rest of the proposal better. And is it only possible to lock existing rows, or can you prevent new records from being created? There's no way to lock yet to be created rows since until a transaction ends, its effects cannot be made visible to other transactions. So if you have an objectStore with auto-incrementing indexes, there is the possibility that two dynamic transactions
Re: [IndexedDB] Current editor's draft
On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. - Overlapping between statically and dynamically scoped transactions follows the same rules as static-static overlaps; they can only overlap on compatible scopes. The only difference is that dynamic transactions may need to block mid-flight until it can grab the resources it needs to proceed. This is the intention with the timeout interval and asynchronous nature of the openObjectStore on a dynamic transaction. Note: for some databases having multiple transactions active on a single connection may be an unsupported thing. This could probably be handled in the IndexedDB layer though by using multiple connections under the covers. -pablo
Re: [IndexedDB] Current editor's draft
On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? If so, how does the author specify which rows are locked? And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? And is it only possible to lock existing rows, or can you prevent new records from being created? And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? / Jonas
RE: [IndexedDB] Current editor's draft
From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 11:27 AM On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? If so, how does the author specify which rows are locked? And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? And is it only possible to lock existing rows, or can you prevent new records from being created? And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? That's my interpretation, dynamic transactions don't lock whole object stores. To me dynamic transactions are the same as what typical SQL databases do today. The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). Regarding deadlocks, that's right, the implementation cannot determine if a deadlock will occur ahead of time. Sophisticated implementations could track locks/owners and do deadlock detection, although a simple timeout-based mechanism is probably enough for IndexedDB. As for locking only existing rows, that depends on how much isolation we
Re: [IndexedDB] Current editor's draft
On Thu, Jul 22, 2010 at 4:41 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 11:27 AM On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? If so, how does the author specify which rows are locked? And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? And is it only possible to lock existing rows, or can you prevent new records from being created? And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? That's my interpretation, dynamic transactions don't lock whole object stores. To me dynamic transactions are the same as what typical SQL databases do today. The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). What counts as see? If you iterate using an index-cursor all the rows that have some value between A and B, but another, not yet committed, transaction changes a row such that its value now is between A and B, what happens? Regarding
Re: [IndexedDB] Current editor's draft
On Thu, Jul 22, 2010 at 7:41 PM, Pablo Castro pablo.cas...@microsoft.comwrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 11:27 AM On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? If so, how does the author specify which rows are locked? And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? And is it only possible to lock existing rows, or can you prevent new records from being created? And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? That's my interpretation, dynamic transactions don't lock whole object stores. To me dynamic transactions are the same as what typical SQL databases do today. The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). Regarding deadlocks, that's right, the implementation cannot determine if a deadlock will occur ahead of time. Sophisticated implementations could track locks/owners and do deadlock detection, although a
Re: [IndexedDB] Current editor's draft
On Thu, Jul 22, 2010 at 5:18 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 22, 2010 at 7:41 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 11:27 AM On Thu, Jul 22, 2010 at 3:43 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 16, 2010, at 5:41 AM, Pablo Castro wrote: From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) Dynamic transactions should be able to lock as little as necessary and as late as required. So dynamic transactions, as defined in your proposal, didn't lock on a whole-objectStore level? If so, how does the author specify which rows are locked? And why is then openObjectStore a asynchronous operation that could possibly fail, since at the time when openObjectStore is called, the implementation doesn't know which rows are going to be accessed and so can't determine if a deadlock is occurring? And is it only possible to lock existing rows, or can you prevent new records from being created? And is it possible to only use read-locking for some rows, but write-locking for others, in the same objectStore? That's my interpretation, dynamic transactions don't lock whole object stores. To me dynamic transactions are the same as what typical SQL databases do today. The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). Regarding deadlocks, that's right, the implementation
RE: [IndexedDB] Current editor's draft
From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 5:18 PM The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). What counts as see? If you iterate using an index-cursor all the rows that have some value between A and B, but another, not yet committed, transaction changes a row such that its value now is between A and B, what happens? We need to design something a bit more formal that covers the whole spectrum. As a short answer, assuming we want to have serializable as our isolation level, then we'd have a range lock that goes from the start of a cursor to the point you've reached, so if you were to start another cursor you'd be guaranteed the exact same view of the world. In that case it wouldn't be possible for other transaction to insert a row between two rows you scanned through with a cursor. -pablo
Re: [IndexedDB] Current editor's draft
On Thu, Jul 22, 2010 at 5:26 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 5:18 PM The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). What counts as see? If you iterate using an index-cursor all the rows that have some value between A and B, but another, not yet committed, transaction changes a row such that its value now is between A and B, what happens? We need to design something a bit more formal that covers the whole spectrum. As a short answer, assuming we want to have serializable as our isolation level, then we'd have a range lock that goes from the start of a cursor to the point you've reached, so if you were to start another cursor you'd be guaranteed the exact same view of the world. In that case it wouldn't be possible for other transaction to insert a row between two rows you scanned through with a cursor. How would you prevent that? Would a call to .modify() or .put() block until the other transaction finishes? With appropriate timeouts on deadlocks of course. / Jonas
RE: [IndexedDB] Current editor's draft
From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 5:25 PM Regarding deadlocks, that's right, the implementation cannot determine if a deadlock will occur ahead of time. Sophisticated implementations could track locks/owners and do deadlock detection, although a simple timeout-based mechanism is probably enough for IndexedDB. Simple implementations will not deadlock because they're only doing object store level locking in a constant locking order. Well, it's not really simple vs sophisticated, but whether they do dynamically scoped transactions or not, isn't it? If you do dynamic transactions, then regardless of the granularity of your locks, code will grow the lock space in a way that you cannot predict so you can't use a well-known locking order, so deadlocks are not avoidable. Sophisticated implementations will be doing key level (IndexedDB's analog to row level) locking with deadlock detection or using methods to completely avoid it. I'm not sure I'm comfortable with having one or two in-between implementations relying on timeouts to resolve deadlocks. Deadlock detection is quite a bit to ask from the storage engine. From the developer's perspective, the difference between deadlock detection and timeouts for deadlocks is the fact that the timeout approach will take a bit longer, and the error won't be as definitive. I don't think this particular difference is enough to require deadlock detection. Of course, if we're breaking deadlocks that means that web developers need to handle this error case on every async request they make. As such, I'd rather that we require implementations to make deadlocks impossible. This means that they either need to be conservative about locking or to do MVCC (or something similar) so that transactions can continue on even beyond the point where we know they can't be serialized. This would be consistent with our usual policy of trying to put as much of the burden as is practical on the browser developers rather than web developers. Same as above...MVCC is quite a bit to mandate from all implementations. For example, I'm not sure but from my basic understanding of SQLite I think it always does straight up locking and doesn't have support for versioning. As for locking only existing rows, that depends on how much isolation we want to provide. If we want serializable, then we'd have to put in things such as range locks and locks on non-existing keys so reads are consistent w.r.t. newly created rows. For the record, I am completely against anything other than serializable being the default. Everything a web developer deals with follows run to completion. If you want to have optional modes that relax things in terms of serializability, maybe we should start a new thread? Agreed. I was against dynamic transactions even when they used whole-objectStore locking. So I'm even more so now that people are proposing row-level locking. But I'd like to understand what people are proposing, and make sure that what is being proposed is a coherent solution, so that we can correctly evaluate it's risks versus benefits. The way I see the risk/benefit tradeoff of dynamic transactions: they bring better concurrency and more flexibility at the cost of new failure modes. I think that weighing them in those terms is more important than the specifics such as whether it's okay to have timeouts versus explicit deadlock errors. -pablo
RE: [IndexedDB] Current editor's draft
From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 5:30 PM On Thu, Jul 22, 2010 at 5:26 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 5:18 PM The author doesn't explicitly specify which rows to lock. All rows that you see become locked (e.g. through get(), put(), scanning with a cursor, etc.). If you start the transaction as read-only then they'll all have shared locks. If you start the transaction as read-write then we can choose whether the implementation should always attempt to take exclusive locks or if it should take shared locks on read, and attempt to upgrade to an exclusive lock on first write (this affects failure modes a bit). What counts as see? If you iterate using an index-cursor all the rows that have some value between A and B, but another, not yet committed, transaction changes a row such that its value now is between A and B, what happens? We need to design something a bit more formal that covers the whole spectrum. As a short answer, assuming we want to have serializable as our isolation level, then we'd have a range lock that goes from the start of a cursor to the point you've reached, so if you were to start another cursor you'd be guaranteed the exact same view of the world. In that case it wouldn't be possible for other transaction to insert a row between two rows you scanned through with a cursor. How would you prevent that? Would a call to .modify() or .put() block until the other transaction finishes? With appropriate timeouts on deadlocks of course. That's right, calls would block if they need to acquire a lock for a key or a range and there is an incompatible lock present that overlaps somehow with that. -pablo
Re: [IndexedDB] Current editor's draft
On Thu, Jul 22, 2010 at 8:39 PM, Pablo Castro pablo.cas...@microsoft.comwrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Thursday, July 22, 2010 5:25 PM Regarding deadlocks, that's right, the implementation cannot determine if a deadlock will occur ahead of time. Sophisticated implementations could track locks/owners and do deadlock detection, although a simple timeout-based mechanism is probably enough for IndexedDB. Simple implementations will not deadlock because they're only doing object store level locking in a constant locking order. Well, it's not really simple vs sophisticated, but whether they do dynamically scoped transactions or not, isn't it? If you do dynamic transactions, then regardless of the granularity of your locks, code will grow the lock space in a way that you cannot predict so you can't use a well-known locking order, so deadlocks are not avoidable. As I've mentioned before, you can simply not run more than one dynamic transaction at a time (and only start locking for a static transaction when all locks are available and doing the locking atomically) to implement the dynamic transactions from an API perspective. Sophisticated implementations will be doing key level (IndexedDB's analog to row level) locking with deadlock detection or using methods to completely avoid it. I'm not sure I'm comfortable with having one or two in-between implementations relying on timeouts to resolve deadlocks. Deadlock detection is quite a bit to ask from the storage engine. From the developer's perspective, the difference between deadlock detection and timeouts for deadlocks is the fact that the timeout approach will take a bit longer, and the error won't be as definitive. I don't think this particular difference is enough to require deadlock detection. This means that some web apps on some platforms will hang for seconds (or minutes?) at a time in a hard to debug fashion. I don't think this is acceptable for a web standard. Of course, if we're breaking deadlocks that means that web developers need to handle this error case on every async request they make. As such, I'd rather that we require implementations to make deadlocks impossible. This means that they either need to be conservative about locking or to do MVCC (or something similar) so that transactions can continue on even beyond the point where we know they can't be serialized. This would be consistent with our usual policy of trying to put as much of the burden as is practical on the browser developers rather than web developers. Same as above...MVCC is quite a bit to mandate from all implementations. For example, I'm not sure but from my basic understanding of SQLite I think it always does straight up locking and doesn't have support for versioning. As I mentioned, there's a simpler behavior that implementations can implement if they feel MVCC is too complicated. If dynamic transactions are included in v1 of the spec, this will almost certainly be what we do initially in Chromium. Of course, I'd rather we just take it out of v1 for reasons like what's coming up in this thread. As for locking only existing rows, that depends on how much isolation we want to provide. If we want serializable, then we'd have to put in things such as range locks and locks on non-existing keys so reads are consistent w.r.t. newly created rows. For the record, I am completely against anything other than serializable being the default. Everything a web developer deals with follows run to completion. If you want to have optional modes that relax things in terms of serializability, maybe we should start a new thread? Agreed. I was against dynamic transactions even when they used whole-objectStore locking. So I'm even more so now that people are proposing row-level locking. But I'd like to understand what people are proposing, and make sure that what is being proposed is a coherent solution, so that we can correctly evaluate it's risks versus benefits. The way I see the risk/benefit tradeoff of dynamic transactions: they bring better concurrency and more flexibility at the cost of new failure modes. I think that weighing them in those terms is more important than the specifics such as whether it's okay to have timeouts versus explicit deadlock errors. I think we should only add additional failure modes when there are very strong reasons why they're worth it. And simplifying things for implementors is not an acceptable reason to add (fairly complex, non-deterministic) failure modes. J
Re: [IndexedDB] Current editor's draft
On Thu, Jul 15, 2010 at 2:37 AM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 14, 2010 at 6:05 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 5:43 PM On Wed, Jul 14, 2010 at 5:03 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 12:07 AM I think what I'm struggling with is how dynamic transactions will help since they are still doing whole-objectStore locking. I'm also curious how you envision people dealing with deadlock hazards. Nikunjs examples in the beginning of this thread simply throw up their hands and report an error if there was a deadlock. That is obviously not good enough for an actual application. So in short, looking forward to an example :) I'll try to come up with one, although I doubt the code itself will be very interesting in this particular case. Not sure what you mean by they are still doing whole-objectStore locking. The point of dynamic transactions is that they *don't* lock the whole store, but instead have the freedom to choose the granularity (e.g. you could do row-level locking). My understanding is that the currently specced dynamic transactions are still whole-objectStore. My understanding is that of Pablo's. I'm not aware of anything in the spec that'd limit you to object-store wide locks. Whats more, if this were true then I'd be _very_ against adding dynamic transactions in v1 since they'd offer us very little in turn for a lot of complexity. This misunderstanding would definitely explain a lot of confusion within our discussions though. :-) Once you call openObjectStore and successfully receive the objectStore through the 'success' event, a lock is held on the whole objectStore until the transaction is committed. No other transaction, dynamic or static, can open the objectStore in the meantime. I base this on the sentence: There MAY not be any overlap among the scopes of all open connections to a given database from the spec. But I might be misunderstanding things entirely. Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. As for deadlocks, whenever you're doing an operation you need to be ready to handle errors (out of disk, timeout, etc.). I'm not sure why deadlocks are different. If the underlying implementation has deadlock detection then you may get a specific error, otherwise you'll just get a timeout. Well, I agree that while you have to handle errors to prevent dataloss, I suspect that most authors won't. Thus the more error conditions that we introduce, the more I think the difference is that deadlocks will happen often enough that they are a real concern. Out of disk space makes most desktop applications freak out enough that they generally cause dataloss, thus OSs tend to warn when you're running low on disk space. As for timeouts, I think we should make the defaults not be to have a timeout. Only if authors specifically specify a timeout parameter should we use one. My main line of thinking is that authors are going to generally be very bad at even looking for errors. Even less so at successfully handling those errors in a way that is satisfactory for the user. So I think the default behavior is that any time an error occurs, we'll end up rolling back the transaction and there will be dataloss. We should absolutely still provide good error handling opportunities so that authors can at least try to deal with it. However I'm not too optimistic that people will actually use them correctly. I agree with all of this. I'd also like to note that, as far as I can tell, without dynamic transactions there's no possible way to deadlock. And even with dynamic transactions, it should be possible to implement things in a way that static transactions always win. Which I think would be a good idea to ensure that anyone using them gets the simplest possible API. This will likely be extra bad for transactions where no write operations are done. In this case failure to call a 'commit()' function won't result in any broken behavior. The transaction will just sit open for a long time and eventually rolled back, though since no changes were done, the rollback is transparent, and the only noticeable effect is that the application halts for a while while the transaction is waiting to time out. I should add that the WebSQLDatabase uses automatically committing transactions very similar to what we're proposing, and it seems to have worked fine there. I find this a bit scary, although it could be that I'm permanently tainted with traditional database stuff. Typical databases follow a presumed abort protocol, where if your code is interrupted
Re: [IndexedDB] Current editor's draft
On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 2:37 AM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 14, 2010 at 6:05 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 5:43 PM On Wed, Jul 14, 2010 at 5:03 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 12:07 AM I think what I'm struggling with is how dynamic transactions will help since they are still doing whole-objectStore locking. I'm also curious how you envision people dealing with deadlock hazards. Nikunjs examples in the beginning of this thread simply throw up their hands and report an error if there was a deadlock. That is obviously not good enough for an actual application. So in short, looking forward to an example :) I'll try to come up with one, although I doubt the code itself will be very interesting in this particular case. Not sure what you mean by they are still doing whole-objectStore locking. The point of dynamic transactions is that they *don't* lock the whole store, but instead have the freedom to choose the granularity (e.g. you could do row-level locking). My understanding is that the currently specced dynamic transactions are still whole-objectStore. My understanding is that of Pablo's. I'm not aware of anything in the spec that'd limit you to object-store wide locks. Whats more, if this were true then I'd be _very_ against adding dynamic transactions in v1 since they'd offer us very little in turn for a lot of complexity. This misunderstanding would definitely explain a lot of confusion within our discussions though. :-) Once you call openObjectStore and successfully receive the objectStore through the 'success' event, a lock is held on the whole objectStore until the transaction is committed. No other transaction, dynamic or static, can open the objectStore in the meantime. I base this on the sentence: There MAY not be any overlap among the scopes of all open connections to a given database from the spec. But I might be misunderstanding things entirely. Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? -- fail with a specific error. Thanks, Andrei
Re: [IndexedDB] Current editor's draft
On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). -- fail with a specific error. To be clear, this means that any async request inside a dynamic transaction could fail and the developer would need to handle this. Given that we're already concerned about users handling errors on commit, I'd definitely be weary of requiring such a thing. But yes, the other option means that implementations need to either lock more conservatively or be able to continue on even if they know failure is certain. Btw, is there any reason you talked only about running multiple static or dynamic transactions at once? As far as I can tell, we should be able to run multiple at the same time as long as a dynamic transaction always fails if it tries to access something that a static transaction has locked. (Which gets us back to the same 2 scenarios that you were asking about for dynamic transactions). J
Re: [IndexedDB] Current editor's draft
On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. -- fail with a specific error. To be clear, this means that any async request inside a dynamic transaction could fail and the developer would need to handle this. Given that we're already concerned about users handling errors on commit, I'd definitely be weary of requiring such a thing. But yes, the other option means that implementations need to either lock more conservatively or be able to continue on even if they know failure is certain. Agreed. Btw, is there any reason you talked only about running multiple static or dynamic transactions at once? As far as I can tell, we should be able to run multiple at the same time as long as a dynamic transaction always fails if it tries to access something that a static transaction has locked. Ah, sorry I wasn't clear: you can certainly have multiple static and dynamic at the same time. Thanks, Andrei
Re: [IndexedDB] Current editor's draft
On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. -- fail with a specific error. To be clear, this means that any async request inside a dynamic transaction could fail and the developer would need to handle this. Given that we're already concerned about users handling errors on commit, I'd definitely be weary of requiring such a thing. But yes, the other option means that implementations need to either lock more conservatively or be able to continue on even if they know failure is certain. Agreed. Btw, is there any reason you talked only about running multiple static or dynamic transactions at once? As far as I can tell, we should be able to run multiple at the same time as long as a dynamic transaction always fails if it tries to access something that a static transaction has locked. Ah, sorry I wasn't clear: you can certainly have multiple static and dynamic at the same time. Thanks, Andrei
RE: [IndexedDB] Current editor's draft
From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Thursday, July 15, 2010 8:41 AM On Thu, Jul 15, 2010 at 4:30 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 3:24 PM, Jeremy Orlow jor...@chromium.org wrote: On Thu, Jul 15, 2010 at 3:09 PM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 15, 2010 at 9:50 AM, Jeremy Orlow jor...@chromium.org wrote: Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? I'd definitely like to hear what Nikunj originally intended here. Hmm, after re-reading the current spec, my understanding is that: - Scope consists in a set of object stores that the transaction operates on. - A connection may have zero or one active transactions. - There may not be any overlap among the scopes of all active transactions (static or dynamic) in a given database. So you cannot have two READ_ONLY static transactions operating simultaneously over the same object store. - The granularity of locking for dynamic transactions is not specified (all the spec says about this is do not acquire locks on any database objects now. Locks are obtained as the application attempts to access those objects). - Using dynamic transactions can lead to dealocks. Given the changes in 9975, here's what I think the spec should say for now: - There can be multiple active static transactions, as long as their scopes do not overlap, or the overlapping objects are locked in modes that are not mutually exclusive. - [If we decide to keep dynamic transactions] There can be multiple active dynamic transactions. TODO: Decide what to do if they start overlapping: -- proceed anyway and then fail at commit time in case of conflicts. However, I think this would require implementing MVCC, so implementations that use SQLite would be in trouble? Such implementations could just lock more conservatively (i.e. not allow other transactions during a dynamic transaction). Umm, I am not sure how useful dynamic transactions would be in that case...Ben Turner made the same comment earlier in the thread and I agree with him. Yes, dynamic transactions would not be useful on those implementations, but the point is that you could still implement the spec without a MVCC backend--though it would limit the concurrency that's possible. Thus implementations that use SQLite would NOT necessarily be in trouble. Interesting, I'm glad this conversation came up so we can sync up on assumptions...mine where: - There can be multiple transactions of any kind active against a given database session (see note below) - Multiple static transactions may overlap as long as they have compatible modes, which in practice means they are all READ_ONLY - Dynamic transactions have arbitrary granularity for scope (implementation specific, down to row-level locking/scope) - Overlapping between statically and dynamically scoped transactions follows the same rules as static-static overlaps; they can only overlap on compatible scopes. The only difference is that dynamic transactions may need to block mid-flight until it can grab the resources it needs to proceed. Note: for some databases having multiple transactions active on a single connection may be an unsupported thing. This could probably be handled in the IndexedDB layer though by using multiple connections under the covers. -pablo
Re: [IndexedDB] Current editor's draft
Hi Pablo, First off, thanks for your comments! (Probably too much) details below. On Tue, Jul 13, 2010 at 7:52 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: public-webapps-requ...@w3.org [mailto:public-webapps-requ...@w3.org] On Behalf Of Andrei Popescu Sent: Monday, July 12, 2010 5:23 AM Sorry I disappeared for a while. Catching up with this discussion was an interesting exercise...there is no particular message in this thread I can respond to, so I thought I'd just reply to the last one. Overall I think the new proposal is shaping up well and is being effective in simplifying scenarios. I do have a few suggestions and questions for things I'm not sure I see all the way. READ_ONLY vs READ_WRITE as defaults for transactions: To be perfectly honest, I think this discussion went really deep over an issue that won't be a huge deal for most people. My perspective, trying to avoid performance or usage frequency speculation, is around what's easier to detect. Concurrency issues are hard to see. On the other hand, whenever we can throw an exception and give explicit guidance that unblocks people right away. For this case I suspect it's best to default to READ_ONLY, because if someone doesn't read or think about it and just uses the stuff and tries to change something they'll get a clear error message saying if you want to change stuff, use READ_WRITE please. The error is not data- or context-dependent, so it'll fail on first try at most once per developer and once they fix it they'll know for all future cases. Yup, this was exactly my thinking. Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. I don't understand enough of the details here to be able to make a decision. The use cases you are bringing up I definitely agree are important, but I would love to look at even a rough draft of what code you are expecting people will need to write. What I suggest is that we keep dynamic transactions in the spec for now, but separate the API from static transactions, start a separate thread and try to hammer out the details and see what we arrive at. I do want to clarify that I don't think dynamic transactions are particularly hard to implement, I just suspect they are hard to use correctly. Implicit commit: Does this really work? I need to play with sample app code more, it may just be that I'm old-fashioned. For example, if I'm downloading a bunch of data form somewhere and pushing rows into the store within a transaction, wouldn't it be reasonable to do the whole thing in a transaction? In that case I'm likely to have to unwind while I wait for the next callback from XmlHttpRequest with the next chunk of data. You definitely want to do it in a transaction. In our proposal there is no way to even call .get or .put if you aren't inside a transaction. For the case you are describing, you'd download the data using XMLHttpRequest first. Once the data has been downloaded you start a transaction, parse the data, and make the desired modifications. Once that is done the transaction is automatically committed. The idea here is to avoid keeping transactions open for long periods of time, while at the same time making the API easier to work with. I'm very concerned that any API that requires people to do: startOperation(); ... do lots of stuff here ... endOperation(); people will forget to do the endOperation call. This is especially true if the startOperation/endOperation calls are spread out over multiple different asynchronously called functions, which seems to be the use case you're concerned about above. One very easy way to forget to call endOperation is if something inbetween the two function calls throw an exception. This will likely be extra bad for transactions where no write operations are done. In this case failure to call a 'commit()' function won't result in any broken behavior.
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 3:52 AM, Pablo Castro pablo.cas...@microsoft.comwrote: From: public-webapps-requ...@w3.org [mailto:public-webapps-requ...@w3.org] On Behalf Of Andrei Popescu Sent: Monday, July 12, 2010 5:23 AM Sorry I disappeared for a while. Catching up with this discussion was an interesting exercise... Yes, Indeed. :-) there is no particular message in this thread I can respond to, so I thought I'd just reply to the last one. Probably a good idea. I was trying to respond hixie style--which is harder than it looks on stuff like this. Overall I think the new proposal is shaping up well and is being effective in simplifying scenarios. I do have a few suggestions and questions for things I'm not sure I see all the way. READ_ONLY vs READ_WRITE as defaults for transactions: To be perfectly honest, I think this discussion went really deep over an issue that won't be a huge deal for most people. My perspective, trying to avoid performance or usage frequency speculation, is around what's easier to detect. Concurrency issues are hard to see. On the other hand, whenever we can throw an exception and give explicit guidance that unblocks people right away. For this case I suspect it's best to default to READ_ONLY, because if someone doesn't read or think about it and just uses the stuff and tries to change something they'll get a clear error message saying if you want to change stuff, use READ_WRITE please. The error is not data- or context-dependent, so it'll fail on first try at most once per developer and once they fix it they'll know for all future cases. Couldn't have said it better myself. Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. Well, the application should never freeze in terms of the UI locking up, but in what you described I could see it taking a while for data to show up on the screen. This is something that can be fixed by doing smaller updates on the background thread, sending a message to the background thread that it should abort for now, doing all database access on the background thread, etc. One point that I never saw made in the thread that I think is really important is that dynamic transactions can make concurrency worse in some cases. For example, with dynamic transactions you can get into live-lock situations. Also, using Pablo's example, you could easily get into a situation where the long running transaction on the worker keeps hitting serialization issues and thus it's never able to make progress. I do see that there are use cases where having dynamic transactions would be much nicer, but the amount of non-determinism they add (including to performance) has me pretty worried. I pretty firmly believe we should look into adding them in v2 and remove them for now. If we do leave them in, it should definitely be in its own method to make it quite clear that the semantics are more complex. Implicit commit: Does this really work? I need to play with sample app code more, it may just be that I'm old-fashioned. For example, if I'm downloading a bunch of data form somewhere and pushing rows into the store within a transaction, wouldn't it be reasonable to do the whole thing in a transaction? In that case I'm likely to have to unwind while I wait for the next callback from XmlHttpRequest with the next chunk of data. I understand that avoiding it results in nicer patterns (e.g. db.objectStores(foo).get(123).onsuccess = ...), but in practice I'm not sure if that will hold given that you still need error callbacks and such. I believe your example of doing XHRs in the middle of a transaction is something we were explicitly trying to avoid making possible. In this case, you should do all of your XHRs first and then do your transaction. If you need to read form the ObjectStore, do a XHR, and then write to the ObjectStore, you can implement it with 2 transactions and have the
Re: [IndexedDB] Current editor's draft
Hi, I would like to propose that we update the current spec to reflect all the changes we have agreement on. We can then iteratively review and make edits as soon as the remaining issues are solved. Concretely, I would like to check in a fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=9975 with the following two exceptions which, based on the feedback in this thread, require more discussion: - leave in support for dynamic transactions but add a separate API for it, as suggested by Jonas earlier in this thread. - leave in the explicit transaction commit - leave in nested transactions The changes in 9975 have been debated for more than two month now, so I feel it's about time to update the specification so that it's in line with what we're actually discussing. Thanks, Andrei On Wed, Jul 14, 2010 at 8:10 AM, Jeremy Orlow jor...@chromium.org wrote: On Wed, Jul 14, 2010 at 3:52 AM, Pablo Castro pablo.cas...@microsoft.com wrote: From: public-webapps-requ...@w3.org [mailto:public-webapps-requ...@w3.org] On Behalf Of Andrei Popescu Sent: Monday, July 12, 2010 5:23 AM Sorry I disappeared for a while. Catching up with this discussion was an interesting exercise... Yes, Indeed. :-) there is no particular message in this thread I can respond to, so I thought I'd just reply to the last one. Probably a good idea. I was trying to respond hixie style--which is harder than it looks on stuff like this. Overall I think the new proposal is shaping up well and is being effective in simplifying scenarios. I do have a few suggestions and questions for things I'm not sure I see all the way. READ_ONLY vs READ_WRITE as defaults for transactions: To be perfectly honest, I think this discussion went really deep over an issue that won't be a huge deal for most people. My perspective, trying to avoid performance or usage frequency speculation, is around what's easier to detect. Concurrency issues are hard to see. On the other hand, whenever we can throw an exception and give explicit guidance that unblocks people right away. For this case I suspect it's best to default to READ_ONLY, because if someone doesn't read or think about it and just uses the stuff and tries to change something they'll get a clear error message saying if you want to change stuff, use READ_WRITE please. The error is not data- or context-dependent, so it'll fail on first try at most once per developer and once they fix it they'll know for all future cases. Couldn't have said it better myself. Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. Well, the application should never freeze in terms of the UI locking up, but in what you described I could see it taking a while for data to show up on the screen. This is something that can be fixed by doing smaller updates on the background thread, sending a message to the background thread that it should abort for now, doing all database access on the background thread, etc. One point that I never saw made in the thread that I think is really important is that dynamic transactions can make concurrency worse in some cases. For example, with dynamic transactions you can get into live-lock situations. Also, using Pablo's example, you could easily get into a situation where the long running transaction on the worker keeps hitting serialization issues and thus it's never able to make progress. I do see that there are use cases where having dynamic transactions would be much nicer, but the amount of non-determinism they add (including to performance) has me pretty worried. I pretty firmly believe we should look into adding them in v2 and remove them for now. If we do leave them in, it should definitely be in its own method to make it quite clear that the semantics are more complex. Implicit commit: Does this really work? I need to play with sample app code more, it may just be that I'm
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 1:20 PM, Andrei Popescu andr...@google.com wrote: Hi, I would like to propose that we update the current spec to reflect all the changes we have agreement on. We can then iteratively review and make edits as soon as the remaining issues are solved. Concretely, I would like to check in a fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=9975 with the following two exceptions which, based on the feedback in this thread, require more discussion: - leave in support for dynamic transactions but add a separate API for it, as suggested by Jonas earlier in this thread. - leave in the explicit transaction commit - leave in nested transactions The changes in 9975 have been debated for more than two month now, so I feel it's about time to update the specification so that it's in line with what we're actually discussing. Agreed. In the future I think we should never let things stay this out of sync for this long, but I understand how this was a bit of a special case because of the scope of the changes. But yeah, let's make these changes and then iterate. And hopefully we can resolve the dynamic transaction, explicit commit, and nested transaction issues in the near future. Thanks, Andrei On Wed, Jul 14, 2010 at 8:10 AM, Jeremy Orlow jor...@chromium.org wrote: On Wed, Jul 14, 2010 at 3:52 AM, Pablo Castro pablo.cas...@microsoft.com wrote: From: public-webapps-requ...@w3.org [mailto: public-webapps-requ...@w3.org] On Behalf Of Andrei Popescu Sent: Monday, July 12, 2010 5:23 AM Sorry I disappeared for a while. Catching up with this discussion was an interesting exercise... Yes, Indeed. :-) there is no particular message in this thread I can respond to, so I thought I'd just reply to the last one. Probably a good idea. I was trying to respond hixie style--which is harder than it looks on stuff like this. Overall I think the new proposal is shaping up well and is being effective in simplifying scenarios. I do have a few suggestions and questions for things I'm not sure I see all the way. READ_ONLY vs READ_WRITE as defaults for transactions: To be perfectly honest, I think this discussion went really deep over an issue that won't be a huge deal for most people. My perspective, trying to avoid performance or usage frequency speculation, is around what's easier to detect. Concurrency issues are hard to see. On the other hand, whenever we can throw an exception and give explicit guidance that unblocks people right away. For this case I suspect it's best to default to READ_ONLY, because if someone doesn't read or think about it and just uses the stuff and tries to change something they'll get a clear error message saying if you want to change stuff, use READ_WRITE please. The error is not data- or context-dependent, so it'll fail on first try at most once per developer and once they fix it they'll know for all future cases. Couldn't have said it better myself. Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. Well, the application should never freeze in terms of the UI locking up, but in what you described I could see it taking a while for data to show up on the screen. This is something that can be fixed by doing smaller updates on the background thread, sending a message to the background thread that it should abort for now, doing all database access on the background thread, etc. One point that I never saw made in the thread that I think is really important is that dynamic transactions can make concurrency worse in some cases. For example, with dynamic transactions you can get into live-lock situations. Also, using Pablo's example, you could easily get into a situation where the long running transaction on the worker keeps hitting serialization issues and thus it's
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 3:10 AM, Jeremy Orlow jor...@chromium.org wrote: For example, with dynamic transactions you can get into live-lock situations. I'm particularly opposed to dynamic transactions for just this reason. We would clearly have to throw an exception or call the error callback if we detect livelock. I doubt that most web authors would recognize the potential hazard, and even if they did I think it would be extremely difficult for a web author to test such a scenario or write code to handle it properly. The hardware running the web app and the browser's transaction scheduling algorithm would of course affect the frequency of these collisions making proper tests even more difficult. If we do leave them in, it should definitely be in its own method to make it quite clear that the semantics are more complex. I completely agree. So, as I've said, I'm very opposed to leaving dynamic transactions in the spec. However, one thing we could do if everyone really wanted this feature I guess is to set a limit of only a single dynamic transaction per database at a time. That would remove the livelock hazard but it may diminish the utility of the feature enough to be useless.
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 5:21 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 14, 2010 at 5:20 AM, Andrei Popescu andr...@google.com wrote: Hi, I would like to propose that we update the current spec to reflect all the changes we have agreement on. We can then iteratively review and make edits as soon as the remaining issues are solved. Concretely, I would like to check in a fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=9975 with the following two exceptions which, based on the feedback in this thread, require more discussion: - leave in support for dynamic transactions but add a separate API for it, as suggested by Jonas earlier in this thread. - leave in the explicit transaction commit - leave in nested transactions The changes in 9975 have been debated for more than two month now, so I feel it's about time to update the specification so that it's in line with what we're actually discussing. When you say leave in the explicit transaction commit, do you mean in addition to the implicit commit one there are no more requests on a transaction, or instead of it? In addition. In the current editor draft we have both: Implicit commit is described at: http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#dfn-transaction Explicit commit is defined at http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#widl-IDBTransaction-commit I was saying I would not remove the explicit one pending further discussion. Thanks, Andrei
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 9:28 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 14, 2010 at 5:21 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 14, 2010 at 5:20 AM, Andrei Popescu andr...@google.com wrote: Hi, I would like to propose that we update the current spec to reflect all the changes we have agreement on. We can then iteratively review and make edits as soon as the remaining issues are solved. Concretely, I would like to check in a fix for http://www.w3.org/Bugs/Public/show_bug.cgi?id=9975 with the following two exceptions which, based on the feedback in this thread, require more discussion: - leave in support for dynamic transactions but add a separate API for it, as suggested by Jonas earlier in this thread. - leave in the explicit transaction commit - leave in nested transactions The changes in 9975 have been debated for more than two month now, so I feel it's about time to update the specification so that it's in line with what we're actually discussing. When you say leave in the explicit transaction commit, do you mean in addition to the implicit commit one there are no more requests on a transaction, or instead of it? In addition. In the current editor draft we have both: Implicit commit is described at: http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#dfn-transaction Explicit commit is defined at http://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#widl-IDBTransaction-commit I was saying I would not remove the explicit one pending further discussion. Makes sense, thanks for clarifying. / Jonas
RE: [IndexedDB] Current editor's draft
From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 12:07 AM Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. I don't understand enough of the details here to be able to make a decision. The use cases you are bringing up I definitely agree are important, but I would love to look at even a rough draft of what code you are expecting people will need to write. I'll try and hack up and example. In general any scenario that has a worker and the UI thread working on the same database will be quite a challenge, because the worker will have to a) split the work in small pieces, even if it was naturally a bigger chunk and b) consider interleaving implications with the UI thread, otherwise even when split in chunks you're not guaranteed that one of the two will starve the other one (the worker running on a tight loop will effectively always have an active transaction, it'll be just changing the actual transaction from time to time). This can certainly happen with dynamic transactions as well, the only difference is that since the locking granularity is different, it may be that what you're working on in the worker and in the UI threads is independent enough that they don't interfere too much, allowing for some more concurrency. What I suggest is that we keep dynamic transactions in the spec for now, but separate the API from static transactions, start a separate thread and try to hammer out the details and see what we arrive at. I do want to clarify that I don't think dynamic transactions are particularly hard to implement, I just suspect they are hard to use correctly. Sounds reasonable. Implicit commit: Does this really work? I need to play with sample app code more, it may just be that I'm old-fashioned. For example, if I'm downloading a bunch of data form somewhere and pushing rows into the store within a transaction, wouldn't it be reasonable to do the whole thing in a transaction? In that case I'm likely to have to unwind while I wait for the next callback from XmlHttpRequest with the next chunk of data. You definitely want to do it in a transaction. In our proposal there is no way to even call .get or .put if you aren't inside a transaction. For the case you are describing, you'd download the data using XMLHttpRequest first. Once the data has been downloaded you start a transaction, parse the data, and make the desired modifications. Once that is done the transaction is automatically committed. The idea here is to avoid keeping transactions open for long periods of time, while at the same time making the API easier to work with. I'm very concerned that any API that requires people to do: startOperation(); ... do lots of stuff here ... endOperation(); people will forget to do the endOperation call. This is especially true if the startOperation/endOperation calls are spread out over multiple different asynchronously called functions, which seems to be the use case you're concerned about above. One very easy way to forget to call endOperation is if something inbetween the two function calls throw an exception. Fair enough, maybe I need to think of this scenario differently, and if someone needs to download a bunch of data and then put it in the database atomically the right way is to download to work tables first over a long time and independent transactions, and then use a transaction only to move the data around into its final spot. This will likely be extra bad for transactions where no write operations are done. In this case failure to call a 'commit()' function won't result in any broken behavior. The transaction will just sit open for a long time and eventually rolled back, though since no changes were done, the rollback is transparent, and the only noticeable effect is that the application halts for a while while the
RE: [IndexedDB] Current editor's draft
From: jor...@google.com [mailto:jor...@google.com] On Behalf Of Jeremy Orlow Sent: Wednesday, July 14, 2010 12:10 AM On Wed, Jul 14, 2010 at 3:52 AM, Pablo Castro pablo.cas...@microsoft.com wrote: From: public-webapps-requ...@w3.org [mailto:public-webapps-requ...@w3.org] On Behalf Of Andrei Popescu Sent: Monday, July 12, 2010 5:23 AM Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. Well, the application should never freeze in terms of the UI locking up, but in what you described I could see it taking a while for data to show up on the screen. This is something that can be fixed by doing smaller updates on the background thread, sending a message to the background thread that it should abort for now, doing all database access on the background thread, etc. This is an issue regardless, isn't it? Let's say you have a worker churning on the database somehow. The worker has no UI or user to wait for, so it'll run in a tight loop at full speed. If it splits the work in small transactions, in cases where it doesn't have to wait for something external there will still be a small gap between transactions. That could easily starve the UI thread that needs to find an opportunity to get in and do a quick thing against the database. As you say the difference between freezing and locking up at this point is not that critical, as the end user in the end is just waiting. One point that I never saw made in the thread that I think is really important is that dynamic transactions can make concurrency worse in some cases. For example, with dynamic transactions you can get into live-lock situations. Also, using Pablo's example, you could easily get into a situation where the long running transaction on the worker keeps hitting serialization issues and thus it's never able to make progress. While it could certainly happen, I don't remember seeing something like a live-lock in a long, long time. Deadlocks are common, but a simple timeout will kill one of the transactions and let the other make progress. A bit violent, but always effective. I do see that there are use cases where having dynamic transactions would be much nicer, but the amount of non-determinism they add (including to performance) has me pretty worried. I pretty firmly believe we should look into adding them in v2 and remove them for now. If we do leave them in, it should definitely be in its own method to make it quite clear that the semantics are more complex. Let's explore a bit more and see where we land. I'm not pushing for dynamic transactions themselves, but more for the scenarios they enable (background processing and such). If we find other ways of doing that, then all the better. Having different entry points is reasonable. Nested transactions: Not sure why we're considering this an advanced scenario. To be clear about what the feature means to me: make it legal to start a transaction when one is already in progress, and the nested one is effectively a no-op, just refcounts the transaction, so you need equal amounts of commit()'s, implicit or explicit, and an abort() cancels all nested transactions. The purpose of this is to allow composition, where a piece of code that needs a transaction can start one locally, independently of whether the caller had already one going. I believe it's actually a bit more tricky than what you said. For example, if we only support static transactions, will we require that any nested transaction only request a subset of the locks the outer one took? What if we try to start a dynamic transaction inside of a static one? Etc. But I agree it's not _that_ tricky and I'm also not convinced it's an advanced feature. I'd suggest we take it out for now and look at re-adding it when the basics of the async API are more solidified. I hope we can
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 5:03 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 12:07 AM Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. I don't understand enough of the details here to be able to make a decision. The use cases you are bringing up I definitely agree are important, but I would love to look at even a rough draft of what code you are expecting people will need to write. I'll try and hack up and example. In general any scenario that has a worker and the UI thread working on the same database will be quite a challenge, because the worker will have to a) split the work in small pieces, even if it was naturally a bigger chunk and b) consider interleaving implications with the UI thread, otherwise even when split in chunks you're not guaranteed that one of the two will starve the other one (the worker running on a tight loop will effectively always have an active transaction, it'll be just changing the actual transaction from time to time). This can certainly happen with dynamic transactions as well, the only difference is that since the locking granularity is different, it may be that what you're working on in the worker and in the UI threads is independent enough that they don't interfere too much, allowing for some more concurrency. I think what I'm struggling with is how dynamic transactions will help since they are still doing whole-objectStore locking. I'm also curious how you envision people dealing with deadlock hazards. Nikunjs examples in the beginning of this thread simply throw up their hands and report an error if there was a deadlock. That is obviously not good enough for an actual application. So in short, looking forward to an example :) Implicit commit: Does this really work? I need to play with sample app code more, it may just be that I'm old-fashioned. For example, if I'm downloading a bunch of data form somewhere and pushing rows into the store within a transaction, wouldn't it be reasonable to do the whole thing in a transaction? In that case I'm likely to have to unwind while I wait for the next callback from XmlHttpRequest with the next chunk of data. You definitely want to do it in a transaction. In our proposal there is no way to even call .get or .put if you aren't inside a transaction. For the case you are describing, you'd download the data using XMLHttpRequest first. Once the data has been downloaded you start a transaction, parse the data, and make the desired modifications. Once that is done the transaction is automatically committed. The idea here is to avoid keeping transactions open for long periods of time, while at the same time making the API easier to work with. I'm very concerned that any API that requires people to do: startOperation(); ... do lots of stuff here ... endOperation(); people will forget to do the endOperation call. This is especially true if the startOperation/endOperation calls are spread out over multiple different asynchronously called functions, which seems to be the use case you're concerned about above. One very easy way to forget to call endOperation is if something inbetween the two function calls throw an exception. Fair enough, maybe I need to think of this scenario differently, and if someone needs to download a bunch of data and then put it in the database atomically the right way is to download to work tables first over a long time and independent transactions, and then use a transaction only to move the data around into its final spot. Yeah, I think that this is what we want to encourage. This will likely be extra bad for transactions where no write operations are done. In this case failure to call a 'commit()' function won't result in any broken behavior. The transaction will just
RE: [IndexedDB] Current editor's draft
From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 5:43 PM On Wed, Jul 14, 2010 at 5:03 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 12:07 AM I think what I'm struggling with is how dynamic transactions will help since they are still doing whole-objectStore locking. I'm also curious how you envision people dealing with deadlock hazards. Nikunjs examples in the beginning of this thread simply throw up their hands and report an error if there was a deadlock. That is obviously not good enough for an actual application. So in short, looking forward to an example :) I'll try to come up with one, although I doubt the code itself will be very interesting in this particular case. Not sure what you mean by they are still doing whole-objectStore locking. The point of dynamic transactions is that they *don't* lock the whole store, but instead have the freedom to choose the granularity (e.g. you could do row-level locking). As for deadlocks, whenever you're doing an operation you need to be ready to handle errors (out of disk, timeout, etc.). I'm not sure why deadlocks are different. If the underlying implementation has deadlock detection then you may get a specific error, otherwise you'll just get a timeout. This will likely be extra bad for transactions where no write operations are done. In this case failure to call a 'commit()' function won't result in any broken behavior. The transaction will just sit open for a long time and eventually rolled back, though since no changes were done, the rollback is transparent, and the only noticeable effect is that the application halts for a while while the transaction is waiting to time out. I should add that the WebSQLDatabase uses automatically committing transactions very similar to what we're proposing, and it seems to have worked fine there. I find this a bit scary, although it could be that I'm permanently tainted with traditional database stuff. Typical databases follow a presumed abort protocol, where if your code is interrupted by an exception, a process crash or whatever, you can always assume transactions will be rolled back if you didn't reach an explicit call to commit. The implicit commit here takes that away, and I'm not sure how safe that is. For example, if I don't have proper exception handling in place, an illegal call to some other non-indexeddb related API may throw an exception causing the whole thing to unwind, at which point nothing will be pending to do in the database and thus the currently active transaction will be committed. Using the same line of thought we used for READ_ONLY, forgetting to call commit() is easy to detect the first time you try out your code. Your changes will simply not stick. It's not as clear as the READ_ONLY example because there is no opportunity to throw an explicit exception with an explanation, but the data not being around will certainly prompt developers to look for the issue :) Ah, I see where we are differing in thinking. My main concern has been that of rollbacks, and associated dataloss, in the non-error case. For example people forget to call commit() in some branch of their code, thus causing dataloss when the transaction is rolled back. Your concern seems to be that of lack of rollback in the error case, for example when an exception is thrown and not caught somewhere in the code. In this case you'd want to have the transaction rolled back. One way to handle this is to try to detect unhandled errors and implicitly roll back the transaction. Two situations where we could do this is: 1. When an 'error' event is fired, but where .preventDefault() has is not called by any handler. The result is that if an error is ever fired, but no one explicitly handles it, we roll back the transaction. See also below. 2. When a success handler is called, but the handler throws an exception. The second is a bit of a problem from a spec point of view. I'm not sure it is allowed by the DOM Events spec, or by all existing DOM Events implementations. I do still think we can pull it off though. This is something I've been thinking about raising for a while, but I wanted to nail down the raised issues first. Would you feel more comfortable with implicit commits if we did the above? It does make it better, although this seems to introduce quite moving parts to the process. I still think an explicit commit() would be better, but I'm open to explore more options. And as you say, you still usually need error callbacks. In fact, we have found while writing examples using our implementation, that you almost always want to add a generic error handler. It's very easy to make a mistake, and if you don't add error handlers then these just go by silently, offering no help as to why your program
Re: [IndexedDB] Current editor's draft
On Wed, Jul 14, 2010 at 6:05 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 5:43 PM On Wed, Jul 14, 2010 at 5:03 PM, Pablo Castro pablo.cas...@microsoft.com wrote: From: Jonas Sicking [mailto:jo...@sicking.cc] Sent: Wednesday, July 14, 2010 12:07 AM I think what I'm struggling with is how dynamic transactions will help since they are still doing whole-objectStore locking. I'm also curious how you envision people dealing with deadlock hazards. Nikunjs examples in the beginning of this thread simply throw up their hands and report an error if there was a deadlock. That is obviously not good enough for an actual application. So in short, looking forward to an example :) I'll try to come up with one, although I doubt the code itself will be very interesting in this particular case. Not sure what you mean by they are still doing whole-objectStore locking. The point of dynamic transactions is that they *don't* lock the whole store, but instead have the freedom to choose the granularity (e.g. you could do row-level locking). My understanding is that the currently specced dynamic transactions are still whole-objectStore. Once you call openObjectStore and successfully receive the objectStore through the 'success' event, a lock is held on the whole objectStore until the transaction is committed. No other transaction, dynamic or static, can open the objectStore in the meantime. I base this on the sentence: There MAY not be any overlap among the scopes of all open connections to a given database from the spec. But I might be misunderstanding things entirely. Nikunj, could you clarify how locking works for the dynamic transactions proposal that is in the spec draft right now? As for deadlocks, whenever you're doing an operation you need to be ready to handle errors (out of disk, timeout, etc.). I'm not sure why deadlocks are different. If the underlying implementation has deadlock detection then you may get a specific error, otherwise you'll just get a timeout. Well, I agree that while you have to handle errors to prevent dataloss, I suspect that most authors won't. Thus the more error conditions that we introduce, the more I think the difference is that deadlocks will happen often enough that they are a real concern. Out of disk space makes most desktop applications freak out enough that they generally cause dataloss, thus OSs tend to warn when you're running low on disk space. As for timeouts, I think we should make the defaults not be to have a timeout. Only if authors specifically specify a timeout parameter should we use one. My main line of thinking is that authors are going to generally be very bad at even looking for errors. Even less so at successfully handling those errors in a way that is satisfactory for the user. So I think the default behavior is that any time an error occurs, we'll end up rolling back the transaction and there will be dataloss. We should absolutely still provide good error handling opportunities so that authors can at least try to deal with it. However I'm not too optimistic that people will actually use them correctly. This will likely be extra bad for transactions where no write operations are done. In this case failure to call a 'commit()' function won't result in any broken behavior. The transaction will just sit open for a long time and eventually rolled back, though since no changes were done, the rollback is transparent, and the only noticeable effect is that the application halts for a while while the transaction is waiting to time out. I should add that the WebSQLDatabase uses automatically committing transactions very similar to what we're proposing, and it seems to have worked fine there. I find this a bit scary, although it could be that I'm permanently tainted with traditional database stuff. Typical databases follow a presumed abort protocol, where if your code is interrupted by an exception, a process crash or whatever, you can always assume transactions will be rolled back if you didn't reach an explicit call to commit. The implicit commit here takes that away, and I'm not sure how safe that is. For example, if I don't have proper exception handling in place, an illegal call to some other non-indexeddb related API may throw an exception causing the whole thing to unwind, at which point nothing will be pending to do in the database and thus the currently active transaction will be committed. Using the same line of thought we used for READ_ONLY, forgetting to call commit() is easy to detect the first time you try out your code. Your changes will simply not stick. It's not as clear as the READ_ONLY example because there is no opportunity to throw an explicit exception with an explanation, but the data not being around will certainly prompt developers to look for the
RE: [IndexedDB] Current editor's draft
From: public-webapps-requ...@w3.org [mailto:public-webapps-requ...@w3.org] On Behalf Of Andrei Popescu Sent: Monday, July 12, 2010 5:23 AM Sorry I disappeared for a while. Catching up with this discussion was an interesting exercise...there is no particular message in this thread I can respond to, so I thought I'd just reply to the last one. Overall I think the new proposal is shaping up well and is being effective in simplifying scenarios. I do have a few suggestions and questions for things I'm not sure I see all the way. READ_ONLY vs READ_WRITE as defaults for transactions: To be perfectly honest, I think this discussion went really deep over an issue that won't be a huge deal for most people. My perspective, trying to avoid performance or usage frequency speculation, is around what's easier to detect. Concurrency issues are hard to see. On the other hand, whenever we can throw an exception and give explicit guidance that unblocks people right away. For this case I suspect it's best to default to READ_ONLY, because if someone doesn't read or think about it and just uses the stuff and tries to change something they'll get a clear error message saying if you want to change stuff, use READ_WRITE please. The error is not data- or context-dependent, so it'll fail on first try at most once per developer and once they fix it they'll know for all future cases. Dynamic transactions: I see that most folks would like to see these going away. While I like the predictability and simplifications that we're able to make by using static scopes for transactions, I worry that we'll close the door for two scenarios: background tasks and query processors. Background tasks such as synchronization and post-processing of content would seem to be almost impossible with the static scope approach, mostly due to the granularity of the scope specification (whole stores). Are we okay with saying that you can't for example sync something in the background (e.g. in a worker) while your app is still working? Am I missing something that would enable this class of scenarios? Query processors are also tricky because you usually take the query specification in some form after the transaction started (especially if you want to execute multiple queries with later queries depending on the outcome of the previous ones). The background tasks issue in particular looks pretty painful to me if we don't have a way to achieve it without freezing the application while it happens. Implicit commit: Does this really work? I need to play with sample app code more, it may just be that I'm old-fashioned. For example, if I'm downloading a bunch of data form somewhere and pushing rows into the store within a transaction, wouldn't it be reasonable to do the whole thing in a transaction? In that case I'm likely to have to unwind while I wait for the next callback from XmlHttpRequest with the next chunk of data. I understand that avoiding it results in nicer patterns (e.g. db.objectStores(foo).get(123).onsuccess = ...), but in practice I'm not sure if that will hold given that you still need error callbacks and such. Nested transactions: Not sure why we're considering this an advanced scenario. To be clear about what the feature means to me: make it legal to start a transaction when one is already in progress, and the nested one is effectively a no-op, just refcounts the transaction, so you need equal amounts of commit()'s, implicit or explicit, and an abort() cancels all nested transactions. The purpose of this is to allow composition, where a piece of code that needs a transaction can start one locally, independently of whether the caller had already one going. Schema versioning: It's unfortunate that we need to have explicit elements in the page for the versioning protocol to work, but the fact that we can have a reliable mechanism for pages to coordinate a version bump is really nice. For folks that don't know about this the first time they build it, an explicit error message on the schema change timeout can explain where to start. I do think that there may be a need for non-breaking changes to the schema to happen without a version dance. For example, query processors regularly create temporary tables during sorts and such. Those shouldn't require any coordination (maybe we allow non-versioned additions, or we just introduce temporary, unnamed tables that evaporate on commit() or database close()...). Thanks -pablo
Re: [IndexedDB] Current editor's draft
Nikunj,
On Fri, Jul 9, 2010 at 8:21 PM, Nikunj Mehta nik...@o-micron.com wrote:
From my examples, it was clear that we need different object stores to be
opened in different modes. Currently dynamic scope supports this use case,
i.e., allow mode specification on a per object-store basis. Therefore, unless
we decide to de-support this use case, we would need to add this ability to
static scope transactions if dynamic scope transactions go out of v1.
I would be very grateful if you could help me understand the statement
above. Looking at your examples, we have:
function processShipment(shipment, callback) {
// we need to validate the part exists in this city first and that the
supplier is known
var txn = db.transaction(); //synchronous because requesting locks as I go along
var parts = txn.objectStore(part, IDBObjectStore.READ_ONLY);
var partRequest = parts.get(shipment.part);
partRequest.onerror = shipmentProcessingError;
partRequest.onsuccess = function(event) {
// the required part exists and we have now locked at least that key-value
// so that it won't disappear when we add the shipment.
var suppliers = txn.objectStore(supplier, IDBObjectStore.READ_ONLY);
var supplierRequest = suppliers.get(shipment.supplier);
supplierRequest.onerror = shipmentProcessingError;
supplierRequest.onsuccess = function(event) {
// the required supplier exists and we have now locked that key-value
// so that it won't disappear when we add the shipment.
var shipments = db.objectStore(shipment, IDBObjectStore.READ_WRITE);
var shipmentRequest = shipments.add(shipment);
supplierRequest.onerror = shipmentProcessingError;
shipmentRequest.onsuccess = function(event) {
var txnRequest = event.transaction.commit();
// before the callback, commit the stored record
var key = event.result;
txnRequest.oncommit = function() {
callback(key); // which is the key generated during storage
}
txnRequest.onerror = shipmentProcessingError;
}
}
If I understand things right, this example processes a new shipment:
it checks that the part and supplier exist and then adds the new
shipment to the appropriate object store. And you are claiming that if
we leave dynamic transactions out of v1, then we need to de-support
this use case. Is this correct?
Now, would the code below support the same use case?
function processShipment(shipment, callback) {
// We open a READ_WRITE transaction since we need to update the
shipments object store.
var txnRequest = db.openTtransaction(part, supplier,
shipments, IDBObjectStore.READ_TRANSACTION);
txnRequest.onsuccess = function(event) {
var txn = event.transaction;
var parts = txn.objectStore(part);
var partRequest = parts.get(shipment.part);
partRequest.onsuccess = function(event) {
// the required part exists
var suppliers = txn.objectStore(supplier);
var supplierRequest = suppliers.get(shipment.supplier);
supplierRequest.onsuccess = function(event) {
// the required supplier exists
var shipments = db.objectStore(shipment);
var shipmentRequest = shipments.add(shipment);
shipmentRequest.onsuccess = function(event) {
var key = event.result;
txnRequest.oncommit = function() {
callback(key); // which is the key generated during storage
}
}
}
}
}
So if the above supports the same use case (albeit by allowing less
concurrency), then we dropping dynamic transactions doesn't mean we
lose this use case. Is this right? Are there any other use cases you
think we could lose? I could not find them in your examples.
Thanks,
Andrei
Re: [IndexedDB] Current editor's draft
On Fri, Jul 9, 2010 at 12:50 PM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 10, 2010, at 12:29 AM, Jonas Sicking wrote: On Fri, Jul 9, 2010 at 11:05 AM, Nikunj Mehta nik...@o-micron.com wrote: We would not make dynamic transactions be the default since they would produce more concurrency than static scoped transactions, correct? On Jul 7, 2010, at 12:57 PM, Jonas Sicking wrote: I'm not sure I understand the question. We would use separate functions for creating dynamic and static transactions so there is no such thing as default. The point is that we are talking of leaving out dynamic scope in v1, while, in the same vein, talking of making READ_ONLY the default _because_ it produces good performance. That is, IMHO, contradictory. I think you are using flawed logic. Just because I want to remove one feature that can be used to increase performance doesn't mean that I hate performance and want to design all other features to be as slow as possible. I'm not arguing for removing dynamic transactions because I don't care about performance. I'm arguing for removing dynamic transactions because I think it's too hard for authors to use them correctly. I absolutely care about performance, that is why I think we should use READ_ONLY as default. Unless we're planning on making all transactions dynamic (I hope not), locks have to be grabbed when the transaction is created, right? If a transaction is holding a READ_ONLY lock for a given objectStore, then attempting to open that objectStore as READ_WRITE should obviously fail. Consecutively, if a transaction is holding a READ_WRITE lock for a given objectStore, then opening that objectStore as READ_ONLY doesn't seem to have any benefit over opening it as READ_WRITE. In short, I can't see any situation when you'd want to open an objectStore in a different mode than what was used when the transaction was created. Finally, I would stronly prefer to have READ_ONLY be the default transaction type if none is specified by the author. It is better to default to what results in higher performance, and have people notice when they need to switch to the slower mode. This is because people will very quickly notice if they use READ_ONLY when they need to use READ_WRITE, since the code will never work. However if people use READ_WRITE when all they need is READ_ONLY, then the only effect is likely to be an application that runs somewhat slower, when they will unlikely detect. This approach is also likely to cause exceptions upon put, remove, and add. I would prefer to not cause exceptions as the default behavior. If we use READ_WRITE as default behavior then it's extremely likely that people will use the wrong lock type and not realize. The downside will be that sites will run less concurrently, and thus slower, than they could. All along our primary objective with IndexedDB is to assist programmers who are not well versed with database programming to be able to write simple programs without errors. By that token, reducing the effort required for their use of IndexedDB seems to be the primary criteria and not great concurrency. As far as I can see this does not significantly complicate development. This seems to be conveniently justified. A strict interpretation of the objective would not require the programmer to specify READ_WRITE even though that involves less mental (cognitive) and physical (typing) effort. I simply don't use as strict interpretation of the objective of simplifying development. Not every single decision we make will come down in favor of what makes a simpler API. Every decision we make is a judgement call as there are advantages and disadvantages with everything. I enumerated those advantages and disadvantages above and I think think the advantages of using READ_ONLY are stronger. In fact, in the process of writing test cases I have several times forgotten to specify lock type. For the cases when I needed a READ_WRITE lock, the code didn't work. It should have worked right the first time. Why wait for a programmer to find out why their code didn't work? We can't ever prevent all mistakes. My point is that this mistake did not meaningfully affect my ability to write working code. I think we simply have to agree to disagree. Based on that, I'm curious to know what other people think. So far it seems like mozilla and google has come out in favor of using READ_ONLY as default. / Jonas
Re: [IndexedDB] Current editor's draft
On Thu, Jul 8, 2010 at 8:27 PM, Jonas Sicking jo...@sicking.cc wrote: On Thu, Jul 8, 2010 at 8:22 AM, Andrei Popescu andr...@google.com wrote: Hi Jonas, On Wed, Jul 7, 2010 at 8:08 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary changes, it only saves data when the user explicitly requests it, for example by
Re: [IndexedDB] Current editor's draft
On Fri, Jul 9, 2010 at 3:21 AM, Andrei Popescu andr...@google.com wrote: On Thu, Jul 8, 2010 at 8:27 PM, Jonas Sicking jo...@sicking.cc wrote: On Thu, Jul 8, 2010 at 8:22 AM, Andrei Popescu andr...@google.com wrote: Hi Jonas, On Wed, Jul 7, 2010 at 8:08 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary
Re: [IndexedDB] Current editor's draft
We would not make dynamic transactions be the default since they would produce more concurrency than static scoped transactions, correct? On Jul 7, 2010, at 12:57 PM, Jonas Sicking wrote: Unless we're planning on making all transactions dynamic (I hope not), locks have to be grabbed when the transaction is created, right? If a transaction is holding a READ_ONLY lock for a given objectStore, then attempting to open that objectStore as READ_WRITE should obviously fail. Consecutively, if a transaction is holding a READ_WRITE lock for a given objectStore, then opening that objectStore as READ_ONLY doesn't seem to have any benefit over opening it as READ_WRITE. In short, I can't see any situation when you'd want to open an objectStore in a different mode than what was used when the transaction was created. Finally, I would stronly prefer to have READ_ONLY be the default transaction type if none is specified by the author. It is better to default to what results in higher performance, and have people notice when they need to switch to the slower mode. This is because people will very quickly notice if they use READ_ONLY when they need to use READ_WRITE, since the code will never work. However if people use READ_WRITE when all they need is READ_ONLY, then the only effect is likely to be an application that runs somewhat slower, when they will unlikely detect. This approach is also likely to cause exceptions upon put, remove, and add. I would prefer to not cause exceptions as the default behavior. If we use READ_WRITE as default behavior then it's extremely likely that people will use the wrong lock type and not realize. The downside will be that sites will run less concurrently, and thus slower, than they could. All along our primary objective with IndexedDB is to assist programmers who are not well versed with database programming to be able to write simple programs without errors. By that token, reducing the effort required for their use of IndexedDB seems to be the primary criteria and not great concurrency. Another downside is that authors should specify lock-type more often, for optimal performance, if we think that READ_ONLY is more common. You haven't provided any evidence about this yet. If we are using READ_ONLY as default behavior, then it's extremely likely that people will use the wrong lock type, notice that their code isn't working, and fix it. The downside is that people will have to fix their bugs. Another downside is that authors will have to specify lock-type more often if we think that READ_WRITE is more common. It is quite common in various languages to specify as a performance or safety hint when someone desires a shared lock and use a read-write version by default. To me the downsides of using READ_WRITE as a default are much worse than the downsides of using READ_ONLY. For all we know, programmers would lock the entire database when they create a transaction. If dynamic transactions appear to be a non-v1 feature, then READ_ONLY being default appears out of place.
Re: [IndexedDB] Current editor's draft
On Jul 7, 2010, at 12:57 PM, Jonas Sicking wrote:
2. Provide a catalog object that can be used to atomically add/remove
object stores and indexes as well as modify version.
It seems to me that a catalog object doesn't really provide any
functionality over the proposal in bug 10052? The advantage that I see
with the syntax proposal in bug 10052 is that it is simpler.
http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052
Can you elaborate on what the advantages are of catalog objects?
To begin with, 10052 shuts down the users of the database completely when
only one is changing its structure, i.e., adding or removing an object
store.
This is not the case. Check the steps defined for setVersion in [1].
At no point are databases shut down automatically. Only once all
existing database connections are manually closed, either by calls to
IDBDatabase.close() or by the user leaving the page, is the 'success'
event from setVersion fired.
Can you justify why one should be forced to stop using the database when
someone else is adding an object store or an index? This is what I meant by
draconian.
[1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0
How can we make it less draconian?
The 'versionchange' event allows pages that are currently using the
database to handle the change. The page can inspect the new version
number supplied by the 'versionchange' event, and if it knows that it
is compatible with a given upgrade, all it needs to do is to call
db.close() and then immediately reopen the database using
indexedDB.open(). The open call won't complete until the upgrade is
finished.
Secondly, I don't see how that
approach can produce atomic changes to the database.
When the transaction created in step 4 of setVersion defined in [1] is
created, only one IDBDatabase object to the database is open. As long
as that transaction is running, no requests returned from
IDBFactory.open will receive a 'success' event. Only once the
transaction is committed, or aborted, will those requests succeed.
This guarantees that no other IDBDatabase object can see a partial
update.
Further, only once the transaction created by setVersion is committed,
are the requested objectStores and indexes created/removed. This
guarantees that the database is never left with a partial update.
That means that the changes are atomic, right?
Atomic is not the same is isolated. Merely the fact that no other use of the
database was being made when you are changing its structure doesn't mean that
you will get all of the changes or none. What happens, for example, if the
browser crashes in the middle of the versionRequest.onsuccess handler?
Thirdly, we shouldn't
need to change version in order to perform database changes.
First off, note that if the upgrade is compatible, you can just pass
the existing database version to setVersion. So no version *change* is
actually needed.
Second, I don't think there is much difference between
var txn = db.transaction();
db.openCatalog(txn).onsuccess = ...
vs
db.setVersion(5).onsuccess = ...
I don't see that telling people that they have to use the former is a big win.
The problem that I see with the catalog proposal, if I understand it
correctly, is that it means that a page that has a IDBDatabase object
open has to always be prepared for calls to
openObjectStore/openTransaction failing. I.e. the page can't ever know
that another page was opened which at any point created a catalog and
removed an objectStore. This forces pages to at every single call
either check that the version is still the same, or that each and
every call to openObjectStore/openTransaction succeeds. This seems
very error prone to me.
We could easily add a condition check to removing an object store so that there
are no open transactions holding a lock on that object store. This would
prevent any errors of the kind you describe.
Looking at your example, it also seems like it contains a race
condition. There is a risk that when someone opens a database, the
first transaction, which uses a catalog to create the necessary
objectStores and indexes, is committed, but the second transaction,
which populates the objectStores with data, has not yet started.
I purposely wrote my example to allow database to be populated separately from
the creation of the database. There is no reason why the two couldn't be done
in the same transaction, though.
Finally, I am
not sure why you consider the syntax proposal simpler. Note that I am not
averse to the version change event notification.
Compare to how your code would look like with the proposals in bugs
9975 and 10052:
var db;
var dbRequest = indexedDB.open(parts, 'Part database');
dbRequest.onsuccess = function(event) {
db = event.result;
if (db.version != 1) {
versionRequest = db.setVersion(1);
versionRequest.ontimeout = function(event) {
throw new Error(timeout
Re: [IndexedDB] Current editor's draft
Andrei, Pejorative remarks about normative text don't help anyone. If you think that the spec text is not clear or that you are unable to interpret it, please say so. The text about dynamic scope has been around for long enough and no one so far mentioned a problem with them. Nikunj On Jul 7, 2010, at 11:11 PM, Andrei Popescu wrote: In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names).
Re: [IndexedDB] Current editor's draft
On Fri, Jul 9, 2010 at 11:05 AM, Nikunj Mehta nik...@o-micron.com wrote: We would not make dynamic transactions be the default since they would produce more concurrency than static scoped transactions, correct? On Jul 7, 2010, at 12:57 PM, Jonas Sicking wrote: I'm not sure I understand the question. We would use separate functions for creating dynamic and static transactions so there is no such thing as default. Unless we're planning on making all transactions dynamic (I hope not), locks have to be grabbed when the transaction is created, right? If a transaction is holding a READ_ONLY lock for a given objectStore, then attempting to open that objectStore as READ_WRITE should obviously fail. Consecutively, if a transaction is holding a READ_WRITE lock for a given objectStore, then opening that objectStore as READ_ONLY doesn't seem to have any benefit over opening it as READ_WRITE. In short, I can't see any situation when you'd want to open an objectStore in a different mode than what was used when the transaction was created. Finally, I would stronly prefer to have READ_ONLY be the default transaction type if none is specified by the author. It is better to default to what results in higher performance, and have people notice when they need to switch to the slower mode. This is because people will very quickly notice if they use READ_ONLY when they need to use READ_WRITE, since the code will never work. However if people use READ_WRITE when all they need is READ_ONLY, then the only effect is likely to be an application that runs somewhat slower, when they will unlikely detect. This approach is also likely to cause exceptions upon put, remove, and add. I would prefer to not cause exceptions as the default behavior. If we use READ_WRITE as default behavior then it's extremely likely that people will use the wrong lock type and not realize. The downside will be that sites will run less concurrently, and thus slower, than they could. All along our primary objective with IndexedDB is to assist programmers who are not well versed with database programming to be able to write simple programs without errors. By that token, reducing the effort required for their use of IndexedDB seems to be the primary criteria and not great concurrency. As far as I can see this does not significantly complicate development. In fact, in the process of writing test cases I have several times forgotten to specify lock type. For the cases when I needed a READ_WRITE lock, the code didn't work. As always when things don't work my first reaction was to go look at the error console which showed a uncaught exception which immediately showed what the problem was. So I argue that this does not meaningfully increase the effort required to use IndexedDB. Using the other lock type as default does however meaningfully increase the effort required to get optimal performance, which I think we should take into account. Another downside is that authors should specify lock-type more often, for optimal performance, if we think that READ_ONLY is more common. You haven't provided any evidence about this yet. Certainly. I was just enumerating all the reasons I could think of why either default would be preferable. I similarly haven't seen any evidence why write transactions are more common. Though I will note that both the example programs that you have written, as well as ones we have written for a few demos, use more read transactions than write transactions. (I can attach those if anyone is interested, though note that they are very specific to the API we currently have implemented). If we are using READ_ONLY as default behavior, then it's extremely likely that people will use the wrong lock type, notice that their code isn't working, and fix it. The downside is that people will have to fix their bugs. Another downside is that authors will have to specify lock-type more often if we think that READ_WRITE is more common. It is quite common in various languages to specify as a performance or safety hint when someone desires a shared lock and use a read-write version by default. I don't understand what you mean here. Could you elaborate? To me the downsides of using READ_WRITE as a default are much worse than the downsides of using READ_ONLY. For all we know, programmers would lock the entire database when they create a transaction. If dynamic transactions appear to be a non-v1 feature, then READ_ONLY being default appears out of place. I don't understand why the lack of dynamic transactions would affect what default lock type we should use for static transactions. In fact, if you are concerned that lack of dynamic transactions will cause people to lock the entire database and thus reduce concurrency (is that your concern?), then it seems like using READ_ONLY as default addresses that concern better than using READ_WRITE. Or am I misunderstanding something? / Jonas
Re: [IndexedDB] Current editor's draft
On Jul 8, 2010, at 12:38 AM, Jonas Sicking wrote: On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary changes, it only saves data when the user explicitly requests it, for example by pressing a 'save' button. This means that there can be a bunch of potentially important data living outside of indexedDB, in other parts of the application, such as in textfields and javascript variables. If we
Re: [IndexedDB] Current editor's draft
On Jul 8, 2010, at 4:17 AM, Shawn Wilsher wrote: On 7/6/2010 6:31 PM, Nikunj Mehta wrote: To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. How can we make it less draconian? Secondly, I don't see how that approach can produce atomic changes to the database. Thirdly, we shouldn't need to change version in order to perform database changes. Finally, I am not sure why you consider the syntax proposal simpler. Note that I am not averse to the version change event notification. In what use case would you want to change the database structure without modifying the version? That almost seems like a footgun for consumers. Can you justify your conclusion?
Re: [IndexedDB] Current editor's draft
On Jul 8, 2010, at 12:38 AM, Jonas Sicking wrote:
One of our main points was to make getting objectStore
objects a synchronous operation as to avoid having to nest multiple
levels of asynchronous calls. Compare
var req = db.openObjectStore(foo, trans);
req.onerror = errorHandler;
req.onsuccess = function(e) {
var fooStore = e.result;
var req = fooStore.get(12);
req.onerror = errorHandler;
req.onsuccess = resultHandler;
}
to
var fooStore = db.openObjectStore(foo, trans);
var req = fooStore.get(12);
req.onerror = errorHandler;
req.onsuccess = resultHandler;
I also don't understand the advantage of having the transaction as an
argument to openObjectStore rather than having openObjectStore live on
transaction. Compare
db.openObjectStore(foo, trans);
to
trans.openObjectStore(foo);
I also don't understand the meaning of specifying a mode when a
objectStore is opened, rather than specifying the mode when the
transaction is created.
Have you reviewed the examples? Different object stores in a transaction
are
used in different modes, and that requires us to identify the mode when
opening the object store. This also increases concurrency. This is
particularly useful for dynamic transactions.
I'm following you better now. I do see how this can work for dynamic
transactions where locks are not acquired upon creation of the
transaction. But I don't see how this makes sense for static
transactions. And it indeed seems like you are not using this feature
for static transactions.
The feature is targeted for use in dynamic scope.
I don't think it's even possible with the current API since
openTransaction() takes a list of objectStore names but a single mode.
Indeed. We could allow static transactions to use different lock
levels for different objectStores, all specified when the
IDBTransaction is initially created. It's just a matter of syntax to
the IDBDatabase.transaction() function. However so far I've thought
that we should leave this for v2. But if people would feel more easy
about dropping dynamic transactions if we add this ability, then I
would be ok with it.
From my examples, it was clear that we need different object stores to be
opened in different modes. Currently dynamic scope supports this use case,
i.e., allow mode specification on a per object-store basis. Therefore, unless
we decide to de-support this use case, we would need to add this ability to
static scope transactions if dynamic scope transactions go out of v1.
If it is the case that specifying a mode when opening an objectStore
only makes sense on dynamic transactions, then I think we should only
expose that argument on dynamic transactions.
Now that I understand your proposal better, I don't understand how
IDBTransaction.objectStore works for dynamically scoped transactions
in your proposal. It seems to require synchronously grabbing a lock
which I thought we wanted to avoid at all cost.
See below.
This is rather confusing: is IDBTransaction::objectStore() creating an
object store, now? If yes, then it must lock it synchronously. If it just
returns an object store that was previously added to the transaction, what
is the 'mode' parameter for?
Looking at Nikunj's example code, it seems like you can request a new
objectStore to be locked using IDBTransaction.objectStore(). Not sure
if that is a bug or not though?
That was a bug. For dynamic transactions, obtaining an object store would have
to be asynchronous as it involves obtaining a lock.
I also did not hear from you about explicit commits. Did that mean that you
agree with that part of my proposal? There are several examples where it makes
sense to explicitly commit, although it is automatic in some cases.
Re: [IndexedDB] Current editor's draft
On Jul 10, 2010, at 12:29 AM, Jonas Sicking wrote: On Fri, Jul 9, 2010 at 11:05 AM, Nikunj Mehta nik...@o-micron.com wrote: We would not make dynamic transactions be the default since they would produce more concurrency than static scoped transactions, correct? On Jul 7, 2010, at 12:57 PM, Jonas Sicking wrote: I'm not sure I understand the question. We would use separate functions for creating dynamic and static transactions so there is no such thing as default. The point is that we are talking of leaving out dynamic scope in v1, while, in the same vein, talking of making READ_ONLY the default _because_ it produces good performance. That is, IMHO, contradictory. Unless we're planning on making all transactions dynamic (I hope not), locks have to be grabbed when the transaction is created, right? If a transaction is holding a READ_ONLY lock for a given objectStore, then attempting to open that objectStore as READ_WRITE should obviously fail. Consecutively, if a transaction is holding a READ_WRITE lock for a given objectStore, then opening that objectStore as READ_ONLY doesn't seem to have any benefit over opening it as READ_WRITE. In short, I can't see any situation when you'd want to open an objectStore in a different mode than what was used when the transaction was created. Finally, I would stronly prefer to have READ_ONLY be the default transaction type if none is specified by the author. It is better to default to what results in higher performance, and have people notice when they need to switch to the slower mode. This is because people will very quickly notice if they use READ_ONLY when they need to use READ_WRITE, since the code will never work. However if people use READ_WRITE when all they need is READ_ONLY, then the only effect is likely to be an application that runs somewhat slower, when they will unlikely detect. This approach is also likely to cause exceptions upon put, remove, and add. I would prefer to not cause exceptions as the default behavior. If we use READ_WRITE as default behavior then it's extremely likely that people will use the wrong lock type and not realize. The downside will be that sites will run less concurrently, and thus slower, than they could. All along our primary objective with IndexedDB is to assist programmers who are not well versed with database programming to be able to write simple programs without errors. By that token, reducing the effort required for their use of IndexedDB seems to be the primary criteria and not great concurrency. As far as I can see this does not significantly complicate development. This seems to be conveniently justified. A strict interpretation of the objective would not require the programmer to specify READ_WRITE even though that involves less mental (cognitive) and physical (typing) effort. In fact, in the process of writing test cases I have several times forgotten to specify lock type. For the cases when I needed a READ_WRITE lock, the code didn't work. It should have worked right the first time. Why wait for a programmer to find out why their code didn't work? As always when things don't work my first reaction was to go look at the error console which showed a uncaught exception which immediately showed what the problem was. So I argue that this does not meaningfully increase the effort required to use IndexedDB. Using the other lock type as default does however meaningfully increase the effort required to get optimal performance, which I think we should take into account. There are many ways to get performance improvement, including dynamic transactions, which you seem not to be favorable towards. I don't see why READ_ONLY should be given special treatment. Another downside is that authors should specify lock-type more often, for optimal performance, if we think that READ_ONLY is more common. You haven't provided any evidence about this yet. Certainly. I was just enumerating all the reasons I could think of why either default would be preferable. I similarly haven't seen any evidence why write transactions are more common. Though I will note that both the example programs that you have written, as well as ones we have written for a few demos, use more read transactions than write transactions. (I can attach those if anyone is interested, though note that they are very specific to the API we currently have implemented). If we are using READ_ONLY as default behavior, then it's extremely likely that people will use the wrong lock type, notice that their code isn't working, and fix it. The downside is that people will have to fix their bugs. Another downside is that authors will have to specify lock-type more often if we think that READ_WRITE is more common. It is quite common in various languages to specify as a performance or safety hint when someone desires a shared lock and use a
Re: [IndexedDB] Current editor's draft
Hi Nikunj, On Fri, Jul 9, 2010 at 7:31 PM, Nikunj Mehta nik...@o-micron.com wrote: Andrei, Pejorative remarks about normative text don't help anyone. If you think that the spec text is not clear or that you are unable to interpret it, please say so. The text about dynamic scope has been around for long enough and no one so far mentioned a problem with them. I didn't mean anything disrespectful, I'm sorry if it sounded that way. I was just stating that, as far as I can tell, the spec is not clear about dynamic transactions. Thanks, Andrei
Re: [IndexedDB] Current editor's draft
On 7/9/2010 11:05 AM, Nikunj Mehta wrote: We would not make dynamic transactions be the default since they would produce more concurrency than static scoped transactions, correct? I'm still of the opinion that dynamic transactions are a bad idea because it's too easy to hold a transaction open for a long period of time, making it easy for programmers to write incorrect/buggy code. All along our primary objective with IndexedDB is to assist programmers who are not well versed with database programming to be able to write simple programs without errors. By that token, reducing the effort required for their use of IndexedDB seems to be the primary criteria and not great concurrency. Er, I thought we were targeting libraries but still wanted this to be simple to use. Regardless, allowing for more concurrency doesn't hurt ease of use in any of what we've discussed so far (as far as I can tell). Another downside is that authors should specify lock-type more often, for optimal performance, if we think that READ_ONLY is more common. You haven't provided any evidence about this yet. I can assert that all the demos I've written (admittedly not many), simply reading from the database has been far more common than writing to it. It's pretty much write data in but then do all operations on the local database. It is quite common in various languages to specify as a performance or safety hint when someone desires a shared lock and use a read-write version by default. You seem to have contradictory statements here. Earlier you argued that reducing the effort required for their [programmers] use of IndexedDB seems to be the primary criteria, but having them to read and know performance or safety hints seems to me like we are making the API more complicated. Having simple clear API calls with sensible error messages on misuse is far better than having an API that you can use one way (the most common way) but would be more efficient if you use it another way. For all we know, programmers would lock the entire database when they create a transaction. If dynamic transactions appear to be a non-v1 feature, then READ_ONLY being default appears out of place. You asked Jonas for data backing up his claims, and now I'm going to ask the same of you. It's been my experience and Ben Turner's experience that defaulting to READ_ONLY results in less code being written using this API. Reads are far more common than writes (in databases in general, I think although edge cases certainly exist), so it makes sense to make that the default. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Current editor's draft
On 7/9/2010 12:50 PM, Nikunj Mehta wrote: The point is that we are talking of leaving out dynamic scope in v1, while, in the same vein, talking of making READ_ONLY the default _because_ it produces good performance. That is, IMHO, contradictory. Dynamic scope == dynamic transactions, correct? Can you please elaborate on how dynamic transactions improve performance? This seems to be conveniently justified. A strict interpretation of the objective would not require the programmer to specify READ_WRITE even though that involves less mental (cognitive) and physical (typing) effort. FWIW, this isn't constructive. Your argument also seems to assume that writes are more common than reads, which hasn't been my experience with this API. It should have worked right the first time. Why wait for a programmer to find out why their code didn't work? Why make writing code with good performance characteristics the uncommon case? We clearly have two different schools of thoughts here, and I'm not sure we are going to find a consensus... There are many ways to get performance improvement, including dynamic transactions, which you seem not to be favorable towards. I don't see why READ_ONLY should be given special treatment. It's unclear to me why you think READ_ONLY is getting special treatment (or what that even means in this context). Various hints are used in SQL syntax, e.g., [1], to manage locks, a certain kind of B-tree behavior, or a level of isolation. These are all aimed at improving performance, but they are set as default behavior. My point is that expecting good performance from a single variable in database systems out of possibly hundreds is not all that helpful. It is also a slippery slope because it confuses performance with options. The database's job is to be fast at what it does, not play performance tricks using default values. While I think this argument makes sense in some cases, I really don't feel like it applies at all in this situation. Even the page you linked to says Because the SQL Server query optimizer typically selects the best execution plan for a query... which in most cases would be READ_ONLY (based on evidence the Mozilla team has from demos written). The default should represent the commonly used case. I don't know that we can make the right performance decisions for people. What we can do is make things perform well and provide tools to improve performance. We aren't making performance decisions though; we are just picking the default to be the most commonly used option. It just happens to be the one to allow more concurrency. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Current editor's draft
On Fri, Jul 9, 2010 at 11:27 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 7, 2010, at 12:57 PM, Jonas Sicking wrote: 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. Can you justify why one should be forced to stop using the database when someone else is adding an object store or an index? This is what I meant by draconian. In the thread [IndexedDB] Atomic schema changes the conclusion was that while it's possible to create API that allows compatible schema changes without calling setVersion, it wasn't worth the added API complexity. It's something that can wait for v2. Are calls to add indexes and objectStores without further incompatible changes common enough that we need to optimize the case when users have multiple tabs open? Remember that calling setVersion really is no overhead if only one copy of the application is running. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. Secondly, I don't see how that approach can produce atomic changes to the database. When the transaction created in step 4 of setVersion defined in [1] is created, only one IDBDatabase object to the database is open. As long as that transaction is running, no requests returned from IDBFactory.open will receive a 'success' event. Only once the transaction is committed, or aborted, will those requests succeed. This guarantees that no other IDBDatabase object can see a partial update. Further, only once the transaction created by setVersion is committed, are the requested objectStores and indexes created/removed. This guarantees that the database is never left with a partial update. That means that the changes are atomic, right? Atomic is not the same is isolated. Merely the fact that no other use of the database was being made when you are changing its structure doesn't mean that you will get all of the changes or none. What happens, for example, if the browser crashes in the middle of the versionRequest.onsuccess handler? Yeah, I forgot one critical piece of information. Like with any other transaction. If the transaction isn't successfully committed, it is fully rolled back and no changes whatsoever are made to the database. Thirdly, we shouldn't need to change version in order to perform database changes. First off, note that if the upgrade is compatible, you can just pass the existing database version to setVersion. So no version *change* is actually needed. Second, I don't think there is much difference between var txn = db.transaction(); db.openCatalog(txn).onsuccess = ... vs db.setVersion(5).onsuccess = ... I don't see that telling people that they have to use the former is a big win. The problem that I see with the catalog proposal, if I understand it correctly, is that it means that a page that has a IDBDatabase object open has to always be prepared for calls to openObjectStore/openTransaction failing. I.e. the page can't ever know that another page was opened which at any point created a catalog and removed an objectStore. This forces pages to at every single call either check that the version is still the same, or that each and every call to openObjectStore/openTransaction succeeds. This seems very error prone to me. We could easily add a condition check to removing an object store so that there are no open transactions holding a lock on that object store. This would prevent any errors of the kind you describe. I don't see that that would help. You're still in the situation that every time you're starting a transaction you have to check that all the objectStores you were expecting to be there still are there. Am I missing something? Looking
Re: [IndexedDB] Current editor's draft
On Fri, Jul 9, 2010 at 11:44 AM, Nikunj Mehta nik...@o-micron.com wrote: On Jul 8, 2010, at 12:38 AM, Jonas Sicking wrote: On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary changes, it only saves data when the user explicitly requests it, for example by pressing a 'save' button. This means that there can be a bunch of potentially important data living outside of indexedDB, in other parts of the application, such as in
Re: [IndexedDB] Current editor's draft
Hi Jonas, On Wed, Jul 7, 2010 at 8:08 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary changes, it only saves data when the user explicitly requests it, for example by pressing a 'save' button. This means that there can be a bunch of potentially important data living outside of indexedDB, in other parts of the
Re: [IndexedDB] Current editor's draft
On Thu, Jul 8, 2010 at 8:22 AM, Andrei Popescu andr...@google.com wrote: Hi Jonas, On Wed, Jul 7, 2010 at 8:08 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary changes, it only saves data when the user explicitly requests it, for example by pressing a 'save' button. This means that there can be a bunch of
Re: [IndexedDB] Current editor's draft
On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. Secondly, I don't see how that approach can produce atomic changes to the database. When the transaction created in step 4 of setVersion defined in [1] is created, only one IDBDatabase object to the database is open. As long as that transaction is running, no requests returned from IDBFactory.open will receive a 'success' event. Only once the transaction is committed, or aborted, will those requests succeed. This guarantees that no other IDBDatabase object can see a partial update. Further, only once the transaction created by setVersion is committed, are the requested objectStores and indexes created/removed. This guarantees that the database is never left with a partial update. That means that the changes are atomic, right? Thirdly, we shouldn't need to change version in order to perform database changes. First off, note that if the upgrade is compatible, you can just pass the existing database version to setVersion. So no version *change* is actually needed. Second, I don't think there is much difference between var txn = db.transaction(); db.openCatalog(txn).onsuccess = ... vs db.setVersion(5).onsuccess = ... I don't see that telling people that they have to use the former is a big win. The problem that I see with the catalog proposal, if I understand it correctly, is that it means that a page that has a IDBDatabase object open has to always be prepared for calls to openObjectStore/openTransaction failing. I.e. the page can't ever know that another page was opened which at any point created a catalog and removed an objectStore. This
Re: [IndexedDB] Current editor's draft
On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. Secondly, I don't see how that approach can produce atomic changes to the database. When the transaction created in step 4 of setVersion defined in [1] is created, only one IDBDatabase object to the database is open. As long as that transaction is running, no requests returned from IDBFactory.open will receive a 'success' event. Only once the transaction is committed, or aborted, will those requests succeed. This guarantees that no other IDBDatabase object can see a partial update. Further, only once the transaction created by setVersion is committed, are the requested objectStores and indexes
Re: [IndexedDB] Current editor's draft
On Wed, Jul 7, 2010 at 10:41 AM, Andrei Popescu andr...@google.com wrote: On Wed, Jul 7, 2010 at 8:27 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 6:31 PM, Nikunj Mehta nik...@o-micron.com wrote: On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote: On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote: Hi folks, There are several unimplemented proposals on strengthening and expanding IndexedDB. The reason I have not implemented them yet is because I am not convinced they are necessary in toto. Here's my attempt at explaining why. I apologize in advance for not responding to individual proposals due to personal time constraints. I will however respond in detail on individual bug reports, e.g., as I did with 9975. I used the current editor's draft asynchronous API to understand where some of the remaining programming difficulties remain. Based on this attempt, I find several areas to strengthen, the most prominent of which is how we use transactions. Another is to add the concept of a catalog as a special kind of object store. Hi Nikunj, Thanks for replying! I'm very interested in getting this stuff sorted out pretty quickly as almost all other proposals in one way or another are affected by how this stuff develops. Here are the main areas I propose to address in the editor's spec: 1. It is time to separate the dynamic and static scope transaction creation so that they are asynchronous and synchronous respectively. I don't really understand what this means. What are dynamic and static scope transaction creation? Can you elaborate? This is the difference in the API in my email between openTransaction and transaction. Dynamic and static scope have been defined in the spec for a long time. In fact, dynamic transactions aren't explicitly specified anywhere. They are just mentioned. You need some amount of guessing to find out what they are or how to create one (i.e. pass an empty list of store names). Yes, that has been a big problem for us too. Ah, I think I'm following you now. I'm actually not sure that we should have dynamic scope at all in the spec, I know Jeremy has expressed similar concerns. However if we are going to have dynamic scope, I agree it is a good idea to have separate APIs for starting dynamic-scope transactions from static-scope transactions. I think it would simplify matters a lot if we were to drop dynamic transactions altogether. And if we do that, then we can also safely move the 'mode' to parameter to the Transaction interface, since all the object stores in a static transaction can be only be open in the same mode. Agreed. 2. Provide a catalog object that can be used to atomically add/remove object stores and indexes as well as modify version. It seems to me that a catalog object doesn't really provide any functionality over the proposal in bug 10052? The advantage that I see with the syntax proposal in bug 10052 is that it is simpler. http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052 Can you elaborate on what the advantages are of catalog objects? To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. This is not the case. Check the steps defined for setVersion in [1]. At no point are databases shut down automatically. Only once all existing database connections are manually closed, either by calls to IDBDatabase.close() or by the user leaving the page, is the 'success' event from setVersion fired. [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052#c0 How can we make it less draconian? The 'versionchange' event allows pages that are currently using the database to handle the change. The page can inspect the new version number supplied by the 'versionchange' event, and if it knows that it is compatible with a given upgrade, all it needs to do is to call db.close() and then immediately reopen the database using indexedDB.open(). The open call won't complete until the upgrade is finished. I had a question here: why does the page need to call 'close'? Any pending transactions will run to completion and new ones should not be allowed to start if a VERSION_CHANGE transaction is waiting to start. From the description of what 'close' does in 10052, I am not entirely sure it is needed. The problem we're trying to solve is this: Imagine an editor which stores documents in indexedDB. However in order to not overwrite the document using temporary changes, it only saves data when the user explicitly requests it, for example by pressing a 'save' button. This means that there can be a bunch of potentially important data living outside of indexedDB, in other parts of the application, such as in textfields and javascript variables. If we were to automatically
Re: [IndexedDB] Current editor's draft
On 7/6/2010 6:31 PM, Nikunj Mehta wrote: To begin with, 10052 shuts down the users of the database completely when only one is changing its structure, i.e., adding or removing an object store. How can we make it less draconian? Secondly, I don't see how that approach can produce atomic changes to the database. Thirdly, we shouldn't need to change version in order to perform database changes. Finally, I am not sure why you consider the syntax proposal simpler. Note that I am not averse to the version change event notification. In what use case would you want to change the database structure without modifying the version? That almost seems like a footgun for consumers. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Current editor's draft
On 7/7/2010 12:27 AM, Jonas Sicking wrote: This interface allows asynchronously requesting more objectStores to be locked. The author must take care whenever calling openObjectStores that the request might fail due to deadlocks. But as previously stated, I think this adds too much complexity and too much racyness to the API. And so I'd prefer to not add this. I feel like we should not be creating an API that allows for deadlocks to happen. Especially with an API that allows for races to happen (which we have) such that it will be hard for web developers to test to ensure they do not have deadlocks. Cheers, Shawn smime.p7s Description: S/MIME Cryptographic Signature
Re: [IndexedDB] Current editor's draft
On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote:
Hi folks,
There are several unimplemented proposals on strengthening and
expanding IndexedDB. The reason I have not implemented them yet is
because I am not convinced they are necessary in toto. Here's my
attempt at explaining why. I apologize in advance for not responding
to individual proposals due to personal time constraints. I will
however respond in detail on individual bug reports, e.g., as I did
with 9975.
I used the current editor's draft asynchronous API to understand where
some of the remaining programming difficulties remain. Based on this
attempt, I find several areas to strengthen, the most prominent of
which is how we use transactions. Another is to add the concept of a
catalog as a special kind of object store.
Hi Nikunj,
Thanks for replying! I'm very interested in getting this stuff sorted
out pretty quickly as almost all other proposals in one way or another
are affected by how this stuff develops.
Here are the main areas I propose to address in the editor's spec:
1. It is time to separate the dynamic and static scope transaction
creation so that they are asynchronous and synchronous respectively.
I don't really understand what this means. What are dynamic and static
scope transaction creation? Can you elaborate?
2. Provide a catalog object that can be used to atomically add/remove
object stores and indexes as well as modify version.
It seems to me that a catalog object doesn't really provide any
functionality over the proposal in bug 10052? The advantage that I see
with the syntax proposal in bug 10052 is that it is simpler.
http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052
Can you elaborate on what the advantages are of catalog objects?
3. Cursors may produce a null key or a null value. I don't see how
this is valid signaling for non-preloaded cursors. I think we need to
add a new flag on the cursor to find out if the cursor is exhausted.
Our proposal was that IDBEvent.result would normally contain the
cursor object, but once the end is reached it returns null. To be
clear:
When a value is found:
event.result; // returns cursor object, never null
event.result.key; // returns key, may be null
event.result.value; // returns value, may be null
When end is reached:
event.result; // returns null
A couple of additional points:
1. I did not see any significant benefits of preloaded cursors in
terms of programming ease.
Yes, there seems to be agreement that preloaded cursors should be
removed. I've removed them from our proposal.
2. *_NO_DUPLICATE simplifies programming as well as aids in good
performance. I have shown one example that illustrates this.
I'll have to analyze the examples below. My gut instinct though is
that I agree with you that they are needed.
3. Since it seems continue is acceptable to implementers, I am also
suggesting we use delete instead of remove, for consistency sake.
Agreed.
--- IDL
[NoInterfaceObject]
interface IDBDatabase {
readonly attribute DOMString name;
readonly attribute DOMString description;
readonly attribute DOMStringList objectStores;
/*
Open an object store in the specified transaction. The transaction can be
dynamic scoped, or the requested object store must be in the static scope.
Returns IDBRequest whose success event of IDBTransactionEvent type contains
a result with IDBObjectStore and transaction is an IDBTransactionRequest.
*/
IDBRequest openObjectStore(in DOMString name, in IDBTransaction txn,
in optional unsigned short mode /* defaults to READ_WRITE */);
I don't understand the advantage of this proposal over mozillas
proposal. One of our main points was to make getting objectStore
objects a synchronous operation as to avoid having to nest multiple
levels of asynchronous calls. Compare
var req = db.openObjectStore(foo, trans);
req.onerror = errorHandler;
req.onsuccess = function(e) {
var fooStore = e.result;
var req = fooStore.get(12);
req.onerror = errorHandler;
req.onsuccess = resultHandler;
}
to
var fooStore = db.openObjectStore(foo, trans);
var req = fooStore.get(12);
req.onerror = errorHandler;
req.onsuccess = resultHandler;
I also don't understand the advantage of having the transaction as an
argument to openObjectStore rather than having openObjectStore live on
transaction. Compare
db.openObjectStore(foo, trans);
to
trans.openObjectStore(foo);
I also don't understand the meaning of specifying a mode when a
objectStore is opened, rather than specifying the mode when the
transaction is created. Unless we're planning on making all
transactions dynamic (I hope not), locks have to be grabbed when the
transaction is created, right? If a transaction is holding a READ_ONLY
lock for a given objectStore, then attempting to open that objectStore
as READ_WRITE should obviously fail. Consecutively, if a transaction
is holding a READ_WRITE lock for a given
Re: [IndexedDB] Current editor's draft
On Wed, Jul 7, 2010 at 5:57 AM, Jonas Sicking jo...@sicking.cc wrote:
On Tue, Jul 6, 2010 at 9:36 AM, Nikunj Mehta nik...@o-micron.com wrote:
Hi folks,
There are several unimplemented proposals on strengthening and
expanding IndexedDB. The reason I have not implemented them yet is
because I am not convinced they are necessary in toto. Here's my
attempt at explaining why. I apologize in advance for not responding
to individual proposals due to personal time constraints. I will
however respond in detail on individual bug reports, e.g., as I did
with 9975.
I used the current editor's draft asynchronous API to understand where
some of the remaining programming difficulties remain. Based on this
attempt, I find several areas to strengthen, the most prominent of
which is how we use transactions. Another is to add the concept of a
catalog as a special kind of object store.
Hi Nikunj,
Thanks for replying! I'm very interested in getting this stuff sorted
out pretty quickly as almost all other proposals in one way or another
are affected by how this stuff develops.
Here are the main areas I propose to address in the editor's spec:
1. It is time to separate the dynamic and static scope transaction
creation so that they are asynchronous and synchronous respectively.
I don't really understand what this means. What are dynamic and static
scope transaction creation? Can you elaborate?
This is the difference in the API in my email between openTransaction and
transaction. Dynamic and static scope have been defined in the spec for a
long time.
2. Provide a catalog object that can be used to atomically add/remove
object stores and indexes as well as modify version.
It seems to me that a catalog object doesn't really provide any
functionality over the proposal in bug 10052? The advantage that I see
with the syntax proposal in bug 10052 is that it is simpler.
http://www.w3.org/Bugs/Public/show_bug.cgi?id=10052
Can you elaborate on what the advantages are of catalog objects?
To begin with, 10052 shuts down the users of the database completely when
only one is changing its structure, i.e., adding or removing an object
store. How can we make it less draconian? Secondly, I don't see how that
approach can produce atomic changes to the database. Thirdly, we shouldn't
need to change version in order to perform database changes. Finally, I am
not sure why you consider the syntax proposal simpler. Note that I am not
averse to the version change event notification.
3. Cursors may produce a null key or a null value. I don't see how
this is valid signaling for non-preloaded cursors. I think we need to
add a new flag on the cursor to find out if the cursor is exhausted.
Our proposal was that IDBEvent.result would normally contain the
cursor object, but once the end is reached it returns null. To be
clear:
When a value is found:
event.result; // returns cursor object, never null
event.result.key; // returns key, may be null
event.result.value; // returns value, may be null
When end is reached:
event.result; // returns null
Got it. I will try out this approach.
A couple of additional points:
1. I did not see any significant benefits of preloaded cursors in
terms of programming ease.
Yes, there seems to be agreement that preloaded cursors should be
removed. I've removed them from our proposal.
2. *_NO_DUPLICATE simplifies programming as well as aids in good
performance. I have shown one example that illustrates this.
I'll have to analyze the examples below. My gut instinct though is
that I agree with you that they are needed.
3. Since it seems continue is acceptable to implementers, I am also
suggesting we use delete instead of remove, for consistency sake.
Agreed.
--- IDL
[NoInterfaceObject]
interface IDBDatabase {
readonly attribute DOMString name;
readonly attribute DOMString description;
readonly attribute DOMStringList objectStores;
/*
Open an object store in the specified transaction. The transaction can
be
dynamic scoped, or the requested object store must be in the static
scope.
Returns IDBRequest whose success event of IDBTransactionEvent type
contains
a result with IDBObjectStore and transaction is an
IDBTransactionRequest.
*/
IDBRequest openObjectStore(in DOMString name, in IDBTransaction txn,
in optional unsigned short mode /* defaults to READ_WRITE */);
I don't understand the advantage of this proposal over mozillas
proposal.
The above proposal allows opening multiple object stores in the same
transaction in dynamic scope, even without having explicitly identified each
one of them at the time of creating the transaction. Secondly, where static
scope is desired, in order to run to completion without being aborted due to
unavailable objects, this proposal ensures that locks are obtained at the
time of creating the transaction.
My
