Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Tue, Mar 19, 2013 at 3:25 PM, Alec Flett wrote:
> Sorry in advance for the long reply here..
> TL;DR is:
> 1) I still don't think transactions are a requirement
> 2) I think fixing the platform's motley crew of async apis, and giving
> developers better control over transaction commits (When you do use them) is
> probably most important.
I agree with 2, at least mostly. But I don't think that means that we
need to give up on transactions. But I suspect we need to agree to
disagree on the meta-point and see if we can reach agreement on the
API level instead.
> On Tue, Mar 19, 2013 at 1:52 PM, Jonas Sicking wrote:
>>
>>
>> var a;
>> trans = db.transaction();
>> trans.get("a").onsuccess = function(e) {
>> a = e.target.result;
>> }
>> trans.get("b").onsuccess = function(e) {
>> trans.set("b", e.target.result + a);
>> }
>>
> I'll be honest: as a web developer who has been bitten numerous times with
> coordinating setTimeout and XHR callbacks, I'd never trust that the above
> worked at all, even with explicit transactions.
If we don't trust implementations to follow the specs, then I think
we're hosed no matter what :(
> I just don't think you'd ever write code like that because you're relying
> with the subtleties of how scoping works in JS, in addition to the order
> guarantees of IndexedDB.
>
> you'd write this instead:
>
> db.objectStore("foo").get("a").onsuccess = function(e) {
> var a = e.target.result;
>
> db.objectStore("foo").get("b").onsuccess = function(e) {
> db.objectStore("foo").set("b", e.target.result + a);
> }
> }
>
> But this is still a degenerate contrived example that I just don't believe
> is representative of real-world code. We'd be optimizing for a fairly
> specific pattern and I think people are far more often bit by
> auto-committing transactions than they are by races like this. If anything,
> XHR and setTimeout (and all the new-fangled HTML5 APIs) have taught people
> to be careful about races in async apis.
I agree it's somewhat of an edge case. But my point is that it's very
easy and subtle to slide from the "ok" code patterns to the "racy"
code patterns.
But like I said, I think the explicit transaction doesn't need to be
meaningfully more complex than the implicit one.
>> But I don't think that requires that we get rid of transactions from
>> the simple API. And I suspect that that doesn't need to meaningfully
>> need to make the simple API that much more complicated.
>
> It seems like there are two kinds of "races" that we're talking about here:
> database races (i.e. read/writes not being atomic, the "A" in ACID) and
> event races (i.e. any two arbitrary operations not having guaranteed order,
> the "I" in ACID) - I think the latter is often solved with a better
> asynchronous API abstraction like Futures/Promises - i.e. an async pattern
> that lets you be explicit about your ordering rather than relying on a
> containing abstraction like transactions.
I'm generally talking about the "A" part.
I agree that we should try to solve the "I" part using promises. But
even with promises I think we should guarantee a callback order.
People will come to rely on callback order unintentionally, even if we
use Futures. And in many cases it will enable simpler code patterns.
People can choose not to rely on it if they want to. I'm all for code clarity.
Also, I'm worried that solving the "I" part using promises will force
us to sacrifice the "A" part. I hope we don't.
> The following pattern going to be far more common:
>
> var key = ...
> xhr1.open("/url?key=" + key);
> xhr1.onsuccess = function(e) {
> var xhrValue = xhr1.responseText;
> indexedDB.get(key).onsuccess = function(e) {
> if (keyValue.value != e.target.result) {
>// update my cache...
> }
>}
> }
>
> but ultimately this is still ugly because you're serializing your operations
> and it's complicated to write code that runs them both in parallel and only
> compares them when both callbacks have fired. (Nevermind the fact that if we
> were dealing with our current auto-committing transactions, any open
> transaction would have committed while we were waiting for the XHR response)
>
> but with futures/promises and nice libraries like q.js you can imagine stuff
> like:
>
> Q.all([
> xhr1.open("/get?key=" + key)
> indexedDB.get(key)
> ])
> .spread(function(responseText, idbValue) {
> if (responseText != idbValue) {
> // update my cache...
> }
> });
>
> Bam. Ordering races are gone.
Like I said above, the ordering races is not what I'm trying to solve.
As far as I can tell IDB handles those just fine and I've never heard
complaints from people that are having IDB code with ordering races.
You'll note that the original complaint in this thread was that code
that *looks* like it's suffering from "I" races actually isn't. That's
a problem that I can live with :)
/ Jonas
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Tue, Mar 19, 2013 at 2:25 PM, Robert Ginda wrote: > One option that will probably never happen: Stop making jank-inducing API's > available on the main thread. I think we're already doing this. As far as I know all browser vendors are on board with the idea of not having APIs which run on the main thread and which are slow enough that they can introduce jank. Including anything that does IO, but also things like encryption and image decoding/encoding. Unfortunately we already have a pretty large debt in form of janky APIs :( > Do them all as synchronous APIs only, and > available only to workers. This part is more controversial I think. Forcing people to use workers even in simple cases might not be popular. There's also the issue that synchronous APIs generally have worse performance. But it would need to be measured. / Jonas / Jonas
Re: IndexedDB, what were the issues? How do we stop it from happening again?
Sorry in advance for the long reply here..
TL;DR is:
1) I still don't think transactions are a requirement
2) I think fixing the platform's motley crew of async apis, and giving
developers better control over transaction commits (When you do use them)
is probably most important.
On Tue, Mar 19, 2013 at 1:52 PM, Jonas Sicking wrote:
>
> var a;
> trans = db.transaction();
> trans.get("a").onsuccess = function(e) {
> a = e.target.result;
> }
> trans.get("b").onsuccess = function(e) {
> trans.set("b", e.target.result + a);
> }
>
> I'll be honest: as a web developer who has been bitten numerous times with
coordinating setTimeout and XHR callbacks, I'd never trust that the above
worked at all, even with explicit transactions.
I just don't think you'd ever write code like that because you're relying
with the subtleties of how scoping works in JS, in addition to the order
guarantees of IndexedDB.
you'd write this instead:
db.objectStore("foo").get("a").onsuccess = function(e) {
var a = e.target.result;
db.objectStore("foo").get("b").onsuccess = function(e) {
db.objectStore("foo").set("b", e.target.result + a);
}
}
But this is still a degenerate contrived example that I just don't believe
is representative of real-world code. We'd be optimizing for a fairly
specific pattern and I think people are far more often bit by
auto-committing transactions than they are by races like this. If anything,
XHR and setTimeout (and all the new-fangled HTML5 APIs) have taught people
to be careful about races in async apis.
> But I don't think that requires that we get rid of transactions from
> the simple API. And I suspect that that doesn't need to meaningfully
> need to make the simple API that much more complicated.
>
>
It seems like there are two kinds of "races" that we're talking about here:
database races (i.e. read/writes not being atomic, the "A" in ACID) and
event races (i.e. any two arbitrary operations not having guaranteed order,
the "I" in ACID) - I think the latter is often solved with a better
asynchronous API abstraction like Futures/Promises - i.e. an async pattern
that lets you be explicit about your ordering rather than relying on a
containing abstraction like transactions.
I mean imagine your same identical code with XHR (drastically simplified,
but hopefully you get the idea)
xhr1.open("/url?key=key1");
xhr1.onsuccess = function(e) {
a = xhr1.responseText;
}
xhr2.open("/url?key=key2");
xhr2.onsuccess = function(e) {
xhr3.open("/update?first=" + a + "&second=" + xhr2.responseText);
}
in the context of XHR, it's now obvious to everyone watching that this is a
race condition.. just a very crude one. I guess I'm doing this to
demonstrate why no developer worth their salt would purposefully write the
race condition that you're afraid may happen without transactions.
No other Async api has a notion of "transactions" to work around races due
to async responses. If that's our concern, then we should be focusing on
getting to Futures/Promises. Having transactions doesn't solve races
between async subsystems, like when using XHR + IDB together.
The following pattern going to be far more common:
var key = ...
xhr1.open("/url?key=" + key);
xhr1.onsuccess = function(e) {
var xhrValue = xhr1.responseText;
indexedDB.get(key).onsuccess = function(e) {
if (keyValue.value != e.target.result) {
// update my cache...
}
}
}
but ultimately this is still ugly because you're serializing your
operations and it's complicated to write code that runs them both in
parallel and only compares them when both callbacks have fired. (Nevermind
the fact that if we were dealing with our current auto-committing
transactions, any open transaction would have committed while we were
waiting for the XHR response)
but with futures/promises and nice libraries like q.js you can imagine
stuff like:
Q.all([
xhr1.open("/get?key=" + key)
indexedDB.get(key)
])
.spread(function(responseText, idbValue) {
if (responseText != idbValue) {
// update my cache...
}
});
Bam. Ordering races are gone.
Alec
Re: IndexedDB, what were the issues? How do we stop it from happening again?
One option that will probably never happen: Stop making jank-inducing API's available on the main thread. Do them all as synchronous APIs only, and available only to workers. Your APIs stay simple, and you don't have to make double the surface area. Building a library that marshals sync APIs to the main thread over async MessagePorts is tractable, and may the best implementation win. With enough standardization across APIs you may even be able to make a generalized broker. Rob. On Thu, Mar 14, 2013 at 6:58 PM, Tab Atkins Jr. wrote: > On Thu, Mar 14, 2013 at 6:36 PM, Glenn Maynard wrote: > > On Thu, Mar 14, 2013 at 1:54 PM, Alex Russell > > wrote: > >> I don't understand why that's true. Workers have a message-oriented API > >> that's inherently async. They can get back to their caller "whenevs". > What's > >> the motivator for needing this? > > > > Being able to write synchronous code is one of the basic uses for > Workers in > > the first place. Synchronously creating streams is useful in the same > way > > that other synchronous APIs are useful, such as FileReaderSync. > > > > That doesn't necessarily mean having a synchronous API for a complex > > interface like this is the ideal approach (there are other ways to do > it), > > but that's the end goal. > > Yes, this seems to be missing the point of Workers entirely. If all > you have are async apis, you don't need Workers in the first place, as > you can just use them in the main thread without jank. Workers exist > explicitly to allow you to do expensive synchronous stuff without > janking the main thread. (Often, the expensive synchronous stuff will > just be a bunch of calculations, so you don't have to explicitly break > it up into setTimeout-able chunks.) > > The entire reason for most async (all?) APIs is thus irrelevant in a > Worker, and it may be a good idea to provide sync versions, or do > something else that negates the annoyance of dealing with async code. > > > (FYI, the messaging in Workers isn't inherently async; it just happens to > > only have an async interface. There's been discussion about adding a > > synchronous interface to messaging.) > > Specifically, this was for workers to be able to synchronously wait > for messages from their sub-workers. Again, the whole point for async > worker messaging is to prevent the main thread from janking, which is > irrelevant inside of a worker. > > ~TJ > >
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Mon, Mar 18, 2013 at 5:42 PM, Alec Flett wrote:
>> transactions - Also should be optional. Vital to complex apps, but totally
>>
>> > not necessary for many.. there should be a default transaction, like
>> > db.objectStore("foo").get("bar")
>>
>> I disagree. This would have made it too trivial to create pages that
>> have race conditions. I.e. people would write code like:
>>
>> db.objectStore("foo").get("bar").onsuccess = function(e) {
>> db.objectStore("foo").set("bar", e.target.result + 1)
>> }
>>
>> without realizing that that contains a race condition.
>>
> Its always possible for users to hang themselves - the web platform has lots
> of rope. I could name a dozen gotchas like that in the JavaScript language
> alone. The fact that we introduced shared workers introduces a whole mess of
> issues like that. Not to criticize either - I think its just something that
> happens as you introduce more flexible capabilities into the platform.
>
> In the above example, you could approach this with automatic transactions -
> all operations that run in the callback of another IDB operation run in the
> same transaction. So the set() and the get() are in the same transaction.
That stops working pretty quickly. If you simply want to add property
'a' to property 'b' you'd get something like:
var a;
db.objectStore("foo").get("a").onsuccess = function(e) {
a = e.target.result;
}
db.objectStore("foo").get("b").onsuccess = function(e) {
db.objectStore("foo").set("b", e.target.result + a);
}
And yes, it's easy to write that code correctly, but I don't think it
passes the following design goal:
* Make it easy to create pages that are race-free even if opened in
multiple tabs at the same time. Ideally it should be easier to create
a race-free page than a page that has race hazards.
> When you need explicit transactional control then you use the transaction()
> API.
My point is that it's very common to want transactions, even when you
don't think you want to.
I'd much rather spend effort on making it so easy to create
transactions that it's something people don't mind doing, than to
create a transaction-less syntax.
If we enable the use of a "default" objectStore then your example code
would become
var trans = db.transaction();
trans.get("bar").onsuccess = function(e) {
trans.set("bar", e.target.result + 1)
}
And my example code
var a;
trans = db.transaction();
trans.get("a").onsuccess = function(e) {
a = e.target.result;
}
trans.get("b").onsuccess = function(e) {
trans.set("b", e.target.result + a);
}
Which in both cases is just one additional line of code, but fewer
total number of characters typed. I think that's a win compared to
something that will result in more race conditions.
>> > named object stores - frankly, for many use cases, a single objectStore
>> > is
>> > all you need. a simple db.get("foo") would be sufficient. Simply naming
>> > a
>> > "default" isn't bad - whats bad is all the onupgradeneeded scaffolding
>> > required to create the objectstore in the first place.
>>
>> I think we should do this as part of a simple API. Similar to something
>> like
>>
>> https://github.com/slightlyoff/async-local-storage
>>
>
> Yes! I mean that's kind of where this conversation took off... I just don't
> think there should be an obvious distinction between "the API with the
> transactions and versions" and "the one without." - if anything presenting
> them in a unified fashion allows for developers to migrate as they need
> individual features (Transactions, versions, etc)
I definitely agree that it would be cool if indexedDB could have a
smooth transition curve from a simple API up to the full feature set
that it currently has. That's definitely something that I think we
failed on.
But I don't think that requires that we get rid of transactions from
the simple API. And I suspect that that doesn't need to meaningfully
need to make the simple API that much more complicated.
/ Jonas
Re: IndexedDB, what were the issues? How do we stop it from happening again?
> transactions - Also should be optional. Vital to complex apps, but totally
> > not necessary for many.. there should be a default transaction, like
> > db.objectStore("foo").get("bar")
>
> I disagree. This would have made it too trivial to create pages that
> have race conditions. I.e. people would write code like:
>
> db.objectStore("foo").get("bar").onsuccess = function(e) {
> db.objectStore("foo").set("bar", e.target.result + 1)
> }
>
> without realizing that that contains a race condition.
>
> Its always possible for users to hang themselves - the web platform has
lots of rope. I could name a dozen gotchas like that in the JavaScript
language alone. The fact that we introduced shared workers introduces a
whole mess of issues like that. Not to criticize either - I think its just
something that happens as you introduce more flexible capabilities into the
platform.
In the above example, you could approach this with automatic transactions -
all operations that run in the callback of another IDB operation run in the
same transaction. So the set() and the get() are in the same transaction.
When you need explicit transactional control then you use the transaction()
API.
> > transaction scoping - even when you do want transactions, the api is just
> > too verbose and repetitive for "get one key from one object store" -
> > db.transaction("foo").objectStore("foo").get("bar") - there should be
> > implicit (lightweight) transactions like db.objectStore("foo").get("bar")
>
> We used to have this exact syntax, but it got everyone confused about
> what how the implicit transaction actually worked.
>
> This is surprising to me - *shrug* - I assume it was like the automatic
transactions I mentioned above.
>
> > named object stores - frankly, for many use cases, a single objectStore
> is
> > all you need. a simple db.get("foo") would be sufficient. Simply naming a
> > "default" isn't bad - whats bad is all the onupgradeneeded scaffolding
> > required to create the objectstore in the first place.
>
> I think we should do this as part of a simple API. Similar to something
> like
>
> https://github.com/slightlyoff/async-local-storage
>
>
Yes! I mean that's kind of where this conversation took off... I just don't
think there should be an obvious distinction between "the API with the
transactions and versions" and "the one without." - if anything presenting
them in a unified fashion allows for developers to migrate as they need
individual features (Transactions, versions, etc)
Alec
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wed, Mar 6, 2013 at 10:14 AM, Alec Flett wrote:
> My primary takeaway from both working on IDB and working with IDB for some
> demo apps is that IDB has just the right amount of complexity for really
> large, robust database use.. but for a "welcome to noSQL in the browser" it
> is way too complicated.
>
> Specifically:
>
> versioning - The reason this exists in IDB is to guarantee a schema (read: a
> fixed set of objectStores + indexes) for a given set of operations.
> Versioning should be optional. And if versioning is optional, so should
> opening - the only reason you need to "open" a database is so that you have
> a handle to a versioned database. You can almost implement versioning in JS
> if you really care about it...(either keep an explicit key, or auto-detect
> the state of the schema) its one of those cases where 80% of versioning is
> dirt simple and the complicated stuff is really about maintaining version
> changes across multiply-opened windows. (i.e. one window opens an idb, the
> next window opens it and changes the schema, the first window may need to
> know that and be able to adapt without breaking any in-flight transactions)
Yeah, the versioning is the part of the API that I'm least happy with.
The reason we need it is because we're synchronously exposing the
"schema" of the database. Instead we could have made the set of object
stores and indexes entirely dynamic and created on-demand.
However that would have made it tricky to do schema which has
constraints (like the "unique" index-flag) or that affects how data is
stored (for example the ability to set a collation on an objectStore
or index as has been discussed).
> transactions - Also should be optional. Vital to complex apps, but totally
> not necessary for many.. there should be a default transaction, like
> db.objectStore("foo").get("bar")
I disagree. This would have made it too trivial to create pages that
have race conditions. I.e. people would write code like:
db.objectStore("foo").get("bar").onsuccess = function(e) {
db.objectStore("foo").set("bar", e.target.result + 1)
}
without realizing that that contains a race condition.
> transaction scoping - even when you do want transactions, the api is just
> too verbose and repetitive for "get one key from one object store" -
> db.transaction("foo").objectStore("foo").get("bar") - there should be
> implicit (lightweight) transactions like db.objectStore("foo").get("bar")
We used to have this exact syntax, but it got everyone confused about
what how the implicit transaction actually worked.
> forced versioning - when versioning is optional, it should be then possible
> to change the schema during a regular transaction. Yes, this is a lot of
> rope but this is actually for much more complex apps, rather than simple
> ones. In particular, it's not uncommon for more complex database systems to
> dynamically create indexes based on observed behavior of the API, or
> observed data (i.e. when data with a particular key becomes prevalent,
> generate an index for it) and then dynamically use them if present. At the
> moment you have to do a manual close/open/version change to dynamically bump
> up the version - effectively rendering fixed-value versions moot (i.e. the
> schema for version 23 in my browser may look totally different than the
> schema for version 23 in your browser) and drastically complicating all your
> code (Because if you try to close/open while transactions are in flight,
> they will be aborted - so you have to temporarily pause all new
> transactions, wait for all in-flight transactions to finish, do a
> close/open, then start running all pending/paused transactions.) This last
> case MIGHT be as simple as adding db.reopen(newVersion) to the existing
> spec.
Yeah, this is something that I've run into as well. It's tricky to
solve this without removing the ability to synchronously access the
list of objectStores and indexes, or without introducing race
conditions if a page is open in multiple tabs.
> named object stores - frankly, for many use cases, a single objectStore is
> all you need. a simple db.get("foo") would be sufficient. Simply naming a
> "default" isn't bad - whats bad is all the onupgradeneeded scaffolding
> required to create the objectstore in the first place.
I think we should do this as part of a simple API. Similar to something like
https://github.com/slightlyoff/async-local-storage
> I do think that the IDBRequest model needs tweaking, and Futures seem like
> the obvious direction to head in.
Yup. Though see the list of constraints in my previous email in this thread.
> FWIW, the "sync" version of the API is more or less dead - nobody has
> actually implemented it.
We're actually working on an implementation and hope to have it ready mid year.
/ Jonas
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wed, Mar 6, 2013 at 6:01 AM, Alex Russell wrote: > I've avoided weighing in on this thread until I had more IDB experience. > I've been wrestling with it on two fronts of late: > > A re-interpretation of the API based on Futures: > https://github.com/slightlyoff/DOMFuture/tree/master/reworked_APIs/IndexedDB > A new async LocalStorage design + p(r)olyfill that's bootstrapped on IDB: > https://github.com/slightlyoff/async-local-storage > > While you might be right that it's unlikely that the API can be > "simplified", I think it's trivial to extend it in ways that make it easier > to reason about and use. > > This thread started out with a discussion of what might be done to keep > IDB's perceived mistakes from reoccurring. Here's a quick stab at both an > outline of the mistakes and what can be done to avoid them: > > Abuse of events > The current IDB design models one-time operations using events. This can > make sense insofar as events can occur zero or more times in the future, but > it's not a natural fit. What does it mean for oncomplete to happen more than > once? Is that an error? Are onsuccess and onerror exclusive? Can they both > be dispatched for an operation? The API isn't clear. Events don't lead to > good design here as they don't encapsulate these concerns. Similarly, event > handlers don't chain. This is natural, as they could be invoked multiple > times (conceptually), but it's not a good fit for data access. It's great > that IDB as async, and events are the existing DOM model for this, but IDB's > IDBRequest object is calling out for a different kind of abstraction. I'll > submit Futures for the job, but others might work (explicit callback, > whatever) so long as they maintain chainability + async. Whether it's an "abuse" of events or not I guess is a matter of opinion. DOM Events have always been used in situations when the Event fired either 0 or 1 time. They've even been used in situations when an eventual success/error has been signaled. In particular the "load" and "error" events for Documents were among some of the first Events created. That said, I agree that Events are generally a better fit for situations when you have a reoccurring "thing" that happen. And yes, if we had had Futures when we designed IDB it might have lead to a much easier to use API. We had a hunch that that was the case when we designed the API. However there weren't then, and there still aren't, a standardized promise API. We felt then, and I still feel that way now, that it would have been a mistake to standardize a promise library as part of a database API. This is why I've been pushing for someone to step up and take on creating a standardized promise library that we can rely on for future APIs. https://twitter.com/SickingJ/status/20215262974322 > Implicitness > IDB is implicit in a number of places that cause confusion for folks not > intimately familiar with the contract(s) that IDB expects you to enter into. > First, the use of events for delivery of notifications means that > sequential-looking code that you might expect to have timing issues doesn't. > Why not? Because IDB operates in some vaguely async way; you can't reason at > all about events that have occurred in the past (they're not values, they're > points in time). You seem to be under the impression that Events are only used/intended for situations when something will happen "zero or more times at some point in the future". I agree that this is the scenario when Events really shine. Especially when that something is connected to DOM Nodes. However the way they are actually used in the DOM platform is as a generic way of doing callbacks. So despite the fact that IDB uses Events, it still has quite strict requirements in which order they fire. Having a strict order of delivering results was a quite intentional design decision. > I can't find anywhere in the spec that the explicit > gaurantees about delivery timing are noted > (http://www.w3.org/TR/IndexedDB/#async-api), https://dvcs.w3.org/hg/IndexedDB/raw-file/tip/Overview.html#steps-for-asynchronously-executing-a-request See in particular step 4 which guarantees that requests are run and deliver their result in the order they were scheduled. > so one could read IDB code that > registers two callbacks as having a temporal dead-zone: a space in code > where something might have happened but which your code might not have a > chance to hear about. I realize that in practice this isn't the case; event > delivery for these is asynchronous, but the soonest timing isn't defined: > end of turn? next turn? end-of-microtask? This means that it's possible to > have implementations the differ on delivery timing, astonishing those who > register event handlers at the wrong time. This is part DOM-ish use of > events for things they're not suited to and a lack of specificity in the > spec. Both can be fixed. I agree that this is something that is vague right now. The intent is that a
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Friday, March 15, 2013 at 4:11 AM, Jarred Nicholls wrote: > On Thu, Mar 14, 2013 at 10:19 PM, Alex Russell (mailto:slightly...@google.com)> wrote: > > > On Thu, Mar 14, 2013 at 6:36 PM, Glenn Maynard wrote: > > > > > > > Workers exist > > > explicitly to allow you to do expensive synchronous stuff without > > > janking the main thread. (Often, the expensive synchronous stuff will > > > just be a bunch of calculations, so you don't have to explicitly break > > > it up into setTimeout-able chunks.) > > > > > > The entire reason for most async (all?) APIs is thus irrelevant in a > > > Worker, and it may be a good idea to provide sync versions, or do > > > something else that negates the annoyance of dealing with async code. > > > > My *first* approach to this annoyance would be to start adding some async > > primitives to the platform that don't suck so hard; e.g., Futures/Promises. > > +1. Libraries cover that fairly well; albeit I think we all would enjoy such > things to be first-class citizens of the platform. I've seen some good > looking implementations and some decent control flow libraries. I use > https://github.com/caolan/async a lot in node projects. > > > Saying that you should do something does not imply that doubling up on API > > surface area for a corner-case is the right solution. > > I agree. It may have seemed like a good and simple idea at first - well > intentioned for sure - but upon reflection we have to admit it's sloppy, a > greater surface area to maintain, and the antithesis of DRY. It's not what I > personally would expect from a modern, quality JS api, and I'm probably not > the only web dev to share that feeling. At the risk of making a blanketed > statement using anecdotal evidence, I would claim that overindulgence from > modern libraries in existence today has raised the expectations of web devs > in how the web platform architects new apis. Node.js comes with sync and async APIs (for good reasons) and I haven't heard anyone complain that this wasn't DRY. --tobie
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thu, Mar 14, 2013 at 10:19 PM, Alex Russell wrote: > > > On Thursday, March 14, 2013, Tab Atkins Jr. wrote: > >> On Thu, Mar 14, 2013 at 6:36 PM, Glenn Maynard wrote: >> > On Thu, Mar 14, 2013 at 1:54 PM, Alex Russell >> > wrote: >> >> I don't understand why that's true. Workers have a message-oriented API >> >> that's inherently async. They can get back to their caller "whenevs". >> What's >> >> the motivator for needing this? >> > >> > Being able to write synchronous code is one of the basic uses for >> Workers in >> > the first place. Synchronously creating streams is useful in the same >> way >> > that other synchronous APIs are useful, such as FileReaderSync. >> > >> > That doesn't necessarily mean having a synchronous API for a complex >> > interface like this is the ideal approach (there are other ways to do >> it), >> > but that's the end goal. >> >> Yes, this seems to be missing the point of Workers entirely. If all >> you have are async apis, you don't need Workers in the first place, as >> you can just use them in the main thread without jank. > > I wouldn't say that. Async code will eventually be scheduled to execute on the UI thread, which can cause contention with other tasks that must run on that same UI thread. Using a worker thread to perform (async) XHR requests and JSON decoding while the UI thread focused on other rendering tasks was one of the methods we (Sencha) used to increase News Feed performance for Fastbook. I agree with a lot of what you're saying re: workers, but I don't agree that they wouldn't be needed if all we had were async apis. > Workers exist >> explicitly to allow you to do expensive synchronous stuff without >> janking the main thread. (Often, the expensive synchronous stuff will >> just be a bunch of calculations, so you don't have to explicitly break >> it up into setTimeout-able chunks.) >> >> The entire reason for most async (all?) APIs is thus irrelevant in a >> Worker, and it may be a good idea to provide sync versions, or do >> something else that negates the annoyance of dealing with async code. >> > > My *first* approach to this annoyance would be to start adding some async > primitives to the platform that don't suck so hard; e.g., Futures/Promises. > +1. Libraries cover that fairly well; albeit I think we all would enjoy such things to be first-class citizens of the platform. I've seen some good looking implementations and some decent control flow libraries. I use https://github.com/caolan/async a lot in node projects. > Saying that you should do something does not imply that doubling up on API > surface area for a corner-case is the right solution. > I agree. It may have seemed like a good and simple idea at first - well intentioned for sure - but upon reflection we have to admit it's sloppy, a greater surface area to maintain, and the antithesis of DRY. It's not what I personally would expect from a modern, quality JS api, and I'm probably not the only web dev to share that feeling. At the risk of making a blanketed statement using anecdotal evidence, I would claim that overindulgence from modern libraries in existence today has raised the expectations of web devs in how the web platform architects new apis. > >> > (FYI, the messaging in Workers isn't inherently async; it just happens >> to >> > only have an async interface. There's been discussion about adding a >> > synchronous interface to messaging.) >> >> Specifically, this was for workers to be able to synchronously wait >> for messages from their sub-workers. Again, the whole point for async >> worker messaging is to prevent the main thread from janking, which is >> irrelevant inside of a worker. >> >> ~TJ >> >
Re: IndexedDB, what were the issues? How do we stop it from happening again?
* Alex Russell wrote: >My *first* approach to this annoyance would be to start adding some async >primitives to the platform that don't suck so hard; e.g., Futures/Promises. >Saying that you should do something does not imply that doubling up on API >surface area for a corner-case is the right solution. http://lists.w3.org/Archives/Public/www-archive/2008Jul/0009.html was my "first approach". Workers of course have it much easier, they just need a single waiting primitive to make an asynchronous API synchronous. I've http://lists.w3.org/Archives/Public/public-webapps/2011OctDec/1016.html argued against duplicating APIs for workers as proposed here, but so far without much success, it would seem... -- Björn Höhrmann · mailto:bjo...@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thu, Mar 14, 2013 at 9:41 PM, Alex Russell wrote: > I didn't imply they were. But addressing the pain point of asynchronous > code that's hard to use doesn't imply that the only answer is a synchronous > version. > The asynchronous programming model is often inherently inconvenient compared to synchronous code, and synchronous APIs (whether at the individual API level, or at another level such as synchronous messaging or yieldable coroutines) are indeed the only practical solution that's been proposed so far. I believe this is a fundamental issue, but if you have a concrete alternative proposal to make then by all means do so (in another thread). Otherwise, this just isn't helpful. This is not a particularly hard or subtle point. > (Let's try to remain civil.) -- Glenn Maynard
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thursday, March 14, 2013, Glenn Maynard wrote: > On Thu, Mar 14, 2013 at 8:58 PM, Tab Atkins Jr. > > > wrote: > >> The entire reason for most async (all?) APIs is thus irrelevant in a >> > Worker, and it may be a good idea to provide sync versions, or do >> something else that negates the annoyance of dealing with async code. >> > > I agree, except that async APIs are also useful and relevant in workers. > Sometimes you want synchronous code and sometimes you want asynchronous > code, depending on what you're doing. > > > On Thu, Mar 14, 2013 at 9:19 PM, Alex Russell > > > wrote: > >> My *first* approach to this annoyance would be to start adding some async >> primitives to the platform that don't suck so hard; e.g., Futures/Promises. >> Saying that you should do something does not imply that doubling up on API >> surface area for a corner-case is the right solution. > > > "Futures" are nothing but a different async API. They're in no way > comparable to synchronous code. > I didn't imply they were. But addressing the pain point of asynchronous code that's hard to use doesn't imply that the only answer is a synchronous version. This is not a particularly hard or subtle point. > But, as I said, it's true that a second synchronous interface isn't > necessarily the best solution for complex APIs like IndexedDB. At least in > this particular case, if we have a synchronous messaging API I might call > the synchronous IDB interface unnecessary. > > -- > Glenn Maynard > >
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thu, Mar 14, 2013 at 8:58 PM, Tab Atkins Jr. wrote: > The entire reason for most async (all?) APIs is thus irrelevant in a > Worker, and it may be a good idea to provide sync versions, or do > something else that negates the annoyance of dealing with async code. > I agree, except that async APIs are also useful and relevant in workers. Sometimes you want synchronous code and sometimes you want asynchronous code, depending on what you're doing. On Thu, Mar 14, 2013 at 9:19 PM, Alex Russell wrote: > My *first* approach to this annoyance would be to start adding some async > primitives to the platform that don't suck so hard; e.g., Futures/Promises. > Saying that you should do something does not imply that doubling up on API > surface area for a corner-case is the right solution. "Futures" are nothing but a different async API. They're in no way comparable to synchronous code. But, as I said, it's true that a second synchronous interface isn't necessarily the best solution for complex APIs like IndexedDB. At least in this particular case, if we have a synchronous messaging API I might call the synchronous IDB interface unnecessary. -- Glenn Maynard
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thursday, March 14, 2013, Tab Atkins Jr. wrote: > On Thu, Mar 14, 2013 at 6:36 PM, Glenn Maynard > > wrote: > > On Thu, Mar 14, 2013 at 1:54 PM, Alex Russell > > > > > > wrote: > >> I don't understand why that's true. Workers have a message-oriented API > >> that's inherently async. They can get back to their caller "whenevs". > What's > >> the motivator for needing this? > > > > Being able to write synchronous code is one of the basic uses for > Workers in > > the first place. Synchronously creating streams is useful in the same > way > > that other synchronous APIs are useful, such as FileReaderSync. > > > > That doesn't necessarily mean having a synchronous API for a complex > > interface like this is the ideal approach (there are other ways to do > it), > > but that's the end goal. > > Yes, this seems to be missing the point of Workers entirely. If all > you have are async apis, you don't need Workers in the first place, as > you can just use them in the main thread without jank. Workers exist > explicitly to allow you to do expensive synchronous stuff without > janking the main thread. (Often, the expensive synchronous stuff will > just be a bunch of calculations, so you don't have to explicitly break > it up into setTimeout-able chunks.) > > The entire reason for most async (all?) APIs is thus irrelevant in a > Worker, and it may be a good idea to provide sync versions, or do > something else that negates the annoyance of dealing with async code. > My *first* approach to this annoyance would be to start adding some async primitives to the platform that don't suck so hard; e.g., Futures/Promises. Saying that you should do something does not imply that doubling up on API surface area for a corner-case is the right solution. > > (FYI, the messaging in Workers isn't inherently async; it just happens to > > only have an async interface. There's been discussion about adding a > > synchronous interface to messaging.) > > Specifically, this was for workers to be able to synchronously wait > for messages from their sub-workers. Again, the whole point for async > worker messaging is to prevent the main thread from janking, which is > irrelevant inside of a worker. > > ~TJ >
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thu, Mar 14, 2013 at 6:36 PM, Glenn Maynard wrote: > On Thu, Mar 14, 2013 at 1:54 PM, Alex Russell > wrote: >> I don't understand why that's true. Workers have a message-oriented API >> that's inherently async. They can get back to their caller "whenevs". What's >> the motivator for needing this? > > Being able to write synchronous code is one of the basic uses for Workers in > the first place. Synchronously creating streams is useful in the same way > that other synchronous APIs are useful, such as FileReaderSync. > > That doesn't necessarily mean having a synchronous API for a complex > interface like this is the ideal approach (there are other ways to do it), > but that's the end goal. Yes, this seems to be missing the point of Workers entirely. If all you have are async apis, you don't need Workers in the first place, as you can just use them in the main thread without jank. Workers exist explicitly to allow you to do expensive synchronous stuff without janking the main thread. (Often, the expensive synchronous stuff will just be a bunch of calculations, so you don't have to explicitly break it up into setTimeout-able chunks.) The entire reason for most async (all?) APIs is thus irrelevant in a Worker, and it may be a good idea to provide sync versions, or do something else that negates the annoyance of dealing with async code. > (FYI, the messaging in Workers isn't inherently async; it just happens to > only have an async interface. There's been discussion about adding a > synchronous interface to messaging.) Specifically, this was for workers to be able to synchronously wait for messages from their sub-workers. Again, the whole point for async worker messaging is to prevent the main thread from janking, which is irrelevant inside of a worker. ~TJ
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thu, Mar 14, 2013 at 1:54 PM, Alex Russell wrote: > On Wednesday, March 6, 2013, Tobie Langel wrote: > >> On Wednesday, March 6, 2013 at 5:51 PM, Jarred Nicholls wrote: >> > This is an entirely different conversation though. I don't know the >> answer to why sync interfaces are there and expected, except that some >> would argue that it makes the code easier to read/write for some devs. >> Since this is mirrored throughout other platform APIs, I wouldn't count >> this as a fault in IDB specifically. >> >> Sync APIs are useful to do I/O inside of a Worker. >> > > I don't understand why that's true. Workers have a message-oriented API > that's inherently async. They can get back to their caller "whenevs". > What's the motivator for needing this? > Being able to write synchronous code is one of the basic uses for Workers in the first place. Synchronously creating streams is useful in the same way that other synchronous APIs are useful, such as FileReaderSync. That doesn't necessarily mean having a synchronous API for a complex interface like this is the ideal approach (there are other ways to do it), but that's the end goal. (FYI, the messaging in Workers isn't inherently async; it just happens to only have an async interface. There's been discussion about adding a synchronous interface to messaging.) -- Glenn Maynard
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thursday, March 14, 2013 at 7:54 PM, Alex Russell wrote: > On Wednesday, March 6, 2013, Tobie Langel wrote: > > Sync APIs are useful to do I/O inside of a Worker. > > > I don't understand why that's true. Workers have a message-oriented API > that's inherently async. They can get back to their caller "whenevs". What's > the motivator for needing this? There's no need per se. Sync API are easier to handle, and given you're already out of the UI thread, blocking in that context isn't much of an issue. > > They're also critical for data consistency in some scenarios, e.g. updating > > the database after a successful xhr request when a worker is about to be > > terminated. > > Unload-catching is a known bug in much o the web platform. Why would we > enable it here? Nevermind. The Web Worker termination process (now?) says scripts get aborted anyway. --tobie
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wednesday, March 6, 2013, Tobie Langel wrote: > On Wednesday, March 6, 2013 at 5:51 PM, Jarred Nicholls wrote: > > This is an entirely different conversation though. I don't know the > answer to why sync interfaces are there and expected, except that some > would argue that it makes the code easier to read/write for some devs. > Since this is mirrored throughout other platform APIs, I wouldn't count > this as a fault in IDB specifically. > > Sync APIs are useful to do I/O inside of a Worker. > I don't understand why that's true. Workers have a message-oriented API that's inherently async. They can get back to their caller "whenevs". What's the motivator for needing this? > They're also critical for data consistency in some scenarios, e.g. > updating the database after a successful xhr request when a worker is about > to be terminated. > Unload-catching is a known bug in much o the web platform. Why would we enable it here?
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wed, Mar 6, 2013 at 1:02 PM, Ian Fette (イアンフェッティ) wrote: > I seem to recall we contemplated people writing libraries on top of IDB > from the beginning. I'm not sure why this is a bad thing. > Expecting libraries providing higher-level abstractions is fine, but it's bad if an API is inconvenient to use directly for common cases. For example, it's natural to expect people to use a game engine library wrapping Canvas to write a game, but Canvas itself is easy to use directly most of the time, for lots of use cases. The only API on the platform that I regularly use which I honestly find unreasonable to use without a wrapper of some kind is cookies, which is one of the worst APIs we've got. Other than that, I can't think of any web API that I actually need a wrapper for. This is very good, since it means everyone else reading my code already understands the APIs I'm using. We originally shipped "web sql" / sqlite, which was a familiar interface > for many and relatively easy to use, but had a sufficiently large API > surface area that no one felt they wanted to document the whole thing such > that we could have an inter-operable standard. (Yes, I'm simplifying a bit.) > (Not to get sidetracked on this, but this seems oversimplified to the point of being confusing. http://lists.w3.org/Archives/Public/public-webapps/2011AprJun/0025.html) As a result, we came up with an approach of "What are the fundamental > primitives that we need?", spec'd that out, and shipped it. We had > discussions at the time that we expected library authors to produce > abstraction layers that made IDB easier to use, as the "fundamental > primitives" approach was not necessarily intended to produce an API that > was as straightforward and easy to use as what we were trying to replace. > If that's now what is happening, that seems like a good thing, not a > failure. > It's fine to not try to be as simple to use as localStorage. That's not an attainable goal; it's not a database in any practical sense and never tried to be. But if we've added a new API to the platform that typical developers wouldn't want to use directly without any wrapper library, we've made an error. -- Glenn Maynard
Re: IndexedDB, what were the issues? How do we stop it from happening again?
+1 to be able to use easily the default API without requiring third party
libraries.
Sent from my Android cell phone, please forgive the lack of format on the
text, and my fat thumbs :-P
El 07/03/2013 21:30, "Shwetank Dixit" escribió:
> On Thu, 07 Mar 2013 13:01:20 +0100, Alex Russell
> wrote:
>
> On Wednesday, March 6, 2013, Ian Fette (イアンフェッティ) wrote:
>>
>> I seem to recall we contemplated people writing libraries on top of IDB
>>> from the beginning. I'm not sure why this is a bad thing.
>>>
>>>
>> It's not bad as an assumption, but it can quickly turn into an excuse for
>> API design malpractice because it often leads to the (mistaken) assumption
>> that user-provided code is as cheap as browser-provided API. Given that
>> users pull their libraries from the network more often than from disk (and
>> must parse/compile, etc.), the incentives of these two API-providers could
>> not be more different. That's why it's critical that API designers try to
>> forestall the need for libraries for as long as possible when it comes to
>> web features.
>>
> +1
>
> Libraries are important in many areas, but the goal should be to have a
> spec which doesn't *require* it. It should be easy to understand and
> implement without it. I would rather learn the spec and write a few lines
> of code and have it run - rather than learn the spec, then learn a library,
> and then use that library in every required page (increasing my bandwidth
> costs and the costs to my users who are accessing my site on mobile, often
> on limited data plans). The former option should be the design goal
> whenever possible.
>
> Also, Alec's points were spot on.
>
>
>>
>> We originally shipped "web sql" / sqlite, which was a familiar interface
>>> for many and relatively easy to use, but had a sufficiently large API
>>> surface area that no one felt they wanted to document the whole thing
>>> such
>>> that we could have an inter-operable standard. (Yes, I'm simplifying a
>>> bit.)
>>>
>>>
>> Yeah, I recall that the SQLite semantics were the big obstacle.
>>
>>
>> As a result, we came up with an approach of "What are the fundamental
>>> primitives that we need?", spec'd that out, and shipped it. We had
>>> discussions at the time that we expected library authors to produce
>>> abstraction layers that made IDB easier to use, as the "fundamental
>>> primitives" approach was not necessarily intended to produce an API that
>>> was as straightforward and easy to use as what we were trying to replace.
>>> If that's now what is happening, that seems like a good thing, not a
>>> failure.
>>>
>>>
>> It's fine in the short run to provide just the low-level stuff and work up
>> to the high-level things -- but only when you can't predict what the
>> high-level needs will be. Assuming that's what the WG's view was, you're
>> right; feature not bug, although there's now more work to do.
>>
>> Anyhow, IDB is incredibly high-level in many places and primitive in
>> others. ISTM that it's not easy to get a handle on it's intended level of
>> abstraction.
>>
>>
>> On Wed, Mar 6, 2013 at 10:14 AM, Alec Flett >> >wrote:
>>>
>>> My primary takeaway from both working on IDB and working with IDB for
>>> some
>>> demo apps is that IDB has just the right amount of complexity for really
>>> large, robust database use.. but for a "welcome to noSQL in the browser"
>>> it
>>> is way too complicated.
>>>
>>> Specifically:
>>>
>>>1. *versioning* - The reason this exists in IDB is to guarantee a
>>>schema (read: a fixed set of objectStores + indexes) for a given set
>>> of
>>>operations. Versioning should be optional. And if versioning is
>>> optional,
>>>so should *opening* - the only reason you need to "open" a database is
>>>so that you have a handle to a versioned database. You can *almost*
>>> implement
>>>versioning in JS if you really care about it...(either keep an
>>> explicit
>>>key, or auto-detect the state of the schema) its one of those cases
>>> where
>>>80% of versioning is dirt simple and the complicated stuff is really
>>> about
>>>maintaining version changes across multiply-opened windows. (i.e. one
>>>window opens an idb, the next window opens it and changes the schema,
>>> the
>>>first window *may* need to know that and be able to adapt without
>>>breaking any in-flight transactions) -
>>>2. *transactions* - Also should be optional. Vital to complex apps,
>>>but totally not necessary for many.. there should be a default
>>> transaction,
>>>like db.objectStore("foo").get("**bar")
>>>3. *transaction scoping* - even when you do want transactions, the api
>>>is just too verbose and repetitive for "get one key from one object
>>> store"
>>>- db.transaction("foo").**objectStore("foo").get("bar") - there
>>> should be
>>>implicit (lightweight) transactions like db.objectStore("foo").get("*
>>> *bar")
>>>4. *forced versioning* - when versioning is optional, it should be
>>>
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Thu, 07 Mar 2013 13:01:20 +0100, Alex Russell
wrote:
On Wednesday, March 6, 2013, Ian Fette (イアンフェッティ) wrote:
I seem to recall we contemplated people writing libraries on top of IDB
from the beginning. I'm not sure why this is a bad thing.
It's not bad as an assumption, but it can quickly turn into an excuse for
API design malpractice because it often leads to the (mistaken)
assumption
that user-provided code is as cheap as browser-provided API. Given that
users pull their libraries from the network more often than from disk
(and
must parse/compile, etc.), the incentives of these two API-providers
could
not be more different. That's why it's critical that API designers try to
forestall the need for libraries for as long as possible when it comes to
web features.
+1
Libraries are important in many areas, but the goal should be to have a
spec which doesn't *require* it. It should be easy to understand and
implement without it. I would rather learn the spec and write a few lines
of code and have it run - rather than learn the spec, then learn a
library, and then use that library in every required page (increasing my
bandwidth costs and the costs to my users who are accessing my site on
mobile, often on limited data plans). The former option should be the
design goal whenever possible.
Also, Alec's points were spot on.
We originally shipped "web sql" / sqlite, which was a familiar interface
for many and relatively easy to use, but had a sufficiently large API
surface area that no one felt they wanted to document the whole thing
such
that we could have an inter-operable standard. (Yes, I'm simplifying a
bit.)
Yeah, I recall that the SQLite semantics were the big obstacle.
As a result, we came up with an approach of "What are the fundamental
primitives that we need?", spec'd that out, and shipped it. We had
discussions at the time that we expected library authors to produce
abstraction layers that made IDB easier to use, as the "fundamental
primitives" approach was not necessarily intended to produce an API that
was as straightforward and easy to use as what we were trying to
replace.
If that's now what is happening, that seems like a good thing, not a
failure.
It's fine in the short run to provide just the low-level stuff and work
up
to the high-level things -- but only when you can't predict what the
high-level needs will be. Assuming that's what the WG's view was, you're
right; feature not bug, although there's now more work to do.
Anyhow, IDB is incredibly high-level in many places and primitive in
others. ISTM that it's not easy to get a handle on it's intended level of
abstraction.
On Wed, Mar 6, 2013 at 10:14 AM, Alec Flett
wrote:
My primary takeaway from both working on IDB and working with IDB for
some
demo apps is that IDB has just the right amount of complexity for really
large, robust database use.. but for a "welcome to noSQL in the
browser" it
is way too complicated.
Specifically:
1. *versioning* - The reason this exists in IDB is to guarantee a
schema (read: a fixed set of objectStores + indexes) for a given set
of
operations. Versioning should be optional. And if versioning is
optional,
so should *opening* - the only reason you need to "open" a database
is
so that you have a handle to a versioned database. You can *almost*
implement
versioning in JS if you really care about it...(either keep an
explicit
key, or auto-detect the state of the schema) its one of those cases
where
80% of versioning is dirt simple and the complicated stuff is
really about
maintaining version changes across multiply-opened windows. (i.e. one
window opens an idb, the next window opens it and changes the
schema, the
first window *may* need to know that and be able to adapt without
breaking any in-flight transactions) -
2. *transactions* - Also should be optional. Vital to complex apps,
but totally not necessary for many.. there should be a default
transaction,
like db.objectStore("foo").get("bar")
3. *transaction scoping* - even when you do want transactions, the
api
is just too verbose and repetitive for "get one key from one object
store"
- db.transaction("foo").objectStore("foo").get("bar") - there should
be
implicit (lightweight) transactions like
db.objectStore("foo").get("bar")
4. *forced versioning* - when versioning is optional, it should be
then possible to change the schema during a regular transaction.
Yes, this
is a lot of rope but this is actually for much more complex apps,
rather
than simple ones. In particular, it's not uncommon for more complex
database systems to dynamically create indexes based on observed
behavior
of the API, or observed data (i.e. when data with a particular key
becomes
prevalent, generate an index for it) and then dynamically use them if
present. At the moment you have to do a manual
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wednesday, March 6, 2013, Ian Fette (イアンフェッティ) wrote:
> I seem to recall we contemplated people writing libraries on top of IDB
> from the beginning. I'm not sure why this is a bad thing.
>
It's not bad as an assumption, but it can quickly turn into an excuse for
API design malpractice because it often leads to the (mistaken) assumption
that user-provided code is as cheap as browser-provided API. Given that
users pull their libraries from the network more often than from disk (and
must parse/compile, etc.), the incentives of these two API-providers could
not be more different. That's why it's critical that API designers try to
forestall the need for libraries for as long as possible when it comes to
web features.
> We originally shipped "web sql" / sqlite, which was a familiar interface
> for many and relatively easy to use, but had a sufficiently large API
> surface area that no one felt they wanted to document the whole thing such
> that we could have an inter-operable standard. (Yes, I'm simplifying a bit.)
>
Yeah, I recall that the SQLite semantics were the big obstacle.
> As a result, we came up with an approach of "What are the fundamental
> primitives that we need?", spec'd that out, and shipped it. We had
> discussions at the time that we expected library authors to produce
> abstraction layers that made IDB easier to use, as the "fundamental
> primitives" approach was not necessarily intended to produce an API that
> was as straightforward and easy to use as what we were trying to replace.
> If that's now what is happening, that seems like a good thing, not a
> failure.
>
It's fine in the short run to provide just the low-level stuff and work up
to the high-level things -- but only when you can't predict what the
high-level needs will be. Assuming that's what the WG's view was, you're
right; feature not bug, although there's now more work to do.
Anyhow, IDB is incredibly high-level in many places and primitive in
others. ISTM that it's not easy to get a handle on it's intended level of
abstraction.
> On Wed, Mar 6, 2013 at 10:14 AM, Alec Flett wrote:
>
> My primary takeaway from both working on IDB and working with IDB for some
> demo apps is that IDB has just the right amount of complexity for really
> large, robust database use.. but for a "welcome to noSQL in the browser" it
> is way too complicated.
>
> Specifically:
>
>1. *versioning* - The reason this exists in IDB is to guarantee a
>schema (read: a fixed set of objectStores + indexes) for a given set of
>operations. Versioning should be optional. And if versioning is optional,
>so should *opening* - the only reason you need to "open" a database is
>so that you have a handle to a versioned database. You can *almost*
> implement
>versioning in JS if you really care about it...(either keep an explicit
>key, or auto-detect the state of the schema) its one of those cases where
>80% of versioning is dirt simple and the complicated stuff is really about
>maintaining version changes across multiply-opened windows. (i.e. one
>window opens an idb, the next window opens it and changes the schema, the
>first window *may* need to know that and be able to adapt without
>breaking any in-flight transactions) -
>2. *transactions* - Also should be optional. Vital to complex apps,
>but totally not necessary for many.. there should be a default transaction,
>like db.objectStore("foo").get("bar")
>3. *transaction scoping* - even when you do want transactions, the api
>is just too verbose and repetitive for "get one key from one object store"
>- db.transaction("foo").objectStore("foo").get("bar") - there should be
>implicit (lightweight) transactions like db.objectStore("foo").get("bar")
>4. *forced versioning* - when versioning is optional, it should be
>then possible to change the schema during a regular transaction. Yes, this
>is a lot of rope but this is actually for much more complex apps, rather
>than simple ones. In particular, it's not uncommon for more complex
>database systems to dynamically create indexes based on observed behavior
>of the API, or observed data (i.e. when data with a particular key becomes
>prevalent, generate an index for it) and then dynamically use them if
>present. At the moment you have to do a manual close/open/version change to
>dynamically bump up the version - effectively rendering fixed-value
>versions moot (i.e. the schema for version 23 in my browser may look
>totally different than the schema for version 23 in your browser) and
>drastically complicating all your code (Because if you try to close/open
>while transactions are in flight, they will be aborted - so you have to
>temporarily pause all new transactions, wait for all in-flight transactions
>to finish, do a close/open, then start running all pending/paused
>transactions.) This last case MIGHT be as simple as ad
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wed, Mar 6, 2013 at 10:18 AM, Alex Russell wrote: > So which part do you disagree with? That events are a bad model for a > one-time action? Or that it's not clear what the expected contract is? > Going by what you've written below, I have to assume the latter, so I'll > just say this: try sitting a non-webdev down with IDB or any other DOM API > that works this way and try to get them to figure it out from code samples. > Yes, yes, being a webdev means knowing the platform idioms, but if we can > agree they're confusing and difficult, we can start to do something about > it. And in any case, you haven't refuted the former; events are simply a > bad model here. > I disagree that events are bad for one-time actions, and I disagree DOM events are confusing or difficult. There's currently no other way to allow an API to be synchronous in workers >> but only async in the UI thread. >> > > Of course not...but what does that have to do with the price of fish? The > core question is what's motivating a sync API here in the first place. > One of the primary reasons Web Workers exist is to allow writing linear, synchronous code, because it's often inherently more convenient than asynchronous code. That requires synchronous APIs, so asynchronous APIs typically gain a synchronous version when exposed to workers. This isn't the only way to go, and complex APIs like this may warrant giving the "sync API for workers" thing another go, but it's the pattern today. I won't be responding to the rest of your message. > (FYI, if you're not replying to something, you don't have to say you're not replying to it. Just snip the quotes.) On Wed, Mar 6, 2013 at 10:51 AM, Jarred Nicholls wrote: > I'm not understanding how this refutes the fact that single-fired events > is an odd model for the job. Sure it may be consistent, but it's > consistently bad. There are plenty of criteria for judging an API, one of > which is consistency amongst the rest of the platform idioms. But it > should be no wonder that solid API developers come along and create > comprehensible wrappers around the platform. > It's up to people claiming that it doesn't work well to demonstrate it; the discussion started at "since DOM Events are bad for this...", assuming everyone will automatically agree and skipping the step of actually arguing that they're bad. I find events to be simple and straightforward, so it seems like a strange assumption to me. I think this would be very desirable on all fronts to avoid duplication of > interfaces; maybe something like signal events on which a caller can wait: > > var signal = someAsyncAction(), > result = signal.wait(); > The discussion starts here, if you want to revive it (but it's fairly long, with a few false starts, if I remember correctly): http://lists.w3.org/Archives/Public/public-webapps/2012JulSep/0629.html -- Glenn Maynard
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wed, Mar 6, 2013 at 11:02 AM, Ian Fette (イアンフェッティ) wrote:
> I seem to recall we contemplated people writing libraries on top of IDB
> from the beginning. I'm not sure why this is a bad thing. We originally
> shipped "web sql" / sqlite, which was a familiar interface for many and
> relatively easy to use, but had a sufficiently large API surface area that
> no one felt they wanted to document the whole thing such that we could have
> an inter-operable standard. (Yes, I'm simplifying a bit.) As a result, we
> came up with an approach of "What are the fundamental primitives that we
> need?", spec'd that out, and shipped it. We had discussions at the time
> that we expected library authors to produce abstraction layers that made
> IDB easier to use, as the "fundamental primitives" approach was not
> necessarily intended to produce an API that was as straightforward and easy
> to use as what we were trying to replace. If that's now what is happening,
> that seems like a good thing, not a failure.
>
>
That's fine for building up, but I guess what I'm saying is that the
primitives are too complicated to allow you to get started. There is an
excellent HTML5rocks page on indexeddb describing a very simple usecase.
But if I were a web developer, I'd say "screw that, back to localStorage".
Most of the html5 APIs seem almost too simple to be useful, and then people
chain primitives together into useful APIs with libraries. For example, the
DOM APIs are woefully verbose and primitive, but people have built much
better APIs around them. But fundamentally they are primitive.
IndexedDB seems like its build the other way - we looked at what would be
hard for developers to implement themselves (transactions, versions) and
built *primitives* that required them. At the same time, I'd argue that
versioning and transactions *could* be built upon a transactionless,
versionless keystore in JavaScript, using a library. Even the notion of
multiple objectStores is an abstraction that could be implemented on a
single keystore. To be fair, implementing transactions against a
transactionless keystore would not be *performant*, but thats a separate
issue. Now that we have transactions and versions, I wouldn't eliminate
them from IDB by any means, but they're not required.
Alec
> -Ian
>
>
> On Wed, Mar 6, 2013 at 10:14 AM, Alec Flett wrote:
>
>> My primary takeaway from both working on IDB and working with IDB for
>> some demo apps is that IDB has just the right amount of complexity for
>> really large, robust database use.. but for a "welcome to noSQL in the
>> browser" it is way too complicated.
>>
>> Specifically:
>>
>>1. *versioning* - The reason this exists in IDB is to guarantee a
>>schema (read: a fixed set of objectStores + indexes) for a given set of
>>operations. Versioning should be optional. And if versioning is optional,
>>so should *opening* - the only reason you need to "open" a database
>>is so that you have a handle to a versioned database. You can *almost*
>> implement
>>versioning in JS if you really care about it...(either keep an explicit
>>key, or auto-detect the state of the schema) its one of those cases where
>>80% of versioning is dirt simple and the complicated stuff is really
>> about
>>maintaining version changes across multiply-opened windows. (i.e. one
>>window opens an idb, the next window opens it and changes the schema, the
>>first window *may* need to know that and be able to adapt without
>>breaking any in-flight transactions) -
>>2. *transactions* - Also should be optional. Vital to complex apps,
>>but totally not necessary for many.. there should be a default
>> transaction,
>>like db.objectStore("foo").get("bar")
>>3. *transaction scoping* - even when you do want transactions, the
>>api is just too verbose and repetitive for "get one key from one object
>>store" - db.transaction("foo").objectStore("foo").get("bar") - there
>> should
>>be implicit (lightweight) transactions like
>> db.objectStore("foo").get("bar")
>>4. *forced versioning* - when versioning is optional, it should be
>>then possible to change the schema during a regular transaction. Yes, this
>>is a lot of rope but this is actually for much more complex apps, rather
>>than simple ones. In particular, it's not uncommon for more complex
>>database systems to dynamically create indexes based on observed behavior
>>of the API, or observed data (i.e. when data with a particular key becomes
>>prevalent, generate an index for it) and then dynamically use them if
>>present. At the moment you have to do a manual close/open/version change
>> to
>>dynamically bump up the version - effectively rendering fixed-value
>>versions moot (i.e. the schema for version 23 in my browser may look
>>totally different than the schema for version 23 in your browser) and
>>drastically complicating all your code (Because if y
Re: IndexedDB, what were the issues? How do we stop it from happening again?
I seem to recall we contemplated people writing libraries on top of IDB
from the beginning. I'm not sure why this is a bad thing. We originally
shipped "web sql" / sqlite, which was a familiar interface for many and
relatively easy to use, but had a sufficiently large API surface area that
no one felt they wanted to document the whole thing such that we could have
an inter-operable standard. (Yes, I'm simplifying a bit.) As a result, we
came up with an approach of "What are the fundamental primitives that we
need?", spec'd that out, and shipped it. We had discussions at the time
that we expected library authors to produce abstraction layers that made
IDB easier to use, as the "fundamental primitives" approach was not
necessarily intended to produce an API that was as straightforward and easy
to use as what we were trying to replace. If that's now what is happening,
that seems like a good thing, not a failure.
-Ian
On Wed, Mar 6, 2013 at 10:14 AM, Alec Flett wrote:
> My primary takeaway from both working on IDB and working with IDB for some
> demo apps is that IDB has just the right amount of complexity for really
> large, robust database use.. but for a "welcome to noSQL in the browser" it
> is way too complicated.
>
> Specifically:
>
>1. *versioning* - The reason this exists in IDB is to guarantee a
>schema (read: a fixed set of objectStores + indexes) for a given set of
>operations. Versioning should be optional. And if versioning is optional,
>so should *opening* - the only reason you need to "open" a database is
>so that you have a handle to a versioned database. You can *almost*
> implement
>versioning in JS if you really care about it...(either keep an explicit
>key, or auto-detect the state of the schema) its one of those cases where
>80% of versioning is dirt simple and the complicated stuff is really about
>maintaining version changes across multiply-opened windows. (i.e. one
>window opens an idb, the next window opens it and changes the schema, the
>first window *may* need to know that and be able to adapt without
>breaking any in-flight transactions) -
>2. *transactions* - Also should be optional. Vital to complex apps,
>but totally not necessary for many.. there should be a default transaction,
>like db.objectStore("foo").get("bar")
>3. *transaction scoping* - even when you do want transactions, the api
>is just too verbose and repetitive for "get one key from one object store"
>- db.transaction("foo").objectStore("foo").get("bar") - there should be
>implicit (lightweight) transactions like db.objectStore("foo").get("bar")
>4. *forced versioning* - when versioning is optional, it should be
>then possible to change the schema during a regular transaction. Yes, this
>is a lot of rope but this is actually for much more complex apps, rather
>than simple ones. In particular, it's not uncommon for more complex
>database systems to dynamically create indexes based on observed behavior
>of the API, or observed data (i.e. when data with a particular key becomes
>prevalent, generate an index for it) and then dynamically use them if
>present. At the moment you have to do a manual close/open/version change to
>dynamically bump up the version - effectively rendering fixed-value
>versions moot (i.e. the schema for version 23 in my browser may look
>totally different than the schema for version 23 in your browser) and
>drastically complicating all your code (Because if you try to close/open
>while transactions are in flight, they will be aborted - so you have to
>temporarily pause all new transactions, wait for all in-flight transactions
>to finish, do a close/open, then start running all pending/paused
>transactions.) This last case MIGHT be as simple as adding
>db.reopen(newVersion) to the existing spec.
>5. *named object stores* - frankly, for *many* use cases, a single
>objectStore is all you need. a simple db.get("foo") would be sufficient.
>Simply naming a "default" isn't bad - whats bad is all the onupgradeneeded
>scaffolding required to create the objectstore in the first place.
>
> I do think that the IDBRequest model needs tweaking, and Futures seem like
> the obvious direction to head in.
>
> FWIW, the "sync" version of the API is more or less dead - nobody has
> actually implemented it.
>
> I think there is a very specialized set of applications that absolutely
> need the features that IDB has right now. Google Docs is a perfect example
> - long lived complicated application that needs to keep absolute integrity
> of schema across multiple tabs over a long period of time.. but for 99% of
> usecases out there, I think they're unnecessary.
>
> I think ultimately, a simplified IDB would allow progressive use of the
> api as your application grows.
>
> // basic interaction - some objectStore named 'default' gets crated under
> the hood.
Re: IndexedDB, what were the issues? How do we stop it from happening again?
My primary takeaway from both working on IDB and working with IDB for some
demo apps is that IDB has just the right amount of complexity for really
large, robust database use.. but for a "welcome to noSQL in the browser" it
is way too complicated.
Specifically:
1. *versioning* - The reason this exists in IDB is to guarantee a schema
(read: a fixed set of objectStores + indexes) for a given set of
operations. Versioning should be optional. And if versioning is optional,
so should *opening* - the only reason you need to "open" a database is
so that you have a handle to a versioned database. You can *almost*
implement
versioning in JS if you really care about it...(either keep an explicit
key, or auto-detect the state of the schema) its one of those cases where
80% of versioning is dirt simple and the complicated stuff is really about
maintaining version changes across multiply-opened windows. (i.e. one
window opens an idb, the next window opens it and changes the schema, the
first window *may* need to know that and be able to adapt without
breaking any in-flight transactions) -
2. *transactions* - Also should be optional. Vital to complex apps, but
totally not necessary for many.. there should be a default transaction,
like db.objectStore("foo").get("bar")
3. *transaction scoping* - even when you do want transactions, the api
is just too verbose and repetitive for "get one key from one object store"
- db.transaction("foo").objectStore("foo").get("bar") - there should be
implicit (lightweight) transactions like db.objectStore("foo").get("bar")
4. *forced versioning* - when versioning is optional, it should be then
possible to change the schema during a regular transaction. Yes, this is a
lot of rope but this is actually for much more complex apps, rather than
simple ones. In particular, it's not uncommon for more complex database
systems to dynamically create indexes based on observed behavior of the
API, or observed data (i.e. when data with a particular key becomes
prevalent, generate an index for it) and then dynamically use them if
present. At the moment you have to do a manual close/open/version change to
dynamically bump up the version - effectively rendering fixed-value
versions moot (i.e. the schema for version 23 in my browser may look
totally different than the schema for version 23 in your browser) and
drastically complicating all your code (Because if you try to close/open
while transactions are in flight, they will be aborted - so you have to
temporarily pause all new transactions, wait for all in-flight transactions
to finish, do a close/open, then start running all pending/paused
transactions.) This last case MIGHT be as simple as adding
db.reopen(newVersion) to the existing spec.
5. *named object stores* - frankly, for *many* use cases, a single
objectStore is all you need. a simple db.get("foo") would be sufficient.
Simply naming a "default" isn't bad - whats bad is all the onupgradeneeded
scaffolding required to create the objectstore in the first place.
I do think that the IDBRequest model needs tweaking, and Futures seem like
the obvious direction to head in.
FWIW, the "sync" version of the API is more or less dead - nobody has
actually implemented it.
I think there is a very specialized set of applications that absolutely
need the features that IDB has right now. Google Docs is a perfect example
- long lived complicated application that needs to keep absolute integrity
of schema across multiple tabs over a long period of time.. but for 99% of
usecases out there, I think they're unnecessary.
I think ultimately, a simplified IDB would allow progressive use of the api
as your application grows.
// basic interaction - some objectStore named 'default' gets crated under
the hood.
indexedDB.get("mykey");
// named database, auto-create the 'first' objectStore named 'default', no
need to 'close' anything
indexedDB.database("mydb").get("mykey")
// now we need multiple objectstores:
indexedDB.database("mydb").objectStore("default").get("mykey")
// time for versioning, but using 'default'
indexedDB.open("mydb", 12).onupgradeneeded(function (db) {...}).get("bar")
etc...
Alec
On Wed, Mar 6, 2013 at 6:01 AM, Alex Russell wrote:
> Comments inline. Adding some folks from the IDB team at Google to the
> thread as well as public-webapps.
>
> On Sunday, February 17, 2013, Miko Nieminen wrote:
>
>>
>>
>> 2013/2/15 Shwetank Dixit
>>
>>> Why did you feel it was necessary to write a layer on top of IndexedDB?
>>>
>>> I think this is the main issue here.
>>>
>>> As it stands, IDB is great in terms of features and power it offers, but
>>> the feedback I recieved from other devs was that writing raw IndexedDB
>>> requires an uncomfortable amount of verbosity even for some simple tasks
>>> (This can be disputed, but that is the views I got from some of the
>>> developers I i
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wednesday, March 6, 2013 at 5:51 PM, Jarred Nicholls wrote: > This is an entirely different conversation though. I don't know the answer to > why sync interfaces are there and expected, except that some would argue that > it makes the code easier to read/write for some devs. Since this is mirrored > throughout other platform APIs, I wouldn't count this as a fault in IDB > specifically. Sync APIs are useful to do I/O inside of a Worker. They're also critical for data consistency in some scenarios, e.g. updating the database after a successful xhr request when a worker is about to be terminated. --tobie
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wednesday, March 6, 2013, Glenn Maynard wrote: > On Wed, Mar 6, 2013 at 8:01 AM, Alex Russell > > > wrote: > >> Comments inline. Adding some folks from the IDB team at Google to the >> thread as well as public-webapps. >> > > (I don't want to cold CC so many people, and anybody working on an IDB > implementation should be on -webapps already, so I've trimmed the CC to > that. I'm not subscribed to -tag, so a mail there would probably bounce > anyway.) > >> >>- *Abuse of events* >>The current IDB design models one-time operations using events. This * >>can* make sense insofar as events can occur zero or more times in the >>future, but it's not a natural fit. What does it mean for oncomplete to >>happen more than once? Is that an error? Are onsuccess and onerror >>exclusive? Can they both be dispatched for an operation? The API isn't >>clear. Events don't lead to good design here as they don't encapsulate >>these concerns. Similarly, event handlers don't chain. This is natural, as >>they could be invoked multiple times (conceptually), but it's not a good >>fit for data access. It's great that IDB as async, and events are the >>existing DOM model for this, but IDB's IDBRequest object is calling out >> for >>a different kind of abstraction. I'll submit Futures for the job, but >>others might work (explicit callback, whatever) so long as they maintain >>chainability + async. >> >> > I disagree. DOM events are used this way across the entire platform. > Everybody understands it, it works well, and coming up with something > different can only add more complexity and inconsistency to the platform by > having additional ways to model the same job. I disagree both that we need > a new way of handling this, and that IDB made a mistake in using the > standard mechanism in an ordinary, well-practiced way. > I'm not understanding how this refutes the fact that single-fired events is an odd model for the job. Sure it may be consistent, but it's consistently bad. There are plenty of criteria for judging an API, one of which is consistency amongst the rest of the platform idioms. But it should be no wonder that solid API developers come along and create comprehensible wrappers around the platform. > > >>- *Doubled API surface for sync version* >>I assume I just don't understand why this choice was made, but the >>explosion of API surface area combined with the conditional availability >> of >>this version of the API make it an odd beast (to be charitable). >> >> There's currently no other way to allow an API to be synchronous in > workers but only async in the UI thread. > > There was some discussion about a generalized way to allow workers to > block on a message from another thread, which would make it possible to > implement a synchronous shim for any async API in JavaScript. In theory > this could make it unnecessary for each API to have its own synchronous > interface. It wouldn't be as convenient, and probably wouldn't be suitable > for every API, but for big, complex interfaces like IDB it might make > sense. There might also be other ways to express synchronous APIs based on > their async interfaces without having a whole second interface (eg. maybe > something like a method to block until an event is received). > I think this would be very desirable on all fronts to avoid duplication of interfaces; maybe something like signal events on which a caller can wait: var signal = someAsyncAction(), result = signal.wait(); This is an entirely different conversation though. I don't know the answer to why sync interfaces are there and expected, except that some would argue that it makes the code easier to read/write for some devs. Since this is mirrored throughout other platform APIs, I wouldn't count this as a fault in IDB specifically. > >>- *The idea that this is all going to be wrapped up by libraries >>anyway* >> >> I don't have an opinion about IDB specifically yet, but I agree that this > is wrong. > > People have become so used to using wrappers around APIs that they've come > to think of them as normal, and that we should design APIs assuming people > will keep doing that. > > People wrap libraries when they're hard to use, and if they're hard to use > then they're badly designed. Just because people wrap bad APIs isn't an > excuse for designing more bad APIs. Wrappers for basic usage are always a > bad thing: you always end up with lots of them, which means everyone is > using different APIs. When everyone uses the provided APIs directly, we > can all read each others' code and all of our code interoperates much more > naturally. > > (As you said, this is only referring to wrappers at the same level of > abstraction, of course, not libraries providing higher-level abstractions.) > > -- > Glenn Maynard > >
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wednesday, March 6, 2013, Glenn Maynard wrote: > On Wed, Mar 6, 2013 at 8:01 AM, Alex Russell > > > wrote: > >> Comments inline. Adding some folks from the IDB team at Google to the >> thread as well as public-webapps. >> > > (I don't want to cold CC so many people, and anybody working on an IDB > implementation should be on -webapps already, so I've trimmed the CC to > that. I'm not subscribed to -tag, so a mail there would probably bounce > anyway.) > >> >>- *Abuse of events* >>The current IDB design models one-time operations using events. This * >>can* make sense insofar as events can occur zero or more times in the >>future, but it's not a natural fit. What does it mean for oncomplete to >>happen more than once? Is that an error? Are onsuccess and onerror >>exclusive? Can they both be dispatched for an operation? The API isn't >>clear. Events don't lead to good design here as they don't encapsulate >>these concerns. Similarly, event handlers don't chain. This is natural, as >>they could be invoked multiple times (conceptually), but it's not a good >>fit for data access. It's great that IDB as async, and events are the >>existing DOM model for this, but IDB's IDBRequest object is calling out >> for >>a different kind of abstraction. I'll submit Futures for the job, but >>others might work (explicit callback, whatever) so long as they maintain >>chainability + async. >> >> > I disagree. DOM events are used this way across the entire platform. > So which part do you disagree with? That events are a bad model for a one-time action? Or that it's not clear what the expected contract is? Going by what you've written below, I have to assume the latter, so I'll just say this: try sitting a non-webdev down with IDB or any other DOM API that works this way and try to get them to figure it out from code samples. Yes, yes, being a webdev means knowing the platform idioms, but if we can agree they're confusing and difficult, we can start to do something about it. And in any case, you haven't refuted the former; events are simply a bad model here. > Everybody understands it, it works well, and coming up with something > different can only add more complexity and inconsistency to the platform by > having additional ways to model the same job. I disagree both that we need > a new way of handling this, and that IDB made a mistake in using the > standard mechanism in an ordinary, well-practiced way. > > >>- *Doubled API surface for sync version* >>I assume I just don't understand why this choice was made, but the >>explosion of API surface area combined with the conditional availability >> of >>this version of the API make it an odd beast (to be charitable). >> >> There's currently no other way to allow an API to be synchronous in > workers but only async in the UI thread. > Of course not...but what does that have to do with the price of fish? The core question is what's motivating a sync API here in the first place. I won't be responding to the rest of your message. > There was some discussion about a generalized way to allow workers to > block on a message from another thread, which would make it possible to > implement a synchronous shim for any async API in JavaScript. In theory > this could make it unnecessary for each API to have its own synchronous > interface. It wouldn't be as convenient, and probably wouldn't be suitable > for every API, but for big, complex interfaces like IDB it might make > sense. There might also be other ways to express synchronous APIs based on > their async interfaces without having a whole second interface (eg. maybe > something like a method to block until an event is received). > > >>- *The idea that this is all going to be wrapped up by libraries >>anyway* >> >> I don't have an opinion about IDB specifically yet, but I agree that this > is wrong. > > People have become so used to using wrappers around APIs that they've come > to think of them as normal, and that we should design APIs assuming people > will keep doing that. > > People wrap libraries when they're hard to use, and if they're hard to use > then they're badly designed. Just because people wrap bad APIs isn't an > excuse for designing more bad APIs. Wrappers for basic usage are always a > bad thing: you always end up with lots of them, which means everyone is > using different APIs. When everyone uses the provided APIs directly, we > can all read each others' code and all of our code interoperates much more > naturally. > > (As you said, this is only referring to wrappers at the same level of > abstraction, of course, not libraries providing higher-level abstractions.) > > -- > Glenn Maynard > >
Re: IndexedDB, what were the issues? How do we stop it from happening again?
On Wed, Mar 6, 2013 at 8:01 AM, Alex Russell wrote: > Comments inline. Adding some folks from the IDB team at Google to the > thread as well as public-webapps. > (I don't want to cold CC so many people, and anybody working on an IDB implementation should be on -webapps already, so I've trimmed the CC to that. I'm not subscribed to -tag, so a mail there would probably bounce anyway.) > >- *Abuse of events* >The current IDB design models one-time operations using events. This * >can* make sense insofar as events can occur zero or more times in the >future, but it's not a natural fit. What does it mean for oncomplete to >happen more than once? Is that an error? Are onsuccess and onerror >exclusive? Can they both be dispatched for an operation? The API isn't >clear. Events don't lead to good design here as they don't encapsulate >these concerns. Similarly, event handlers don't chain. This is natural, as >they could be invoked multiple times (conceptually), but it's not a good >fit for data access. It's great that IDB as async, and events are the >existing DOM model for this, but IDB's IDBRequest object is calling out for >a different kind of abstraction. I'll submit Futures for the job, but >others might work (explicit callback, whatever) so long as they maintain >chainability + async. > > I disagree. DOM events are used this way across the entire platform. Everybody understands it, it works well, and coming up with something different can only add more complexity and inconsistency to the platform by having additional ways to model the same job. I disagree both that we need a new way of handling this, and that IDB made a mistake in using the standard mechanism in an ordinary, well-practiced way. >- *Doubled API surface for sync version* >I assume I just don't understand why this choice was made, but the >explosion of API surface area combined with the conditional availability of >this version of the API make it an odd beast (to be charitable). > > There's currently no other way to allow an API to be synchronous in workers but only async in the UI thread. There was some discussion about a generalized way to allow workers to block on a message from another thread, which would make it possible to implement a synchronous shim for any async API in JavaScript. In theory this could make it unnecessary for each API to have its own synchronous interface. It wouldn't be as convenient, and probably wouldn't be suitable for every API, but for big, complex interfaces like IDB it might make sense. There might also be other ways to express synchronous APIs based on their async interfaces without having a whole second interface (eg. maybe something like a method to block until an event is received). >- *The idea that this is all going to be wrapped up by libraries anyway >* > > I don't have an opinion about IDB specifically yet, but I agree that this is wrong. People have become so used to using wrappers around APIs that they've come to think of them as normal, and that we should design APIs assuming people will keep doing that. People wrap libraries when they're hard to use, and if they're hard to use then they're badly designed. Just because people wrap bad APIs isn't an excuse for designing more bad APIs. Wrappers for basic usage are always a bad thing: you always end up with lots of them, which means everyone is using different APIs. When everyone uses the provided APIs directly, we can all read each others' code and all of our code interoperates much more naturally. (As you said, this is only referring to wrappers at the same level of abstraction, of course, not libraries providing higher-level abstractions.) -- Glenn Maynard
Re: IndexedDB, what were the issues? How do we stop it from happening again?
I wrote a quick overview of the issues I have had using the indexedDB API http://arandomurl.com/2013/02/21/thoughts-on-indexeddb.html Most of them are just implementation details, however I still havent met a webdev who understands the transaction model without having one of the indexeddb implementors explain it to them, and past being confusing it also makes things much harder to implement due to being tied to the event loop. and I agree with pretty much everything that Alex wrote, at least the parts that I understood. Cheers Dale On 6 March 2013 15:01, Alex Russell wrote: > Comments inline. Adding some folks from the IDB team at Google to the > thread as well as public-webapps. > > On Sunday, February 17, 2013, Miko Nieminen wrote: > >> >> >> 2013/2/15 Shwetank Dixit >> >>> Why did you feel it was necessary to write a layer on top of IndexedDB? >>> >>> I think this is the main issue here. >>> >>> As it stands, IDB is great in terms of features and power it offers, but >>> the feedback I recieved from other devs was that writing raw IndexedDB >>> requires an uncomfortable amount of verbosity even for some simple tasks >>> (This can be disputed, but that is the views I got from some of the >>> developers I interacted with). Adding that much amount of code (once again, >>> im talking of raw IndexedDB) makes it less readable and understandable. For >>> beginners, this all seemed very intimidating, and for some people more >>> experienced, it was a bit frustrating. >>> >>> >> After my experiments with IDB, I don't feel that it is particularly >> verbose. I have to admit that often I prefer slightly verbose syntax over >> shorter one when it makes reading the code easier. In IDB's case, I think >> this is the case. >> >> >> >>> For the latter bit, I reckon it would be a good practice for groups working on low-level APIs to more or less systematically produce a library that operates at a higher level. This would not only help developers in that they could pick that up instead of the lower-level stuff, but more importantly (at least in terms of goals) it would serve to validate that the lower-level design is indeed appropriate for librarification. >>> >>> I think that would be a good idea. Also, people making those low level >>> APIs should still keep in mind that the resulting code should not be too >>> verbose or complex. Librarification should be an advantage, but not a de >>> facto requirement for developers when it comes to such APIs. It should >>> still be feasable for them to write code in the raw low level API without >>> writing uncomfortably verbose or complex code for simple tasks. Spec >>> designers of low level APIs should not take this as a license to make >>> things so complex that only they and a few others understand it, and then >>> hope that some others will go ahead and make it simple for the 'common >>> folk' through an abstraction library. >> >> >> I quite don't see how to simplify IDB syntax much more. >> > > I've avoided weighing in on this thread until I had more IDB experience. > I've been wrestling with it on two fronts of late: > > >- A re-interpretation of the API based on Futures: > > > https://github.com/slightlyoff/DOMFuture/tree/master/reworked_APIs/IndexedDB >- A new async LocalStorage design + p(r)olyfill that's bootstrapped on >IDB: >https://github.com/slightlyoff/async-local-storage > > While you might be right that it's unlikely that the API can be > "simplified", I think it's trivial to extend it in ways that make it easier > to reason about and use. > > This thread started out with a discussion of what might be done to keep > IDB's perceived mistakes from reoccurring. Here's a quick stab at both an > outline of the mistakes and what can be done to avoid them: > > >- *Abuse of events* >The current IDB design models one-time operations using events. This * >can* make sense insofar as events can occur zero or more times in the >future, but it's not a natural fit. What does it mean for oncomplete to >happen more than once? Is that an error? Are onsuccess and onerror >exclusive? Can they both be dispatched for an operation? The API isn't >clear. Events don't lead to good design here as they don't encapsulate >these concerns. Similarly, event handlers don't chain. This is natural, as >they could be invoked multiple times (conceptually), but it's not a good >fit for data access. It's great that IDB as async, and events are the >existing DOM model for this, but IDB's IDBRequest object is calling out for >a different kind of abstraction. I'll submit Futures for the job, but >others might work (explicit callback, whatever) so long as they maintain >chainability + async. > >- *Implicitness* >IDB is implicit in a number of places that cause confusion for folks >not intimately familiar with the contract(s) that IDB expects you to enter >into. First, t
IndexedDB, what were the issues? How do we stop it from happening again?
Comments inline. Adding some folks from the IDB team at Google to the thread as well as public-webapps. On Sunday, February 17, 2013, Miko Nieminen wrote: > > > 2013/2/15 Shwetank Dixit > >> Why did you feel it was necessary to write a layer on top of IndexedDB? >>> >> >> I think this is the main issue here. >> >> As it stands, IDB is great in terms of features and power it offers, but >> the feedback I recieved from other devs was that writing raw IndexedDB >> requires an uncomfortable amount of verbosity even for some simple tasks >> (This can be disputed, but that is the views I got from some of the >> developers I interacted with). Adding that much amount of code (once again, >> im talking of raw IndexedDB) makes it less readable and understandable. For >> beginners, this all seemed very intimidating, and for some people more >> experienced, it was a bit frustrating. >> >> > After my experiments with IDB, I don't feel that it is particularly > verbose. I have to admit that often I prefer slightly verbose syntax over > shorter one when it makes reading the code easier. In IDB's case, I think > this is the case. > > > >> For the latter bit, I reckon it would be a good practice for groups >>> working on low-level APIs to more or less systematically produce a library >>> that operates at a higher level. This would not only help developers in >>> that they could pick that up instead of the lower-level stuff, but more >>> importantly (at least in terms of goals) it would serve to validate that >>> the lower-level design is indeed appropriate for librarification. >>> >> >> I think that would be a good idea. Also, people making those low level >> APIs should still keep in mind that the resulting code should not be too >> verbose or complex. Librarification should be an advantage, but not a de >> facto requirement for developers when it comes to such APIs. It should >> still be feasable for them to write code in the raw low level API without >> writing uncomfortably verbose or complex code for simple tasks. Spec >> designers of low level APIs should not take this as a license to make >> things so complex that only they and a few others understand it, and then >> hope that some others will go ahead and make it simple for the 'common >> folk' through an abstraction library. > > > I quite don't see how to simplify IDB syntax much more. > I've avoided weighing in on this thread until I had more IDB experience. I've been wrestling with it on two fronts of late: - A re-interpretation of the API based on Futures: https://github.com/slightlyoff/DOMFuture/tree/master/reworked_APIs/IndexedDB - A new async LocalStorage design + p(r)olyfill that's bootstrapped on IDB: https://github.com/slightlyoff/async-local-storage While you might be right that it's unlikely that the API can be "simplified", I think it's trivial to extend it in ways that make it easier to reason about and use. This thread started out with a discussion of what might be done to keep IDB's perceived mistakes from reoccurring. Here's a quick stab at both an outline of the mistakes and what can be done to avoid them: - *Abuse of events* The current IDB design models one-time operations using events. This *can * make sense insofar as events can occur zero or more times in the future, but it's not a natural fit. What does it mean for oncomplete to happen more than once? Is that an error? Are onsuccess and onerror exclusive? Can they both be dispatched for an operation? The API isn't clear. Events don't lead to good design here as they don't encapsulate these concerns. Similarly, event handlers don't chain. This is natural, as they could be invoked multiple times (conceptually), but it's not a good fit for data access. It's great that IDB as async, and events are the existing DOM model for this, but IDB's IDBRequest object is calling out for a different kind of abstraction. I'll submit Futures for the job, but others might work (explicit callback, whatever) so long as they maintain chainability + async. - *Implicitness* IDB is implicit in a number of places that cause confusion for folks not intimately familiar with the contract(s) that IDB expects you to enter into. First, the use of events for delivery of notifications means that sequential-looking code that you might expect to have timing issues doesn't. Why not? Because IDB operates in some vaguely async way; you can't reason at all about events that have occurred in the past (they're not values, they're points in time). I can't find anywhere in the spec that the explicit gaurantees about delivery timing are noted ( http://www.w3.org/TR/IndexedDB/#async-api), so one could read IDB code that registers two callbacks as having a temporal dead-zone: a space in code where something might have happened but which your code might not have a chance to hear about. I realize that in practice this isn't the cas
Fwd: IndexedDB, what were the issues? How do we stop it from happening again?
FYI, Marcos started a thread about IDB and API design on the www-tag list. If you have any replies, please send them to www-...@w3.org <http://lists.w3.org/Archives/Public/www-tag/2013Feb/0003.html> Original Message Subject: IndexedDB, what were the issues? How do we stop it from happening again? Resent-Date:Thu, 7 Feb 2013 14:47:58 + Resent-From: Date: Thu, 7 Feb 2013 14:47:28 + From: ext Marcos Caceres To: www-...@w3.org Hi, Ever since IndexedDB was released in browsers, there has been a lot of criticism about the API from the developer community: mainly people saying "it sucks", but not clearly articulating *why* it sucks. For example: Dear IndexedDB. You SUCK!! https://twitter.com/Paul_Kinlan/status/72993324306411520 [[ Mikeal: IndexDB is a great go-to example of "here's how awful a committee can make an API" Jan Lehnardt @janl it’s an unfortunate history and the resulting API sucks, while the idea is still great. ]] https://twitter.com/tobie/status/236196232463257601 (Aside: the above twitter thread is a good read, actually… as it brings up gripes also with the accessibility of WebIDL to what I would consider highly-influential people in the developer community). Patrick O'Reilly It's no secret that I think #IndexedDB sucks, so what about porting +MongoDB for client-side use with a #localStorage backend? #HTML5 https://plus.google.com/103349380055842606923/posts/TjCKpCpLVa5 "I'm firmly in the camp that sees IndexedDB as a solution that'll result in more complex, more difficult to maintain, code, that'll encourage dependencies on bloated third party libraries. I'm not seeing the upside." http://blog.harritronics.com/2011/03/on-web-sql-database-and-indexeddb.html "In my opinion it is a terribly designed API and we lost a lot of power when we lost SQL, but I suppose JavaScript libraries can help make up for this." http://badassjs.com/post/42434864654/idbwrapper-a-cross-browser-indexeddb-wrapper-for Comments from: http://www.html5rocks.com/en/tutorials/indexeddb/todo/ [[ You've got to be kidding. It's a joke, right? This is the state of the art in databases? No, it's a bunch of children recreating database assembly code from the 1960's. This is awful, awful stuff…. Looking at the example, I see spaghetti control flow and data flying in all directions, like a spoiled child throwing his broccoli at the wall. The key facts of the program are obscured under layers of callbacks and come-froms. I was hoping for a usable API for regular programmers. What's shown here seems useful only to graduate students with unlimited time and no requirement to produce maintainable code on schedule. === i agree with you 100%. this is f*cking GARBAGE, and i will not use it. ]] And even from our own Alex Russell: " since 2007 and in that time I’ve seen how the dysfunctional relationship with the W3C enables terrible API design. Needless to say, the results haven’t been great. For a small taste, go look at some IndexedDB samples." http://infrequently.org/2012/12/reforming-the-w3c-tag/ I'm wondering if we can discuss what some of the issues are and get a coherent understanding about what actually sucks about IndexedDB. I'm personally concerned that we are seeing similar API patterns to IndexedDB being standardized in the System Apps APIs. If you take a look at the Alarm API [1], you see what appears to be similar patterns to those used in IndexedDB (in particular the AlarmRequest interface). You see the same XRequest patterns being applied to all of the "System APIs" (see list of specs at [2]). I'm told that this XRequest pattern is based on IndexedDB (and Mozilla's B2G Project's nonstandard "DOMRequest" object [3]). I could be wrong, and the SysApps APIs might actually be great. But, for the sanity of developers and so we avoid a barrage of "THIS SUCKS!" again, I'm wondering if this is something the TAG can look into (specially "the reformists", who made pledges to look out for the interest of developers!;)). And one more question I have, is it a goal or failure to have to rely on other's to build JS libraries on top of a standardised API to make it useful? It seems that it was the goal of IndexDB that it would not be developer friendly: "The goal of IndexedDB was to allow multiple different query processors to be built in JS. We expected few people to use it directly." https://twitter.com/adrianba/status/236249951049502720 What do you think? Kind regards, Marcos [1] http://www.w3.org/TR/web-alarms/ [2] http://www.w3.org/2012/sysapps/ [3] https://developer.mozilla.org/en-US/docs/DOM/DOMRequest -- Marcos Caceres
