Splitting into its own thread since this isn't really connected to the new Async interface and that thread is already pretty big.
On Wed, Jun 9, 2010 at 10:36 PM, Mikeal Rogers <mikeal.rog...@gmail.com>wrote: > I've been looking through the current spec and all the proposed changes. > > Great work. I'm going to be building a CouchDB compatible API on top > of IndexedDB that can support peer-to-peer replication without other > CouchDB instances. > > One of the things that will entail is a by-sequence index for all the > changes in a give "database" (in my case a database will be scoped to > more than one ObjectStore). In order to accomplish this I'll need to > keep the last known sequence around so that each new write can create > a new entry in the by-sequence index. The problem is that if another > tab/window writes to the database it'll increment that sequence and I > won't be notified so I would have to start every transaction with a > check on the sequence index for the last sequence which seems like a > lot of extra cursor calls. > It would be a lot of extra calls, but I'm a bit hesitant to add much more API surface area to v1, and the fall back plan doesn't seem too unreasonable. > What I really need is an event listener on an ObjectStore that fires > after a transaction is committed to the store but before the next > transaction is run that gives me information about the commits to the > ObjectStore. > > Thoughts? > To do this, we could specify an IndexedDatabaseRequest.ontransactioncommitted event that would be guaranteed to fire after every commit and before we started the next transaction. I think that'd meet your needs and not add too much additional surface area... What do others think? -Mikeal > > On Wed, Jun 9, 2010 at 11:40 AM, Jeremy Orlow <jor...@chromium.org> wrote: > > On Wed, Jun 9, 2010 at 7:25 PM, Jonas Sicking <jo...@sicking.cc> wrote: > >> > >> On Wed, Jun 9, 2010 at 7:42 AM, Jeremy Orlow <jor...@chromium.org> > wrote: > >> > On Tue, May 18, 2010 at 8:34 PM, Jonas Sicking <jo...@sicking.cc> > wrote: > >> >> > >> >> On Tue, May 18, 2010 at 12:10 PM, Jeremy Orlow <jor...@chromium.org> > >> >> wrote: > >> >> > I'm not sure I like the idea of offering sync cursors either since > >> >> > the > >> >> > UA > >> >> > will either need to load everything into memory before starting or > >> >> > risk > >> >> > blocking on disk IO for large data sets. Thus I'm not sure I > support > >> >> > the > >> >> > idea of synchronous cursors. But, at the same time, I'm concerned > >> >> > about > >> >> > the > >> >> > overhead of firing one event per value with async cursors. Which > is > >> >> > why I > >> >> > was suggesting an interface where the common case (the data is in > >> >> > memory) is > >> >> > done synchronously but the uncommon case (we'd block if we had to > >> >> > respond > >> >> > synchronously) has to be handled since we guarantee that the first > >> >> > time > >> >> > will > >> >> > be forced to be asynchronous. > >> >> > Like I said, I'm not super happy with what I proposed, but I think > >> >> > some > >> >> > hybrid async/sync interface is really what we need. Have you guys > >> >> > spent > >> >> > any > >> >> > time thinking about something like this? How dead-set are you on > >> >> > synchronous cursors? > >> >> > >> >> The idea is that synchronous cursors load all the required data into > >> >> memory, yes. I think it would help authors a lot to be able to load > >> >> small chunks of data into memory and read and write to it > >> >> synchronously. Dealing with asynchronous operations constantly is > >> >> certainly possible, but a bit of a pain for authors. > >> >> > >> >> I don't think we should obsess too much about not keeping things in > >> >> memory, we already have things like canvas and the DOM which adds up > >> >> to non-trivial amounts of memory. > >> >> > >> >> Just because data is loaded from a database doesn't mean it's huge. > >> >> > >> >> I do note that you're not as concerned about getAll(), which actually > >> >> have worse memory characteristics than synchronous cursors since you > >> >> need to create the full JS object graph in memory. > >> > > >> > I've been thinking about this off and on since the original proposal > was > >> > made, and I just don't feel right about getAll() or synchronous > cursors. > >> > You make some good points about there already being many ways to > >> > overwhelm > >> > ram with webAPIs, but is there any place we make it so easy? You're > >> > right > >> > that just because it's a database doesn't mean it needs to be huge, > but > >> > often times they can get quite big. And if a developer doesn't spend > >> > time > >> > making sure they test their app with the upper ends of what users may > >> > possibly see, it just seems like this is a recipe for problems. > >> > Here's a concrete example: structured clone allows you to store image > >> > data. > >> > Lets say I'm building an image hosting site and that I cache all the > >> > images > >> > along with their thumbnails locally in an IndexedDB entity store. > Lets > >> > say > >> > each thumbnail is a trivial amount, but each image is 1MB. I have an > >> > album > >> > with 1000 images. I do |var photos = > >> > albumIndex.getAllObjects(albumName);| > >> > and then iterate over that to get the thumbnails. But I've just > loaded > >> > over > >> > 1GB of stuff into ram (assuming no additional inefficiency/blowup). I > >> > suppose it's possible JavaScript engines could build mechanisms to > fetch > >> > this stuff lazily (like you could even with a synchronous cursor) but > >> > that > >> > will take time/effort and introduce lag in the page (while fetching > >> > additional info from disk). > >> > > >> > I'm not completely against the idea of getAll/sync cursors, but I do > >> > think > >> > they should be de-coupled from this proposed API. I would also > suggest > >> > that > >> > we re-consider them only after at least one implementation has normal > >> > cursors working and there's been some experimentation with it. Until > >> > then, > >> > we're basing most of our arguments on intuition and assumptions. > >> > >> I'm not married to the concept of sync cursors. However I pretty > >> strongly feel that getAll is something we need. If we just allow > >> cursors for getting multiple results I think we'll see an extremely > >> common pattern of people using a cursor to loop through a result set > >> and put values into an array. > >> > >> Yes, it can be misused, but I don't see a reason why people wouldn't > >> misuse a cursor just as much. If they don't think about the fact that > >> a range contains lots of data when using getAll, why would they think > >> about it when using cursors? > > > > Once again, I feel like there is a lot of speculation (more than normal) > > happening here. I'd prefer we take the Async API without the sync > cursors > > or getAll and give the rest of the API some time to bake before > considering > > it again. Ideally by then we'd have at least one or two early adopters > that > > can give their perspective on the issue. > > J > > >
