Re: [eventsource] Moving Server-sent Events spec back to Last Call
On Fri, 25 Feb 2011 16:49:28 +0100, Ian Hickson i...@hixie.ch wrote: On Fri, 25 Feb 2011, Anne van Kesteren wrote: Test suite is pretty much finished (I have a few things left to clean up and need to put it on w3c-test.org now that it accepts PHP): I think I will postpone doing this. It only increases the amount of places I have to keep updated while giving very little in return. http://tc.labs.opera.com/apis/EventSource/ The source is still accessible here: http://tc.labs.opera.com/svn/apis/EventSource/ At least this test needs updating for recent changes: http://tc.labs.opera.com/apis/EventSource/request-2xx.htm It has been removed and request-status-error.htm has been updated to make sure 2xx response codes behave similar to other failures. (Thanks to Pablo Flouret.) The test suite seems very short for the spec... in particular, it doesn't seem to test the parsing of the event format very well. Anything in particular you think needs testing? But it's hard to say for sure without seeing the source of the support files. See above. Also, the tests seem to rely on a big external file, which is generally bad practice, but I'm not familiar enough with the test harness to be able to tell whether this is a real problem or not. Personally I prefer test harnesses that keep well out of the way, e.g. by using parent.test() to report results -- for example, see: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/001.html ...which works fine as a stand-alone test (and has no external dependencies) but can also work in a harness: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/all.html I probably shouldn't be whining about tests though given that I didn't write any for this spec. :-) I do not really care about the format that much. For automated testing a harness is useful and vendors seem to be somewhat willing to converge on something that can be used for most of our tests. -- Anne van Kesteren http://annevankesteren.nl/
Re: Overview of W3C technologies for mobile Web applications
Hi Ben, Le vendredi 25 février 2011 à 14:04 +, Ben Laurie a écrit : As part of a European research project I'm involved in [1], I've compiled a report on the existing technologies in development (or in discussion) at W3C for building Web applications and that are particularly relevant on mobile devices: http://www.w3.org/2011/02/mobile-web-app-state.html Nothing on security? It does mention the work on CORS and the work around widgets security, but there is no dedicated section on security — I'm not sure what would appear there that would be particularly relevant on mobile devices, any suggestion? Thanks, Dom
RE: Overview of W3C technologies for mobile Web applications
(trimming CC) Hi Somnath, Le samedi 26 février 2011 à 12:45 +0530, Somnath Chandra a écrit : This document is an excellent document. It gives present state-of-the art and roadmap ahead for development of Mobile Web. Implementation of Mobile Web with South Asian complex scripts is a challenging task. In India we have 22 constitutionally recognized languages and 12 scripts. Therefore , as a lead in W3C Mobile Web , Internationalization requirements and associcated complexities for Mobile Web implementation may also be a part of your document. If you desire , Swaran Lata , Director W3C India Country Manager and myself Dr. Somnath Chandra , Dy. country manager , W3C India would be happy to contribute. It would be indeed very useful to get your perspectives on what standards could help in getting mobile Web applications deployed across many more languages and scripts. I know Richard Ishida already provided some input on the bricks needed to display non-Latin 1 pages in a blog post: http://rishida.net/blog/?p=445 I wonder if this could be a starting point for something more formal that could help in this space? (I expect I'll get to hear some of your thoughts on this at the Mobile Web Apps camp in WWW2011 http://www.w3.org/2011/03/w3c-track.html) Thanks, Dom
Re: Overview of W3C technologies for mobile Web applications
Hi Paul, Le vendredi 25 février 2011 à 16:53 +0100, Paul Libbrecht a écrit : I definitely agree this is a useful deliverable; I wish more EU projects be as careful in their survey as that! Thanks! I was looking to see if MathML was mentioned (I think it should as a future technology but it has almost zero coverage on the mobile world yet). But I realize that HTML is also not decomposed there. It seems to be assumed and mentioned in many places. Am I right? The approach I've taken is to highlight the most relevant features in specs across the large number of W3C groups for developing applications on mobile devices. While there are certainly use cases for display maths as part of mobile app (e.g. for education), it hasn't struck me as a particularly mobile-relevant feature, which is why I haven't included it. The fact that it has very little implementation on mobile browsers doesn't help either. But I'd be happy to reconsider that position if you have further input on this :) I would think a feature-by-feature analysis of what's in HTML4/5/XHTML and works on mobiles would be rather useful. I would certainly love that as well, but I think it is also beyond what I can possibly manage :) That said, part of the MobiWebApp project is also to help on the development of test suites for the relevant technologies, so maybe we'll get some of that picture when this makes more progress. Thanks for the feedback, Dom
Re: Overview of W3C technologies for mobile Web applications
Hi Dom, On Mar 7, 2011, at 11:57 AM, Dominique Hazael-Massieux wrote: Hi Ben, Le vendredi 25 février 2011 à 14:04 +, Ben Laurie a écrit : As part of a European research project I'm involved in [1], I've compiled a report on the existing technologies in development (or in discussion) at W3C for building Web applications and that are particularly relevant on mobile devices: http://www.w3.org/2011/02/mobile-web-app-state.html Nothing on security? It does mention the work on CORS and the work around widgets security, but there is no dedicated section on security — I'm not sure what would appear there that would be particularly relevant on mobile devices, any suggestion? For example, mobile devices are usually correlated with a single individual, or at most a small group of people. The data contained on them is often personal. As such, identifiers related to mobile devices (phone number, IMEI) constitute sensitive information. In addition, they carry an increasing array of sensors again closely related to a single individual (e.g. GPS). By providing Javascript APIs to device functionality, we are opening up a mechanism which allows unidentified (or, identified mostly only by unreliable technologies) access to personal and/or sensitive information. There are some security benefits to doing so with Javascript APIs accessible only to the recipient of an HTTP request initiated by the user, but also some potential pitfalls. Of course, I can't tell if that's what Ben was alluding to with his question ;) Regards, - John
CfC: publish Last Call Working Draft of HTML5 Web Messaging; deadline March 14
This is a Call for Consensus (CfC) to publish a new Last Call Working Draft of the HTML5 Web Messaging spec based on the following version of the spec (copied from ED version 1.77): http://dev.w3.org/html5/postmsg/publish/LCWD-webmessaging-201103TBD.html This CfC satisfies the group's requirement to record the group's decision to request advancement for this LCWD. Note the Process Document states the following regarding the significance/meaning of a LCWD: [[ http://www.w3.org/2005/10/Process-20051014/tr.html#last-call Purpose: A Working Group's Last Call announcement is a signal that: * the Working Group believes that it has satisfied its relevant technical requirements (e.g., of the charter or requirements document) in the Working Draft; * the Working Group believes that it has satisfied significant dependencies with other groups; * other groups SHOULD review the document to confirm that these dependencies have been satisfied. In general, a Last Call announcement is also a signal that the Working Group is planning to advance the technical report to later maturity levels. ]] Positive response to this CfC is preferred and encouraged and silence will be assumed to mean agreement with the proposal. The deadline for comments is March 14. Please send all comments to: public-webapps@w3.org Assuming there is consensus to publish this LCWD, the tentative plan is to publish it on March 17 with a the LC comment period ending June 1. -Art Barstow
Re: [eventsource] Moving Server-sent Events spec back to Last Call
On Mon, 7 Mar 2011, Anne van Kesteren wrote: The test suite seems very short for the spec... in particular, it doesn't seem to test the parsing of the event format very well. Anything in particular you think needs testing? Zero, one or two BOMs before an event. Parsing of comments with zero, one, or 2049 bytes of content, on either side of a real field. Handling of unknown fields. Handling of fields with names similar to but not identical to valid fields, in particular with spaces, hyphens, underscares, nulls where they're not expected; with uppercase names instead of lowercase ones; with turkish upper-case dotted Is where a lowercase i is expected. Fields with and without value parts, for each allowed field and for illegal fields. Fields with and without spaces after their colons. End of line handling: cr/lf/cr, lf/cr/lf, etc. Event dispatch when the last field ends with eof. Nulls in values. Surrogate halves in field names and values. UTF-8 errors in field names and values. Retry field with leading zeros that are binary and octal (011, 071), with hex values (0x01), with non-numeric trailing garbage (123x). Also, the tests seem to rely on a big external file, which is generally bad practice, but I'm not familiar enough with the test harness to be able to tell whether this is a real problem or not. Personally I prefer test harnesses that keep well out of the way, e.g. by using parent.test() to report results -- for example, see: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/001.html ...which works fine as a stand-alone test (and has no external dependencies) but can also work in a harness: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/all.html I probably shouldn't be whining about tests though given that I didn't write any for this spec. :-) I do not really care about the format that much. For automated testing a harness is useful and vendors seem to be somewhat willing to converge on something that can be used for most of our tests. I've no problem with a harness, my problem is just that the tests require the harness to run. I prefer tests that are standalone but when used in a harness can report errors to their harness. But that's just a personal preference. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
[Bug 11567] IndexedDB should utilize the new WebIDL static feature
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11567 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|NEW |RESOLVED CC||jo...@sicking.cc Resolution||FIXED -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 12261] New: Update cursor API
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12261 Summary: Update cursor API Product: WebAppsWG Version: unspecified Platform: PC OS/Version: All Status: NEW Severity: normal Priority: P2 Component: Indexed Database API AssignedTo: dave.n...@w3.org ReportedBy: jo...@sicking.cc QAContact: member-webapi-...@w3.org CC: m...@w3.org, public-webapps@w3.org This was discussed on the list. Goals are: * Give access to the current key in the objectStore when iterating indexes * Make it more clear that index key-cursors don't give access to the entry value -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
Using ArrayBuffer as payload for binary data to/from Web Workers
Now that ArrayBuffer has made its way into XHR, I think it would be reasonable to somehow use this new object type as a way to pass data to and from Workers without copying. I've seen hints and thoughts about this here and there, but I've never seen a formal discussion. I'm not even sure if webapps is the place for this discussion, although it seems like a reasonable place. Please let me know if there is a better place. Has there been discussion anywhere that I've missed? The idea is that the buffer owned by the ArrayBuffer object could be passed to and from Web Workers with some mechanism (probably part of ArrayBuffer) controlling access to the data. Pass the data to the Web Worker and it would be accessible by the Worker, but not to the main thread. Pass it back and now the main thread has access and the Worker can no longer touch the data. Thoughts? Am I bringing up an issue that has already been beaten to death elsewhere? - ~Chris cmar...@apple.com
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 6:05 PM, Chris Marrin cmar...@apple.com wrote: Now that ArrayBuffer has made its way into XHR, I think it would be reasonable to somehow use this new object type as a way to pass data to and from Workers without copying. I've seen hints and thoughts about this here and there, but I've never seen a formal discussion. I'm not even sure if webapps is the place for this discussion, although it seems like a reasonable place. Please let me know if there is a better place. ArrayBuffer is the most obvious use for zero-copy messaging, but I don't think it should be limited to it... Has there been discussion anywhere that I've missed? Probably not the only one, but check the WebWorkers and images thread on whatwg. -- Glenn Maynard
[Bug 12114] Blocked setVersion transactions should be aborted when their database is closed
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12114 Jeremy Orlow jor...@chromium.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 12016] After page unload, all IDBDatabases attached to the document should have .close() called on them
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12016 Jeremy Orlow jor...@chromium.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Comment #2 from Jeremy Orlow jor...@chromium.org 2011-03-08 00:52:28 UTC --- This is already specced, actually. -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 11375] [IndexedDB] Error codes need to be assigned new numbers
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11375 Jeremy Orlow jor...@chromium.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Comment #1 from Jeremy Orlow jor...@chromium.org 2011-03-08 01:19:26 UTC --- This was somewhat done at some point. And any further cleanup should probably wait until exception/error stuff upstream in WebIDL is settled. -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 12261] Update cursor API
http://www.w3.org/Bugs/Public/show_bug.cgi?id=12261 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 11747] openCursor's text contradicts itself
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11747 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|NEW |RESOLVED CC||jo...@sicking.cc Resolution||FIXED -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mar 7, 2011, at 4:46 PM, Kenneth Russell wrote: On Mon, Mar 7, 2011 at 3:54 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 7, 2011 at 6:05 PM, Chris Marrin cmar...@apple.com wrote: Now that ArrayBuffer has made its way into XHR, I think it would be reasonable to somehow use this new object type as a way to pass data to and from Workers without copying. I've seen hints and thoughts about this here and there, but I've never seen a formal discussion. I'm not even sure if webapps is the place for this discussion, although it seems like a reasonable place. Please let me know if there is a better place. ArrayBuffer is the most obvious use for zero-copy messaging, but I don't think it should be limited to it... Has there been discussion anywhere that I've missed? Probably not the only one, but check the WebWorkers and images thread on whatwg. There's definitely interest among the editors of the Typed Array spec in revising the spec to support zero-copy data transfers to and from web workers. In informal offline discussions, there was a tentative plan to put up a new draft for discussion within the next month or so. A goal was to prototype it before solidifying a spec so that we can be assured it will work well for real-world use cases. Yeah, I guess the question is whether we should put the functionality into ArrayBuffer, or into a wrapper class which would part of the Web Worker spec. The latter might make it easier to add other resources (like image and canvas) at some point. But I agree, it should be implemented before finalizing anything. Did I hear you volunteer to add a strawman proposal to the Typed Array spec? :-) - ~Chris cmar...@apple.com
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 5:55 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 7, 2011 at 8:04 PM, Chris Marrin cmar...@apple.com wrote: Probably not the only one, but check the WebWorkers and images thread on whatwg. Yeah, I thought about that case. The extra complication there is that images are rendering resources, which muddies the issues some. If the 2D Canvas API were able to use ArrayBuffer data as an image source like WebGL can, then we could kill 2 birds with one stone. Pass the ArrayBuffer to the worker and have it generate some image data, then pass it back and render it to a canvas. You could even split the array buffer into tiles and have multiple workers operate on the tiles. I'd expect CanvasPixelArray to allow optimizations that ArrayBuffer doesn't, since the implementation can use the native surface format, translating to RGBA for the script transparently. This can matter for streaming textures to OpenGL/D3D, too; creating BGRA textures on nVidia hardware is typically much faster than RGBA ones. I don't recall if this has been brought up: are there cases where explicit zero-copy messaging is better than transparent copy-on-write? Yes. Copy on write does not efficiently handle the case where large amounts of data are continually produced by workers and posted to the main thread for display. Each time the worker posts a block of data to the main thread, the next time it attempts to update its version of the block for the next iteration, a copy will need to be made so the main thread's version appears immutable. Bi-directional, zero-copy messaging is needed to preserve ECMAScript's shared-nothing semantic while avoiding a continuous stream of memory allocation for producer/consumer queues between workers and the main thread. The Vertex Buffer Object demo at http://khronos.org/webgl/wiki/Demo_Repository provides an example where multiple workers should be able to produce independent portions of the mesh for rendering by the main thread, and where a nearly linear speedup is possible. -Ken
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 5:18 PM, Chris Marrin cmar...@apple.com wrote: On Mar 7, 2011, at 4:46 PM, Kenneth Russell wrote: On Mon, Mar 7, 2011 at 3:54 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 7, 2011 at 6:05 PM, Chris Marrin cmar...@apple.com wrote: Now that ArrayBuffer has made its way into XHR, I think it would be reasonable to somehow use this new object type as a way to pass data to and from Workers without copying. I've seen hints and thoughts about this here and there, but I've never seen a formal discussion. I'm not even sure if webapps is the place for this discussion, although it seems like a reasonable place. Please let me know if there is a better place. ArrayBuffer is the most obvious use for zero-copy messaging, but I don't think it should be limited to it... Has there been discussion anywhere that I've missed? Probably not the only one, but check the WebWorkers and images thread on whatwg. There's definitely interest among the editors of the Typed Array spec in revising the spec to support zero-copy data transfers to and from web workers. In informal offline discussions, there was a tentative plan to put up a new draft for discussion within the next month or so. A goal was to prototype it before solidifying a spec so that we can be assured it will work well for real-world use cases. Yeah, I guess the question is whether we should put the functionality into ArrayBuffer, or into a wrapper class which would part of the Web Worker spec. The latter might make it easier to add other resources (like image and canvas) at some point. But I agree, it should be implemented before finalizing anything. Did I hear you volunteer to add a strawman proposal to the Typed Array spec? :-) Yes, you did. :-) -Ken
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 9:07 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 3/7/11 8:55 PM, Glenn Maynard wrote: I'd expect CanvasPixelArray to allow optimizations that ArrayBuffer doesn't, since the implementation can use the native surface format, translating to RGBA for the script transparently. This can matter for streaming textures to OpenGL/D3D, too; creating BGRA textures on nVidia hardware is typically much faster than RGBA ones. But modifying the ImageData is not supposed to modify what the graphics card sees, right? So you have to make a copy here on putImageData (or on the next write to the image data), right? Right, either way you need to make a copy when you send the data back to the canvas. But, if the surface format of the memory buffer matches a native format of the graphics card, the copy is very fast--probably a DMA, for hardware-accelerated surfaces. (The exact mechanics of this are hidden behind the drivers, of course, but that's what it looks like in my experience.) For example, nVidia cards typically are BGRA natively, and if you give them RGBA with OpenGL it needs to be converted before (or as) it's copied, which is much slower. (Of course, CanvasPixelData itself always has to *appear* RGBA, but that doesn't mean the backing store needs to actually be RGBA.) I don't recall if this has been brought up: are there cases where explicit zero-copy messaging is better than transparent copy-on-write? Transparent copy on write requires that each write do a should I copy? check, right? Yeah. The question is probably whether that check can be moved out of inner loops. It couldn't be if function calls are made that might have side-effects to cause a private object to become a COW object (eg. non-pure functions). For page-aligned buffers, an implementation could optimize these checks away entirely by marking the buffer read-only, so if a write is made it'll raise an exception that the engine can catch. Yes. Copy on write does not efficiently handle the case where large amounts of data are continually produced by workers and posted to the main thread for display. Each time the worker posts a block of data to the main thread, the next time it attempts to update its version of the block for the next iteration, a copy will need to be made so the main thread's version appears immutable. But how is this any different than just throwing away the ArrayBuffer and creating a fresh one after each postMessage? I guess one issue is that double-buffering would run into GC problems. With explicit zero-copy semantics, you can maintain two buffers: one for the worker as it creates the next frame, and one for the main thread as it updates the canvas. On each frame, the worker posts its new frame to the main thread, and the main thread posts the old buffer back to the worker. (This avoids reallocating large buffers every frame.) That doesn't work with COW, since there's no way to deterministically throw away the old buffer so it knows that it doesn't need to make a copy on the other side. That could be dealt with by adding methods like ArrayBuffer.discard(), though... -- Glenn Maynard
[Bug 11528] We should add some form of dynamic transaction to IndexedDB
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11528 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|NEW |RESOLVED CC||jo...@sicking.cc Resolution||LATER --- Comment #1 from Jonas Sicking jo...@sicking.cc 2011-03-08 03:15:29 UTC --- Resolving this as LATER so that we can pick it up once we start working on v2 -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 11351] [IndexedDB] Should we have a maximum key size (or something like that)?
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11351 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|NEW |RESOLVED CC||jo...@sicking.cc Resolution||WONTFIX --- Comment #1 from Jonas Sicking jo...@sicking.cc 2011-03-08 03:43:29 UTC --- This was discussed on the list and it seems like there was agreement not to put in any hard limits here. -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 9796] IndexedDB's async interface should be available within workers
http://www.w3.org/Bugs/Public/show_bug.cgi?id=9796 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|ASSIGNED|RESOLVED CC||jo...@sicking.cc Resolution||FIXED -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 11553] Ensure indexedDBSync is on the right worker interface
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11553 Jeremy Orlow jor...@chromium.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
[Bug 11164] There is no way to get from an error event to other objectStores
http://www.w3.org/Bugs/Public/show_bug.cgi?id=11164 Jonas Sicking jo...@sicking.cc changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED AssignedTo|nikunj.me...@oracle.com |jo...@sicking.cc --- Comment #2 from Jonas Sicking jo...@sicking.cc 2011-03-08 04:19:58 UTC --- Checked in a fix -- Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug.
Re: [IndexedDB] Two Real World Use-Cases
On Thu, Mar 3, 2011 at 4:15 AM, Joran Greef jo...@ronomon.com wrote: Hi Jonas I have been trying out your suggestion of using a separate object store to do manual indexing (and so support compound indexes or index object properties with arrays as values). There are some problems with this approach: 1. It's far too slow. To put an object and insert 50 index records (typical when updating an inverted index) this way takes 100ms using IDB versus 10ms using WebSQL (with a separate indexes table and compound primary key on index name and object key). For instance, my application has a real requirement to replicate 4,000,000 emails between client and server and I would not be prepared to accept latencies of 100ms to store each object. That's more than the network latency. This doesn't seem right. Assuming your WebSQL implementation had all the same indexes isn't it doing pretty much the same things as using separate objectStores in IDB? Why would it be an order of magnitude slower? I'm sure whatever implementation you're using hasn't seen much optimization but you seem to be implying there's something more fundamental? The only thing I can think of to blame would be the fat in the objectStore interface -- like, for instance, the index building facilities. It seems to me your proposed solution is to add yet more fat to the interface (more complex indexing), but wouldn't it be just as suitable to instead strip down objectStores to their bare essentials to make them more suitable to act as indexes? Then the indexing functionality and all the hard decisions could be punted to libraries where they'd be free to innovate. This would simplify the spec greatly by lopping things like keyranges and schema enforcement out completely, not to mention the entire index cursor API that is similar to but slightly different than objectStore cursors. And yes, in combination with some fundamental relational operations (as you already proposed), and some stats, you could easily implement a relational db. And I don't see anything wrong with that. You could implement all kinds of databases -- and isn't that the point of this API? Is there any real limitation that I'm missing to making objectStores suitable as indexes -- performance or otherwise? Namespace collisions come mind (because of all the new objectStores that would have to be managed) but conventions seem to have served the world of sql just fine for this. 2. It's a waste of space. Using a separate object store to do manual indexing may work in theory but it does not work in practice. I do not think it can even be remotely suggested as a panacea, however temporary it may be. Why? You wouldn't necessarily have to store the whole object in each index, just the index key, a value and some pointer to the original source object. Something to resolve this pointer to the source would need to be spec'd (a la couchdb's include_docs), but that's simple. Even better, say it were possible to define a link relation on an object store that can resolve to its source object -- you could define a source link relation and the property to use -- and this would have the added bonus of being more broadly applicable than just linking an index record to its source instance. We can fix all of this right now very simply: 1. Enable objectStore.put and objectStore.delete to accept a setIndexes option and an unsetIndexes option. The value passed for either option would be an array (string list) of index references. This would only work for indexes arrays of strings, right? Things can get much more complicated than that, and when they do you'd have to use an objectStore to do your indexing anyway, right? 2. The object would first be removed as a member from any indexes referenced by the unsetIndexes option. Any referenced indexes which would be empty thereafter would be removed. 3. The object would then be added as a member to any indexes referenced by the setIndexes option. Any referenced indexes which do not yet exist would be created. This would provide the much-needed indexing capabilities presently lacking in IDB without sacrificing performance. Why is it more theoretically performant than using objectStores in the raw? It would also enable developers to use IDB statefully (MySQL-like pre-defined schemas with the DB taking on the complexities of schema migration and data migration) or statelessly (See Berkeley DB with the application responsible for the complexities of data maintenance) rather than enforcing an assumption at such an early stage. I don't necessarily understand the stateful vs. stateless distinction here. I don't see how your proposed solution removes the requirement for IDB to enforce constraints when certain indexes are present. Developers would already be able to use IDB statefully (with predefined schemas) -- they'd just use a library that has a schema mechanism. I doubt such a library for IDB already exists, but it'd
[IndexedDB] What should be allowed as a key path?
As far as I recall, we never settled on how key path should be specified. Right now in Chrome, we allow any combination of .'s and static array lookups. So, for example, we allow foo.bar[1][2].baz. I don't remember any specific use cases for the array lookups though, so I'm wondering if we should just specify property [. property]* to be the key path specification. Where property is whatever's allowed in JavaScript. It seems to me that this should support most use cases, should be easy to understand and implement, and should leave us a lot of flexibility to support other use cases as we may come up with them. Thoughts? J
Re: [IndexedDB] Two Real World Use-Cases
On 08 Mar 2011, at 7:23 AM, Dean Landolt wrote: This doesn't seem right. Assuming your WebSQL implementation had all the same indexes isn't it doing pretty much the same things as using separate objectStores in IDB? Why would it be an order of magnitude slower? I'm sure whatever implementation you're using hasn't seen much optimization but you seem to be implying there's something more fundamental? The only thing I can think of to blame would be the fat in the objectStore interface -- like, for instance, the index building facilities. It seems to me your proposed solution is to add yet more fat to the interface (more complex indexing), but wouldn't it be just as suitable to instead strip down objectStores to their bare essentials to make them more suitable to act as indexes? Then the indexing functionality and all the hard decisions could be punted to libraries where they'd be free to innovate. Exactly. It's not what one would expect, and indication of the poor state of the IDB implementation (which is essentially a wrapper around SQLite anyway). If someone is advising that object stores be used to handle indexes then may I be the first to raise a red flag and say that IDB is failing us (and it would have been better for the spec team to provide a locking mechanism for LocalStorage so it could be used in that way). The whole point of IDB as far as I can see is to provide transactional indexed access to a key value store. Why? You wouldn't necessarily have to store the whole object in each index, just the index key, a value and some pointer to the original source object. Something to resolve this pointer to the source would need to be spec'd (a la couchdb's include_docs), but that's simple. Even better, say it were possible to define a link relation on an object store that can resolve to its source object -- you could define a source link relation and the property to use -- and this would have the added bonus of being more broadly applicable than just linking an index record to its source instance. Think of the object creation and JSON serialization/deserialization overhead for putting 50 indexes and you have got more than enough waste there already. We can fix all of this right now very simply: 1. Enable objectStore.put and objectStore.delete to accept a setIndexes option and an unsetIndexes option. The value passed for either option would be an array (string list) of index references. This would only work for indexes arrays of strings, right? Things can get much more complicated than that, and when they do you'd have to use an objectStore to do your indexing anyway, right? No it would work for pretty much anything. The application would be free to determine the indexes, and also to convert query parameters into indexes when querying. It's essentially computed indexes without the hassles of IDB trying to do it (there was an interesting thread last year on the challenges of storing am index computing function in IDB). Why is it more theoretically performant than using objectStores in the raw? It's a more direct interface. Think about it for a second. Using objectStores in the raw is interpolating O(n) complexity with multiple function calls, to give just one reason. If IDB can receive a list of indexes to add and remove an object to and from, then it can also do things like perform a set difference first to save unnecessary IO. I have written a database or two with this technique and it's certainly faster. I don't necessarily understand the stateful vs. stateless distinction here. I don't see how your proposed solution removes the requirement for IDB to enforce constraints when certain indexes are present. Developers would already be able to use IDB statefully (with predefined schemas) -- they'd just use a library that has a schema mechanism. I doubt such a library for IDB already exists, but it'd be quite easy to port perstore, for instance, which is derived from the IDB API and already has this functionality using json-schema. There will no doubt be many ORM-like libraries that will pop up as soon as IDB starts to stabilize (or as soon as it gets a node.js implementation). The trouble is you always think a database would be quite easy until you actually try to do it yourself. At first when I dug into IDB I didn't think there would be any problems that could not be handled in some way. I have actually switched back to WebSQL now and will encourage my users to use Safari or Chrome as long as these browsers support WebSQL (and I hope Chrome will at least finish up by adding a quota interface for WebSQL). IDB right now is like a completely neutered slower SQLite without any of the benefits to be expected of a transactional indexed KV store. It's really sad. For examples of stateless databases see the interfaces for Redis (the best example, and a perfect target for IDB), Berkeley, Tokyo. For a statefull
Re: [IndexedDB] Compound and multiple keys
On Fri, Jan 21, 2011 at 1:41 AM, Jeremy Orlow jor...@chromium.org wrote:
On Thu, Jan 20, 2011 at 6:29 PM, Tab Atkins Jr. jackalm...@gmail.comwrote:
On Thu, Jan 20, 2011 at 10:12 AM, Keean Schupke ke...@fry-it.com wrote:
Compound primary keys are commonly used afaik.
Indeed. It's one of the common themes in the debate between natural
and synthetic keys.
Fair enough.
Should we allow explicit compound keys? I.e myOS.put({...}, ['first name',
'last name'])? I feel pretty strongly that if we do, we should require this
be specified up-front when creating the objectStore. I.e. add some
additional parameter to the optional options object. Otherwise, we'll force
implementations to handle variable compound keys for just this one case,
which seems kind of silly.
The other option is to just disallow them.
After thinking about it a bunch and talking to others, I'm actually leaning
towards both option A and B. Although this will be a little harder for
implementors, it seems like there are solid reasons why some users would
want to use A and solid reasons why others would want to use B.
Any objections to us going that route?
Thanks,
J
