Re: Two years on and still no sensible web storage solutions exist
I'd like to propose as a constructive strategy not to flame/offend everybody right off the bat. I'm sure there's reasons, I'd like to hear them, too. On Fri, Nov 9, 2012 at 7:24 PM, Todd Blanchard toddvblanch...@gmail.comwrote: It has been two years since the following little note was attached to the Web SQL Spec This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. This move has left the web browser world in disarray and has been widely misconstrued by readers to mean Web SQL is deprecated and will not be supported in the future - better port to IndexedDB. Today, TWO YEARS LATER, we have SQLite on iOS, Android, Chrome, and Safari but no IndexedDB. On Firefox we have IndexedDB with SQLite available only via a browser extension https://addons.mozilla.org/en-US/firefox/addon/html5-websql-for-firefox/ (annoying but liveable), and on IE[89] only DOM storage with IndexedDB expected on IE 10. Sources: http://caniuse.com/indexeddb http://caniuse.com/sql-storage As someone who is trying to build an offline web app the works both on browsers and smart phones and needs to store a lot of client side complex data that will require lots of joins - let me just say WTF? Why do we have standards again? You're not helping. I look around at information on the state of storage options and I read stuff like this: Since November 18, 2010, the W3C announced that Web SQL database is a deprecated specification. This is a recommendation for web developers to no longer use the technology as effectively, the spec will receive no new updates and browser vendors aren't encouraged to support this technology. The new alternative is IndexedDB which is already available on Chrome 12+ and Firefox 5+, and, soon, in IE 10 as well. Was it really the intent to abandon SQL as a concept because everybody is using the same well tested and portable library? Are we doomed to never ever having a stable and consistent platform to work on?Because there are no competing implementations for browser accessible SQL database access - everybody settled on one nice bit of code to fulfill this requirement - the specification is dropped and the browser developers drop even trying to implement SQL database access and there is even talk of removing it? WTF is wrong with you people? IndexedDB is fine - add it. But don't for a second try to tell me it is anywhere near as powerful as having a real SQL database on hand. Also, the hand waving about how it should be possible to add SQL on top of IndexedDB rings hollow. If it were easy we would have it. OTOH, going the other way seems pretty easy. https://github.com/axemclion/IndexedDBShim maybe the developers should just pursue this approach with SQLite and call it a day. Absolutely disgusted. -Todd Blanchard
Re: Two years on and still no sensible web storage solutions exist
My recollection is that some vendors refused to ever support SQLite, making a SQL-based standard not really feasible. You can undoubtedly review the w3c archives to find out more details if you want to know the rationale they expressed at the time. It's unfortunate that IndexedDB is still not widely supported by browsers - I share your frustration. However, as you say, you can write to the IndexedDB APIs, and use a shim to get support on platforms that do not support it yet. -atw On Mon, Nov 12, 2012 at 9:50 AM, Florian Bösch pya...@gmail.com wrote: I'd like to propose as a constructive strategy not to flame/offend everybody right off the bat. I'm sure there's reasons, I'd like to hear them, too. On Fri, Nov 9, 2012 at 7:24 PM, Todd Blanchard toddvblanch...@gmail.comwrote: It has been two years since the following little note was attached to the Web SQL Spec This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. This move has left the web browser world in disarray and has been widely misconstrued by readers to mean Web SQL is deprecated and will not be supported in the future - better port to IndexedDB. Today, TWO YEARS LATER, we have SQLite on iOS, Android, Chrome, and Safari but no IndexedDB. On Firefox we have IndexedDB with SQLite available only via a browser extension https://addons.mozilla.org/en-US/firefox/addon/html5-websql-for-firefox/ (annoying but liveable), and on IE[89] only DOM storage with IndexedDB expected on IE 10. Sources: http://caniuse.com/indexeddb http://caniuse.com/sql-storage As someone who is trying to build an offline web app the works both on browsers and smart phones and needs to store a lot of client side complex data that will require lots of joins - let me just say WTF? Why do we have standards again? You're not helping. I look around at information on the state of storage options and I read stuff like this: Since November 18, 2010, the W3C announced that Web SQL database is a deprecated specification. This is a recommendation for web developers to no longer use the technology as effectively, the spec will receive no new updates and browser vendors aren't encouraged to support this technology. The new alternative is IndexedDB which is already available on Chrome 12+ and Firefox 5+, and, soon, in IE 10 as well. Was it really the intent to abandon SQL as a concept because everybody is using the same well tested and portable library? Are we doomed to never ever having a stable and consistent platform to work on?Because there are no competing implementations for browser accessible SQL database access - everybody settled on one nice bit of code to fulfill this requirement - the specification is dropped and the browser developers drop even trying to implement SQL database access and there is even talk of removing it? WTF is wrong with you people? IndexedDB is fine - add it. But don't for a second try to tell me it is anywhere near as powerful as having a real SQL database on hand. Also, the hand waving about how it should be possible to add SQL on top of IndexedDB rings hollow. If it were easy we would have it. OTOH, going the other way seems pretty easy. https://github.com/axemclion/IndexedDBShim maybe the developers should just pursue this approach with SQLite and call it a day. Absolutely disgusted. -Todd Blanchard
Re: Two years on and still no sensible web storage solutions exist
Don't care if it is SQLite so long as it is a real relational database. IndexedDB or some other key-value store is no substitute for the power of a relational database and is unlikely to be performant given the number of different views we support. I don't mind too much if IE used mssql, Firefox settled on mysql, or postgresql or anything else so long as it is a proper relational database - switching sql dialects is already a fact of life. I've settled on using SQLite on smart phones (Android and iOS - we will not support phones lacking this capability) and I'm going to just rely on calls to the server for desktop apps. I think fetching from the server will be no worse for the average desktop user than iterating a key-value store trying to calculate joins in javascript. -Todd Blanchard On Nov 12, 2012, at 1:14 AM, Andrew Wilson wrote: My recollection is that some vendors refused to ever support SQLite, making a SQL-based standard not really feasible. You can undoubtedly review the w3c archives to find out more details if you want to know the rationale they expressed at the time. It's unfortunate that IndexedDB is still not widely supported by browsers - I share your frustration. However, as you say, you can write to the IndexedDB APIs, and use a shim to get support on platforms that do not support it yet. -atw
Mail List Etiquette [Was: Re: Two years on and still no sensible web storage solutions exist]
Todd - your e-mail fails the respectful manner requirement of the group's e-mail policy (#Mail-List-Etiqutte). Please restrict the content of your e-mails to respectful technical comments. If you continue to fail to abide with this policy, your posting rights to this list will be removed. -Thanks, AB #Mail-List-Etiqutte http://www.w3.org/2008/webapps/wiki/WorkMode#Mail_List_Policy.2C_Usage.2C_Etiquette.2C_etc. On 11/9/12 1:24 PM, ext Todd Blanchard wrote: http://lists.w3.org/Archives/Public/public-webapps/2012OctDec/0430.html
Re: Two years on and still no sensible web storage solutions exist
On Mon, 12 Nov 2012 10:14:47 +0100, Andrew Wilson atwil...@google.com wrote:My recollection is that some vendors refused to ever support SQLite, making a SQL-based standard not really feasible. You can undoubtedly review the w3c archives to find out more details if you want to know the rationale they expressed at the time.Right. You'll find a lot of the discussion took place at the TPAC meeting in November 2008 and again in 2010. Note that when implementors say "never", you can be pretty sure it means "at least not for a while, then we might re-evaluate".It's unfortunate that IndexedDB is still not widely supported by browsers - I share your frustration.Likewise. It is unfortunate that the industry has not managed to agree on any widely-supported standard for offline data storage beyond key-value pairs. However, as you say, you can write to the IndexedDB APIs, and use a shim to get support on platforms that do not support it yet. -atwOn Mon, Nov 12, 2012 at 9:50 AM, Florian Bösch pya...@gmail.com wrote: I'd like to propose as a constructive strategy not to flame/offend everybody right off the bat.As co-chair of the working group I'd like to point out that flaming is generally considered an unacceptable way to communicate in this group, and request a more civil tone in future messages. On Fri, Nov 9, 2012 at 7:24 PM, Todd Blanchard toddvblanch...@gmail.com wrote: It has been two years since the following little note was attached to the Web SQL SpecThis document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. This move has left the web browser world in disarray and has been widely misconstrued by readers to mean "Web SQL is deprecated and will not be supported in the future - better port to IndexedDB".I don't think that is misconstruing what the general sense of the gathered implementors was. As you note elsewhere, in your own case you are no more forced to agree with a standard than any other implementor. We're just documenting a view of reality here.Today, TWO YEARS LATER, we have SQLite on iOS, Android, Chrome, and Safari but no IndexedDB. On Firefox we have IndexedDB with SQLite available only via a browser extensionhttps://addons.mozilla.org/en-US/firefox/addon/html5-websql-for-firefox/(annoying but liveable), and on IE[89] only DOM storage with IndexedDB expected on IE 10. Sources:http://caniuse.com/indexeddbhttp://caniuse.com/sql-storage As someone who is trying to build an offline web app the works both on browsers and smart phones and needs to store a lot of client side complex data that will require lots of joins - let me just say WTF?I understand your frustration. I will point out that many people volunteer their own time, on top of time that many companies pay for, in order to help provide standards that make it easier for you to make a living or just do something useful. But this is not a guarantee that other people are required to solve your problems.Why do we have standards again? You're not helping.To be clear, standards are documents that describe what the industry has agreed on. There is no formal power of enforcement beyond market adoption. If your preferred tool developer doesn't support your preferred standard, then you as a customer probably have a lot more power than we as a forum where they can listen to what others are doing and make their own proposals.If you are providing solutions to their customers, then your ability to conI look around at information on the state of storage options and I read stuff like this:"Since November 18, 2010, the W3C announced that Web SQL database is a deprecated specification. This is a recommendation for web developers to no longer use the technology as effectively, the spec will receive no new updates and browser vendors aren't encouraged to support this technology. The new alternative is IndexedDB which is already available on Chrome 12+ and Firefox 5+, and, soon, in IE 10 as well. "Was it really the intent to abandon SQL as a concept because everybody is using the same well tested and portable library?No. It was the intent to abandon SQL because significant parts (that is of course a udegement call, and you're entitled to judge it differently) of the industry were not using it and not interested in using it - and because most of those who *did* implement it agreed that it was a bad idea. Incidentally, a half-proposal was recently floated by Ashok Malhotra, to introduce another SQL specification which as well as using SQL for local storage is intended to simplify the problem of synchronising data between online and offline applications. I don't know what the follow-up for that is.The Webapps group will continue to try and reach industry agreement on ways to handle data storage for offline applications. Help is appreciated.
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 1:12 PM, Charles McCathie Nevile cha...@yandex-team.ru wrote: The Webapps group will continue to try and reach industry agreement on ways to handle data storage for offline applications. Help is appreciated. Trying to help is generally appreciated too. A slight tangent, but coming from WebGL where we're dealing with a lot of binary data, the inability of WebSQL and IndexedDB to deal with Array Buffers is quite an obstacle. The File storage API however isn't supported anywhere but Chrome. I don't think it's mandatory for a query based API to also support binary blobs. Traditionally this has often been solved in similar situations by storing keys (paths) to an appropriate provider of binary storage (such as a filesystem). Similar uses could be made if browsers supported both a query based DB and a filesystem like API for lookups by key/value. On a related note, key/value backing stores might not be terribly fancy compared to query based backing stores. But a surprising number of data modelling problems does not neatly fit into relational logic well (usually graphs, see also the popularity of key/value backing stores and the whole NoSQL stuff). So for me a query based backing store would be nice to have. But a good key/value backing store with the ability to store binary isn't really optional.
Re: Two years on and still no sensible web storage solutions exist
On Nov 12, 2012, at 4:30 AM, Florian Bösch wrote: On Mon, Nov 12, 2012 at 1:12 PM, Charles McCathie Nevile cha...@yandex-team.ru wrote: The Webapps group will continue to try and reach industry agreement on ways to handle data storage for offline applications. Help is appreciated. Trying to help is generally appreciated too. A slight tangent, but coming from WebGL where we're dealing with a lot of binary data, the inability of WebSQL and IndexedDB to deal with Array Buffers is quite an obstacle. Wondering what is wrong with the blob type in SQLite? http://www.sqlite.org/datatype3.html
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 1:36 PM, Todd Blanchard toddvblanch...@gmail.comwrote: Wondering what is wrong with the blob type in SQLite? A couple of things 1) Array buffers can't be passed to websql 2) websql is subject to the hard quota (5mb) and I'm talking about gigabytes of data 3) traditionally blobs in RDBMSes are slow Of course you could use base64 encoding, unfortunately atob btoa do not support array buffers, and the methods are also not standardized, and there don't exist native methods for base64 encoding/decoding to array buffers. Base64 encoding in JS of hundreds of megabytes or a couple gigabytes is unfortunately not quite a feasible prospect, especially not on weaker machines.
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 4:30 AM, Florian Bösch pya...@gmail.com wrote: On Mon, Nov 12, 2012 at 1:12 PM, Charles McCathie Nevile cha...@yandex-team.ru wrote: The Webapps group will continue to try and reach industry agreement on ways to handle data storage for offline applications. Help is appreciated. Trying to help is generally appreciated too. A slight tangent, but coming from WebGL where we're dealing with a lot of binary data, the inability of WebSQL and IndexedDB to deal with Array Buffers is quite an obstacle. Er, IndexedDB should handle ArrayBuffers just fine. If you're seeing problems with that that's an implementation bug, not a case that the spec failed to handle. - Kyle
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 3:49 PM, Kyle Huey m...@kylehuey.com wrote: Er, IndexedDB should handle ArrayBuffers just fine. If you're seeing problems with that that's an implementation bug, not a case that the spec failed to handle. You're right it does, I just checked. Nevertheless there are some issues. This testsite: http://codeflow.org/issues/indexeddb.html Putting 50mb takes between 2.5 to 8 seconds. Getting it between 2 to 4 seconds. Putting 100mb doesn't work in Chrome at all and throws an exception on the put after half a second. For comparison, in python: Writing 50mb takes between 40-80ms. Reading takes around 50ms.
Re: Two years on and still no sensible web storage solutions exist
Your best bet is to report implementation/performance issues to the browser vendors, rather than here, unless you believe that there is something inherent in the spec that results in slow implementations. I'd say in the specific case of Chrome, we probably have not optimized for the case of writing out a 100MB ArrayBuffer - logging a bug at http://crbug.com describing your use case would be a useful way to have this addressed, at least on that platform. On Mon, Nov 12, 2012 at 5:17 PM, Florian Bösch pya...@gmail.com wrote: On Mon, Nov 12, 2012 at 3:49 PM, Kyle Huey m...@kylehuey.com wrote: Er, IndexedDB should handle ArrayBuffers just fine. If you're seeing problems with that that's an implementation bug, not a case that the spec failed to handle. You're right it does, I just checked. Nevertheless there are some issues. This testsite: http://codeflow.org/issues/indexeddb.html Putting 50mb takes between 2.5 to 8 seconds. Getting it between 2 to 4 seconds. Putting 100mb doesn't work in Chrome at all and throws an exception on the put after half a second. For comparison, in python: Writing 50mb takes between 40-80ms. Reading takes around 50ms.
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 9:52 AM, Joshua Bell jsb...@google.com wrote: Per the spec, anything the structured cloning algorithm [1] handles can be used as record values in IndexedDB. ArrayBuffers are not on that list, but Chrome does support them in IndexedDB. The TypedArray spec specifies how to structured clone ArrayBuffers. http://www.khronos.org/registry/typedarray/specs/latest/#9.1 - Kyle
Re: Two years on and still no sensible web storage solutions exist
For anyone that's confused, I sent from the wrong email address so non-list recipients received my reply but the list did not. And Kyle's right, as I realized when following up before re-sending. On Mon, Nov 12, 2012 at 9:56 AM, Kyle Huey m...@kylehuey.com wrote: On Mon, Nov 12, 2012 at 9:52 AM, Joshua Bell jsb...@google.com wrote: Per the spec, anything the structured cloning algorithm [1] handles can be used as record values in IndexedDB. ArrayBuffers are not on that list, but Chrome does support them in IndexedDB. The TypedArray spec specifies how to structured clone ArrayBuffers. http://www.khronos.org/registry/typedarray/specs/latest/#9.1 - Kyle
Re: Two years on and still no sensible web storage solutions exist
On Nov 12, 2012, at 4:43 AM, Florian Bösch wrote: On Mon, Nov 12, 2012 at 1:36 PM, Todd Blanchard toddvblanch...@gmail.com wrote: Wondering what is wrong with the blob type in SQLite? A couple of things 1) Array buffers can't be passed to websql Could be fixed. 2) websql is subject to the hard quota (5mb) and I'm talking about gigabytes of data Gigabytes of data? Why would one subject WebSQL to a hard quote of 5M and think using gigabytes of user disk space in a different scenario is OK? Something seems out of balance here. Why limit one but not the other?
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 7:19 PM, Todd Blanchard toddvblanch...@gmail.comwrote: Gigabytes of data? Why would one subject WebSQL to a hard quote of 5M and think using gigabytes of user disk space in a different scenario is OK? Something seems out of balance here. Why limit one but not the other? The hard quota is not my idea. It's what those browsers who do implement WebSQL implement. Unlike IndexedDB which triggers a do you want to allow yadda yadda or whatever, at least sometimes, WebSQL is hard limited to 5mb. And as to why large sizes do matter. Let's take the sponza atrium in binary form including normals, positions, texcoords and material IDs (100mb). If you chuck out a bunch of things you can compute, structure and otherwise make smaller, you can get out at some 12mb. But you can't send that to the GPU. You've got to convert it to something that's usable for the GPU, which takes up around 1-2 seconds of CPU time in JS if you wrote a super-optimized piece of code. Now reading 100mb from disk to a binary representation shouldn't take more than about 80ms or so worst case. So It would be kind of really desirable to be able to store temporary calculated assets. Another usecase for large asset storage is productivity software that deals with images/meshes. The amount of data you quickly generate with that kind of software is staggering, and in the interest to keep things easy to use it would be kinda nice to be able to manage assets on the client other than file drop.
Re: Two years on and still no sensible web storage solutions exist
On Mon, 12 Nov 2012 19:27:33 +0100, Florian Bösch pya...@gmail.com wrote: The hard quota is not my idea. It's what those browsers who do implement WebSQL implement. Unlike IndexedDB which triggers a do you want to allow yadda yadda or whatever, at least sometimes, WebSQL is hard limited to 5mb. You're again confusing implementation details with the specification. User agents are free to implement whatever quota management they want, as long as it's transparent and respects the visible effects on the webpage side.
Re: Two years on and still no sensible web storage solutions exist
On Mon, Nov 12, 2012 at 7:42 PM, João Eiras jo...@opera.com wrote: You're again confusing implementation details with the specification. User agents are free to implement whatever quota management they want, as long as it's transparent and respects the visible effects on the webpage side. That may be so, it still means WebSQL is universally hard limited without escape to 5mb. So it doesn't really matter if that's what's specced or not, it makes it equally unsuitable for data beyond 5mb.
Re: Two years on and still no sensible web storage solutions exist
On Nov 12, 2012, at 10:27 AM, Florian Bösch wrote: The hard quota is not my idea. It's what those browsers who do implement WebSQL implement. Unlike IndexedDB which triggers a do you want to allow yadda yadda or whatever, at least sometimes, WebSQL is hard limited to 5mb. Actually, asking for more than 5M triggers a I can haz mor? dialog as well. The limit is equally soft. At least IME. But I don't get the philosophy disconnect. If the user is going to object to 5M of user data in a database, he certainly isn't going to be happy with giant files all over the place. I certainly wouldn't be happy about it. Disk bytes is disk bytes.
Re: Two years on and still no sensible web storage solutions exist
On Mon, 12 Nov 2012 19:50:20 +0100, Florian Bösch pya...@gmail.com wrote: On Mon, Nov 12, 2012 at 7:42 PM, João Eiras jo...@opera.com wrote: You're again confusing implementation details with the specification. User agents are free to implement whatever quota management they want, as long as it's transparent and respects the visible effects on the webpage side. That may be so, it still means WebSQL is universally hard limited without escape to 5mb. So it doesn't really matter if that's what's specced or not, it makes it equally unsuitable for data beyond 5mb. Again, not a spec issue. Go to your favorite browser bug tracker and make a feature request.
Re: Two years on and still no sensible web storage solutions exist
Similarly the vendors deciding not wanting to support WebSQL isn't a spec issue and it should be filed in the bug tracker of your favorite vendor. On Mon, Nov 12, 2012 at 8:02 PM, João Eiras jo...@opera.com wrote: On Mon, 12 Nov 2012 19:50:20 +0100, Florian Bösch pya...@gmail.com wrote: On Mon, Nov 12, 2012 at 7:42 PM, João Eiras jo...@opera.com wrote: You're again confusing implementation details with the specification. User agents are free to implement whatever quota management they want, as long as it's transparent and respects the visible effects on the webpage side. That may be so, it still means WebSQL is universally hard limited without escape to 5mb. So it doesn't really matter if that's what's specced or not, it makes it equally unsuitable for data beyond 5mb. Again, not a spec issue. Go to your favorite browser bug tracker and make a feature request.
Re: Two years on and still no sensible web storage solutions exist
On 11/12/12 8:17 AM, Florian Bösch wrote: This testsite: http://codeflow.org/issues/indexeddb.html Putting 50mb takes between 2.5 to 8 seconds. Getting it between 2 to 4 seconds. Putting 100mb doesn't work in Chrome at all and throws an exception on the put after half a second. After fixing your testcase to work in Gecko (by updating to current indexeddb spec and fixing all the webkit-specific things in there), I see it take about 60ms to put 50MB of data. About 150ms to read it. Obvious your mileage may vary depending on your machine's I/O subsystem. -Boris
Re: Two years on and still no sensible web storage solutions exist
Care to paste that one? I'll test it in firefox. On Mon, Nov 12, 2012 at 9:25 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 11/12/12 8:17 AM, Florian Bösch wrote: This testsite: http://codeflow.org/issues/**indexeddb.htmlhttp://codeflow.org/issues/indexeddb.html Putting 50mb takes between 2.5 to 8 seconds. Getting it between 2 to 4 seconds. Putting 100mb doesn't work in Chrome at all and throws an exception on the put after half a second. After fixing your testcase to work in Gecko (by updating to current indexeddb spec and fixing all the webkit-specific things in there), I see it take about 60ms to put 50MB of data. About 150ms to read it. Obvious your mileage may vary depending on your machine's I/O subsystem. -Boris
Re: Two years on and still no sensible web storage solutions exist
On 11/12/12 12:31 PM, Florian Bösch wrote: Care to paste that one? I'll test it in firefox. http://pastebin.mozilla.org/1937689 -Boris
Re: Two years on and still no sensible web storage solutions exist
In FF 16.0.2 on linux, 50mb put takes about 100ms and get takes about 500ms. On Mon, Nov 12, 2012 at 10:50 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 11/12/12 12:31 PM, Florian Bösch wrote: Care to paste that one? I'll test it in firefox. http://pastebin.mozilla.org/**1937689http://pastebin.mozilla.org/1937689 -Boris
Re: Two years on and still no sensible web storage solutions exist
On 11/12/12 1:56 PM, Florian Bösch wrote: In FF 16.0.2 on linux, 50mb put takes about 100ms and get takes about 500ms. For what it's worth, I did some measuring, and the main performance cost centers for the write (the CPU-intensive ones, not the disk access) seem to be: 1) Making a copy of the data to put in the database (because the database write happens on a background thread and we want to return control to JS before we do that to avoid blocking the main thread, so have to copy the data). 2) Compressing the data (we store it compressed on disk). For the read it's decompressing and then copying out of the resulting data structure into the JS engine's heap for the typed array. It might be possible to mitigate the copying costs, especially if indexeddb had a way to indicate that things should be transferred, not copied, similar to postMessage. -Boris
Re: Two years on and still no sensible web storage solutions exist
A-ha. Filled payload with random. Now it's FF 4.4s put, 5.5s get. In line with chromes perf. On Mon, Nov 12, 2012 at 11:46 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 11/12/12 1:56 PM, Florian Bösch wrote: In FF 16.0.2 on linux, 50mb put takes about 100ms and get takes about 500ms. For what it's worth, I did some measuring, and the main performance cost centers for the write (the CPU-intensive ones, not the disk access) seem to be: 1) Making a copy of the data to put in the database (because the database write happens on a background thread and we want to return control to JS before we do that to avoid blocking the main thread, so have to copy the data). 2) Compressing the data (we store it compressed on disk). For the read it's decompressing and then copying out of the resulting data structure into the JS engine's heap for the typed array. It might be possible to mitigate the copying costs, especially if indexeddb had a way to indicate that things should be transferred, not copied, similar to postMessage. -Boris
Re: Two years on and still no sensible web storage solutions exist
On 11/12/12 2:59 PM, Florian Bösch wrote: A-ha. Filled payload with random. Now it's FF 4.4s put, 5.5s get. In line with chromes perf. Hmm. This I can't reproduce. I tried filling the array with randoms, and the times end up about 30% higher, but not 20x higher, for me. -Boris
