Re: Storage in Gecko

2013-05-16 Thread David Rajchenbach-Teller
On 5/16/13 2:26 AM, Robert Kaiser wrote: David Rajchenbach-Teller schrieb: I'd even go as far as limiting it to 16kb. (possibly with a transition phase during which going above 16kb only prints warnings) I think most of us agree, but the problem is that apparently a number of add-ons rely

Re: Storage in Gecko

2013-05-15 Thread Robert Kaiser
David Rajchenbach-Teller schrieb: I'd even go as far as limiting it to 16kb. (possibly with a transition phase during which going above 16kb only prints warnings) I think most of us agree, but the problem is that apparently a number of add-ons rely on large prefs atm, so right now we did set

Re: Storage in Gecko

2013-05-09 Thread Robert Kaiser
Gregory Szorc schrieb: Perhaps this should be advertised more, especially to the add-on community. Looking at about:config of my main profile, about 2/3 of my preferences are user set. There are hundreds of preferences apparently being used for key-value storage by add-ons (not to pick on one,

Re: Storage in Gecko

2013-05-06 Thread Jed Davis
On Mon, May 06, 2013 at 09:41:08AM -0700, David Dahl wrote: KyotoCabinet might make a good backend for a new storage API: http://fallabs.com/kyotocabinet/ It's released under the GPL, so it's MPL-incompatible, if I understand correctly. As for the Kyoto Products Specific FOSS Library Linking

Re: Storage in Gecko

2013-05-03 Thread Benjamin Smedberg
On 5/2/2013 7:43 PM, Gregory Szorc wrote: On 5/2/2013 4:40 PM, Kyle Huey wrote: On Thu, May 2, 2013 at 4:36 PM, Gregory Szorc g...@mozilla.com mailto:g...@mozilla.com wrote: We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with

Re: Storage in Gecko

2013-05-03 Thread Ehsan Akhgari
=yJWrGKmbSi6S Did someone post a summary of this meeting? Is there a link to share? Notes at https://etherpad.mozilla.org/storage-in-gecko We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flushing, etc guarantees. Clients would

Re: Storage in Gecko

2013-05-03 Thread Honza Bambas
If you guys don't need transactions and only a simple key+value storage, the new localstorage code be used. Only thing needed to make it work in a completely non-blocking way is to expose API telling the consumer that localstorage has loaded from disk and is fully cached in memory and thus

Re: Storage in Gecko

2013-05-02 Thread Lawrence Mandel
- Original Message - Great post, Taras! Per IRC conversations, we'd like to move subsequent discussion of actions into a meeting so we can more quickly arrive at a resolution. Please meet in Gregory Szorc's Vidyo Room at 1400 PDT Tuesday, April 30. That's 2200 UTC. Apologies to

Re: Storage in Gecko

2013-05-02 Thread Gregory Szorc
at https://etherpad.mozilla.org/storage-in-gecko We seemed to converge on a (presumably C++-based) storage service that has named branches/buckets with specific consistency, flushing, etc guarantees. Clients would obtain a handle on a branch, and perform basic I/O operations, including

Re: Storage in Gecko

2013-05-02 Thread David Teller
...@mozilla.com To: Lawrence Mandel lman...@mozilla.com Cc: David Rajchenbach-Teller dtel...@mozilla.com, Taras Glek tg...@mozilla.com, dev-platform dev-platform@lists.mozilla.org Sent: Friday, May 3, 2013 1:36:15 AM Subject: Re: Storage in Gecko On 5/2/2013 4:13 PM, Lawrence Mandel wrote

Re: Storage in Gecko

2013-04-30 Thread Joshua Cranmer 
On 4/30/2013 12:33 AM, Ehsan Akhgari wrote: On 2013-04-29 1:51 PM, Taras Glek wrote: * How to robustly write/update small datasets? #3 above is it for small datasets. The correct way to do this is to write blobs of JSON to disk. End of discussion. For an API that is meant to be used by

Re: Storage in Gecko

2013-04-30 Thread Dave Townsend
On 4/30/2013 8:37 AM, Joshua Cranmer  wrote: On 4/30/2013 12:33 AM, Ehsan Akhgari wrote: On 2013-04-29 1:51 PM, Taras Glek wrote: Writes of data = ~64K should just be implemented as atomic whole-file read/write operations. Those are almost always single blocks on disk. Writing a whole file

Re: Storage in Gecko

2013-04-30 Thread Marco Bonardo
On 26/04/2013 22:18, Andrew Sutherland wrote: Specifically, it does not look like IndexedDB's model; it still does a lot of fsync's in order to maintain the requisite SQL ACID semantics. Right, we can't expect miracles just by moving from SQLite3 to SQLite4, though it still uses an enhanced

Re: Storage in Gecko

2013-04-29 Thread Taras Glek
So there is no general 'good for performance' way of doing IO. However I think most people who need this need to write small bits of data and there is a good way to do that. Gregory Szorc wrote: I'd like to start a discussion about the state of storage in Gecko. Currently when you

Re: Storage in Gecko

2013-04-29 Thread Andrew McCreight
- Original Message - On Apr 27, 9:37 am, Mounir Lamouri mou...@lamouri.fr wrote: Why? Wouldn't be the idea of such component to make sure it is usable from C++? Perhaps some day, but IndexedDB was always designed with JS in mind. To use it you pass special JS dictionaries for

Re: Storage in Gecko

2013-04-29 Thread Gregory Szorc
Great post, Taras! Per IRC conversations, we'd like to move subsequent discussion of actions into a meeting so we can more quickly arrive at a resolution. Please meet in Gregory Szorc's Vidyo Room at 1400 PDT Tuesday, April 30. That's 2200 UTC. Apologies to the European and east coast

Re: Storage in Gecko

2013-04-27 Thread Mounir Lamouri
On 26/04/13 20:42, bent wrote: IndexedDB is our answer for this for JS... C++ folks are still pretty much on their own! Why? Wouldn't be the idea of such component to make sure it is usable from C++? -- Mounir ___ dev-platform mailing list

Storage in Gecko

2013-04-26 Thread Gregory Szorc
I'd like to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices: 1) Preferences 2) SQLite 3) Manual file I/O Preferences are arguably the easiest. However, they have a number of setbacks: a) Poor

Re: Storage in Gecko

2013-04-26 Thread Kyle Huey
On Fri, Apr 26, 2013 at 11:17 AM, Gregory Szorc g...@mozilla.com wrote: I'd like to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices: 1) Preferences 2) SQLite 3) Manual file I/O

Re: Storage in Gecko

2013-04-26 Thread Andreas Gal
to start a discussion about the state of storage in Gecko. Currently when you are writing a feature that needs to store data, you have roughly 3 choices: 1) Preferences 2) SQLite 3) Manual file I/O Preferences are arguably the easiest. However, they have a number of setbacks: a) Poor

Re: Storage in Gecko

2013-04-26 Thread Justin Lebar
The current level of flakiness in the IndexedDB test suite (especially on OSX) makes me concerned about what to expect if it starts getting heavier use across the various platforms. Is that just in the OOP tests, or everywhere? ___ dev-platform

Re: Storage in Gecko

2013-04-26 Thread Benjamin Smedberg
On 4/26/2013 2:50 PM, Gavin Sharp wrote: On Fri, Apr 26, 2013 at 11:36 AM, Andreas Gal g...@mozilla.com wrote: Preferences are as the name implies intended for preferences. There is no sane use case for storing data in preferences. I would give any patch I come across doing that an automatic

Re: Storage in Gecko

2013-04-26 Thread Kyle Huey
Resending to list. On Fri, Apr 26, 2013 at 12:02 PM, Gregory Szorc g...@mozilla.com wrote: On 4/26/2013 11:52 AM, Kyle Huey wrote: Could you please point me at a good implementation of a Gecko consumer of IndexedDB? If you don't know which are good, an MXR search URL will suffice :) I

Re: Storage in Gecko

2013-04-26 Thread Kyle Huey
On Fri, Apr 26, 2013 at 11:57 AM, Ryan VanderMeulen rya...@gmail.comwrote: The current level of flakiness in the IndexedDB test suite (especially on OSX) makes me concerned about what to expect if it starts getting heavier use across the various platforms. Of the 24 open intermittent failure

Re: Storage in Gecko

2013-04-26 Thread Ryan VanderMeulen
On 4/26/2013 3:07 PM, Justin Lebar wrote: The current level of flakiness in the IndexedDB test suite (especially on OSX) makes me concerned about what to expect if it starts getting heavier use across the various platforms. Is that just in the OOP tests, or everywhere? Mostly IPC.

Re: Storage in Gecko

2013-04-26 Thread Gavin Sharp
On Fri, Apr 26, 2013 at 12:10 PM, Benjamin Smedberg benja...@smedbergs.us wrote: I really hope the outcome of this discussion is that we end up storing everything that isn't a true preference in some other datastore, and that is an async-by-default datastore ;-) With a pretty simple JSM

Re: Storage in Gecko

2013-04-26 Thread Gregory Szorc
On 4/26/2013 12:10 PM, Benjamin Smedberg wrote: On 4/26/2013 2:50 PM, Gavin Sharp wrote: On Fri, Apr 26, 2013 at 11:36 AM, Andreas Gal g...@mozilla.com wrote: Preferences are as the name implies intended for preferences. There is no sane use case for storing data in preferences. I would give

Re: Storage in Gecko

2013-04-26 Thread Reuben Morais
We use IndexedDB extensively in a lot of the WebAPIs, see Contacts, Settings, SMS, MMS, Push, NetworkStats… Right now there's a lot of boilerplate[1] involved in setting up IndexedDB, and people end up duplicating a lot of the boilerplate code. It'd be great to see a more polished wrapper

Re: Storage in Gecko

2013-04-26 Thread Andrew Sutherland
On 04/26/2013 03:21 PM, Dirkjan Ochtman wrote: Also, I wonder if SQLite 4 (which is more like a key-value store) SQLite 4 is not actually more like a key-value store. The underlying storage model used by the SQL-interface-that-is-the-interface changed from being a page-centric btree

Re: Storage in Gecko

2013-04-26 Thread Andrew Sutherland
On 04/26/2013 03:30 PM, Gregory Szorc wrote: However, before that happens, I'd like some consensus that IndexedDB is the best solution here. I'd especially like to hear what Performance thinks: I don't want to start creating a preferred storage solution without their blessing. If they have

Re: Storage in Gecko

2013-04-26 Thread Justin Dolske
On 4/26/13 11:17 AM, Gregory Szorc wrote: But, please don't make consumers worry about things like SQL, schema design, and PRAGMA statements. Ideally, yes. But I suspect there will never be a one-size-fits all solution, and so we should probably be clear about what it's appropriate/intended

Re: Storage in Gecko

2013-04-26 Thread Mounir Lamouri
On 26/04/13 11:17, Gregory Szorc wrote: Anyway, I just wanted to see if others have thought about this. Do others feel it is a concern? If so, can we formulate a plan to address it? Who would own this? As others, I believe that we should use IndexedDB for Gecko internal storage. I opened a bug

Re: Storage in Gecko

2013-04-26 Thread Andreas Gal
, can we formulate a plan to address it? Who would own this? As others, I believe that we should use IndexedDB for Gecko internal storage. I opened a bug regarding this quite a while ago: https://bugzilla.mozilla.org/show_bug.cgi?id=766057 We could easily imagine an XPCOM component that would

Experimental Technology in Gecko (Re: Storage in Gecko)

2013-04-26 Thread Matt Brubeck
On 4/26/2013 11:43 AM, Gregory Szorc wrote: Have you explored using IndexedDB? Not seriously. The this is an experimental technology warning on MDN is off-putting. The largest audience for MDN is web developers, so we put that warning on anything that's not ready for widespread use on the

Re: Storage in Gecko

2013-04-26 Thread bent
IndexedDB is our answer for this for JS... C++ folks are still pretty much on their own! IndexedDB handles indexing (hence the rather awkward name), transactions with abort/rollback, object-graph serialization (not just JSON), usage from multiple tabs/windows/components/processes simultaneously,