Marco,
>There's never been that kind of tone, the article has been written to >point >new Mozilla codebase contributors at possible pitfalls we already hit in >the past, and actually help them making informed decisions. That's not how one reads it. Start with only the title: "Performance/Avoid SQLite In Your Next Firefox Feature" Do you read with me? "If you want performance at any rate, AVOID SQLite ..." Then it starts enumerating pretended "SQLite pitfalls", pointing out several topics which are presented as inherent drawbacks to using SQLite from a FF pluggin. This is not "help them making their decision", just simply bashing the library which you seem to expect being almost reserved to core code. >It actually briefly explains in which cases a JSON log can work better Starting at ~1Mb data it advises storing stuff as compressed JSON. Now you're seriously trying to tell me that with a multi-Mb compressed complex JSON: -) locking the compressed JSON file -) reading it up in full, in memory -) unpacking it (with much more memory consumption, of course) -) parsing it (again more memory workspace needed) -) scanning the object tree in search of the data you're after -) reading and changing a single value in the tree -) flattening the whole tree back to JSON form -) repacking it -) rewriting the lot -) releasing the lock ALL of this sequence (in a multi-threaded environment with all the extra burden that comes with it) is going to take less time, waste less CPU cycles and use less RAM than a single SQLite UPDATE wrapped in a safe-to-use API, all in a smaller "footprint"? Seriously? And what does happen when two or more compressed JSON have to be used together to obtain/access the required information? Looks like a can or worms in this case. Note how the footprint argument is hypocritical since the SQLite library is already loaded and ready for work since it's much in use for the main thread, so the footprint is always there, that SQLite be used in pluggins or not. Any decently written safe wrapping API is ridiculously small and doesn't even count. I don't seem the only one to be utterly doubtful about this "compressed JSON > SQLite" assertion, when applied to the general case. >It tries to make people think before doing. Yes a simple flat file can be the best solution in simple cases but that shouldn't trigger the AVOID verb in general cases. And less simple flat files can fit the bill for another range of use cases. It would be wise to shape the arguments (a number of them being valid) in a different way. >It states to evaluate alternatives. Existing alternatives like OS.File >writeAtomic and JSON. Those are already used with success in both the >products and add-ons, when it makes sense. The last part of your sentence, "when it makes sense", clashes with the title (AVOID). Avoiding is not evaluating. >Sure the developer mus think to durability, backups, coherence, but >would it not be the same if he'd use SQLite? Definitely not, because you know that SQLite already takes greatest care of all the possible issues on countless platforms and this comes free: the code is already there, highly optimized and working. Code written by "average" pluggin developper can't compete with years of experience with those issues which are prone to overlook. Whenever a dark corner resulting in a bug in SQLite surfaces, the fix comes in no time and fixes all applications using SQLite at a time. > > After all, it's Mozilla devs themselves who designed pluggin APIs > and let > > "spurious main-thread SQL statements" be possible. If they were sooo > > clever, they would never had allowed that and they also would have > wrapped > > SQLite interface in a strictly limited set of rules enforced by a > safe API. > > That, they won't tell you. > >And they'd have a 100% bug free product with millions lines of code... No, >that's unrealistic. Great! So I've news for you: you can produce proven correct *-code-* using today's formal methods tools. Granted it would actually be irrealistic to dream of switching to a formal development process within months, even years, but such things do exist. Yet I never pretended that offering a safer API would solve all of the issues, just a number of those the text is about. > The Mozilla codebase comes from the 90s Netscape >codebase, at that time the most common thing was a single-threaded and >single-process browser, able to show text, some images and tables. The >reality evolved A LOT and the code had to evolve to cope with it. At the >time mozStorage (the SQLite wrapper) was written, there was still that >kind >of vision, and it was written as a main-thread synchronous API. Sure, now >we all know it was wrong, but at that time it was the right-ish thing to >do. The API grew a purely asynchronous alternative, but when you have >hundreds millions of users and thousands of add-ons using an API, you >can't >just say "sorry, we now break you all". Come on. From the 90's to 2015, that is 25 years (300 months!) and a large number of minor and major releases which I'm too lazy to enumerate right now. It certainly isn't something you can launch as the "new thing" between v1.2.8 and v1.3.0 without provoking a tsunami among the partners. But it definitely is something that can be --and ought have been-- planned over years in small steps along a public semi-final design goal direction. Yet today's FF already *-does-* break existing pluggins, since some of them are denied loading when a new FF version is released. Of course with good reasons, but this is something both developpers and users have now accepted, for it brings them more than running unreliable and unstable programs. >Sure, the Firefox add-ons are the most powerful add-ons around, they >can do >anything, and while this might be (as you point out) a downside, cause >they >can perform poorly, it is also their major selling point. There are >still a >lots of things you can do with a Firefox add-on that you can't do in any >other browser add-on. There is no discussion here, you're plain right and Mozilla can be proud of that. Again, I'm not after Mozilla, at all. It's just unfortunate that some valid points (and some not valid at all) were made in such a disregarding way. >Also, this has nothing to do with the fact SQLite can still be the wrong >choice for certain data store needs (as it's the best for others), that is >still the main purpose of that article. The difference is the word "can". The title and general tone are merely severely dissuasive and not a step by step reasonned guidance to the best choice(s). If it had been worded more thoughtfully, the link posted here wouldn't have caused several dozens of replies (now shifting to something else entirely). Even if english is not my mothertongue, I can rewrite the heart of the thing in a much more friendly and constructive way, exposing the genuine issues that may arise and possible avenues to overcome the problems caused, without ever giving the feeling that I'm bashing or favoring this or that storage strategy. As you say yourself: the purpose is to make people engage brain. Yet the reading is more "we've thought for you since you're unable to do it yourself". Jean-Christophe.
