Re: [sqlite] light weight write barriers
On Mon, Nov 05, 2012 at 05:37:02PM -0500, Richard Hipp wrote: > > Per the docs: "Only the superuser or a process possessing the > CAP_SYS_RESOURCE capability can set or clear this attribute." That > prevents most applications that run SQLite from being able to take > advantage of this, since most such applications lack elevated privileges. If this feature would prove useful to sqllite, that's something we could address. I could image making this available to processes that belong to a specific group that would be specified in the superblock or as a mount option. (We already have something like that which allows a specific uid or gid to use the reserved space in the superblock.) - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Mon, Nov 05, 2012 at 09:03:48PM +0100, Pavel Machek wrote: > > Well, using data journalling with ext3/4 may do what you want. If you > > don't do any fsync, the changes will get written every 5 seconds when > > the automatic journal sync happens (and sub-4k writes will also get > > Hmm. But that would need setting journalling mode per-file, no? > > Like, make it journal data for all the databases, but keep normal mode > for rest of system... You can do that, using "chattr +j file.db". It's apparently not a well known feature of ext3/4 - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Fri, Oct 26, 2012 at 09:54:53PM -0400, Vladislav Bolkhovitin wrote: > What different in our positions is that you are considering storage > as something you can connect to your desktop, while in my view > storage is something, which stores data and serves them the best > possible way with the best performance. I don't get paid to make Linux storage work well for gold-plated storage, and as far as I know, none of the purveyors of said gold plated software systems are currently employing Linux file system developers to make Linux file systems work well on said gold-plated hardware. As for what I might do on my own time, for fun, I can't afford said gold-plated hardware, and personally I get a lot more satisfaction if I know there will be a large number of people who benefit from my work (it was really cool when I found out that millions and millions of Android devices were going to be using ext4 :-), as opposed to a very small number of people who have paid $$$ to storage vendors who don't feel it's worthwhile to pay core Linux file system developers to leverage their hardware. Earlier, you were bemoaning why Linux file system developers weren't paying attention to using said fancy SCSI features. Perhaps now you'll understand better it's not happening? > Price doesn't matter here, because it's completely different topic. It matters if you think I'm going to do it on my own time, out of my own budget. And if you think my employer is going to choose to use said hardware, price definitely matters. I consider engineering to be the art of making tradeoffs, and price is absolutely one of the things that we need to trade off against other goals. It's rare that you get to design something where performance matters above all else. Maybe it's that way if you're paid by folks whose job it is to destablize the world's financial markets by pushing the holes into the right half plane (i.e., high frequency trading :-). But for the rest of the world, price absolutely matters. - Ted P.S. All of the storage I have access to at home is SATA. If someone would like to change that and ship me free hardware, as long as it doesn't require three-phase power (or require some exotic interconnect which is ghastly expensive and which you are also not going to provide me for free), do contact me off-line. :-) ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Thu, Oct 25, 2012 at 11:03:13AM -0700, da...@lang.hm wrote: > I agree, this is why I'm trying to figure out the recommended way to > do this without needing to do full commits. > > Since in most cases it's acceptable to loose the last few chunks > written, if we had some way of specifying ordering, without having > to specify "write this NOW", the solution would be pretty obvious. Well, using data journalling with ext3/4 may do what you want. If you don't do any fsync, the changes will get written every 5 seconds when the automatic journal sync happens (and sub-4k writes will also get coalesced to a 5 second granularity). Even with plain text files, it's pretty easy to tell whether or not the final record is a partially written or not after a crash; just look for a trailing newline. Better yet, if you are writing to multiple log files with data journalling, all of the writes will happen at the same time, and they will be streamed to the file system journal, minimizing random writes for at least the journal writes. - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Wed, Oct 24, 2012 at 11:58:49PM -0700, da...@lang.hm wrote: > The frustrating thing is that when people point out how things like > sqlite are so horribly slow, the reply seems to be "well, that's > what you get for doing so many fsyncs, don't do that", when there is > a 'problem' like the KDE "config loss" problem a few years ago, the > response is "well, that's what you get for not doing fsync" Sure... but the answer is to only do the fsync's when you need to. For example, if GNOME and KDE is rewriting the entire registry file each time the application is changing a single registry key, sure, if you rewrite the entire registry file, and then fsync after each rewrite before you replace the file, you will be safe. And if the application needs to update dozens or hundreds of registry keys (or every time the window gets moved or resized), then yes, it will be slow. But the application didn't have to do that! It could have updated all the registry keys in memory, and then update the registry file periodically instead. Similarly, Firefox didn't need to do a sqllite commit after every single time its history file was written, causing a third of a megabyte of write traffic each time you clicked on a web page. It could have batched its updates to the history file, since most of the time, you don't care about making sure the web history is written to stable store before you're allowed to click on a web page and visit the next web page. Or does rsyslog *really* need to issue an fsync after each log message? Or could it batch updates so that every N seconds, it flushes writes to the disk? (And this is a problem with most Android applications as well. Apparently the framework API's are such that it's easier for an application to treat each sqlite statement as an atomic update, so many/most application writers don't use explicit transaction boundaries, so updates don't get batched even though it would be more efficient if they did so.) Sometimes, the answer is not to try to create exotic database like functionality in the file system --- the answer is to be more intelligent at the application leyer. Not only will the application be more portable, it will also in the end be more efficient, since even with the most exotic database technologies, the most efficient transactional commit is the unneeded commit that you optimize away at the application layer. - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Thu, Oct 25, 2012 at 02:03:25PM +0100, Alan Cox wrote: > > I doubt they care. The profit on high end features from the people who > really need them I would bet far exceeds any other benefit of giving it to > others. Welcome to capitalism 8) Yes, but it's a question of pricing. If they had priced it a just a wee bit higher, then there would have been incentive to add support for TCQ so it could actually be used into various Linux file systems, since there would have been lots of users of it. But as it is, the folks who are purchasing huge, vast number of these drives --- such as at the large cloud providers: Amazon, Facebook, Racespace, et. al. --- will choose to purchase large numbers of commodity drives, and then find ways to work around the missing functionality in userspace. For example, DIF/DIX would be nice, and if it were available for cheap, I could imagine it being used. But you can accomplish the same thing in userspace, and in fact at Google I've implemented a special not-for-mainline patch which spikes out stable writes (required for DIF/DIX) because it has significant performance overhead, and DIF/DIX has zero benefit if you're not willing to shell out $$$ for hardware that supports it. Maybe the HDD manufacturers have been able to price guage a small number enterprise I/T shops with more dollars than sense, but personally, I'm not convinced they picked an optimal pricing strategy Put another way, I accept that Toyota should price a Lexus ES more than a Camry, but if it's priced at say, 3x the price of a Camry instead of 20%, they might find that precious few people are willing to pay that kind of money for what is essentially the same car with minor luxury tweaks added to it. > Plus - spinning rust for those end users is on the way out, SATA to flash > is a bit of hack and people are already putting a lot of focus onto > things like NVM Express. Yeah I don't buy that. One, flash is still too expensive. Two, the capital costs to build enough Silicon foundries to replace the current production volume of HDD's is way too expensive for any company to afford (the cloud providers are buying *huge* numbers of HDD's) --- and that's assuming companies wouldn't chose to use those foundries for products with larger margins --- such as, for example, CPU/GPU chips. :-) And third and finally, if you study the long-term trends in terms of Data Retention Time (going down), Program and Read Disturb (going up), and Write Endurance (going down) as a function of feature size and/or time, you'd be wise to treat flash as nothing more than short-term cache, and not as a long term stable store. If end users completely give up on flash, and store all of their precious family pictures on flash storage, after a couple of years, they are likely going to be very disappointed Speaking personally, I wouldn't want to have anything on flash for more than a few months at *most* before I made sure I had another copy saved on spinning rust platters for long-term retention. - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Thu, Oct 25, 2012 at 12:18:47AM -0500, Nico Williams wrote: > > By trusting fsync(). And if you don't care about immediate Durability > you can run the fsync() in a background thread and mark the associated > transaction as completed in the next transaction to be written after > the fsync() completes. The challenge is when you have entagled metadata updates. That is, you update file A, and file B, and file A and B might share metadata. In order to sync file A, you also have to update part of the metadata for the updates to file B, which means calculating the dependencies of what you have to drag in can get very complicated. You can keep track of what bits of the metadata you have to undo and then redo before writing out the metadata for fsync(A), but that basically means you have to implement soft updates, and all of the complexity this implies: http://lwn.net/Articles/339337/ If you can keep all of the metadata separate, this can be somewhat mitigated, but usually the block allocation records (regardless of whether you use a tree, or a bitmap, or some other data structure) tends of have entanglement problems. It certainly is not impossible; RDBMS's have implemented this. On the other hand, they generally aren't as fast as file systems for non-transactional workloads, and people really care about performance on those sorts of workloads for file systems. (About a decade ago, Oracle tried to claim that you could run file system workloads using an Oracle databsae as a back-end. Everyone laughed at them, and the idea died a quick, merciful death.) Still, if you want to try to implement such a thing, by all means, give it a try. But I think you'll find that creating a file system that can compete with existing file systems for performance, and *then* also supports a transactional model, is going to be quite a challenge. - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Wed, Oct 24, 2012 at 03:03:00PM -0700, da...@lang.hm wrote: > Like what is being described for sqlite, loosing the tail end of the > messages is not a big problem under normal conditions. But there is > a need to be sure that what is there is complete up to the point > where it's lost. > > this is similar in concept to write-ahead-logs done for databases > (without the absolute durability requirement) If that's what you require, and you are using ext3/4, usng data journalling might meet your requirements. It's something you can enable on a per-file basis, via chattr +j; you don't have to force all file systems to use data journaling via the data=journalled mount option. The potential downsides that you may or may not care about for this particular application: (a) This will definitely have a performance impact, especially if you are doing lots of small (less than 4k) writes, since the data blocks will get run through the journal, and will only get written to their final location on disk. (b) You don't get atomicity if the write spans a 4k block boundary. All of the bytes before i_size will be written, so you don't have to worry about "holes"; but the last message written to the log file might be truncated. (c) There will be a performance impact, since the contents of data blocks will be written at least twice (once to the journal, and once to the final location on disk). If you do lots of small, sub-4k writes, the performance might be even worse, since data blocks might be written multiple times to the journal. - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] light weight write barriers
On Tue, Oct 23, 2012 at 03:53:11PM -0400, Vladislav Bolkhovitin wrote: > Yes, SCSI has full support for ordered/simple commands designed > exactly for that task: to have steady flow of commands even in case > when some of them are ordered. SCSI does, yes --- *if* the device actually implements Tagged Command Queuing (TCQ). Not all devices do. More importantly, SATA drives do *not* have this capability, and when you compare the price of SATA drives to uber-expensive "enterprise drives", it's not surprising that most people don't actually use SCSI/SAS drives that have implemented TCQ. SATA's Native Command Queuing (NCQ) is not equivalent; this allows the drive to reorder requests (in particular read requests) so they can be serviced more efficiently, but it does *not* allow the OS to specify a partial, relative ordering of requests. Yes, you can turn off writeback caching, but that has pretty huge performance costs; and there is the FUA bit, but that's just an unconditional high priority bypass of the writeback cache, which is useful in some cases, but which again, does not give the ability for the OS to specify a partial order, while letting the drive reorder other requests for efficiency/performance's sake, since the drive has a lot more information about the optimal way to reorder requests based on the current location of the drive head and where certain blocks may have been remapped due to bad block sparing, etc. > Hopefully, eventually the storage developers will realize the value > behind ordered commands and learn corresponding SCSI facilities to > deal with them. Eventually, drive manufacturers will realize that trying to price guage people who want advanced features such as TCQ, DIF/DIX, is the best way to gaurantee that most people won't bother to purchase them, and hence the features will remain largely unused - Ted ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
