On Friday 26 October 2007, Steve wrote: > I totally get what you mean about being IO bound, however as I said > earlier, there is still CPU time involved, saving 1% may not sound > like a lot but it could add up, and after the initial development cost > is recouped that 1% becomes free. However I believe that by not > incurring the overhead of loading a general purpose utility your > savings are much more than 1%. That said my way using streams was > certainly not ideal, because you do incur the whole stream overhead > penalty, but I figured that was counterbalanced by the speed at which > the development could proceed.
Okay, we need to set this straight. IO bound means there is a physical hardware limit being hit. Yes, the cpu portion of this process can be reduced, but doing so *does not* change the physical limits on IO. Thus all it would do is free up cpu for other tasks, not make this task any faster. > > Additionally, in what real world application of something like this, > would you be able to use a file system hack like that, and/or change > the file system all together? If he's going to do that, why not > upgrade the machine, throw in SATA go with a stripped RAID > configuration and so forth. Because they don't solve the same problem. One walks around the problem, the other just throws resources at it to reduce the wall-time consumed. > > Either way the file system hacking bit seems like a cool idea, but how > does it quantifiably increase IO throughput? Seems like adding more > layers of stuff for the IO operation to pass through would actually > slow things down a bit. It depends on your application. If you don't actually need to *change* the file, just what applications *see*, then masking off the first x bytes of the file with a hack such that applications see it as if those bytes weren't there totally removes the problem--you no longer have to even do any io to alter the file. This overhead would be very minimal--essentially just add an offset to the file pointer behind the scenes. Compared to having to move GB of data, this takes no resources at all. If what you really want to do is alter the file, then there really is no way around moving all the data with current file systems--but you can do so with a choice of methods, some slower, some faster. Unless, you rewrote the filesystem to be a rope container, then it could be very trivial operation to slice off any subset of a file. Maybe not a bad idea for some applications (databases come to mind). /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
