How does HDFS deal with optimization of file streaming? Do data nodes have any optimizations at the disk level for dealing with fragmented files? I assume not, but just curious if this is at all in the works, or if there are java-y ways of dealing with a long running set of files in an HDFS cluster. MAybe, for example, data nodes could log the amount of time spent on I/O for certain files as a way of reporting wether or not defragmentation needed to be run on a particular node in a cluster.
-- Jay Vyas http://jayunit100.blogspot.com
