Hi Akshay, You may be interested in the work carried out on https://issues.apache.org/jira/browse/HADOOP-7714 (For HDFS side, head to https://issues.apache.org/jira/browse/HDFS-2465 as mentioned on it)
On Thu, Apr 26, 2012 at 11:37 AM, Akshay Singh <akshay.i...@gmail.com> wrote: > Hi, > > I was looking for caching mechanisms in Hadoop, and was expecting file/block > caching on Datanodes for frequently accessed file-blocks. > > As it seems, HDFS does not provide any caching below the file system > interface and utilizes DataNode's OS buffer cache for keeping frequently > accessed HDFS-file (stored as local OS files) in memory. Am i missing > anything ? > > Also, is there any extension to HDFS which has implemented file caching at > DataNode level ? I understand that this another level of caching would bring > up issues like data-coherence, but I guess the performance gain may be worth > paying for this consistency overhead. > > P.S. : I am looking for memory based cache on Datanodes, in case it was not > clear. > > Thanks, > Akshay -- Harsh J
