Hi, George, I read the results in your JIRA. Very encouraging. It would be useful to test the improvement on both cold and warm data (warm data likely has larger improvment). There is a simple way to clear the file cache on Linux (http://www.linuxinsight.com/proc_sys_vm_drop_caches.html).
An alternative approach is to build an in-memory caching layer on top of a DFS Client. The advantages are (1) less security issues; (2) probably even better performance since checksum can be avoided once the data is cached in memory; (3) the caching layer can be used anywhere, no just nodes owning a block locally. The disadvantage is that data is buffered twice in memory: once in the caching layer and once in the OS file cache. One can probably limit the OS file cache size (not sure if there is an easy way in Linux). What's your thought on this? Jun george.por...@sun.com wrote on 01/08/2009 10:13:25 AM: > Hi Jun, > > The earlier responses to your email reference the JIRA that I opened > about this issue. Short-circuiting the primary HDFS datapath does > improve throughput, and the amount depends on your workload (random > reads especially). Some initial experimental results are posted to that > JIRA. A second advantage is that since the JVM hosting the HDFS client > is doing the reading, the O/S will satisfy future disk requests from the > cache, which isn't really possible when you read over the network (even > to another JVM on the same host). > > There are several real disadvantages, the largest of which include 1) it > adds a new datapath, and 2) bypasses various security and auditing > features of HDFS. I would certainly like to think through a more clean > interface for achieving this goal, especially since reading local data > should be the common case. Any thoughts you might have would be > appreciated. > > Thanks, > George > > Jun Rao wrote: > > Hi, > > > > Today, HDFS always reads through a socket even when the data is local to > > the client. This adds a lot of overhead, especially for warm reads. It > > should be possible for a dfs client to test if a block to be read is local > > and if so, bypass socket and read through local FS api directly. This > > should improve random access performance significantly (e.g., for HBase). > > Has this been considered in HDFS? Thanks, > > > > Jun > > > >
