If you are pressed for time, you could look at the source code. I believe a huge proportion of the people that could answer your question ( and it isn't a small one ) are sleeping right now. :)
Source code is probably your best answer. James Sent from my mobile. Please excuse the typos. On 2011-05-02, at 5:23 AM, Matthew John <[email protected]> wrote: > someone kindly give some pointers on this!! > > On Mon, May 2, 2011 at 12:46 PM, Matthew John > <[email protected]>wrote: > >> Any documentations on how the different daemons do the write/read on HDFS >> and Local File System (direct), I mean the different protocols used in the >> interactions. I basically wanted to figure out how intricate the coupling >> between the Storage (HDFS + Local) and other processes in the Hadoop >> infrastructure is. >> >> >> >> On Mon, May 2, 2011 at 12:26 PM, Ted Dunning <[email protected]>wrote: >> >>> Yes. There is quite a bit of need for the local file system in clustered >>> mode. >>> >>> For one think, all of the shuffle intermediate files are on local disk. >>> For >>> another, the distributed cache is actually stored on local disk. >>> >>> HFDS is a frail vessel that cannot cope with all the needs. >>> >>> On Sun, May 1, 2011 at 11:48 PM, Matthew John <[email protected] >>>> wrote: >>> >>>> ... >>>> 2) Does the Hadoop system utilize the local storage directly for any >>>> purpose >>>> (without going through the HDFS) in clustered mode? >>>> >>>> >>> >> >>
