Hi Stas, Look at the LocalDirAllocator class. It's not really meant for public consumption, so its API might break in future releases, but that's the class that the hadoop daemons use for round robin behavior.
-Todd On Tue, Sep 15, 2009 at 4:51 AM, Stas Oskin <[email protected]> wrote: > Hi. > > Thanks for the advice, but I actually asked, if I can use the DFS local > files (where the file is written locally, and not to DFS), with the > round-robin mechanism, for my own needs. > > Regards. > > 2009/9/14 Dali Kilani <[email protected]> > > > Just specify multiple directories (where the different local partitions > are > > mounted) for dfs.data.dir (hdfs data) and mapred.local.dir (intermediate > > data) in hdfs-site.xml.Data should then be striped across the different > > partitions/disks. > > > > See here : http://bit.ly/fbUkr > > > > Dali > > > > On Mon, Sep 14, 2009 at 11:50 AM, Stas Oskin <[email protected]> > wrote: > > > > > Hi. > > > > > > Thanks for the explanation. > > > > > > Any idea if I can re-use this round robin mechanism for local disk > > writing? > > > > > > Or it's DFS only? > > > > > > Regards. > > > > > > 2009/9/14 Jason Venner <[email protected]> > > > > > > > When you have multiple partitions specified for hdfs storage, they > are > > > used > > > > for block storage in a round robin fashion. > > > > If a partition has insufficient space it is dropped for the set used > > for > > > > storing new blocks. > > > > > > > > On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <[email protected]> > > > wrote: > > > > > > > > > Hi. > > > > > > > > > > When I specify multiple disks for DFS, does Hadoop distributes the > > > > > concurrent writings over the multiple disks? > > > > > > > > > > I mean, to prevent an utilization of a single disk? > > > > > > > > > > Thanks for any info on subject. > > > > > > > > > > > > > > > > > > > > > -- > > > > Pro Hadoop, a book to guide you from beginner to hadoop mastery, > > > > http://www.amazon.com/dp/1430219424?tag=jewlerymall > > > > www.prohadoopbook.com a community for Hadoop Professionals > > > > > > > > > > > > > > > -- > > Dali Kilani > > =========== > > Phone : (650) 492-5921 (Google Voice) > > E-Fax : (775) 552-2982 > > >
