Hi Todd. Thanks, that's exactly what I looked for.
Regards. 2009/9/15 Todd Lipcon <[email protected]> > Hi Stas, > > Look at the LocalDirAllocator class. It's not really meant for public > consumption, so its API might break in future releases, but that's the > class > that the hadoop daemons use for round robin behavior. > > -Todd > > On Tue, Sep 15, 2009 at 4:51 AM, Stas Oskin <[email protected]> wrote: > > > Hi. > > > > Thanks for the advice, but I actually asked, if I can use the DFS local > > files (where the file is written locally, and not to DFS), with the > > round-robin mechanism, for my own needs. > > > > Regards. > > > > 2009/9/14 Dali Kilani <[email protected]> > > > > > Just specify multiple directories (where the different local partitions > > are > > > mounted) for dfs.data.dir (hdfs data) and mapred.local.dir > (intermediate > > > data) in hdfs-site.xml.Data should then be striped across the different > > > partitions/disks. > > > > > > See here : http://bit.ly/fbUkr > > > > > > Dali > > > > > > On Mon, Sep 14, 2009 at 11:50 AM, Stas Oskin <[email protected]> > > wrote: > > > > > > > Hi. > > > > > > > > Thanks for the explanation. > > > > > > > > Any idea if I can re-use this round robin mechanism for local disk > > > writing? > > > > > > > > Or it's DFS only? > > > > > > > > Regards. > > > > > > > > 2009/9/14 Jason Venner <[email protected]> > > > > > > > > > When you have multiple partitions specified for hdfs storage, they > > are > > > > used > > > > > for block storage in a round robin fashion. > > > > > If a partition has insufficient space it is dropped for the set > used > > > for > > > > > storing new blocks. > > > > > > > > > > On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <[email protected]> > > > > wrote: > > > > > > > > > > > Hi. > > > > > > > > > > > > When I specify multiple disks for DFS, does Hadoop distributes > the > > > > > > concurrent writings over the multiple disks? > > > > > > > > > > > > I mean, to prevent an utilization of a single disk? > > > > > > > > > > > > Thanks for any info on subject. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Pro Hadoop, a book to guide you from beginner to hadoop mastery, > > > > > http://www.amazon.com/dp/1430219424?tag=jewlerymall > > > > > www.prohadoopbook.com a community for Hadoop Professionals > > > > > > > > > > > > > > > > > > > > > -- > > > Dali Kilani > > > =========== > > > Phone : (650) 492-5921 (Google Voice) > > > E-Fax : (775) 552-2982 > > > > > >
