[
https://issues.apache.org/jira/browse/HADOOP-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585812#action_12585812
]
Marco Nicosia commented on HADOOP-2150:
---------------------------------------
> I'm willing to implement as long as we get consensus from at least 1
> committer
+1 (fwiw, ianac)
> dfs.data.dir syntax needs revamping: multiple percentages and weights
> ---------------------------------------------------------------------
>
> Key: HADOOP-2150
> URL: https://issues.apache.org/jira/browse/HADOOP-2150
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Environment: This is likely a cross-platform issue.
> Reporter: Allen Wittenauer
> Priority: Minor
>
> Currently, all filesystems listed in the dfs.data.dir are treated the same
> with respected to the space reservation percentages. This makes sense on
> homogeneous, dedicated machines, but breaks badly on heterogeneous ones and
> creates a bit of a support nightmare.
> In a grid with multiple disk sizes, the admin is either leaving space
> unallocated or is required to slice up the disk. In addition, if Hadoop
> isn't the only application running, there may be unexpected collisions. In
> order to work around this limitation, the administrator must specifically
> partition up filesystem space such that the reservation 'make sense' for all
> of the configured file systems. For example, if someone has 2 small file
> systems and 2 big ones on a single machine, due to various requirements (such
> as the OS being mirrored, systems were built from spare parts, server
> consolidation, whatever). Reserving 10% might make sense on the small file
> systems (say 7G) but 10% may leave a lot more space than desired free on the
> big ones (say 50G).
> Instead, Hadoop should support a more robust syntax for directory layout.
> Ideally, an admin should be able to specify the directory location, the
> amount of space reserved (in either a percentage or a raw size syntax) for
> HDFS, as well as a weighting such that some file systems may be preferred
> over others. In the example above, the two larger file systems would likely
> be preferred over the two smaller ones. Additionally, the reservation on the
> larger file system might be changed such that it matches the 7G on the
> smaller file system.
> Doing so would allow for much more complex configuration scenarios without
> having to shuffle a lot of things around at the operating system level.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
