dfs.data.dir syntax needs revamping: multiple percentages and weights
---------------------------------------------------------------------
Key: HADOOP-2150
URL: https://issues.apache.org/jira/browse/HADOOP-2150
Project: Hadoop
Issue Type: Improvement
Components: dfs
Environment: This is likely a cross-platform issue.
Reporter: Allen Wittenauer
Priority: Minor
Currently, all filesystems listed in the dfs.data.dir are treated the same with
respected to the space reservation percentages. This makes sense on
homogeneous, dedicated machines, but breaks badly on heterogeneous ones and
creates a bit of a support nightmare.
In a grid with multiple disk sizes, the admin is either leaving space
unallocated or is required to slice up the disk. In addition, if Hadoop isn't
the only application running, there may be unexpected collisions. In order to
work around this limitation, the administrator must specifically partition up
filesystem space such that the reservation 'make sense' for all of the
configured filesystems. For example, if someone has 2 small file systems and
2 big ones on a single machine, due to various requirements (such as the OS
being mirrored, systems were built from spare parts, server consolidation,
whatever). Reserving 10% might make sense on the small file systems (say 7G)
but 10% may leave a lot more space than desired free on the big ones (say 50G).
Instead, Hadoop should support a more robust syntax for directory layout.
Ideally, an admin should be able to specify the directory location, the amount
of space reserved--in either a percentage or a raw size syntax--for HDFS, as
well as a weighting such that some file systems may be preferred over others.
In the example above, the two larger file systems would likely be preferred
over the two smaller ones. Additionally, the reservation on the larger file
system might be changed such that it matches the 7G on the smaller file system.
Doing so would allow for much more complex configuration scenarios without
having to shuffle a lot of things around at the operating system level.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
