MiniDFS cluster improvements
----------------------------
Key: HDFS-2209
URL: https://issues.apache.org/jira/browse/HDFS-2209
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Affects Versions: 0.20.203.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
I've been deploying MiniDFSCluster for some testing, and while using it/looking
through the code I made some notes of where there are issues and improvement
opportunities. This is mostly minor as its a test tool, but a risk of
synchronization problems is there and does need addressing; the rest are all
feature creep.
Field {{nameNode}} should be marked as volatile as the shutdown operation can
be in a different thread than startup. Best of all,
add synchronized methods to set and get the field, as well as shutdown.
The data dir is set from from System Properties.
{code}
base_dir = new File(System.getProperty("test.build.data",
"build/test/data"), "dfs/");
data_dir = new File(base_dir, "data");
{code}
This is done in {{formatDataNodeDirs()}} {{corruptBlockOnDataNode()}} and the
constructor.
Improvement: have a test property in the conf file, and only read the system
property if this is unset. This will enable
multiple MiniDFSClusters to come up in the same JVM, and handle
shutdown/startup race conditions better, and avoid the
"java.io.IOException: Cannot lock storage build/test/data/dfs/name1. The
directory is already locked." messages
Messages should log to the commons logging and not {{System.err}} and
{{System.out}}. This enables containers to catch and stream better,
and include more diagnostics such as timestamp and thread Id
Class could benefit from a method to return the FS URI, rather than just the
FS. This currently has to be worked around with some tricks involving a cached
configuration
{{waitActive()}} could get confused if "localhost" maps to an IPv6 address.
Better to ask for 127.0.0.1 as the hostname; Junit
test runs may need to be set up to force in IPv4 too.
{{injectBlocks}} has a spelling error in the IOException, "SumulatedFSDataset"
is the correct spelling
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
