Kihwal Lee created HDFS-6360:
--------------------------------
Summary: MiniDFSCluster can cause unexpected side effects due to
sharing of config
Key: HDFS-6360
URL: https://issues.apache.org/jira/browse/HDFS-6360
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
As noted in HDFS-6329 and HDFS-5522, certain use cases of MiniDFSCluster can
result in unexpected results and falsely failing or passing unit tests.
Since a {{Configuration}} object is shared for all namenode startups, the
modified conf object during a NN startup is passed to the next NN startup. The
effect of the modified conf propagation and subsequent modifications is
different depending on whether it is a single NN cluster, HA cluster or
federation cluster.
It also depends on what test cases are doing with the config. For example,
MiniDFSCluster#getConfiguration(int) returns the saved conf for the specified
NN, but that is not actually the conf object used by the NN. It just contained
the same content one time in the past and it is not guaranteed to be that way.
Restarting the same NN can also cause unexpected results. The new NN will
switch to the conf that was cloned & saved AFTER the last startup. The new NN
will start with a changed config intentionally or unintentionally. The config
variables such as {{fs.defaultFs}}, {{dfs.namenode.rpc-address}} will be
implicitly set differently than the initial condition. Some test cases rely on
this and others occasionally break because of this.
In summary,
* MiniDFSCluster does not properly isolate configs.
* Many test cases happen to work most of times. Correcting MiniDFSCluster
causes mass breakages of test cases and requires fixing them.
* Many test cases rely on broken behavior and might pass when they should have
actually failed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
