[
https://issues.apache.org/jira/browse/HDFS-10723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15409906#comment-15409906
]
Vrushali C commented on HDFS-10723:
-----------------------------------
I updated the test to use a random port number. The test output that I see is
as follows:
{code}
[machine-channapattan hadoop-hdfs (trunk)]$ grep -rni -A2 "nnport="
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:2:Will
now start MiniDFSCluster with nnport=58999
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-3-2016-08-05
11:51:56,081 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:<init>(465)) - starting cluster: numNameNodes=1,
numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-4-2016-08-05
11:51:56,319 [main] DEBUG namenode.NameNode
(NameNode.java:initializeGenericKeys(1670)) - Setting fs.defaultFS to
hdfs://127.0.0.1:58999
--
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:1592:Will
now start MiniDFSCluster with nnport=7009
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-1593-2016-08-05
11:52:09,583 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:<init>(465)) - starting cluster: numNameNodes=1,
numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-1594-2016-08-05
11:52:09,587 [main] DEBUG namenode.NameNode
(NameNode.java:initializeGenericKeys(1670)) - Setting fs.defaultFS to
hdfs://127.0.0.1:7009
--
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:2523:Will
now start MiniDFSCluster with nnport=17516
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-2524-2016-08-05
11:52:20,090 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:<init>(465)) - starting cluster: numNameNodes=1,
numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-2525-2016-08-05
11:52:20,093 [main] DEBUG namenode.NameNode
(NameNode.java:initializeGenericKeys(1670)) - Setting fs.defaultFS to
hdfs://127.0.0.1:17516
--
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt:3450:Will
now start MiniDFSCluster with nnport=28234
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-3451-2016-08-05
11:52:30,808 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:<init>(465)) - starting cluster: numNameNodes=1,
numDataNodes=1
target/surefire-reports/org.apache.hadoop.hdfs.TestRenameWhileOpen-output.txt-3452-2016-08-05
11:52:30,811 [main] DEBUG namenode.NameNode
(NameNode.java:initializeGenericKeys(1670)) - Setting fs.defaultFS to
hdfs://127.0.0.1:28234
[machine-channapattan hadoop-hdfs (trunk)]$
{code}
So I still can't upload a patch to HDFS jiras. Hence pasting my patch here:
{code}
[machine-channapattan hadoop (trunk)]$ cat ../HDFS-10723.001.patch
diff --git
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
index 949fc74..9d6b127 100644
---
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
+++
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
@@ -55,10 +55,16 @@ public void testWhileOpenRenameParent() throws IOException {
conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1);
conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, TestFileCreation.blockSize);
+ System.out.println("Test 1*****************************");
+ // try to get a random port number < 65535
+ // hence take mod 65534
+ final int nnport = (int) (System.currentTimeMillis() % 65534);
+ System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
// create cluster
- System.out.println("Test 1*****************************");
- MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+ .nameNodePort(nnport).format(true).build();
+
FileSystem fs = null;
try {
cluster.waitActive();
@@ -72,8 +78,6 @@ public void testWhileOpenRenameParent() throws IOException {
doNothing().when(spyLog).endCurrentLogSegment(Mockito.anyBoolean());
DFSTestUtil.setEditLogForTesting(cluster.getNamesystem(), spyLog);
- final int nnport = cluster.getNameNodePort();
-
// create file1.
Path dir1 = new Path("/user/a+b/dir1");
Path file1 = new Path(dir1, "file1");
@@ -155,13 +159,18 @@ public void testWhileOpenRenameParentToNonexistentDir()
throws IOException {
conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
System.out.println("Test 2************************************");
+ // try to get a random port number < 65535
+ // hence take mod 65534
+ final int nnport = (int) (System.currentTimeMillis() % 65534);
+ System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
// create cluster
- MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+ .nameNodePort(nnport).format(true).build();
+
FileSystem fs = null;
try {
cluster.waitActive();
fs = cluster.getFileSystem();
- final int nnport = cluster.getNameNodePort();
// create file1.
Path dir1 = new Path("/user/dir1");
@@ -230,13 +239,18 @@ public void testWhileOpenRenameToExistentDirectory()
throws IOException {
conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
System.out.println("Test 3************************************");
+ // try to get a random port number < 65535
+ // hence take mod 65534
+ final int nnport = (int) (System.currentTimeMillis() % 65534);
+ System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
// create cluster
- MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+ .nameNodePort(nnport).format(true).build();
+
FileSystem fs = null;
try {
cluster.waitActive();
fs = cluster.getFileSystem();
- final int nnport = cluster.getNameNodePort();
// create file1.
Path dir1 = new Path("/user/dir1");
@@ -295,13 +309,18 @@ public void testWhileOpenRenameToNonExistentDirectory()
throws IOException {
conf.setInt(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_KEY, 1);
System.out.println("Test 4************************************");
+ // try to get a random port number < 65535
+ // hence take mod 65534
+ final int nnport = (int) (System.currentTimeMillis() % 65534);
+ System.out.println("Will now start MiniDFSCluster with nnport=" + nnport);
// create cluster
- MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+ .nameNodePort(nnport).format(true).build();
+
FileSystem fs = null;
try {
cluster.waitActive();
fs = cluster.getFileSystem();
- final int nnport = cluster.getNameNodePort();
// create file1.
Path dir1 = new Path("/user/dir1");
[machine-channapattan hadoop (trunk)]$
{code}
Would appreciate your feedback.
> TestRenameWhileOpen tries to restart NN with the same rpc port
> --------------------------------------------------------------
>
> Key: HDFS-10723
> URL: https://issues.apache.org/jira/browse/HDFS-10723
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Reporter: Kihwal Lee
> Attachments: testWhileOpenRenameParentToNonexistentDir.txt
>
>
> In {{testWhileOpenRenameParentToNonexistentDir}},
> {code}
> // restart cluster with the same namenode port as before.
> // This ensures that leases are persisted in fsimage.
> cluster.shutdown();
> {code}
> And we are getting test failures in precommit builds.
> {noformat}
> java.net.BindException: Problem binding to [localhost:47395]
> java.net.BindException: Address already in use;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
