[jira] [Commented] (HDFS-6169) Move the address in WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962634#comment-13962634
]
Akira AJISAKA commented on HDFS-6169:
-
Thanks for the comments! Attaching a new patch.
bq. How to differentiate the url "/webhdfs/v1?op=LISTSTATUS" and "/wrongurl"?
I started a NameNode and sent a HTTP request to "/webhdfs/v1?op=liststatus" :
{code}
$ curl -i http://trunk:50070/webhdfs/v1?op=liststatus
HTTP/1.1 404 Not Found
Cache-Control: no-cache
Expires: Tue, 08 Apr 2014 04:36:24 GMT
Date: Tue, 08 Apr 2014 04:36:24 GMT
Pragma: no-cache
Expires: Tue, 08 Apr 2014 04:36:24 GMT
Date: Tue, 08 Apr 2014 04:36:24 GMT
Pragma: no-cache
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File
/webhdfs/v1 does not exist."}}
{code}
Since both "/webhdfs/v1" and "/wrongurl" return 404, I think there is no need
to differentiate them.
> Move the address in WebImageViewer
> --
>
> Key: HDFS-6169
> URL: https://issues.apache.org/jira/browse/HDFS-6169
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-6169.2.patch, HDFS-6169.3.patch, HDFS-6169.4.patch,
> HDFS-6169.5.patch, HDFS-6169.patch
>
>
> Move the endpoint of WebImageViewer from http://hostname:port/ to
> http://hostname:port/webhdfs/v1/ to support {{hdfs dfs -ls}} to
> WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5693) Few NN metrics data points were collected via JMX when NN is under heavy load
[
https://issues.apache.org/jira/browse/HDFS-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962590#comment-13962590
]
Hadoop QA commented on HDFS-5693:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639123/HDFS-5693.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6613//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6613//console
This message is automatically generated.
> Few NN metrics data points were collected via JMX when NN is under heavy load
> -
>
> Key: HDFS-5693
> URL: https://issues.apache.org/jira/browse/HDFS-5693
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Ming Ma
> Attachments: HADOOP-5693.patch, HDFS-5693.patch
>
>
> JMX sometimes doesn’t return any value when NN is under heavy load. However,
> that is when we would like to get metrics to help to diagnosis the issue.
> When NN is under heavy load due to bad application or other reasons, it holds
> FSNamesystem's writer lock for a long period of time. Many of the
> FSNamesystem metrics require FSNamesystem's reader lock and thus can't be
> processed.
> This is a special case to improve the overall NN concurrency.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Attachment: HDFS-6200.004.patch > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962569#comment-13962569 ] Haohui Mai commented on HDFS-6200: -- Rebased > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962565#comment-13962565
]
Hadoop QA commented on HDFS-6200:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639133/HDFS-6200.003.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6614//console
This message is automatically generated.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch,
> HDFS-6200.002.patch, HDFS-6200.003.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962562#comment-13962562 ] Haohui Mai commented on HDFS-6200: -- The v3 patch fixes the findbugs and the unit test issues. > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Attachment: HDFS-6200.003.patch > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962552#comment-13962552
]
Hadoop QA commented on HDFS-6200:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639104/HDFS-6200.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6612//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6612//console
This message is automatically generated.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch,
> HDFS-6200.002.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5693) Few NN metrics data points were collected via JMX when NN is under heavy load
[ https://issues.apache.org/jira/browse/HDFS-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-5693: -- Attachment: HDFS-5693.patch Updated patch for latest trunk. > Few NN metrics data points were collected via JMX when NN is under heavy load > - > > Key: HDFS-5693 > URL: https://issues.apache.org/jira/browse/HDFS-5693 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Ming Ma > Attachments: HADOOP-5693.patch, HDFS-5693.patch > > > JMX sometimes doesn’t return any value when NN is under heavy load. However, > that is when we would like to get metrics to help to diagnosis the issue. > When NN is under heavy load due to bad application or other reasons, it holds > FSNamesystem's writer lock for a long period of time. Many of the > FSNamesystem metrics require FSNamesystem's reader lock and thus can't be > processed. > This is a special case to improve the overall NN concurrency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6143: - Target Version/s: (was: 2.4.1) > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Fix For: 2.5.0 > > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962512#comment-13962512 ] Hudson commented on HDFS-6143: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5467 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5467/]) HDFS-6143. WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths. Contributed by Gera Shegalov. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585639) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Fix For: 2.5.0 > > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6143: - Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed the patch to trunk and branch-2. Thanks [~jira.shegalov] for the contribution. > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Fix For: 2.5.0 > > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962505#comment-13962505 ] Haohui Mai commented on HDFS-6143: -- Test failures are unrelated. I'll commit it shortly. > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[
https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962494#comment-13962494
]
Hadoop QA commented on HDFS-6143:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639083/HDFS-6143-trunk-after-HDFS-5570.v02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.TestFileAppend4
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6609//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6609//console
This message is automatically generated.
> WebHdfsFileSystem open should throw FileNotFoundException for non-existing
> paths
>
>
> Key: HDFS-6143
> URL: https://issues.apache.org/jira/browse/HDFS-6143
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: HDFS-6143-branch-2.4.0.v01.patch,
> HDFS-6143-trunk-after-HDFS-5570.v01.patch,
> HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch,
> HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch,
> HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch
>
>
> WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles
> non-existing paths.
> - 'open', does not really open anything, i.e., it does not contact the
> server, and therefore cannot discover FileNotFound, it's deferred until next
> read. It's counterintuitive and not how local FS or HDFS work. In POSIX you
> get ENOENT on open.
> [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java]
> is an example of the code that's broken because of this.
> - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST
> instead of SC_NOT_FOUND for non-exitsing paths
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Attachment: HDFS-6200.002.patch > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3503) Move LengthInputStream and PositionTrackingInputStream to common
[
https://issues.apache.org/jira/browse/HDFS-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962454#comment-13962454
]
Hadoop QA commented on HDFS-3503:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639073/h3503_20140407.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6608//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6608//console
This message is automatically generated.
> Move LengthInputStream and PositionTrackingInputStream to common
>
>
> Key: HDFS-3503
> URL: https://issues.apache.org/jira/browse/HDFS-3503
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h3503_20140328.patch, h3503_20140407.patch
>
>
> We have LengthInputStream in org.apache.hadoop.hdfs.server.datanode.fsdataset
> and PositionTrackingInputStream in
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader. These two classes
> are generally useful. Let's move them to org.apache.hadoop.io.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962443#comment-13962443
]
Hadoop QA commented on HDFS-6200:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639097/HDFS-6200.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6611//console
This message is automatically generated.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6056) Clean up NFS config settings
[
https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962442#comment-13962442
]
Hadoop QA commented on HDFS-6056:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639069/HDFS-6056.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs-nfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6607//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6607//console
This message is automatically generated.
> Clean up NFS config settings
>
>
> Key: HDFS-6056
> URL: https://issues.apache.org/jira/browse/HDFS-6056
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.3.0
>Reporter: Aaron T. Myers
>Assignee: Brandon Li
> Attachments: HDFS-6056.001.patch
>
>
> As discussed on HDFS-6050, there's a few opportunities to improve the config
> settings related to NFS. This JIRA is to implement those changes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6198: Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this to trunk, branch-2 and branch-2.4. Haohui and Arpit, thank you for the code reviews. > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6198.1.patch, HDFS-6198.2.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962424#comment-13962424 ] Hudson commented on HDFS-6198: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5466 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5466/]) HDFS-6198. DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585627) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockPoolSliceStorage.java > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-6198.1.patch, HDFS-6198.2.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6180) dead node count / listing is very broken in JMX and old GUI
[
https://issues.apache.org/jira/browse/HDFS-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962425#comment-13962425
]
Hudson commented on HDFS-6180:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5466 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5466/])
HDFS-6180. Dead node count / listing is very broken in JMX and old GUI.
Contributed by Haohui Mai. (wheat9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585625)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostFileManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHostFileManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
> dead node count / listing is very broken in JMX and old GUI
> ---
>
> Key: HDFS-6180
> URL: https://issues.apache.org/jira/browse/HDFS-6180
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Haohui Mai
>Priority: Blocker
> Fix For: 2.5.0
>
> Attachments: HDFS-6180.000.patch, HDFS-6180.001.patch,
> HDFS-6180.002.patch, HDFS-6180.003.patch, HDFS-6180.004.patch, dn.log
>
>
> After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on
> the new GUI, but showed up properly in the datanodes tab. Some nodes are
> also being double reported in the deadnode and inservice section (22 show up
> dead, 565 show up alive, 9 duplicated nodes).
> From /jmx (confirmed that it's the same in jconsole):
> {noformat}
> {
> "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : 5477748687372288,
> "CapacityUsed" : 24825720407,
> "CapacityRemaining" : 5477723861651881,
> "TotalLoad" : 565,
> "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
> "BlocksTotal" : 21065,
> "MaxObjects" : 0,
> "FilesTotal" : 25454,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational",
> "NumLiveDataNodes" : 565,
> "NumDeadDataNodes" : 0,
> "NumDecomLiveDataNodes" : 0,
> "NumDecomDeadDataNodes" : 0,
> "NumDecommissioningDataNodes" : 0,
> "NumStaleDataNodes" : 1
> },
> {noformat}
> I'm not going to include deadnode/livenodes because the list is huge, but
> I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Attachment: HDFS-6200.001.patch > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6180) dead node count / listing is very broken in JMX and old GUI
[
https://issues.apache.org/jira/browse/HDFS-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6180:
-
Resolution: Fixed
Fix Version/s: 2.5.0
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks Nicholas for the review.
> dead node count / listing is very broken in JMX and old GUI
> ---
>
> Key: HDFS-6180
> URL: https://issues.apache.org/jira/browse/HDFS-6180
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Haohui Mai
>Priority: Blocker
> Fix For: 2.5.0
>
> Attachments: HDFS-6180.000.patch, HDFS-6180.001.patch,
> HDFS-6180.002.patch, HDFS-6180.003.patch, HDFS-6180.004.patch, dn.log
>
>
> After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on
> the new GUI, but showed up properly in the datanodes tab. Some nodes are
> also being double reported in the deadnode and inservice section (22 show up
> dead, 565 show up alive, 9 duplicated nodes).
> From /jmx (confirmed that it's the same in jconsole):
> {noformat}
> {
> "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : 5477748687372288,
> "CapacityUsed" : 24825720407,
> "CapacityRemaining" : 5477723861651881,
> "TotalLoad" : 565,
> "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
> "BlocksTotal" : 21065,
> "MaxObjects" : 0,
> "FilesTotal" : 25454,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational",
> "NumLiveDataNodes" : 565,
> "NumDeadDataNodes" : 0,
> "NumDecomLiveDataNodes" : 0,
> "NumDecomDeadDataNodes" : 0,
> "NumDecommissioningDataNodes" : 0,
> "NumStaleDataNodes" : 1
> },
> {noformat}
> I'm not going to include deadnode/livenodes because the list is huge, but
> I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962406#comment-13962406
]
Hadoop QA commented on HDFS-6200:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639093/HDFS-6200.000.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6610//console
This message is automatically generated.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962402#comment-13962402
]
Haohui Mai commented on HDFS-6200:
--
The v0 patch moves WebHdfsFileSystem and SWebHdfsFileSystem into a separate
jar. The patch moves the files to a different project except for the following:
# It modifies {{DFSConfigKeys}} so that it no longer depends on the
{{AuthFilter}} and the {{BlockPlacementPolicyDefault}} class.
# It moves some methods from the {{HAUtil}} / {{DFSUtilClient}} in the original
hdfs jar to the {{HAUtilClient}} / {{DFSUtilClient}} in the client side jar.
# It moves some methods from the {{JsonUtils}} in the original hdfs jar to the
{{JsonUtilClient}} in the client side jar. It contains a new function that
translates JSON representation of the {{BlockLocation}} object to the Java
object.
To review this patch, one can use {{git diff -M}} to omit the renames.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Status: Patch Available (was: Open) > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Attachment: HDFS-6200.000.patch > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[
https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962395#comment-13962395
]
Hadoop QA commented on HDFS-6198:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639058/HDFS-6198.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6606//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6606//console
This message is automatically generated.
> DataNode rolling upgrade does not correctly identify current block pool
> directory and replace with trash on Windows.
>
>
> Key: HDFS-6198
> URL: https://issues.apache.org/jira/browse/HDFS-6198
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-6198.1.patch, HDFS-6198.2.patch
>
>
> For rolling upgrade, the DataNode uses a regex to identify the current block
> pool directory and replace it with the equivalent trash directory. This
> regex hard-codes '/' as the file path separator, which doesn't work on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6200) Create a separate jar for hdfs-client
Haohui Mai created HDFS-6200: Summary: Create a separate jar for hdfs-client Key: HDFS-6200 URL: https://issues.apache.org/jira/browse/HDFS-6200 Project: Hadoop HDFS Issue Type: Improvement Reporter: Haohui Mai Assignee: Haohui Mai Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs client. As discussed in the hdfs-dev mailing list (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), downstream projects are forced to bring in additional dependency in order to access hdfs. The additional dependency sometimes can be difficult to manage for projects like Apache Falcon and Apache Oozie. This jira proposes to create a new project, hadoop-hdfs-cliient, which contains the client side of the hdfs code. Downstream projects can use this jar instead of the hadoop-hdfs to avoid unnecessary dependency. Note that it does not break the compatibility of downstream projects. This is because old downstream projects implicitly depend on hadoop-hdfs-client through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962372#comment-13962372 ] Haohui Mai commented on HDFS-6143: -- +1 pending jenkins. > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6199) WebHdfsFileSystem does not treat UnresolvedLinkException
Gera Shegalov created HDFS-6199:
---
Summary: WebHdfsFileSystem does not treat UnresolvedLinkException
Key: HDFS-6199
URL: https://issues.apache.org/jira/browse/HDFS-6199
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Critical
{{o.a.h.hdfs.web.resources.ExceptionHandler.toResponse}} does not treat
{{UnresolvedLinkException}} . This implies the client-side will generate a
generic {{IOException}} or an unchecked exception.
The server should return {{GONE}} that can be translated to
{{FileNotFoundException}} or the actual {{UnresolvedLinkException}} if the API
allows it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated HDFS-6143: Attachment: HDFS-6143-trunk-after-HDFS-5570.v02.patch Removing the ExceptionHandler change to be addressed as a separate JIRA > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v02.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962313#comment-13962313 ] Haohui Mai commented on HDFS-6143: -- bq. That's server-side. On the client side, HttpUrlConnection translates GONE to FileNotFoundException as well... It looks to me that it depends on the fact that the server return invalid json rather than relying on the status code. Since this is marked as a blocker, it might be good to keep the changes minimal. Is it okay to remove this changes from this jira? We can address it into a separate jira though. > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-3503) Move LengthInputStream and PositionTrackingInputStream to common
[ https://issues.apache.org/jira/browse/HDFS-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-3503: -- Attachment: h3503_20140407.patch h3503_20140407.patch: synced with trunk. > Move LengthInputStream and PositionTrackingInputStream to common > > > Key: HDFS-3503 > URL: https://issues.apache.org/jira/browse/HDFS-3503 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h3503_20140328.patch, h3503_20140407.patch > > > We have LengthInputStream in org.apache.hadoop.hdfs.server.datanode.fsdataset > and PositionTrackingInputStream in > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader. These two classes > are generally useful. Let's move them to org.apache.hadoop.io. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6056: - Status: Patch Available (was: Open) > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962290#comment-13962290
]
Hudson commented on HDFS-6197:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5465 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5465/])
HDFS-6197. Rolling upgrade rollback on Windows can fail attempting to rename
edit log segment files to a destination that already exists. Contributed by
Chris Nauroth. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585586)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 3.0.0, 2.4.1
>
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6056: - Attachment: HDFS-6056.001.patch > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6197:
Resolution: Fixed
Fix Version/s: 2.4.1
3.0.0
Status: Resolved (was: Patch Available)
I committed this to trunk, branch-2 and branch-2.4. Nicholas and Jing, thank
you for the code reviews.
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 3.0.0, 2.4.1
>
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962266#comment-13962266
]
Chris Nauroth commented on HDFS-6197:
-
The test failures look to be unrelated. I can't reproduce them.
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6180) dead node count / listing is very broken in JMX and old GUI
[
https://issues.apache.org/jira/browse/HDFS-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6180:
--
Hadoop Flags: Reviewed
+1 patch looks good. Thanks, Haohui.
> dead node count / listing is very broken in JMX and old GUI
> ---
>
> Key: HDFS-6180
> URL: https://issues.apache.org/jira/browse/HDFS-6180
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Haohui Mai
>Priority: Blocker
> Attachments: HDFS-6180.000.patch, HDFS-6180.001.patch,
> HDFS-6180.002.patch, HDFS-6180.003.patch, HDFS-6180.004.patch, dn.log
>
>
> After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on
> the new GUI, but showed up properly in the datanodes tab. Some nodes are
> also being double reported in the deadnode and inservice section (22 show up
> dead, 565 show up alive, 9 duplicated nodes).
> From /jmx (confirmed that it's the same in jconsole):
> {noformat}
> {
> "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
> "CapacityTotal" : 5477748687372288,
> "CapacityUsed" : 24825720407,
> "CapacityRemaining" : 5477723861651881,
> "TotalLoad" : 565,
> "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
> "BlocksTotal" : 21065,
> "MaxObjects" : 0,
> "FilesTotal" : 25454,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "ScheduledReplicationBlocks" : 0,
> "FSState" : "Operational",
> "NumLiveDataNodes" : 565,
> "NumDeadDataNodes" : 0,
> "NumDecomLiveDataNodes" : 0,
> "NumDecomDeadDataNodes" : 0,
> "NumDecommissioningDataNodes" : 0,
> "NumStaleDataNodes" : 1
> },
> {noformat}
> I'm not going to include deadnode/livenodes because the list is huge, but
> I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962252#comment-13962252 ] Arpit Agarwal commented on HDFS-6198: - +1 for the updated patch, pending Jenkins. Thanks for finding and fixing it Chris. > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-6198.1.patch, HDFS-6198.2.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6198: Status: Patch Available (was: Open) > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-6198.1.patch, HDFS-6198.2.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6198: Attachment: HDFS-6198.2.patch Here is patch v2 with the additional test fix mentioned by Arpit. > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-6198.1.patch, HDFS-6198.2.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6178) Decommission on standby NN couldn't finish
[
https://issues.apache.org/jira/browse/HDFS-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962243#comment-13962243
]
Hadoop QA commented on HDFS-6178:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639019/HDFS-6178.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6604//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6604//console
This message is automatically generated.
> Decommission on standby NN couldn't finish
> --
>
> Key: HDFS-6178
> URL: https://issues.apache.org/jira/browse/HDFS-6178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ming Ma
> Attachments: HDFS-6178.patch
>
>
> Currently decommissioning machines in HA-enabled cluster requires running
> refreshNodes in both active and standby nodes. Sometimes decommissioning
> won't finish from standby NN's point of view. Here is the diagnosis of why
> it could happen.
> Standby NN's blockManager manages blocks replication and block invalidation
> as if it is the active NN; even though DNs will ignore block commands coming
> from standby NN. When standby NN makes block operation decisions such as the
> target of block replication and the node to remove excess blocks from, the
> decision is independent of active NN. So active NN and standby NN could have
> different states. When we try to decommission nodes on standby nodes; such
> state inconsistency might prevent standby NN from making progress. Here is an
> example.
> Machine A
> Machine B
> Machine C
> Machine D
> Machine E
> Machine F
> Machine G
> Machine H
> 1. For a given block, both active and standby have 5 replicas on machine A,
> B, C, D, E. So both active and standby decide to pick excess nodes to
> invalidate.
> Active picked D and E as excess DNs. After the next block reports from D and
> E, active NN has 3 active replicas (A, B, C), 0 excess replica.
> {noformat}
> 2014-03-27 01:50:14,410 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (E:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> 2014-03-27 01:50:15,539 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (D:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> {noformat}
> Standby pick C, E as excess DNs. Given DNs ignore commands from standby,
> After the next block reports from C, D, E, standby has 2 active replicas (A,
> B), 1 excess replica (C).
> {noformat}
> 2014-03-27 01:51:49,543 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (E:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> 2014-03-27 01:51:49,894 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (C:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> {noformat}
> 2. Machine A decomm request was sent to standby. Standby only had one live
> replica and picked machine G, H as targets, but given standby commands was
> ignored by DNs, G, H remained in pending replication queue until they are
> timed out. At this point, you have one decommissioning replica (A), 1 active
> replica (B), one excess replica (C).
> {noformat}
> 2014-03-27 04:42:52,258 INFO BlockStateChange: BLOCK* ask A:50010 to
> replicate blk_-5207804474559026159_121186764 to datanode(s) G:50010 H:50010
> {noformat}
> 3. Machine A decomm request was sent to active NN. Active NN picked machine F
> as the target. It finished properly. So active NN had 3 active replicas (B,
> C, F), one decommissioned replica (A).
> {noformat}
> 2014-03-27 04:44:15,239 INFO BlockStateChange: BLOCK* ask 10.42.246.110:50010
> to replicate blk_-5207804474559026159_121186764 to datanode(s) F:50010
> 2014-03-27 04:44:16,083 INFO BlockStateChange: BLOCK* addStoredBl
[jira] [Updated] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6198: Status: Open (was: Patch Available) > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-6198.1.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[
https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962233#comment-13962233
]
Chris Nauroth commented on HDFS-6198:
-
If we were just doing {{File.separator + "BPstuff" + File.separator}}, then the
regex would be invalid. We'd get an exception while trying to parse the regex.
Wrapping it with {{Pattern.quote}} is what makes it safe. This injects {{\Q}}
and {{\E}} for literal character quoting. On Linux/Mac, the final pattern is:
{code}
^(.*)(\Q/\EBP-\d+-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}-\d+\Q/\E)(current)(.*)$
{code}
On Windows, the final pattern is:
{code}
^(.*)(\Q\\EBP-\d+-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}-\d+\Q\\E)(current)(.*)$
{code}
Thanks for the tip on {{TestBlockPoolSliceStorage}}. I'll take a look at that
one and provide a new patch.
> DataNode rolling upgrade does not correctly identify current block pool
> directory and replace with trash on Windows.
>
>
> Key: HDFS-6198
> URL: https://issues.apache.org/jira/browse/HDFS-6198
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-6198.1.patch
>
>
> For rolling upgrade, the DataNode uses a regex to identify the current block
> pool directory and replace it with the equivalent trash directory. This
> regex hard-codes '/' as the file path separator, which doesn't work on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[
https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962230#comment-13962230
]
Arpit Agarwal commented on HDFS-6198:
-
Hi Chris, I think we should also fix {{TestBlockPoolSliceStorage}} in the same
Jira.
It was intended as a targeted unit test for the regex operations and also uses
"/" as the hard-coded separator. I should have known better...
> DataNode rolling upgrade does not correctly identify current block pool
> directory and replace with trash on Windows.
>
>
> Key: HDFS-6198
> URL: https://issues.apache.org/jira/browse/HDFS-6198
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-6198.1.patch
>
>
> For rolling upgrade, the DataNode uses a regex to identify the current block
> pool directory and replace it with the equivalent trash directory. This
> regex hard-codes '/' as the file path separator, which doesn't work on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962228#comment-13962228
]
Hadoop QA commented on HDFS-6197:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639026/HDFS-6197.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDistributedFileSystem
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6603//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6603//console
This message is automatically generated.
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6181) Fix the wrong property names in NFS user guide
[
https://issues.apache.org/jira/browse/HDFS-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6181:
-
Fix Version/s: 2.5.0
> Fix the wrong property names in NFS user guide
> --
>
> Key: HDFS-6181
> URL: https://issues.apache.org/jira/browse/HDFS-6181
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HDFS-6181.002.patch, HDFS-6181.003.patch, HDFS-6181.patch
>
>
> A couple property names are wrong in the NFS user guide, and should be fixed
> as the following:
> {noformat}
>
> - dfs.nfsgateway.keytab.file
> + dfs.nfs.keytab.file
>/etc/hadoop/conf/nfsserver.keytab
>
>
> - dfs.nfsgateway.kerberos.principal
> + dfs.nfs.kerberos.principal
>nfsserver/_h...@your-realm.com
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[
https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962224#comment-13962224
]
Gera Shegalov commented on HDFS-6143:
-
Hi [~wheat9],
bq. Should this be 404 instead of 410? In UNIX, opening a unresolved symlink
return ENOENT.
That's server-side. On the client side, {{HttpUrlConnection}} translates
{{GONE}} to {{FileNotFoundException}} as well.
On another note, if you look at the API for {{AbstractFileSystem}} a potential
future WebHdfs implementation will be able to surface the real cause as
{{UnresolvedLinkException}}
> WebHdfsFileSystem open should throw FileNotFoundException for non-existing
> paths
>
>
> Key: HDFS-6143
> URL: https://issues.apache.org/jira/browse/HDFS-6143
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: HDFS-6143-branch-2.4.0.v01.patch,
> HDFS-6143-trunk-after-HDFS-5570.v01.patch, HDFS-6143.v01.patch,
> HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch,
> HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch
>
>
> WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles
> non-existing paths.
> - 'open', does not really open anything, i.e., it does not contact the
> server, and therefore cannot discover FileNotFound, it's deferred until next
> read. It's counterintuitive and not how local FS or HDFS work. In POSIX you
> get ENOENT on open.
> [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java]
> is an example of the code that's broken because of this.
> - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST
> instead of SC_NOT_FOUND for non-exitsing paths
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[
https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962218#comment-13962218
]
Haohui Mai commented on HDFS-6198:
--
Is {{Pattern.quote(File.separator)}} going to be {{\}} in Windows? If so then
the regex should be invalid. Am I correct?
> DataNode rolling upgrade does not correctly identify current block pool
> directory and replace with trash on Windows.
>
>
> Key: HDFS-6198
> URL: https://issues.apache.org/jira/browse/HDFS-6198
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-6198.1.patch
>
>
> For rolling upgrade, the DataNode uses a regex to identify the current block
> pool directory and replace it with the equivalent trash directory. This
> regex hard-codes '/' as the file path separator, which doesn't work on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[
https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6198:
Attachment: HDFS-6198.1.patch
This patch uses the platform-specific file path separator. I've verified that
{{TestDataNodeRollingUpgrade}} passes on both Mac and Windows after this patch.
> DataNode rolling upgrade does not correctly identify current block pool
> directory and replace with trash on Windows.
>
>
> Key: HDFS-6198
> URL: https://issues.apache.org/jira/browse/HDFS-6198
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-6198.1.patch
>
>
> For rolling upgrade, the DataNode uses a regex to identify the current block
> pool directory and replace it with the equivalent trash directory. This
> regex hard-codes '/' as the file path separator, which doesn't work on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
[ https://issues.apache.org/jira/browse/HDFS-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6198: Status: Patch Available (was: Open) > DataNode rolling upgrade does not correctly identify current block pool > directory and replace with trash on Windows. > > > Key: HDFS-6198 > URL: https://issues.apache.org/jira/browse/HDFS-6198 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-6198.1.patch > > > For rolling upgrade, the DataNode uses a regex to identify the current block > pool directory and replace it with the equivalent trash directory. This > regex hard-codes '/' as the file path separator, which doesn't work on > Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6198) DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows.
Chris Nauroth created HDFS-6198: --- Summary: DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows. Key: HDFS-6198 URL: https://issues.apache.org/jira/browse/HDFS-6198 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0, 2.4.0 Reporter: Chris Nauroth Assignee: Chris Nauroth For rolling upgrade, the DataNode uses a regex to identify the current block pool directory and replace it with the equivalent trash directory. This regex hard-codes '/' as the file path separator, which doesn't work on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6181) Fix the wrong property names in NFS user guide
[ https://issues.apache.org/jira/browse/HDFS-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962154#comment-13962154 ] Hudson commented on HDFS-6181: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5464 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5464/]) HDFS-6181. Fix the wrong property names in NFS user guide. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585563) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/RpcProgramMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/QJournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/GetJournalEditServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/InterDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/JournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetGroups.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStartSecureDataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetImageServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server
[jira] [Commented] (HDFS-6191) Disable quota checks when replaying edit log.
[ https://issues.apache.org/jira/browse/HDFS-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962147#comment-13962147 ] Hudson commented on HDFS-6191: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5463 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5463/]) HDFS-6191. Disable quota checks when replaying edit log. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1585544) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java > Disable quota checks when replaying edit log. > - > > Key: HDFS-6191 > URL: https://issues.apache.org/jira/browse/HDFS-6191 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0, 0.23.11, 2.5.0 > > Attachments: HDFS-6191.branch-0.23.patch, HDFS-6191.patch, > HDFS-6191.patch, HDFS-6191.patch > > > Since the serving NN does quota checks before logging edits, performing quota > checks is unnecessary while replaying edits. I propose disabling quota checks > for 2NN and SBN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6181) Fix the wrong property names in NFS user guide
[
https://issues.apache.org/jira/browse/HDFS-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962144#comment-13962144
]
Brandon Li commented on HDFS-6181:
--
I've committed the patch.
> Fix the wrong property names in NFS user guide
> --
>
> Key: HDFS-6181
> URL: https://issues.apache.org/jira/browse/HDFS-6181
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
>Priority: Trivial
> Attachments: HDFS-6181.002.patch, HDFS-6181.003.patch, HDFS-6181.patch
>
>
> A couple property names are wrong in the NFS user guide, and should be fixed
> as the following:
> {noformat}
>
> - dfs.nfsgateway.keytab.file
> + dfs.nfs.keytab.file
>/etc/hadoop/conf/nfsserver.keytab
>
>
> - dfs.nfsgateway.kerberos.principal
> + dfs.nfs.kerberos.principal
>nfsserver/_h...@your-realm.com
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6181) Fix the wrong property names in NFS user guide
[
https://issues.apache.org/jira/browse/HDFS-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6181:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix the wrong property names in NFS user guide
> --
>
> Key: HDFS-6181
> URL: https://issues.apache.org/jira/browse/HDFS-6181
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
>Priority: Trivial
> Attachments: HDFS-6181.002.patch, HDFS-6181.003.patch, HDFS-6181.patch
>
>
> A couple property names are wrong in the NFS user guide, and should be fixed
> as the following:
> {noformat}
>
> - dfs.nfsgateway.keytab.file
> + dfs.nfs.keytab.file
>/etc/hadoop/conf/nfsserver.keytab
>
>
> - dfs.nfsgateway.kerberos.principal
> + dfs.nfs.kerberos.principal
>nfsserver/_h...@your-realm.com
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6191) Disable quota checks when replaying edit log.
[ https://issues.apache.org/jira/browse/HDFS-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6191: - Resolution: Fixed Fix Version/s: 2.5.0 0.23.11 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2 and branch-0.23. > Disable quota checks when replaying edit log. > - > > Key: HDFS-6191 > URL: https://issues.apache.org/jira/browse/HDFS-6191 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0, 0.23.11, 2.5.0 > > Attachments: HDFS-6191.branch-0.23.patch, HDFS-6191.patch, > HDFS-6191.patch, HDFS-6191.patch > > > Since the serving NN does quota checks before logging edits, performing quota > checks is unnecessary while replaying edits. I propose disabling quota checks > for 2NN and SBN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4564) Webhdfs returns incorrect http response codes for denied operations
[ https://issues.apache.org/jira/browse/HDFS-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962097#comment-13962097 ] Haohui Mai commented on HDFS-4564: -- [~daryn], I wonder, do you have any plans on HADOOP-10453? Thanks! > Webhdfs returns incorrect http response codes for denied operations > --- > > Key: HDFS-4564 > URL: https://issues.apache.org/jira/browse/HDFS-4564 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Fix For: 2.4.0 > > Attachments: HDFS-4564.branch-23.patch, HDFS-4564.branch-23.patch, > HDFS-4564.branch-23.patch, HDFS-4564.patch, HDFS-4564.patch > > > Webhdfs is returning 401 (Unauthorized) instead of 403 (Forbidden) when it's > denying operations. Examples including rejecting invalid proxy user attempts > and renew/cancel with an invalid user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6191) Disable quota checks when replaying edit log.
[ https://issues.apache.org/jira/browse/HDFS-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962095#comment-13962095 ] Kihwal Lee commented on HDFS-6191: -- Thanks, Jing for the review. I will take a look at other verification functions and file a separate jira if necessary. > Disable quota checks when replaying edit log. > - > > Key: HDFS-6191 > URL: https://issues.apache.org/jira/browse/HDFS-6191 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6191.branch-0.23.patch, HDFS-6191.patch, > HDFS-6191.patch, HDFS-6191.patch > > > Since the serving NN does quota checks before logging edits, performing quota > checks is unnecessary while replaying edits. I propose disabling quota checks > for 2NN and SBN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962090#comment-13962090
]
Chris Nauroth commented on HDFS-6197:
-
Thanks for the reviews, Nicholas and Jing! I'll commit this after Jenkins runs
it.
bq. Just want to make sure I understand the issue correctly, so here multiple
attempts means multiple attempts of rollback?
This would be multiple attempts of beginning the rolling upgrade and rolling
back with no additional edits in between. Probably the most specific
explanation would be to look at the code of
{{TestRollingUpgrade#testRollback}}. This test failed on Windows during the
rename before my patch.
bq. I think we may also want to add some new mechanism to cleanup the trashed
edits for rollback of rolling upgrade (e.g., the first checkpoint after the
rollback maybe).
Yes, this seems to make sense too.
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
[ https://issues.apache.org/jira/browse/HDFS-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962088#comment-13962088 ] Mit Desai commented on HDFS-6195: - TestRMRestart is a different issue related to YARN-1906 > TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and > intermittently fails on trunk and branch2 > -- > > Key: HDFS-6195 > URL: https://issues.apache.org/jira/browse/HDFS-6195 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6195.patch > > > The test has 1 containers that it tries to cleanup. > The cleanup has a timeout of 2ms in which the test sometimes cannot do > the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6178) Decommission on standby NN couldn't finish
[
https://issues.apache.org/jira/browse/HDFS-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-6178:
--
Status: Patch Available (was: Open)
> Decommission on standby NN couldn't finish
> --
>
> Key: HDFS-6178
> URL: https://issues.apache.org/jira/browse/HDFS-6178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ming Ma
> Attachments: HDFS-6178.patch
>
>
> Currently decommissioning machines in HA-enabled cluster requires running
> refreshNodes in both active and standby nodes. Sometimes decommissioning
> won't finish from standby NN's point of view. Here is the diagnosis of why
> it could happen.
> Standby NN's blockManager manages blocks replication and block invalidation
> as if it is the active NN; even though DNs will ignore block commands coming
> from standby NN. When standby NN makes block operation decisions such as the
> target of block replication and the node to remove excess blocks from, the
> decision is independent of active NN. So active NN and standby NN could have
> different states. When we try to decommission nodes on standby nodes; such
> state inconsistency might prevent standby NN from making progress. Here is an
> example.
> Machine A
> Machine B
> Machine C
> Machine D
> Machine E
> Machine F
> Machine G
> Machine H
> 1. For a given block, both active and standby have 5 replicas on machine A,
> B, C, D, E. So both active and standby decide to pick excess nodes to
> invalidate.
> Active picked D and E as excess DNs. After the next block reports from D and
> E, active NN has 3 active replicas (A, B, C), 0 excess replica.
> {noformat}
> 2014-03-27 01:50:14,410 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (E:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> 2014-03-27 01:50:15,539 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (D:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> {noformat}
> Standby pick C, E as excess DNs. Given DNs ignore commands from standby,
> After the next block reports from C, D, E, standby has 2 active replicas (A,
> B), 1 excess replica (C).
> {noformat}
> 2014-03-27 01:51:49,543 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (E:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> 2014-03-27 01:51:49,894 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (C:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> {noformat}
> 2. Machine A decomm request was sent to standby. Standby only had one live
> replica and picked machine G, H as targets, but given standby commands was
> ignored by DNs, G, H remained in pending replication queue until they are
> timed out. At this point, you have one decommissioning replica (A), 1 active
> replica (B), one excess replica (C).
> {noformat}
> 2014-03-27 04:42:52,258 INFO BlockStateChange: BLOCK* ask A:50010 to
> replicate blk_-5207804474559026159_121186764 to datanode(s) G:50010 H:50010
> {noformat}
> 3. Machine A decomm request was sent to active NN. Active NN picked machine F
> as the target. It finished properly. So active NN had 3 active replicas (B,
> C, F), one decommissioned replica (A).
> {noformat}
> 2014-03-27 04:44:15,239 INFO BlockStateChange: BLOCK* ask 10.42.246.110:50010
> to replicate blk_-5207804474559026159_121186764 to datanode(s) F:50010
> 2014-03-27 04:44:16,083 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: F:50010 is added to blk_-5207804474559026159_121186764 size
> 7100065
> {noformat}
> 4. Standby NN picked up F as a new replica. Thus standby had one
> decommissioning replica (A), 2 active replicas (B, F), one excess replica
> (C). Standby NN kept trying to schedule replication work, but DNs ignored the
> commands.
> {noformat}
> 2014-03-27 04:44:16,084 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: F:50010 is added to blk_-5207804474559026159_121186764 size
> 7100065
> 2014-03-28 23:06:11,970 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Block:
> blk_-5207804474559026159_121186764, Expected Replicas: 3, live replicas: 2,
> corrupt replicas: 0, decommissioned replicas: 1, excess replicas: 1, Is Open
> File: false, Datanodes having this block: C:50010 B:50010 A:50010 F:50010 ,
> Current Datanode: A:50010, Is current datanode decommissioning: true
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962075#comment-13962075
]
Jing Zhao commented on HDFS-6197:
-
Thanks for the fix, [~cnauroth]! +1 for the patch.
bq. If no new edits arrive in between multiple attempts
Just want to make sure I understand the issue correctly, so here multiple
attempts means multiple attempts of rollback? I think we may also want to add
some new mechanism to cleanup the trashed edits for rollback of rolling upgrade
(e.g., the first checkpoint after the rollback maybe).
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6197:
--
Hadoop Flags: Reviewed
+1 patch looks good.
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6197:
Status: Patch Available (was: Open)
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
[
https://issues.apache.org/jira/browse/HDFS-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6197:
Attachment: HDFS-6197.1.patch
Here is the patch.
# Add a fallback code path in {{FileJournalManager#EditLogFile#renameSelf}} to
try deleting the destination if the initial rename fails. This is the same
strategy that we use for handling this situation in
{{FSImage#renameImageFileInDir}}.
# Shutdown a {{MiniDFSCluster}} that was getting leaked in
{{TestRollingUpgrade}}. This had the effect of disrupting a lot of other tests
on Windows, because the leaked cluster continued to hold file locks.
> Rolling upgrade rollback on Windows can fail attempting to rename edit log
> segment files to a destination that already exists.
> --
>
> Key: HDFS-6197
> URL: https://issues.apache.org/jira/browse/HDFS-6197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-6197.1.patch
>
>
> As part of a rollback from a rolling upgrade in progress, we discard edit log
> segments by renaming the file with suffix ".trash". If no new edits arrive
> in between multiple attempts, then the ".trash" file may still exist. On
> Windows, {{java.io.File#renameTo}} fails if the destination already exists
> though. This is visible right now as a failure in
> {{TestRollingUpgrade#testRollback}} when running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6197) Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists.
Chris Nauroth created HDFS-6197:
---
Summary: Rolling upgrade rollback on Windows can fail attempting
to rename edit log segment files to a destination that already exists.
Key: HDFS-6197
URL: https://issues.apache.org/jira/browse/HDFS-6197
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0, 2.4.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
As part of a rollback from a rolling upgrade in progress, we discard edit log
segments by renaming the file with suffix ".trash". If no new edits arrive in
between multiple attempts, then the ".trash" file may still exist. On Windows,
{{java.io.File#renameTo}} fails if the destination already exists though. This
is visible right now as a failure in {{TestRollingUpgrade#testRollback}} when
running on Windows.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6169) Move the address in WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962057#comment-13962057
]
Haohui Mai commented on HDFS-6169:
--
The overall control flow is more clear in the following way:
{code}
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws
Exception {
String op = getOp(...);
try {
String path = getPath(...);
handleOperation(op, e);
} finally {
e.getFuture().addListener(ChannelFutureListener.CLOSE);
}
}
{code}
Considering the following code in the patch:
{code}
...
+
+ private String trimPath(String path, MessageEvent e) {
+if (path.startsWith("/webhdfs/v1/")) {
+ return path.replaceFirst("/webhdfs/v1", "");
+} else {
+ // invalid path, close the channel
+ LOG.warn("Path: " + path + " should start with \"/webhdfs/v1/\"");
+ HttpResponse response = new DefaultHttpResponse(
+ HttpVersion.HTTP_1_1, HttpResponseStatus.NOT_FOUND);
+ LOG.info("404 method=GET target=" + path);
+ e.getChannel().write(response).addListener(ChannelFutureListener.CLOSE);
+ return "";
+}
+ }
{code}
1. How to differentiate the url "/webhdfs/v1?op=LISTSTATUS" and "/wrongurl"?
You probably need to return null in the invalid case.
2. It is a bad design to have code of reclaiming resources in different places.
I suggest following the control flow described on the top of this comment.
> Move the address in WebImageViewer
> --
>
> Key: HDFS-6169
> URL: https://issues.apache.org/jira/browse/HDFS-6169
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-6169.2.patch, HDFS-6169.3.patch, HDFS-6169.4.patch,
> HDFS-6169.5.patch, HDFS-6169.patch
>
>
> Move the endpoint of WebImageViewer from http://hostname:port/ to
> http://hostname:port/webhdfs/v1/ to support {{hdfs dfs -ls}} to
> WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6191) Disable quota checks when replaying edit log.
[ https://issues.apache.org/jira/browse/HDFS-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962049#comment-13962049 ] Jing Zhao commented on HDFS-6191: - The patch looks good to me. +1 for the patch. In FSDirectory we have several other verification functions, such as verifyFsLimitsForRename. Looks like we can also disable them when replaying edit log (in a separate jira)? > Disable quota checks when replaying edit log. > - > > Key: HDFS-6191 > URL: https://issues.apache.org/jira/browse/HDFS-6191 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6191.branch-0.23.patch, HDFS-6191.patch, > HDFS-6191.patch, HDFS-6191.patch > > > Since the serving NN does quota checks before logging edits, performing quota > checks is unnecessary while replaying edits. I propose disabling quota checks > for 2NN and SBN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[
https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962042#comment-13962042
]
Haohui Mai commented on HDFS-6143:
--
{code}
+} else if (e instanceof UnresolvedLinkException) {
+ s = Response.Status.GONE;
{code}
Should this be 404 instead of 410? In UNIX, opening a unresolved symlink return
{{ENOENT}}.
> WebHdfsFileSystem open should throw FileNotFoundException for non-existing
> paths
>
>
> Key: HDFS-6143
> URL: https://issues.apache.org/jira/browse/HDFS-6143
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: HDFS-6143-branch-2.4.0.v01.patch,
> HDFS-6143-trunk-after-HDFS-5570.v01.patch, HDFS-6143.v01.patch,
> HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch,
> HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch
>
>
> WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles
> non-existing paths.
> - 'open', does not really open anything, i.e., it does not contact the
> server, and therefore cannot discover FileNotFound, it's deferred until next
> read. It's counterintuitive and not how local FS or HDFS work. In POSIX you
> get ENOENT on open.
> [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java]
> is an example of the code that's broken because of this.
> - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST
> instead of SC_NOT_FOUND for non-exitsing paths
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6178) Decommission on standby NN couldn't finish
[
https://issues.apache.org/jira/browse/HDFS-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-6178:
--
Attachment: HDFS-6178.patch
Here is the fix. Couple notes:
1. It seems the code has already prevent SBN from updating excess blocks for
common case like addStoredBlock. The case that is missing is node recommission
scenario. So the fix is at recommission layer instead of fixing it inside
processOverReplicatedBlock.
2. The fix in ReplicationMonitor to not do replication work isn't necessary to
address this specific issue.
3. The fix has been tested manually in a real cluster. The added test case
reproduced the excess replica scenario.
> Decommission on standby NN couldn't finish
> --
>
> Key: HDFS-6178
> URL: https://issues.apache.org/jira/browse/HDFS-6178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ming Ma
> Attachments: HDFS-6178.patch
>
>
> Currently decommissioning machines in HA-enabled cluster requires running
> refreshNodes in both active and standby nodes. Sometimes decommissioning
> won't finish from standby NN's point of view. Here is the diagnosis of why
> it could happen.
> Standby NN's blockManager manages blocks replication and block invalidation
> as if it is the active NN; even though DNs will ignore block commands coming
> from standby NN. When standby NN makes block operation decisions such as the
> target of block replication and the node to remove excess blocks from, the
> decision is independent of active NN. So active NN and standby NN could have
> different states. When we try to decommission nodes on standby nodes; such
> state inconsistency might prevent standby NN from making progress. Here is an
> example.
> Machine A
> Machine B
> Machine C
> Machine D
> Machine E
> Machine F
> Machine G
> Machine H
> 1. For a given block, both active and standby have 5 replicas on machine A,
> B, C, D, E. So both active and standby decide to pick excess nodes to
> invalidate.
> Active picked D and E as excess DNs. After the next block reports from D and
> E, active NN has 3 active replicas (A, B, C), 0 excess replica.
> {noformat}
> 2014-03-27 01:50:14,410 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (E:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> 2014-03-27 01:50:15,539 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (D:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> {noformat}
> Standby pick C, E as excess DNs. Given DNs ignore commands from standby,
> After the next block reports from C, D, E, standby has 2 active replicas (A,
> B), 1 excess replica (C).
> {noformat}
> 2014-03-27 01:51:49,543 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (E:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> 2014-03-27 01:51:49,894 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
> (C:50010, blk_-5207804474559026159_121186764) is added to invalidated blocks
> set
> {noformat}
> 2. Machine A decomm request was sent to standby. Standby only had one live
> replica and picked machine G, H as targets, but given standby commands was
> ignored by DNs, G, H remained in pending replication queue until they are
> timed out. At this point, you have one decommissioning replica (A), 1 active
> replica (B), one excess replica (C).
> {noformat}
> 2014-03-27 04:42:52,258 INFO BlockStateChange: BLOCK* ask A:50010 to
> replicate blk_-5207804474559026159_121186764 to datanode(s) G:50010 H:50010
> {noformat}
> 3. Machine A decomm request was sent to active NN. Active NN picked machine F
> as the target. It finished properly. So active NN had 3 active replicas (B,
> C, F), one decommissioned replica (A).
> {noformat}
> 2014-03-27 04:44:15,239 INFO BlockStateChange: BLOCK* ask 10.42.246.110:50010
> to replicate blk_-5207804474559026159_121186764 to datanode(s) F:50010
> 2014-03-27 04:44:16,083 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: F:50010 is added to blk_-5207804474559026159_121186764 size
> 7100065
> {noformat}
> 4. Standby NN picked up F as a new replica. Thus standby had one
> decommissioning replica (A), 2 active replicas (B, F), one excess replica
> (C). Standby NN kept trying to schedule replication work, but DNs ignored the
> commands.
> {noformat}
> 2014-03-27 04:44:16,084 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: F:50010 is added to blk_-5207804474559026159_121186764 size
> 7100065
> 2014-03-28 23:06:11,970 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Block:
> blk_-5207804474559026159_121186764, Expected Replicas: 3, live replicas: 2,
> corrupt replicas: 0, decommissioned replicas: 1, excess replicas: 1, Is Open
> File: false, Datano
[jira] [Commented] (HDFS-3228) Use fadvise in local read path
[
https://issues.apache.org/jira/browse/HDFS-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962029#comment-13962029
]
Colin Patrick McCabe commented on HDFS-3228:
It's worth noting that BlockReaderLocal already implements a form of readahead,
in that it reads and checksums {{dfs.client.read.shortcircuit.buffer.size}}
bytes at a time. Typically this is 1 MB. (Since HDFS-5634, we will also read
fewer bytes than {{dfs.client.read.shortcircuit.buffer.size}} if readahead is
turned down or turned off.)
Of course, this 1 MB read is done in the foreground (you don't get back any
bytes until it finishes). An fadvise-based system would operate a little bit
differently.
> Use fadvise in local read path
> --
>
> Key: HDFS-3228
> URL: https://issues.apache.org/jira/browse/HDFS-3228
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, performance
>Reporter: Henry Robinson
>Assignee: Henry Robinson
>
> The read path through BlockReaderLocal does not take advantage of readahead
> or drop-behind in the way that BlockSender does. We could arguably stand to
> gain even more from hinting about read patterns to the kernel here, so we
> should add the same mechanisms to BlockReaderLocal.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
[
https://issues.apache.org/jira/browse/HDFS-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962028#comment-13962028
]
Hadoop QA commented on HDFS-6195:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639009/HDFS-6195.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6602//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6602//console
This message is automatically generated.
> TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and
> intermittently fails on trunk and branch2
> --
>
> Key: HDFS-6195
> URL: https://issues.apache.org/jira/browse/HDFS-6195
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6195.patch
>
>
> The test has 1 containers that it tries to cleanup.
> The cleanup has a timeout of 2ms in which the test sometimes cannot do
> the cleanup completely and gives out an Assertion Failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6191) Disable quota checks when replaying edit log.
[
https://issues.apache.org/jira/browse/HDFS-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961999#comment-13961999
]
Hadoop QA commented on HDFS-6191:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638978/HDFS-6191.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6601//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6601//console
This message is automatically generated.
> Disable quota checks when replaying edit log.
> -
>
> Key: HDFS-6191
> URL: https://issues.apache.org/jira/browse/HDFS-6191
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-6191.branch-0.23.patch, HDFS-6191.patch,
> HDFS-6191.patch, HDFS-6191.patch
>
>
> Since the serving NN does quota checks before logging edits, performing quota
> checks is unnecessary while replaying edits. I propose disabling quota checks
> for 2NN and SBN.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
[
https://issues.apache.org/jira/browse/HDFS-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961984#comment-13961984
]
Mit Desai commented on HDFS-6195:
-
While cleaning up the containers,
{code}
while (cleanedSize < allocatedSize && waitCount++ < 200) {
Thread.sleep(100);
resp = nm.nodeHeartbeat(true);
cleaned = resp.getContainersToCleanup();
cleanedSize += cleaned.size();
}
{code}
The test sometimes cannot do the complete cleanup and some of the 1
containers cannot be cleaned up. Resulting an assertion error at
{{Assert.assertEquals(allocatedSize, cleanedSize);}}.
This test has been failing in our nightly builds since couple of days. I was
able to reproduce this consistently on eclipse but not using maven. I think
this is an environment issue so cannot be reproduced everywhere.
As a fix, I have increased the thread sleep time in the while loop. Which will
give some extra time for the container cleanup. And as there is also a check in
the while loop for the allocated size and cleaned size, the test will not
always take up all cycles in the loop.
> TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and
> intermittently fails on trunk and branch2
> --
>
> Key: HDFS-6195
> URL: https://issues.apache.org/jira/browse/HDFS-6195
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6195.patch
>
>
> The test has 1 containers that it tries to cleanup.
> The cleanup has a timeout of 2ms in which the test sometimes cannot do
> the cleanup completely and gives out an Assertion Failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
[ https://issues.apache.org/jira/browse/HDFS-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-6195: Fix Version/s: 2.5.0 3.0.0 Status: Patch Available (was: Open) > TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and > intermittently fails on trunk and branch2 > -- > > Key: HDFS-6195 > URL: https://issues.apache.org/jira/browse/HDFS-6195 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6195.patch > > > The test has 1 containers that it tries to cleanup. > The cleanup has a timeout of 2ms in which the test sometimes cannot do > the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
[ https://issues.apache.org/jira/browse/HDFS-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-6195: Attachment: HDFS-6195.patch Attaching the patch for trunk and branch2 > TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and > intermittently fails on trunk and branch2 > -- > > Key: HDFS-6195 > URL: https://issues.apache.org/jira/browse/HDFS-6195 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-6195.patch > > > The test has 1 containers that it tries to cleanup. > The cleanup has a timeout of 2ms in which the test sometimes cannot do > the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961906#comment-13961906 ] Arun C Murthy commented on HDFS-6143: - One concern: what is the impact on compatibility with 2.2 etc? > WebHdfsFileSystem open should throw FileNotFoundException for non-existing > paths > > > Key: HDFS-6143 > URL: https://issues.apache.org/jira/browse/HDFS-6143 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov >Priority: Blocker > Attachments: HDFS-6143-branch-2.4.0.v01.patch, > HDFS-6143-trunk-after-HDFS-5570.v01.patch, HDFS-6143.v01.patch, > HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch, > HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6196) WEBHDFS: Number of bytes read value is incorrect on Counters page
Rushabh S Shah created HDFS-6196: Summary: WEBHDFS: Number of bytes read value is incorrect on Counters page Key: HDFS-6196 URL: https://issues.apache.org/jira/browse/HDFS-6196 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.9 Reporter: Rushabh S Shah For a sample job (copying data from WebHDFS to HDFS), the value of field "WEBHDFS: Number of bytes read" is not updated (i.e. showing 0) but the value of field "HDFS: Number of bytes written" is correctly shown as the number of bytes written into HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6196) WEBHDFS: Number of bytes read value is incorrect on Counters page
[ https://issues.apache.org/jira/browse/HDFS-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah reassigned HDFS-6196: Assignee: Rushabh S Shah > WEBHDFS: Number of bytes read value is incorrect on Counters page > -- > > Key: HDFS-6196 > URL: https://issues.apache.org/jira/browse/HDFS-6196 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > > For a sample job (copying data from WebHDFS to HDFS), the value of field > "WEBHDFS: Number of bytes read" is not updated (i.e. showing 0) but the > value of field "HDFS: Number of bytes written" is correctly shown as the > number of bytes written into HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
[ https://issues.apache.org/jira/browse/HDFS-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961887#comment-13961887 ] Mit Desai commented on HDFS-6195: - analyzing the cause. Will post the analysis/fix soon > TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and > intermittently fails on trunk and branch2 > -- > > Key: HDFS-6195 > URL: https://issues.apache.org/jira/browse/HDFS-6195 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.5.0 >Reporter: Mit Desai >Assignee: Mit Desai > > The test has 1 containers that it tries to cleanup. > The cleanup has a timeout of 2ms in which the test sometimes cannot do > the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6191) Disable quota checks when replaying edit log.
[ https://issues.apache.org/jira/browse/HDFS-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6191: - Attachment: HDFS-6191.patch The new patch fixes the javac warnings in the test case. > Disable quota checks when replaying edit log. > - > > Key: HDFS-6191 > URL: https://issues.apache.org/jira/browse/HDFS-6191 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6191.branch-0.23.patch, HDFS-6191.patch, > HDFS-6191.patch, HDFS-6191.patch > > > Since the serving NN does quota checks before logging edits, performing quota > checks is unnecessary while replaying edits. I propose disabling quota checks > for 2NN and SBN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6195) TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2
Mit Desai created HDFS-6195: --- Summary: TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails on trunk and branch2 Key: HDFS-6195 URL: https://issues.apache.org/jira/browse/HDFS-6195 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.5.0 Reporter: Mit Desai Assignee: Mit Desai The test has 1 containers that it tries to cleanup. The cleanup has a timeout of 2ms in which the test sometimes cannot do the cleanup completely and gives out an Assertion Failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5215) dfs.datanode.du.reserved is not taking effect as it's not considered while getting the available space
[
https://issues.apache.org/jira/browse/HDFS-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961795#comment-13961795
]
Brahma Reddy Battula commented on HDFS-5215:
Hi [~fahlke],
Thanks for taking look into this issue..
{quote}
Have you considered that the underlying filesystem reserved space (ext3, ext4
-> tune2fs) can also affect this behavior?
{quote}
Yes, I checked in ext3 file system only..
{quote}
I'm not sure if this is a problem in hadoop-3.0.x anymore.
{quote}
As I knew , it is not fixed..problem will be present in hadoop-3.0.x . .Please
let me know your thoughts on this...
> dfs.datanode.du.reserved is not taking effect as it's not considered while
> getting the available space
> --
>
> Key: HDFS-5215
> URL: https://issues.apache.org/jira/browse/HDFS-5215
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0
>
> Attachments: HDFS-5215.patch
>
>
> {code}public long getAvailable() throws IOException {
> long remaining = getCapacity()-getDfsUsed();
> long available = usage.getAvailable();
> if (remaining > available) {
> remaining = available;
> }
> return (remaining > 0) ? remaining : 0;
> }
> {code}
> Here we are not considering the reserved space while getting the Available
> Space.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6143) WebHdfsFileSystem open should throw FileNotFoundException for non-existing paths
[
https://issues.apache.org/jira/browse/HDFS-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961714#comment-13961714
]
Hadoop QA commented on HDFS-6143:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638943/HDFS-6143-branch-2.4.0.v01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6600//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6600//console
This message is automatically generated.
> WebHdfsFileSystem open should throw FileNotFoundException for non-existing
> paths
>
>
> Key: HDFS-6143
> URL: https://issues.apache.org/jira/browse/HDFS-6143
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Blocker
> Attachments: HDFS-6143-branch-2.4.0.v01.patch,
> HDFS-6143-trunk-after-HDFS-5570.v01.patch, HDFS-6143.v01.patch,
> HDFS-6143.v02.patch, HDFS-6143.v03.patch, HDFS-6143.v04.patch,
> HDFS-6143.v04.patch, HDFS-6143.v05.patch, HDFS-6143.v06.patch
>
>
> WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles
> non-existing paths.
> - 'open', does not really open anything, i.e., it does not contact the
> server, and therefore cannot discover FileNotFound, it's deferred until next
> read. It's counterintuitive and not how local FS or HDFS work. In POSIX you
> get ENOENT on open.
> [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java]
> is an example of the code that's broken because of this.
> - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST
> instead of SC_NOT_FOUND for non-exitsing paths
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6169) Move the address in WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961686#comment-13961686
]
Hadoop QA commented on HDFS-6169:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638944/HDFS-6169.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6599//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6599//console
This message is automatically generated.
> Move the address in WebImageViewer
> --
>
> Key: HDFS-6169
> URL: https://issues.apache.org/jira/browse/HDFS-6169
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-6169.2.patch, HDFS-6169.3.patch, HDFS-6169.4.patch,
> HDFS-6169.5.patch, HDFS-6169.patch
>
>
> Move the endpoint of WebImageViewer from http://hostname:port/ to
> http://hostname:port/webhdfs/v1/ to support {{hdfs dfs -ls}} to
> WebImageViewer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6194) Create new tests for {{ByteRangeInputStream}}
[
https://issues.apache.org/jira/browse/HDFS-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned HDFS-6194:
---
Assignee: Akira AJISAKA
> Create new tests for {{ByteRangeInputStream}}
> -
>
> Key: HDFS-6194
> URL: https://issues.apache.org/jira/browse/HDFS-6194
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Akira AJISAKA
>
> HDFS-5570 removes old tests for {{ByteRangeInputStream}}, because the tests
> only are tightly coupled with hftp / hsftp. New tests need to be written
> because the same class is also used by {{WebHdfsFileSystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
