[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: Zipkin Trace a06e941b0172ec73.png Here's an example of what I have currently. I'm still trying to balance what should be instrumented. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, Zipkin Trace a06e941b0172ec73.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-2.patch Instrumented Sender and Receiver (Though some of those code paths are not hit as well). better read side instrumentation. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, Zipkin Trace a06e941b0172ec73.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[
https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782665#comment-13782665
]
Hadoop QA commented on HDFS-5255:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606040/HDFS-5255.05.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHftpDelegationToken
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5070//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5070//console
This message is automatically generated.
Distcp job fails with hsftp when https is enabled
-
Key: HDFS-5255
URL: https://issues.apache.org/jira/browse/HDFS-5255
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Yesha Vora
Assignee: Arpit Agarwal
Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch,
HDFS-5255.04.patch, HDFS-5255.05.patch
Run Distcp job using hsftp when ssl is enabled. The job fails with
java.net.SocketException: Unexpected end of file from server Error
Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5
All the tasks fails with below error.
13/09/23 15:52:38 INFO mapreduce.Job: Task Id :
attempt_1379976241507_0004_m_00_0, Status : FAILED
Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 --
hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more
Caused by:
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
Caused by: java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
at
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782731#comment-13782731
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606051/HDFS-5274-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures
org.apache.hadoop.hdfs.qjournal.TestNNWithQJM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5072//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5072//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
Zipkin Trace a06e941b0172ec73.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5282) Support built in resolution of /~ to user's home directory: /user/{user.name}
[
https://issues.apache.org/jira/browse/HDFS-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782797#comment-13782797
]
Steve Loughran commented on HDFS-5282:
--
# What breaks with this: are people using ~ as a filename/dir name today?
# what would be the policy w.r.t other filesystems?
/~ isnt normally valid on a unix fs:
{code}
$ ls /~
ls: /~: No such file or directory
{code}
It'd be needed in hdfs, so that you could have uris like hdfs://~/data, or it
is only ever allowed as a relative path against something else, which isn't so
easy to use. But, as that relative - absolute resolution is shared across
filesystems, it is where we could put it and have it feed into everything
(supporting \~ as an escape?)
Support built in resolution of /~ to user's home directory: /user/{user.name}
-
Key: HDFS-5282
URL: https://issues.apache.org/jira/browse/HDFS-5282
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Kevin Minder
In many cases it would be very convenient for HDFS (and WebHDFS in
particular) to support the Unix notion of /~ representing the user's home
directory. This would allow for some scripts reusable because they would not
contain hard coded home directories.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
Vinay created HDFS-5283: --- Summary: NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 2.1.1-beta, 3.0.0 Reporter: Vinay Assignee: Vinay Priority: Blocker This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782813#comment-13782813 ] Hudson commented on HDFS-5230: -- FAILURE: Integrated in Hadoop-Yarn-trunk #349 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/349/]) HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527726) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCallCache.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServerHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServerHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/XDR.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcCallCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/RpcProgramMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Introduce RpcInfo to decouple XDR classes from the RPC API -- Key: HDFS-5230 URL: https://issues.apache.org/jira/browse/HDFS-5230 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.1.2-beta Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch, HDFS-5230.004.patch, HDFS-5230.005.patch, HDFS-5230.006.patch, HDFS-5230.007.patch, HDFS-5230.008.patch, HDFS-5230.009.patch The XDR class is one fundamental aspect in the current implementation of NFS server. While the client might potentially have a higher level APIs, it also requires redundant copying since the upstream clients have insufficient information. This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly to the client in order to open the opportunity for avoid copying. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4517) Cover class RemoteBlockReader with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782809#comment-13782809 ] Hudson commented on HDFS-4517: -- FAILURE: Integrated in Hadoop-Yarn-trunk #349 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/349/]) HDFS-4517. Cover class RemoteBlockReader with unit tests. Contributed by Vadim Bondarev and Dennis Y. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527807) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java Cover class RemoteBlockReader with unit tests - Key: HDFS-4517 URL: https://issues.apache.org/jira/browse/HDFS-4517 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Dennis Y Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-4517-branch-0.23-a.patch, HADOOP-4517-branch-2-a.patch, HADOOP-4517-branch-2-b.patch, HADOOP-4517-branch-2c.patch, HADOOP-4517-trunk-a.patch, HADOOP-4517-trunk-b.patch, HADOOP-4517-trunk-c.patch, HDFS-4517-branch-2--N2.patch, HDFS-4517-branch-2--N3.patch, HDFS-4517-branch-2--N4.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5283: Attachment: HDFS-5283.patch Attaching patch contains 2 tests along with initial fix. 1. When only some of the open files are deleted after taking snapshot. -- Attached changes fixes this issue. 2. When the directory containing the open files is deleted. Did not get the exact solution to fix this issue. In this case BlockCollection is of INodeFileUnderConstruction, not INodeFileUnderConstructionWithSnapShot. Please somebody suggest the fix. NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.1-beta Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5283: Status: Patch Available (was: Open) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 2.1.1-beta, 3.0.0 Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4517) Cover class RemoteBlockReader with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782915#comment-13782915 ] Hudson commented on HDFS-4517: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1539 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1539/]) HDFS-4517. Cover class RemoteBlockReader with unit tests. Contributed by Vadim Bondarev and Dennis Y. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527807) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java Cover class RemoteBlockReader with unit tests - Key: HDFS-4517 URL: https://issues.apache.org/jira/browse/HDFS-4517 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Dennis Y Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-4517-branch-0.23-a.patch, HADOOP-4517-branch-2-a.patch, HADOOP-4517-branch-2-b.patch, HADOOP-4517-branch-2c.patch, HADOOP-4517-trunk-a.patch, HADOOP-4517-trunk-b.patch, HADOOP-4517-trunk-c.patch, HDFS-4517-branch-2--N2.patch, HDFS-4517-branch-2--N3.patch, HDFS-4517-branch-2--N4.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782919#comment-13782919 ] Hudson commented on HDFS-5230: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1539 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1539/]) HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527726) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCallCache.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServerHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServerHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/XDR.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcCallCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/RpcProgramMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Introduce RpcInfo to decouple XDR classes from the RPC API -- Key: HDFS-5230 URL: https://issues.apache.org/jira/browse/HDFS-5230 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.1.2-beta Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch, HDFS-5230.004.patch, HDFS-5230.005.patch, HDFS-5230.006.patch, HDFS-5230.007.patch, HDFS-5230.008.patch, HDFS-5230.009.patch The XDR class is one fundamental aspect in the current implementation of NFS server. While the client might potentially have a higher level APIs, it also requires redundant copying since the upstream clients have insufficient information. This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly to the client in order to open the opportunity for avoid copying. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5283: Attachment: HDFS-5283.patch This patch seems to work for me. Please review and validate the fix. I am not much aware of snapshot related code. Thanks NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.1-beta Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4512) Cover package org.apache.hadoop.hdfs.server.common with tests
[ https://issues.apache.org/jira/browse/HDFS-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4512: - Assignee: Vadim Bondarev Cover package org.apache.hadoop.hdfs.server.common with tests - Key: HDFS-4512 URL: https://issues.apache.org/jira/browse/HDFS-4512 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Vadim Bondarev Attachments: HADOOP-4512-branch-0.23-a.patch, HADOOP-4512-branch-2-a.patch, HADOOP-4512-trunk-a.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4512) Cover package org.apache.hadoop.hdfs.server.common with tests
[ https://issues.apache.org/jira/browse/HDFS-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4512: - Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. Thanks for the patch, Vadim. Cover package org.apache.hadoop.hdfs.server.common with tests - Key: HDFS-4512 URL: https://issues.apache.org/jira/browse/HDFS-4512 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Vadim Bondarev Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-4512-branch-0.23-a.patch, HADOOP-4512-branch-2-a.patch, HADOOP-4512-trunk-a.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4512) Cover package org.apache.hadoop.hdfs.server.common with tests
[ https://issues.apache.org/jira/browse/HDFS-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782974#comment-13782974 ] Hudson commented on HDFS-4512: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4504 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4504/]) HDFS-4512. Cover package org.apache.hadoop.hdfs.server.common with tests. Contributed by Vadim Bondarev. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528097) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestJspHelper.java Cover package org.apache.hadoop.hdfs.server.common with tests - Key: HDFS-4512 URL: https://issues.apache.org/jira/browse/HDFS-4512 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Vadim Bondarev Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-4512-branch-0.23-a.patch, HADOOP-4512-branch-2-a.patch, HADOOP-4512-trunk-a.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5230) Introduce RpcInfo to decouple XDR classes from the RPC API
[ https://issues.apache.org/jira/browse/HDFS-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782971#comment-13782971 ] Hudson commented on HDFS-5230: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1565 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1565/]) HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527726) * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Base.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcCallCache.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcResponse.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleTcpServerHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/SimpleUdpServerHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/XDR.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/security/Verifier.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/portmap/RpcProgramPortmap.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestRpcCallCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/RpcProgramMountd.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Introduce RpcInfo to decouple XDR classes from the RPC API -- Key: HDFS-5230 URL: https://issues.apache.org/jira/browse/HDFS-5230 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.1.2-beta Attachments: HDFS-5230.002.patch, HDFS-5230.003.patch, HDFS-5230.004.patch, HDFS-5230.005.patch, HDFS-5230.006.patch, HDFS-5230.007.patch, HDFS-5230.008.patch, HDFS-5230.009.patch The XDR class is one fundamental aspect in the current implementation of NFS server. While the client might potentially have a higher level APIs, it also requires redundant copying since the upstream clients have insufficient information. This JIRA introduces a new class, RpcInfo, which (1) decouples XDR from the APIs, turning it into a utility class, and (2) exposes ChannelBuffer directly to the client in order to open the opportunity for avoid copying. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4517) Cover class RemoteBlockReader with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782967#comment-13782967 ] Hudson commented on HDFS-4517: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1565 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1565/]) HDFS-4517. Cover class RemoteBlockReader with unit tests. Contributed by Vadim Bondarev and Dennis Y. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1527807) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java Cover class RemoteBlockReader with unit tests - Key: HDFS-4517 URL: https://issues.apache.org/jira/browse/HDFS-4517 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Dennis Y Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-4517-branch-0.23-a.patch, HADOOP-4517-branch-2-a.patch, HADOOP-4517-branch-2-b.patch, HADOOP-4517-branch-2c.patch, HADOOP-4517-trunk-a.patch, HADOOP-4517-trunk-b.patch, HADOOP-4517-trunk-c.patch, HDFS-4517-branch-2--N2.patch, HDFS-4517-branch-2--N3.patch, HDFS-4517-branch-2--N4.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[
https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782995#comment-13782995
]
Hadoop QA commented on HDFS-5283:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606091/HDFS-5283.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5073//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5073//console
This message is automatically generated.
NN not coming out of startup safemode due to under construction blocks only
inside snapshots also counted in safemode threshhold
Key: HDFS-5283
URL: https://issues.apache.org/jira/browse/HDFS-5283
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Vinay
Assignee: Vinay
Priority: Blocker
Attachments: HDFS-5283.patch, HDFS-5283.patch
This is observed in one of our env:
1. A MR Job was running which has created some temporary files and was
writing to them.
2. Snapshot was taken
3. And Job was killed and temporary files were deleted.
4. Namenode restarted.
5. After restart Namenode was in safemode waiting for blocks
Analysis
-
1. Since the snapshot taken also includes the temporary files which were
open, and later original files are deleted.
2. UnderConstruction blocks count was taken from leases. not considered the
UC blocks only inside snapshots
3. So safemode threshold count was more and NN did not come out of safemode
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5284) Flatten INode hierarchy
Tsz Wo (Nicholas), SZE created HDFS-5284: Summary: Flatten INode hierarchy Key: HDFS-5284 URL: https://issues.apache.org/jira/browse/HDFS-5284 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Currently, we have a complicated inode hierarchy for representing different states of a file or a directory. For example, when a file is being created, it is represented by an INodeFileUnderConstruction. When a file is being closed, the inode is replaced by an INodeFile. If it is reopened for append, the inode is replaced again by an INodeFileUnderConstruction. This JIRA is to flatten the inode hierarchy. We may also improve the performance by eliminating the inode replacement in runtime. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783009#comment-13783009 ] Binglin Chang commented on HDFS-5276: - bq. Why not keep thread-local read statistics and sum them up periodically? That seems better than disabling this entirely. ThreadLocal variables also has performance penalties in java, although I have not test it, see http://stackoverflow.com/questions/609826/performance-of-threadlocal-variable. Use them frequently in inner loop may also cause performance penalty Since atomic variable or ThreadLocal both have performance impact(big or small), and most applications use hdfs client donot use statistics at all, I think at least we can give them a option to disable it. We can also do optimizations, they are not conflict. Hadoop fs client is too heavyweight now, with to much threads and states. Imagine a NM/TaskTracker with 40+ of tasks, each with several hdfs clients which has multiple threads, we may get thousand threads just for hdfs read/write, it will cause a lot of context switch expenses. FileSystem.Statistics got performance issue on multi-thread read/write. --- Key: HDFS-5276 URL: https://issues.apache.org/jira/browse/HDFS-5276 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Chengxiang Li Attachments: DisableFSReadWriteBytesStat.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG FileSystem.Statistics is a singleton variable for each FS scheme, each read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does not perform well in multi-threads(let's say more than 30 threads). so it may cause serious performance issue. during our spark test profile, 32 threads read data from HDFS, about 70% cpu time is spent on FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5286) Flatten INodeDirectory hierarchy
Tsz Wo (Nicholas), SZE created HDFS-5286: Summary: Flatten INodeDirectory hierarchy Key: HDFS-5286 URL: https://issues.apache.org/jira/browse/HDFS-5286 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Similar to the case of INodeFile (HFDS-5285), we should also flatten the INodeDirectory hierarchy. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5285) Flatten INodeFile hierarchy
Tsz Wo (Nicholas), SZE created HDFS-5285: Summary: Flatten INodeFile hierarchy Key: HDFS-5285 URL: https://issues.apache.org/jira/browse/HDFS-5285 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE For files, there are INodeFile, INodeFileUnderConstruction, INodeFileWithSnapshot and INodeFileUnderConstructionWithSnapshot for representing whether a file is under construction or whether it is in some snapshot. The following are two major problems of the current approach: - Java class does not support multiple inheritances so that INodeFileUnderConstructionWithSnapshot cannot extend both INodeFileUnderConstruction and INodeFileWithSnapshot. - The number of classes is exponential to the number of features. Currently, there are only two features, UnderConstruction and WithSnapshot. The number of classes is 2^2 = 4. It is hard to add one more feature since the number of classes will become 2^3 = 8. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-3983) Hftp should support both SPNEGO and KSSL
[
https://issues.apache.org/jira/browse/HDFS-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley resolved HDFS-3983.
-
Resolution: Won't Fix
Target Version/s: (was: )
KSSL is deprecated and should never be used for secure deployments.
Hftp should support both SPNEGO and KSSL
Key: HDFS-3983
URL: https://issues.apache.org/jira/browse/HDFS-3983
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Blocker
Attachments: hdfs-3983.txt, hdfs-3983.txt
Hftp currently doesn't work against a secure cluster unless you configure
{{dfs.https.port}} to be the http port, otherwise the client can't fetch
tokens:
{noformat}
$ hadoop fs -ls hftp://c1225.hal.cloudera.com:50070/
12/09/26 18:02:00 INFO fs.FileSystem: Couldn't get a delegation token from
http://c1225.hal.cloudera.com:50470 using http.
ls: Security enabled but user not authenticated by filter
{noformat}
This is due to Hftp still using the https port. Post HDFS-2617 it should use
the regular http port. Hsftp should still use the secure port, however now
that we have HADOOP-8581 it's worth considering removing Hsftp entirely. I'll
start a separate thread about that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Closed] (HDFS-3983) Hftp should support both SPNEGO and KSSL
[
https://issues.apache.org/jira/browse/HDFS-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley closed HDFS-3983.
---
Assignee: (was: Eli Collins)
Hftp should support both SPNEGO and KSSL
Key: HDFS-3983
URL: https://issues.apache.org/jira/browse/HDFS-3983
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Priority: Blocker
Attachments: hdfs-3983.txt, hdfs-3983.txt
Hftp currently doesn't work against a secure cluster unless you configure
{{dfs.https.port}} to be the http port, otherwise the client can't fetch
tokens:
{noformat}
$ hadoop fs -ls hftp://c1225.hal.cloudera.com:50070/
12/09/26 18:02:00 INFO fs.FileSystem: Couldn't get a delegation token from
http://c1225.hal.cloudera.com:50470 using http.
ls: Security enabled but user not authenticated by filter
{noformat}
This is due to Hftp still using the https port. Post HDFS-2617 it should use
the regular http port. Hsftp should still use the secure port, however now
that we have HADOOP-8581 it's worth considering removing Hsftp entirely. I'll
start a separate thread about that.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Closed] (HDFS-3699) HftpFileSystem should try both KSSL and SPNEGO when authentication is required
[ https://issues.apache.org/jira/browse/HDFS-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley closed HDFS-3699. --- HftpFileSystem should try both KSSL and SPNEGO when authentication is required -- Key: HDFS-3699 URL: https://issues.apache.org/jira/browse/HDFS-3699 Project: Hadoop HDFS Issue Type: Sub-task Reporter: eric baldeschwieler See discussion in HDFS-2617 (Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution). To handle the transition from Hadoop1.0 systems running KSSL authentication to Hadoop systems running SPNEGO, it would be good to fix the client in both 1 and 2 to try SPNEGO and then fall back to try KSSL. This will allow organizations that are running a lot of Hadoop 1.0 to gradually transition over, without needing to convert all clusters at the same time. They would first need to update their 1.0 HFTP clients (and 2.0/0.23 if they are already running those) and then they could copy data between clusters without needing to move all clusters to SPNEGO in a big bang. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-3699) HftpFileSystem should try both KSSL and SPNEGO when authentication is required
[ https://issues.apache.org/jira/browse/HDFS-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HDFS-3699. - Resolution: Won't Fix Using KSSL is strongly deprecated and should be avoided in secure clusters. HftpFileSystem should try both KSSL and SPNEGO when authentication is required -- Key: HDFS-3699 URL: https://issues.apache.org/jira/browse/HDFS-3699 Project: Hadoop HDFS Issue Type: Sub-task Reporter: eric baldeschwieler See discussion in HDFS-2617 (Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution). To handle the transition from Hadoop1.0 systems running KSSL authentication to Hadoop systems running SPNEGO, it would be good to fix the client in both 1 and 2 to try SPNEGO and then fall back to try KSSL. This will allow organizations that are running a lot of Hadoop 1.0 to gradually transition over, without needing to convert all clusters at the same time. They would first need to update their 1.0 HFTP clients (and 2.0/0.23 if they are already running those) and then they could copy data between clusters without needing to move all clusters to SPNEGO in a big bang. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[
https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783061#comment-13783061
]
Hadoop QA commented on HDFS-5283:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606098/HDFS-5283.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5074//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5074//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5074//console
This message is automatically generated.
NN not coming out of startup safemode due to under construction blocks only
inside snapshots also counted in safemode threshhold
Key: HDFS-5283
URL: https://issues.apache.org/jira/browse/HDFS-5283
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Vinay
Assignee: Vinay
Priority: Blocker
Attachments: HDFS-5283.patch, HDFS-5283.patch
This is observed in one of our env:
1. A MR Job was running which has created some temporary files and was
writing to them.
2. Snapshot was taken
3. And Job was killed and temporary files were deleted.
4. Namenode restarted.
5. After restart Namenode was in safemode waiting for blocks
Analysis
-
1. Since the snapshot taken also includes the temporary files which were
open, and later original files are deleted.
2. UnderConstruction blocks count was taken from leases. not considered the
UC blocks only inside snapshots
3. So safemode threshold count was more and NN did not come out of safemode
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5285) Flatten INodeFile hierarchy
[ https://issues.apache.org/jira/browse/HDFS-5285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5285: - Attachment: h5285_20131001.patch h5285_20131001.patch: preliminary patch which does not compile. Still need to fix the file with snapshot case. Flatten INodeFile hierarchy --- Key: HDFS-5285 URL: https://issues.apache.org/jira/browse/HDFS-5285 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h5285_20131001.patch For files, there are INodeFile, INodeFileUnderConstruction, INodeFileWithSnapshot and INodeFileUnderConstructionWithSnapshot for representing whether a file is under construction or whether it is in some snapshot. The following are two major problems of the current approach: - Java class does not support multiple inheritances so that INodeFileUnderConstructionWithSnapshot cannot extend both INodeFileUnderConstruction and INodeFileWithSnapshot. - The number of classes is exponential to the number of features. Currently, there are only two features, UnderConstruction and WithSnapshot. The number of classes is 2^2 = 4. It is hard to add one more feature since the number of classes will become 2^3 = 8. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783052#comment-13783052 ] Binglin Chang commented on HDFS-5276: - [~chengxiang li], could you provide your test environment? CPU and java version? Looks like AtomicLong has scalability issues under specific CPUs and java versions, it may cause this serious 70% CPU usage, in normal cases, there may be some cache coherency effect, but not so much. http://java.dzone.com/articles/adventures-atomiclong FileSystem.Statistics got performance issue on multi-thread read/write. --- Key: HDFS-5276 URL: https://issues.apache.org/jira/browse/HDFS-5276 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Chengxiang Li Attachments: DisableFSReadWriteBytesStat.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG FileSystem.Statistics is a singleton variable for each FS scheme, each read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does not perform well in multi-threads(let's say more than 30 threads). so it may cause serious performance issue. during our spark test profile, 32 threads read data from HDFS, about 70% cpu time is spent on FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated HDFS-5276: Attachment: ThreadLocalStat.patch A quick hack implementing ThreadLocal version of AtomicLong, you can see from the code that using thread local has two impact: 1. incrementBytesRead/Written using threadlocal can avoid contention, but introduce null check and hashmap lookup. 2. getBytesRead/Written now need to loop over all threads, which takes more time than before(just a heap lookup) FileSystem.Statistics got performance issue on multi-thread read/write. --- Key: HDFS-5276 URL: https://issues.apache.org/jira/browse/HDFS-5276 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Chengxiang Li Attachments: DisableFSReadWriteBytesStat.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG, ThreadLocalStat.patch FileSystem.Statistics is a singleton variable for each FS scheme, each read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does not perform well in multi-threads(let's say more than 30 threads). so it may cause serious performance issue. during our spark test profile, 32 threads read data from HDFS, about 70% cpu time is spent on FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated HDFS-5276: Attachment: (was: ThreadLocalStat.patch) FileSystem.Statistics got performance issue on multi-thread read/write. --- Key: HDFS-5276 URL: https://issues.apache.org/jira/browse/HDFS-5276 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Chengxiang Li Attachments: DisableFSReadWriteBytesStat.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG FileSystem.Statistics is a singleton variable for each FS scheme, each read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does not perform well in multi-threads(let's say more than 30 threads). so it may cause serious performance issue. during our spark test profile, 32 threads read data from HDFS, about 70% cpu time is spent on FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated HDFS-5276: Attachment: ThreadLocalStat.patch Resubmit fix a bug.. FileSystem.Statistics got performance issue on multi-thread read/write. --- Key: HDFS-5276 URL: https://issues.apache.org/jira/browse/HDFS-5276 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Chengxiang Li Attachments: DisableFSReadWriteBytesStat.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG, ThreadLocalStat.patch FileSystem.Statistics is a singleton variable for each FS scheme, each read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does not perform well in multi-threads(let's say more than 30 threads). so it may cause serious performance issue. during our spark test profile, 32 threads read data from HDFS, about 70% cpu time is spent on FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5263) Delegation token is not created generateNodeDataHeader method of NamenodeJspHelper$NodeListJsp
[
https://issues.apache.org/jira/browse/HDFS-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783109#comment-13783109
]
Daryn Sharp commented on HDFS-5263:
---
Looks good with a cursory glance, but I'm a bit concerned about storing the
token in the instance. Is that necessary?
Delegation token is not created generateNodeDataHeader method of
NamenodeJspHelper$NodeListJsp
--
Key: HDFS-5263
URL: https://issues.apache.org/jira/browse/HDFS-5263
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, webhdfs
Reporter: Vasu Mariyala
Attachments: HDFS-5263.patch
When Kerberos authentication is enabled, we are unable to browse to the data
nodes using ( Name node web page -- Live Nodes -- Select any of the data
nodes). The reason behind this is the delegation token is not provided as
part of the url in the method (generateNodeDataHeader method of NodeListJsp)
{code}
String url = HttpConfig.getSchemePrefix() + d.getHostName() + :
+ d.getInfoPort()
+ /browseDirectory.jsp?namenodeInfoPort= + nnHttpPort + dir=
+ URLEncoder.encode(/, UTF-8)
+ JspHelper.getUrlParam(JspHelper.NAMENODE_ADDRESS, nnaddr);
{code}
But browsing the file system using name node web page -- Browse the file
system - any directory is working fine as the redirectToRandomDataNode
method of NamenodeJspHelper creates the delegation token
{code}
redirectLocation = HttpConfig.getSchemePrefix() + fqdn + : +
redirectPort
+ /browseDirectory.jsp?namenodeInfoPort=
+ nn.getHttpAddress().getPort() + dir=/
+ (tokenString == null ? :
JspHelper.getDelegationTokenUrlParam(tokenString))
+ JspHelper.getUrlParam(JspHelper.NAMENODE_ADDRESS, addr);
{code}
I will work on providing a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5037) Active NN should trigger its own edit log rolls
[ https://issues.apache.org/jira/browse/HDFS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-5037: -- Priority: Critical (was: Major) Active NN should trigger its own edit log rolls --- Key: HDFS-5037 URL: https://issues.apache.org/jira/browse/HDFS-5037 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 3.0.0, 2.1.0-beta Reporter: Todd Lipcon Assignee: Aaron T. Myers Priority: Critical We've seen cases where the SBN/2NN went down, and then users accumulated very very large edit log segments. This causes a slow startup time because the last edit log segment must be read fully to recover it before the NN can start up again. Additionally, in the case of QJM, it can trigger timeouts on recovery or edit log syncing because the very-large segment has to get processed within a certain time bound. We could easily improve this by having the NN trigger its own edit log rolls on a configurable size (eg every 256MB) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4885: - Attachment: HDFS-4885-v2.patch Update verifyBlockPlacement() API in BlockPlacementPolicy - Key: HDFS-4885 URL: https://issues.apache.org/jira/browse/HDFS-4885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Labels: BlockPlacementPolicy Attachments: HDFS-4885.patch, HDFS-4885-v2.patch verifyBlockPlacement() has unused parameter -srcPath as its responsibility just verify single block rather than files under a specific path. Also the return value (int) does not make sense as the violation of block placement has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4885: - Status: Open (was: Patch Available) Update verifyBlockPlacement() API in BlockPlacementPolicy - Key: HDFS-4885 URL: https://issues.apache.org/jira/browse/HDFS-4885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Labels: BlockPlacementPolicy Attachments: HDFS-4885.patch, HDFS-4885-v2.patch verifyBlockPlacement() has unused parameter -srcPath as its responsibility just verify single block rather than files under a specific path. Also the return value (int) does not make sense as the violation of block placement has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4885: - Target Version/s: 3.0.0, 2.1.2-beta (was: 3.0.0) Status: Patch Available (was: Open) Address Luke's comments in v2 patch. Update verifyBlockPlacement() API in BlockPlacementPolicy - Key: HDFS-4885 URL: https://issues.apache.org/jira/browse/HDFS-4885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Labels: BlockPlacementPolicy Attachments: HDFS-4885.patch, HDFS-4885-v2.patch verifyBlockPlacement() has unused parameter -srcPath as its responsibility just verify single block rather than files under a specific path. Also the return value (int) does not make sense as the violation of block placement has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5287) JN need not validate finalized log segments in newEpoch
Todd Lipcon created HDFS-5287:
-
Summary: JN need not validate finalized log segments in newEpoch
Key: HDFS-5287
URL: https://issues.apache.org/jira/browse/HDFS-5287
Project: Hadoop HDFS
Issue Type: Bug
Components: qjm
Affects Versions: 2.1.1-beta
Reporter: Todd Lipcon
Priority: Minor
In {{scanStorageForLatestEdits}}, the JN will call {{validateLog}} on the last
log segment, regardless of whether it is finalized. If it's finalized, then
this is a needless pass over the logs which can adversely affect failover time
for a graceful failover.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783163#comment-13783163
]
Luke Lu commented on HDFS-4885:
---
{{BlockPlacementStatus}} impls should just override toString, so that fsck can
just print the string without knowing {{BlockPlacementPolicy}} internal
details. e.g. {{BlockPlacementStatusWithNodeGroup#toString}} can return block
should be replicated on 1 more node group and 1 more rack for replication=2
after data VM movement.
To help thinking more generally, one usecase the API should be able cover is
group (anti-)affinity of blocks, useful for efficient fail-over of HBase
regions, where we'd prefer one replica of blocks belong to the same group
(region) being replaced on the same node (affinity), while the other replicas
of the group should belong to different fault domains (anti-affinity).
Update verifyBlockPlacement() API in BlockPlacementPolicy
-
Key: HDFS-4885
URL: https://issues.apache.org/jira/browse/HDFS-4885
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Junping Du
Assignee: Junping Du
Labels: BlockPlacementPolicy
Attachments: HDFS-4885.patch, HDFS-4885-v2.patch
verifyBlockPlacement() has unused parameter -srcPath as its responsibility
just verify single block rather than files under a specific path. Also the
return value (int) does not make sense as the violation of block placement
has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HDFS-4986) Namenode webUI changes to incorporate storage type
[ https://issues.apache.org/jira/browse/HDFS-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-4986: Assignee: Haohui Mai Namenode webUI changes to incorporate storage type -- Key: HDFS-4986 URL: https://issues.apache.org/jira/browse/HDFS-4986 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Haohui Mai Namenode needs to change cluster storage summary to show storage information per storage type. Datanode list must also include this information. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[
https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5255:
Attachment: HDFS-5255.06.patch
Fix {{TestHftpDelegationToken#testSelectHftpDelegationToken}}, add some
Javadocs.
Distcp job fails with hsftp when https is enabled
-
Key: HDFS-5255
URL: https://issues.apache.org/jira/browse/HDFS-5255
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Yesha Vora
Assignee: Arpit Agarwal
Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch,
HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch
Run Distcp job using hsftp when ssl is enabled. The job fails with
java.net.SocketException: Unexpected end of file from server Error
Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5
All the tasks fails with below error.
13/09/23 15:52:38 INFO mapreduce.Job: Task Id :
attempt_1379976241507_0004_m_00_0, Status : FAILED
Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 --
hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more
Caused by:
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
Caused by: java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at
org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230)
... 16 more
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4510) Cover classes ClusterJspHelper/NamenodeJspHelper with unit tests
[ https://issues.apache.org/jira/browse/HDFS-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783186#comment-13783186 ] Daryn Sharp commented on HDFS-4510: --- I don't think it's ok to hardcode hdfs://localhost.localdomain:45541 in hdfs-site.xml. Won't this cause concurrent test execution problems with port in use? Cover classes ClusterJspHelper/NamenodeJspHelper with unit tests Key: HDFS-4510 URL: https://issues.apache.org/jira/browse/HDFS-4510 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Attachments: HADOOP-4510-branch-0.23-a.patch, HADOOP-4510-branch-0.23-b.patch, HADOOP-4510-branch-0.23-c.patch, HADOOP-4510-branch-2-a.patch, HADOOP-4510-branch-2-b.patch, HADOOP-4510-branch-2-c.patch, HADOOP-4510-trunk-a.patch, HADOOP-4510-trunk-b.patch, HADOOP-4510-trunk-c.patch, HDFS-4510-trunk--N27.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated HDFS-4885: -- Target Version/s: 2.3.0 (was: 3.0.0, 2.1.2-beta) Update verifyBlockPlacement() API in BlockPlacementPolicy - Key: HDFS-4885 URL: https://issues.apache.org/jira/browse/HDFS-4885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Labels: BlockPlacementPolicy Attachments: HDFS-4885.patch, HDFS-4885-v2.patch verifyBlockPlacement() has unused parameter -srcPath as its responsibility just verify single block rather than files under a specific path. Also the return value (int) does not make sense as the violation of block placement has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5263) Delegation token is not created generateNodeDataHeader method of NamenodeJspHelper$NodeListJsp
[
https://issues.apache.org/jira/browse/HDFS-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vasu Mariyala updated HDFS-5263:
Attachment: HDFS-5263-rev1.patch
Attached the patch which removes the need to have the token string as an
instance variable.
Delegation token is not created generateNodeDataHeader method of
NamenodeJspHelper$NodeListJsp
--
Key: HDFS-5263
URL: https://issues.apache.org/jira/browse/HDFS-5263
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, webhdfs
Reporter: Vasu Mariyala
Attachments: HDFS-5263.patch, HDFS-5263-rev1.patch
When Kerberos authentication is enabled, we are unable to browse to the data
nodes using ( Name node web page -- Live Nodes -- Select any of the data
nodes). The reason behind this is the delegation token is not provided as
part of the url in the method (generateNodeDataHeader method of NodeListJsp)
{code}
String url = HttpConfig.getSchemePrefix() + d.getHostName() + :
+ d.getInfoPort()
+ /browseDirectory.jsp?namenodeInfoPort= + nnHttpPort + dir=
+ URLEncoder.encode(/, UTF-8)
+ JspHelper.getUrlParam(JspHelper.NAMENODE_ADDRESS, nnaddr);
{code}
But browsing the file system using name node web page -- Browse the file
system - any directory is working fine as the redirectToRandomDataNode
method of NamenodeJspHelper creates the delegation token
{code}
redirectLocation = HttpConfig.getSchemePrefix() + fqdn + : +
redirectPort
+ /browseDirectory.jsp?namenodeInfoPort=
+ nn.getHttpAddress().getPort() + dir=/
+ (tokenString == null ? :
JspHelper.getDelegationTokenUrlParam(tokenString))
+ JspHelper.getUrlParam(JspHelper.NAMENODE_ADDRESS, addr);
{code}
I will work on providing a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5281) COMMIT request should not block
[ https://issues.apache.org/jira/browse/HDFS-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5281: - Attachment: HDFS-5281.001.patch There are at least two ways to serve COMMIT asynchronously. 1. save the commit requests in OpenFileCtx, and check if any commit is satisfied after each write. 2. save the commit requests in WriteCtx, return response after the corresponding WriteCtx is removed. The uploaded patch implements the first approach. COMMIT request should not block --- Key: HDFS-5281 URL: https://issues.apache.org/jira/browse/HDFS-5281 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-5281.001.patch Currently Commit request is handled synchronously, blocked at most 30 seconds before timeout. This JIRA is to make is asynchronous and thus it won't block other requests coming from the same channel. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5281) COMMIT request should not block
[ https://issues.apache.org/jira/browse/HDFS-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5281: - Status: Patch Available (was: Open) COMMIT request should not block --- Key: HDFS-5281 URL: https://issues.apache.org/jira/browse/HDFS-5281 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-5281.001.patch Currently Commit request is handled synchronously, blocked at most 30 seconds before timeout. This JIRA is to make is asynchronous and thus it won't block other requests coming from the same channel. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HDFS-5274: Attachment: HDFS-5274-3.patch Fix for tests. Add Tracing to HDFS --- Key: HDFS-5274 URL: https://issues.apache.org/jira/browse/HDFS-5274 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 2.1.1-beta Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, HDFS-5274-3.patch, Zipkin Trace a06e941b0172ec73.png Since Google's Dapper paper has shown the benefits of tracing for a large distributed system, it seems like a good time to add tracing to HDFS. HBase has added tracing using HTrace. I propose that the same can be done within HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783252#comment-13783252
]
Hadoop QA commented on HDFS-4885:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606138/HDFS-4885-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5075//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5075//console
This message is automatically generated.
Update verifyBlockPlacement() API in BlockPlacementPolicy
-
Key: HDFS-4885
URL: https://issues.apache.org/jira/browse/HDFS-4885
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Junping Du
Assignee: Junping Du
Labels: BlockPlacementPolicy
Attachments: HDFS-4885.patch, HDFS-4885-v2.patch
verifyBlockPlacement() has unused parameter -srcPath as its responsibility
just verify single block rather than files under a specific path. Also the
return value (int) does not make sense as the violation of block placement
has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[
https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783305#comment-13783305
]
Hadoop QA commented on HDFS-5255:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606145/HDFS-5255.06.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5076//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5076//console
This message is automatically generated.
Distcp job fails with hsftp when https is enabled
-
Key: HDFS-5255
URL: https://issues.apache.org/jira/browse/HDFS-5255
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Yesha Vora
Assignee: Arpit Agarwal
Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch,
HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch
Run Distcp job using hsftp when ssl is enabled. The job fails with
java.net.SocketException: Unexpected end of file from server Error
Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5
All the tasks fails with below error.
13/09/23 15:52:38 INFO mapreduce.Job: Task Id :
attempt_1379976241507_0004_m_00_0, Status : FAILED
Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 --
hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more
Caused by:
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
Caused by: java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187)
at
[jira] [Commented] (HDFS-5281) COMMIT request should not block
[
https://issues.apache.org/jira/browse/HDFS-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783330#comment-13783330
]
Hadoop QA commented on HDFS-5281:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606165/HDFS-5281.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5079//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5079//console
This message is automatically generated.
COMMIT request should not block
---
Key: HDFS-5281
URL: https://issues.apache.org/jira/browse/HDFS-5281
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-5281.001.patch
Currently Commit request is handled synchronously, blocked at most 30 seconds
before timeout. This JIRA is to make is asynchronous and thus it won't block
other requests coming from the same channel.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5263) Delegation token is not created generateNodeDataHeader method of NamenodeJspHelper$NodeListJsp
[
https://issues.apache.org/jira/browse/HDFS-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783381#comment-13783381
]
Hadoop QA commented on HDFS-5263:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606158/HDFS-5263-rev1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5077//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5077//console
This message is automatically generated.
Delegation token is not created generateNodeDataHeader method of
NamenodeJspHelper$NodeListJsp
--
Key: HDFS-5263
URL: https://issues.apache.org/jira/browse/HDFS-5263
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, webhdfs
Reporter: Vasu Mariyala
Attachments: HDFS-5263.patch, HDFS-5263-rev1.patch
When Kerberos authentication is enabled, we are unable to browse to the data
nodes using ( Name node web page -- Live Nodes -- Select any of the data
nodes). The reason behind this is the delegation token is not provided as
part of the url in the method (generateNodeDataHeader method of NodeListJsp)
{code}
String url = HttpConfig.getSchemePrefix() + d.getHostName() + :
+ d.getInfoPort()
+ /browseDirectory.jsp?namenodeInfoPort= + nnHttpPort + dir=
+ URLEncoder.encode(/, UTF-8)
+ JspHelper.getUrlParam(JspHelper.NAMENODE_ADDRESS, nnaddr);
{code}
But browsing the file system using name node web page -- Browse the file
system - any directory is working fine as the redirectToRandomDataNode
method of NamenodeJspHelper creates the delegation token
{code}
redirectLocation = HttpConfig.getSchemePrefix() + fqdn + : +
redirectPort
+ /browseDirectory.jsp?namenodeInfoPort=
+ nn.getHttpAddress().getPort() + dir=/
+ (tokenString == null ? :
JspHelper.getDelegationTokenUrlParam(tokenString))
+ JspHelper.getUrlParam(JspHelper.NAMENODE_ADDRESS, addr);
{code}
I will work on providing a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783405#comment-13783405
]
Luke Lu commented on HDFS-4885:
---
I posted the comment before refreshing to see the v2 patch, which is getting
closer. I'd use something like {{hasMisplacement}} to check misplacement in
fsck instead of {{getMisReplicatedNum}}, as the latter might not be easy
calculate for certain BBPs. {{BlockPlacementStatus#getDescription}} is probably
better than either {{toString}} or {{getMisReplicatedString}}.
One the second thought, I wonder if we should simply pass down the
{{HdfsFileStatus}} instead of just a string to {{verifyBlockPlacement}} and get
rid of the constraints creation in fsck, as the file status will contain the
needed info to construct/enforce proper constraints in the BPP impls. So the
API would be:
{code}
public BlockPlacementStatus verifyBlockPlacement(String srcParent,
HdfsFileStatus srcFile, LocatedBlock blk);
{code}
The advantage of this is that if we have (extended) attributes to implement
various placement grouping policies in the future, no API change is necessary.
We should probably add a new API to calculate/reset the accumulative status of
a BBP instance, so a proper summary can be printed. This is also useful to
verify group (anti-)affinity, which cannot be determined on a block by block
basis.
{code}
public void resetStatus();
public BlockPlacementStatus getStatus();
{code}
Thoughts?
Update verifyBlockPlacement() API in BlockPlacementPolicy
-
Key: HDFS-4885
URL: https://issues.apache.org/jira/browse/HDFS-4885
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Junping Du
Assignee: Junping Du
Labels: BlockPlacementPolicy
Attachments: HDFS-4885.patch, HDFS-4885-v2.patch
verifyBlockPlacement() has unused parameter -srcPath as its responsibility
just verify single block rather than files under a specific path. Also the
return value (int) does not make sense as the violation of block placement
has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Attachment: HDFS-5255.07.patch Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783445#comment-13783445
]
Hadoop QA commented on HDFS-5274:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606168/HDFS-5274-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5078//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5078//console
This message is automatically generated.
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, Zipkin Trace a06e941b0172ec73.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5283: Status: Open (was: Patch Available) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 2.1.1-beta, 3.0.0 Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5283: Attachment: HDFS-5283.000.patch Thanks for the fix Vinay! Your analysis makes sense to me, and I think your patch can fix the file-deletion scenario. For dir-deletion scenario, instead of changing the current snapshot code (i.e., to convert all the INodeFileUC under the deleted dir to INodeFIleUCWithSnapshot), I think maybe we can just check if the full name of the INodeFile retrieved from the blocksMap can still represent an INode in the current fsdir tree, and if yes, whether the corresponding inode is the same with the one in blocksMap. So I tried to provide a patch based on your existing patch, with the extra check mentioned above and some other small fixes. We can continue working on this patch if you think this is the correct path. NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.1-beta Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783465#comment-13783465 ] Jing Zhao commented on HDFS-5283: - Looks like my patch will fail some tests. Will update the patch later. NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.1-beta Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Attachment: (was: HDFS-5255.07.patch) Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Attachment: HDFS-5255.07.patch Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Attachment: HDFS-5255.08.patch Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Target Version/s: 2.1.1-beta Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783489#comment-13783489 ] Suresh Srinivas commented on HDFS-5255: --- [~arpitagarwal], can you please add details of manual tests you have done? Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5137) Improve WriteManager for processing stable write requests and commit requests
[ https://issues.apache.org/jira/browse/HDFS-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5137: - Resolution: Duplicate Status: Resolved (was: Patch Available) Resolve it as a dup since this JIRA was used as a temp solution before HDFS-5281. Improve WriteManager for processing stable write requests and commit requests - Key: HDFS-5137 URL: https://issues.apache.org/jira/browse/HDFS-5137 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor Attachments: HDFS-5137.001.patch, HDFS-5137.patch WriteManager#handleCommit needs to block until all the data before the commitOffset has been received. This jira plans to improve the current concurrency implementation for this blocking call. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783494#comment-13783494 ] Suresh Srinivas commented on HDFS-5255: --- BTW I am +1 for the change. Can you please create a jira for work related to testing hftp and hsftp with secure clusters, if you find any bugs? Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783499#comment-13783499 ] Arpit Agarwal commented on HDFS-5255: - Yes, for a single node cluster I tested the following: # SSL enabled, get file via hsftp # SSL enabled, distcp from hsftp to hdfs # SSL disabled, get file via hftp # SSL disabled, distcp from hftp to hdfs Distcp job fails with hsftp when https is enabled - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled in insecure cluster
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Summary: Distcp job fails with hsftp when https is enabled in insecure cluster (was: Distcp job fails with hsftp when https is enabled) Distcp job fails with hsftp when https is enabled in insecure cluster - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled in insecure cluster
[
https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783510#comment-13783510
]
Hadoop QA commented on HDFS-5255:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606205/HDFS-5255.07.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5080//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5080//console
This message is automatically generated.
Distcp job fails with hsftp when https is enabled in insecure cluster
-
Key: HDFS-5255
URL: https://issues.apache.org/jira/browse/HDFS-5255
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Yesha Vora
Assignee: Arpit Agarwal
Fix For: 2.1.2-beta
Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch,
HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch,
HDFS-5255.07.patch, HDFS-5255.08.patch
Run Distcp job using hsftp when ssl is enabled. The job fails with
java.net.SocketException: Unexpected end of file from server Error
Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5
All the tasks fails with below error.
13/09/23 15:52:38 INFO mapreduce.Job: Task Id :
attempt_1379976241507_0004_m_00_0, Status : FAILED
Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 --
hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more
Caused by:
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
Caused by: java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
at
[jira] [Updated] (HDFS-5255) Distcp job fails with hsftp when https is enabled in insecure cluster
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5255: Resolution: Fixed Fix Version/s: 2.1.2-beta Status: Resolved (was: Patch Available) I have committed this to trunk, branch-2 and branch-2.1-beta. Thanks [~sureshms] for reviewing and your help understanding the background! Distcp job fails with hsftp when https is enabled in insecure cluster - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Fix For: 2.1.2-beta Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:75) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:230) ... 16 more -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled in insecure cluster
[ https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783513#comment-13783513 ] Hudson commented on HDFS-5255: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4509 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4509/]) HDFS-5255. Distcp job fails with hsftp when https is enabled in insecure cluster. (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528279) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileChecksumServlets.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileDataServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpFileSystem.java Distcp job fails with hsftp when https is enabled in insecure cluster - Key: HDFS-5255 URL: https://issues.apache.org/jira/browse/HDFS-5255 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Yesha Vora Assignee: Arpit Agarwal Fix For: 2.1.2-beta Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch, HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch, HDFS-5255.07.patch, HDFS-5255.08.patch Run Distcp job using hsftp when ssl is enabled. The job fails with java.net.SocketException: Unexpected end of file from server Error Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5 All the tasks fails with below error. 13/09/23 15:52:38 INFO mapreduce.Job: Task Id : attempt_1379976241507_0004_m_00_0, Status : FAILED Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 -- hdfs://localhost:19000/f5 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: HTTP_OK expected, received 500 at org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383) at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783528#comment-13783528 ] Junping Du commented on HDFS-4885: -- Thanks Luke for review and great comments! bq. I'd use something like hasMisplacement to check misplacement in fsck instead of getMisReplicatedNum, as the latter might not be easy calculate for certain BBPs. BlockPlacementStatus#getDescription is probably better than either toString or getMisReplicatedString. Sure. will update it. bq. The advantage of this is that if we have (extended) attributes to implement various placement grouping policies in the future, no API change is necessary. Agree. Previous BlockPlacementConstraints is hard to extend to align with different BPP, we should leave this flexibility to implementation of verifyBlockPlacement() in different BPP but provide necessary info like HdfsFileStatus. bq. We should probably add a new API to calculate/reset the accumulative status of a BBP instance, so a proper summary can be printed. This is also useful to verify group (anti-)affinity, which cannot be determined on a block by block basis. That's a great idea. However, I guess it may be better to wait to implement this until the BPP that based on (anti-) affinity group of blocks is ready, then we can see how to extend BlockPlacementStatus (mostly seems to be merge work) to status of a bunch of blocks. Thoughts? Will address these comments soon. Update verifyBlockPlacement() API in BlockPlacementPolicy - Key: HDFS-4885 URL: https://issues.apache.org/jira/browse/HDFS-4885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Labels: BlockPlacementPolicy Attachments: HDFS-4885.patch, HDFS-4885-v2.patch verifyBlockPlacement() has unused parameter -srcPath as its responsibility just verify single block rather than files under a specific path. Also the return value (int) does not make sense as the violation of block placement has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5288) Close idle connections in portmap
Haohui Mai created HDFS-5288: Summary: Close idle connections in portmap Key: HDFS-5288 URL: https://issues.apache.org/jira/browse/HDFS-5288 Project: Hadoop HDFS Issue Type: Improvement Components: nfs Reporter: Haohui Mai Assignee: Haohui Mai Currently the portmap daemon does not close idle connections. The daemon should close any idle connections to save resources. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5283: Attachment: (was: HDFS-5283.000.patch) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.1-beta Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5283: Attachment: HDFS-5283.000.patch Update the patch to fix unit tests. NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.1-beta Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5283: Status: Patch Available (was: Open) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold Key: HDFS-5283 URL: https://issues.apache.org/jira/browse/HDFS-5283 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 2.1.1-beta, 3.0.0 Reporter: Vinay Assignee: Vinay Priority: Blocker Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch This is observed in one of our env: 1. A MR Job was running which has created some temporary files and was writing to them. 2. Snapshot was taken 3. And Job was killed and temporary files were deleted. 4. Namenode restarted. 5. After restart Namenode was in safemode waiting for blocks Analysis - 1. Since the snapshot taken also includes the temporary files which were open, and later original files are deleted. 2. UnderConstruction blocks count was taken from leases. not considered the UC blocks only inside snapshots 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4885: - Attachment: HDFS-4885-v3.patch Address Luke's comments in v3 patch. Update verifyBlockPlacement() API in BlockPlacementPolicy - Key: HDFS-4885 URL: https://issues.apache.org/jira/browse/HDFS-4885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Labels: BlockPlacementPolicy Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch verifyBlockPlacement() has unused parameter -srcPath as its responsibility just verify single block rather than files under a specific path. Also the return value (int) does not make sense as the violation of block placement has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5246) Make Hadoop nfs server port and mount daemon port configurable
[ https://issues.apache.org/jira/browse/HDFS-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-5246: - Issue Type: Sub-task (was: Improvement) Parent: HDFS-4750 Make Hadoop nfs server port and mount daemon port configurable -- Key: HDFS-5246 URL: https://issues.apache.org/jira/browse/HDFS-5246 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Affects Versions: 2.1.0-beta Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 Reporter: Jinghui Wang Assignee: Jinghui Wang Fix For: 3.0.0, 2.1.2-beta Attachments: HDFS-5246-2.patch, HDFS-5246-3.patch, HDFS-5246.patch Hadoop nfs binds the nfs server to port 2049, which is also the default port that Linux nfs uses. If Linux nfs is already running on the machine then Hadoop nfs will not be albe to start. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5255) Distcp job fails with hsftp when https is enabled in insecure cluster
[
https://issues.apache.org/jira/browse/HDFS-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783571#comment-13783571
]
Hadoop QA commented on HDFS-5255:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606220/HDFS-5255.08.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5081//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5081//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5081//console
This message is automatically generated.
Distcp job fails with hsftp when https is enabled in insecure cluster
-
Key: HDFS-5255
URL: https://issues.apache.org/jira/browse/HDFS-5255
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Yesha Vora
Assignee: Arpit Agarwal
Fix For: 2.1.2-beta
Attachments: HDFS-5255.01.patch, HDFS-5255.02.patch,
HDFS-5255.04.patch, HDFS-5255.05.patch, HDFS-5255.06.patch,
HDFS-5255.07.patch, HDFS-5255.08.patch
Run Distcp job using hsftp when ssl is enabled. The job fails with
java.net.SocketException: Unexpected end of file from server Error
Running: hadoop distcp hsftp://localhost:50070/f1 hdfs://localhost:19000/f5
All the tasks fails with below error.
13/09/23 15:52:38 INFO mapreduce.Job: Task Id :
attempt_1379976241507_0004_m_00_0, Status : FAILED
Error: java.io.IOException: File copy failed: hsftp://localhost:50070/f1 --
hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hsftp://127.0.0.1:50070/f1 to hdfs://localhost:19000/f5
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
... 10 more
Caused by:
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:233)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:198)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:134)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:101)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
Caused by: java.io.IOException: HTTP_OK expected, received 500
at
org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderUrlOpener.connect(HftpFileSystem.java:383)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119)
[jira] [Commented] (HDFS-5279) Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem.
[
https://issues.apache.org/jira/browse/HDFS-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783576#comment-13783576
]
Suresh Srinivas commented on HDFS-5279:
---
[~cnauroth], thanks for addressing the comments. +1 for the patch.
Guard against NullPointerException in NameNode JSP pages before
initialization of FSNamesystem.
---
Key: HDFS-5279
URL: https://issues.apache.org/jira/browse/HDFS-5279
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5279.1.patch, HDFS-5279.2.patch
HDFS-4372 added tracking of NameNode startup progress. As part of that
change, the NameNode HTTP server now starts before initialization of the
{{FSNamesystem}}. There are a few code paths remaining in the JSP pages that
are at risk of causing {{NullPointerException}} if accessed when the
{{FSNamesystem}} has not been fully initialized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[
https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783605#comment-13783605
]
Hadoop QA commented on HDFS-5283:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606236/HDFS-5283.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5082//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5082//console
This message is automatically generated.
NN not coming out of startup safemode due to under construction blocks only
inside snapshots also counted in safemode threshhold
Key: HDFS-5283
URL: https://issues.apache.org/jira/browse/HDFS-5283
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Vinay
Assignee: Vinay
Priority: Blocker
Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch
This is observed in one of our env:
1. A MR Job was running which has created some temporary files and was
writing to them.
2. Snapshot was taken
3. And Job was killed and temporary files were deleted.
4. Namenode restarted.
5. After restart Namenode was in safemode waiting for blocks
Analysis
-
1. Since the snapshot taken also includes the temporary files which were
open, and later original files are deleted.
2. UnderConstruction blocks count was taken from leases. not considered the
UC blocks only inside snapshots
3. So safemode threshold count was more and NN did not come out of safemode
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783609#comment-13783609
]
Hadoop QA commented on HDFS-4885:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12606237/HDFS-4885-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5083//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5083//console
This message is automatically generated.
Update verifyBlockPlacement() API in BlockPlacementPolicy
-
Key: HDFS-4885
URL: https://issues.apache.org/jira/browse/HDFS-4885
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Junping Du
Assignee: Junping Du
Labels: BlockPlacementPolicy
Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch
verifyBlockPlacement() has unused parameter -srcPath as its responsibility
just verify single block rather than files under a specific path. Also the
return value (int) does not make sense as the violation of block placement
has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5279) Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem.
[
https://issues.apache.org/jira/browse/HDFS-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13783674#comment-13783674
]
Hudson commented on HDFS-5279:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4513 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4513/])
HDFS-5279. Guard against NullPointerException in NameNode JSP pages before
initialization of FSNamesystem. Contributed by Chris Nauroth. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1528308)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeJspHelper.java
Guard against NullPointerException in NameNode JSP pages before
initialization of FSNamesystem.
---
Key: HDFS-5279
URL: https://issues.apache.org/jira/browse/HDFS-5279
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5279.1.patch, HDFS-5279.2.patch
HDFS-4372 added tracking of NameNode startup progress. As part of that
change, the NameNode HTTP server now starts before initialization of the
{{FSNamesystem}}. There are a few code paths remaining in the JSP pages that
are at risk of causing {{NullPointerException}} if accessed when the
{{FSNamesystem}} has not been fully initialized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5279) Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem.
[
https://issues.apache.org/jira/browse/HDFS-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5279:
Attachment: HDFS-5279-branch-2.1.patch
I ended up needing a slightly different patch when I merged down to
branch-2.1-beta. This is because {{generateSnapshotReport}} doesn't exist in
branch-2.1-beta. This was added in HDFS-4096, which targeted 2.3.0. I'm
attaching the branch-2.1-beta patch, just FYI.
Guard against NullPointerException in NameNode JSP pages before
initialization of FSNamesystem.
---
Key: HDFS-5279
URL: https://issues.apache.org/jira/browse/HDFS-5279
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.0.0, 2.1.2-beta
Attachments: HDFS-5279.1.patch, HDFS-5279.2.patch,
HDFS-5279-branch-2.1.patch
HDFS-4372 added tracking of NameNode startup progress. As part of that
change, the NameNode HTTP server now starts before initialization of the
{{FSNamesystem}}. There are a few code paths remaining in the JSP pages that
are at risk of causing {{NullPointerException}} if accessed when the
{{FSNamesystem}} has not been fully initialized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5279) Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem.
[
https://issues.apache.org/jira/browse/HDFS-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5279:
Resolution: Fixed
Fix Version/s: 2.1.2-beta
3.0.0
Target Version/s: 2.1.1-beta, 3.0.0 (was: 3.0.0, 2.1.1-beta)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have committed this to trunk, branch-2, and branch-2.1-beta. Suresh, thank
you for the code review.
Guard against NullPointerException in NameNode JSP pages before
initialization of FSNamesystem.
---
Key: HDFS-5279
URL: https://issues.apache.org/jira/browse/HDFS-5279
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.0.0, 2.1.2-beta
Attachments: HDFS-5279.1.patch, HDFS-5279.2.patch,
HDFS-5279-branch-2.1.patch
HDFS-4372 added tracking of NameNode startup progress. As part of that
change, the NameNode HTTP server now starts before initialization of the
{{FSNamesystem}}. There are a few code paths remaining in the JSP pages that
are at risk of causing {{NullPointerException}} if accessed when the
{{FSNamesystem}} has not been fully initialized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
