[jira] [Updated] (HDFS-10572) Fix TestOfflineEditsViewer#testGenerated
[
https://issues.apache.org/jira/browse/HDFS-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-10572:
-
Target Version/s: 2.9.0
The test also fails in branch-2.
> Fix TestOfflineEditsViewer#testGenerated
>
>
> Key: HDFS-10572
> URL: https://issues.apache.org/jira/browse/HDFS-10572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiaoyu Yao
>Priority: Blocker
>
> The test has been failing consistently on trunk recently. This ticket is open
> to fix this test to avoid false alarm on Jenkins. Figure out which recent
> commit caused this failure can be a good start.
>
> {code}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.646 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> testGenerated(org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer)
> Time elapsed: 3.623 sec <<< FAILURE!
> java.lang.AssertionError: Generated edits and reparsed (bin to XML to bin)
> should be same
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer.testGenerated(TestOfflineEditsViewer.java:125)
> Results :
> Failed tests:
> TestOfflineEditsViewer.testGenerated:125 Generated edits and reparsed (bin
> to XML to bin) should be same
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10548) Remove the long deprecated BlockReaderRemote
[
https://issues.apache.org/jira/browse/HDFS-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-10548:
-
Release Note: This will obsoletes this configuration property, since the
legacy block reader is removed from the code base.
{{dfs.client.use.legacy.blockreader}} (was: This will obsoletes this
configuration property, since the legacy block reader is removed from the code
base.
{{dfs.client.use.legacy.blockreader}})
> Remove the long deprecated BlockReaderRemote
>
>
> Key: HDFS-10548
> URL: https://issues.apache.org/jira/browse/HDFS-10548
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Fix For: 3.0.0-alpha1
>
> Attachments: HDFS-10548-v1.patch, HDFS-10548-v2.patch,
> HDFS-10548-v3.patch
>
>
> To lessen the maintain burden like raised in HDFS-8901, suggest we remove
> {{BlockReaderRemote}} class that's deprecated very long time ago.
> From {{BlockReaderRemote}} header:
> {quote}
> * @deprecated this is an old implementation that is being left around
> * in case any issues spring up with the new {@link BlockReaderRemote2}
> * implementation.
> * It will be removed in the next release.
> {quote}
> From {{BlockReaderRemote2}} class header:
> {quote}
> * This is a new implementation introduced in Hadoop 0.23 which
> * is more efficient and simpler than the older BlockReader
> * implementation. It should be renamed to BlockReaderRemote
> * once we are confident in it.
> {quote}
> So even further, after getting rid of the old class, we could rename as the
> comment suggested: BlockReaderRemote2 => BlockReaderRemote.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10548) Remove the long deprecated BlockReaderRemote
[
https://issues.apache.org/jira/browse/HDFS-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-10548:
-
Release Note:
This will obsoletes this configuration property, since the legacy block reader
is removed from the code base.
{{dfs.client.use.legacy.blockreader}}
> Remove the long deprecated BlockReaderRemote
>
>
> Key: HDFS-10548
> URL: https://issues.apache.org/jira/browse/HDFS-10548
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Fix For: 3.0.0-alpha1
>
> Attachments: HDFS-10548-v1.patch, HDFS-10548-v2.patch,
> HDFS-10548-v3.patch
>
>
> To lessen the maintain burden like raised in HDFS-8901, suggest we remove
> {{BlockReaderRemote}} class that's deprecated very long time ago.
> From {{BlockReaderRemote}} header:
> {quote}
> * @deprecated this is an old implementation that is being left around
> * in case any issues spring up with the new {@link BlockReaderRemote2}
> * implementation.
> * It will be removed in the next release.
> {quote}
> From {{BlockReaderRemote2}} class header:
> {quote}
> * This is a new implementation introduced in Hadoop 0.23 which
> * is more efficient and simpler than the older BlockReader
> * implementation. It should be renamed to BlockReaderRemote
> * once we are confident in it.
> {quote}
> So even further, after getting rid of the old class, we could rename as the
> comment suggested: BlockReaderRemote2 => BlockReaderRemote.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10548) Remove the long deprecated BlockReaderRemote
[
https://issues.apache.org/jira/browse/HDFS-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-10548:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha1
Status: Resolved (was: Patch Available)
Thanks [~andrew.wang]! I just committed this to trunk, targeting for 3.0
> Remove the long deprecated BlockReaderRemote
>
>
> Key: HDFS-10548
> URL: https://issues.apache.org/jira/browse/HDFS-10548
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Fix For: 3.0.0-alpha1
>
> Attachments: HDFS-10548-v1.patch, HDFS-10548-v2.patch,
> HDFS-10548-v3.patch
>
>
> To lessen the maintain burden like raised in HDFS-8901, suggest we remove
> {{BlockReaderRemote}} class that's deprecated very long time ago.
> From {{BlockReaderRemote}} header:
> {quote}
> * @deprecated this is an old implementation that is being left around
> * in case any issues spring up with the new {@link BlockReaderRemote2}
> * implementation.
> * It will be removed in the next release.
> {quote}
> From {{BlockReaderRemote2}} class header:
> {quote}
> * This is a new implementation introduced in Hadoop 0.23 which
> * is more efficient and simpler than the older BlockReader
> * implementation. It should be renamed to BlockReaderRemote
> * once we are confident in it.
> {quote}
> So even further, after getting rid of the old class, we could rename as the
> comment suggested: BlockReaderRemote2 => BlockReaderRemote.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10548) Remove the long deprecated BlockReaderRemote
[
https://issues.apache.org/jira/browse/HDFS-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359970#comment-15359970
]
Hudson commented on HDFS-10548:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #10047 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10047/])
HDFS-10548. Remove the long deprecated BlockReaderRemote. Contributed by
(kai.zheng: rev 8b281bce85474501868d68f8d5590a6086abb7b7)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderRemote.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderBase.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderRemote2.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestClientBlockVerification.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderRemote2.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java
> Remove the long deprecated BlockReaderRemote
>
>
> Key: HDFS-10548
> URL: https://issues.apache.org/jira/browse/HDFS-10548
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HDFS-10548-v1.patch, HDFS-10548-v2.patch,
> HDFS-10548-v3.patch
>
>
> To lessen the maintain burden like raised in HDFS-8901, suggest we remove
> {{BlockReaderRemote}} class that's deprecated very long time ago.
> From {{BlockReaderRemote}} header:
> {quote}
> * @deprecated this is an old implementation that is being left around
> * in case any issues spring up with the new {@link BlockReaderRemote2}
> * implementation.
> * It will be removed in the next release.
> {quote}
> From {{BlockReaderRemote2}} class header:
> {quote}
> * This is a new implementation introduced in Hadoop 0.23 which
> * is more efficient and simpler than the older BlockReader
> * implementation. It should be renamed to BlockReaderRemote
> * once we are confident in it.
> {quote}
> So even further, after getting rid of the old class, we could rename as the
> comment suggested: BlockReaderRemote2 => BlockReaderRemote.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-9700) DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-9700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9700:
Comment: was deleted
(was: Nice.
+1 (non-binding))
> DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for
> DataTransferProtocol
>
>
> Key: HDFS-9700
> URL: https://issues.apache.org/jira/browse/HDFS-9700
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.7.1, 2.6.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.8.0
>
> Attachments: HDFS-9700-branch-2.7.002.patch,
> HDFS-9700-branch-2.7.003.patch, HDFS-9700-v1.patch, HDFS-9700-v2.patch,
> HDFS-9700.002.patch, HDFS-9700.003.patch, HDFS-9700.004.patch,
> HDFS-9700_branch-2.7-v2.patch, HDFS-9700_branch-2.7.patch
>
>
> In {{DFSClient.connectToDN()}} and
> {{DFSOutputStream.createSocketForPipeline()}}, we never call
> {{setTcpNoDelay()}} on the constructed socket before sending. In both cases,
> we should respect the value of ipc.client.tcpnodelay in the configuration.
> While this applies whether security is enabled or not, it seems to have a
> bigger impact on latency when security is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-9652) TestStartup.testImageChecksum is broken by HDFS-9569
[
https://issues.apache.org/jira/browse/HDFS-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9652:
Comment: was deleted
(was: +1 (non-binding))
> TestStartup.testImageChecksum is broken by HDFS-9569
>
>
> Key: HDFS-9652
> URL: https://issues.apache.org/jira/browse/HDFS-9652
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-9652.001.patch
>
>
> TestStartup.testImageChecksum is broken due to a message change introduced in
> HDFS-9569 fix.
> {quote}
> It seems that this caused TestStartup failing for the past 12 builds; see
> https://builds.apache.org/job/PreCommit-HDFS-Build/14117/testReport/org.apache.hadoop.hdfs.server.namenode/TestStartup/testImageChecksum/
> {quote}
> Thanks [~szetszwo] for noticing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-9566) Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method
[
https://issues.apache.org/jira/browse/HDFS-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9566:
Comment: was deleted
(was: +1 (non-binding) pending on Jenkins.)
> Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State
> state)' method
> ---
>
> Key: HDFS-9566
> URL: https://issues.apache.org/jira/browse/HDFS-9566
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0
>
> Attachments: HDFS-9566.branch-2.patch, HDFS-9566.patch
>
>
> HDFS-5318 added a {{BlocksMap#getStorages(Block, State)}} which is based on
> iterables and predicates. The method is very expensive compared to a simple
> comparison/continue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-9528) Cleanup namenode audit/log/exception messages
[ https://issues.apache.org/jira/browse/HDFS-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-9528: Comment: was deleted (was: +1 (non-binding)) > Cleanup namenode audit/log/exception messages > - > > Key: HDFS-9528 > URL: https://issues.apache.org/jira/browse/HDFS-9528 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.8.0 > > Attachments: h9528_20151208.patch, h9528_20151210.patch, > h9528_20151210b.patch > > > - Cleanup unnecessary long methods for constructing message strings. > - Avoid calling toString() methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-9407) TestFileTruncate fails with BindException
[
https://issues.apache.org/jira/browse/HDFS-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9407:
Comment: was deleted
(was: Thanks for reporting and working on this.)
> TestFileTruncate fails with BindException
> -
>
> Key: HDFS-9407
> URL: https://issues.apache.org/jira/browse/HDFS-9407
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: HDFS-9407-002.patch, HDFS-9407-003.patch, HDFS-9407.patch
>
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/2530/
> {noformat}
> java.net.BindException: Problem binding to [localhost:8020]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:390)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1564)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1247)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1016)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:891)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:823)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:482)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:441)
> at
> org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.setUp(TestFileTruncate.java:103)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8991) Provide information on BPOfferService in DN JMX
[
https://issues.apache.org/jira/browse/HDFS-8991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359854#comment-15359854
]
Mingliang Liu commented on HDFS-8991:
-
Thanks [~jnp] for the review and comment. For the mini-dfs cluster, the
hostname will be localhost and only one entry in the HA case.
Before this patch:
- Non-HA:
{code}
{"localhost":"BP-954094909-10.22.8.141-1467417399231"}
{code}
- HA:
{code}
{"localhost":"BP-21526920-10.22.8.141-1467417356425"}
{code}
After the patch:
- Non-HA:
{code}
{"BP-1939740699-10.22.8.141-1467416841045":{"localhost:57787":"active"}}
{code}
- HA:
{code}
{"BP-1698487347-10.22.8.141-1467417257785":{"localhost:57925":"active","localhost:57927":"standby"}}
{code}
> Provide information on BPOfferService in DN JMX
> ---
>
> Key: HDFS-8991
> URL: https://issues.apache.org/jira/browse/HDFS-8991
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-8991.000.patch, HDFS-8991.001.patch
>
>
> In cases like HDFS-7714 where the BPOfferService thread is missing, which
> require nontrivial effort to debug that which NN that the DN thinks it active
> / standby.
> It would make sense to make the information more accessible through JMX or
> Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-8991) Provide information on BPOfferService in DN JMX
[ https://issues.apache.org/jira/browse/HDFS-8991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359754#comment-15359754 ] Jitendra Nath Pandey edited comment on HDFS-8991 at 7/1/16 10:57 PM: - Could you please paste the output of the jmx before and after this patch, covering the relevant parts of the output? was (Author: jnp): Could you please paste the out of the jmx before and after this patch, covering the relevant parts of the output? > Provide information on BPOfferService in DN JMX > --- > > Key: HDFS-8991 > URL: https://issues.apache.org/jira/browse/HDFS-8991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Haohui Mai >Assignee: Mingliang Liu > Attachments: HDFS-8991.000.patch, HDFS-8991.001.patch > > > In cases like HDFS-7714 where the BPOfferService thread is missing, which > require nontrivial effort to debug that which NN that the DN thinks it active > / standby. > It would make sense to make the information more accessible through JMX or > Web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[
https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359783#comment-15359783
]
Hadoop QA commented on HDFS-9890:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
43s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
29s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
30s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
31s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 51s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed CTEST tests |
test_libhdfs_threaded_hdfspp_test_shim_static |
| | test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static |
| JDK v1.7.0_101 Failed CTEST tests |
test_libhdfs_threaded_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815822/HDFS-9890.HDFS-8707.013.patch
|
| JIRA Issue | HDFS-9890 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 8feed0781337 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-10548) Remove the long deprecated BlockReaderRemote
[
https://issues.apache.org/jira/browse/HDFS-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359780#comment-15359780
]
Andrew Wang commented on HDFS-10548:
SGTM, feel free to commit Kai. +1.
> Remove the long deprecated BlockReaderRemote
>
>
> Key: HDFS-10548
> URL: https://issues.apache.org/jira/browse/HDFS-10548
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HDFS-10548-v1.patch, HDFS-10548-v2.patch,
> HDFS-10548-v3.patch
>
>
> To lessen the maintain burden like raised in HDFS-8901, suggest we remove
> {{BlockReaderRemote}} class that's deprecated very long time ago.
> From {{BlockReaderRemote}} header:
> {quote}
> * @deprecated this is an old implementation that is being left around
> * in case any issues spring up with the new {@link BlockReaderRemote2}
> * implementation.
> * It will be removed in the next release.
> {quote}
> From {{BlockReaderRemote2}} class header:
> {quote}
> * This is a new implementation introduced in Hadoop 0.23 which
> * is more efficient and simpler than the older BlockReader
> * implementation. It should be renamed to BlockReaderRemote
> * once we are confident in it.
> {quote}
> So even further, after getting rid of the old class, we could rename as the
> comment suggested: BlockReaderRemote2 => BlockReaderRemote.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8991) Provide information on BPOfferService in DN JMX
[ https://issues.apache.org/jira/browse/HDFS-8991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359754#comment-15359754 ] Jitendra Nath Pandey commented on HDFS-8991: Could you please paste the out of the jmx before and after this patch, covering the relevant parts of the output? > Provide information on BPOfferService in DN JMX > --- > > Key: HDFS-8991 > URL: https://issues.apache.org/jira/browse/HDFS-8991 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Haohui Mai >Assignee: Mingliang Liu > Attachments: HDFS-8991.000.patch, HDFS-8991.001.patch > > > In cases like HDFS-7714 where the BPOfferService thread is missing, which > require nontrivial effort to debug that which NN that the DN thinks it active > / standby. > It would make sense to make the information more accessible through JMX or > Web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9854) Log cipher suite negotiation more verbosely
[ https://issues.apache.org/jira/browse/HDFS-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-9854: Component/s: encryption > Log cipher suite negotiation more verbosely > --- > > Key: HDFS-9854 > URL: https://issues.apache.org/jira/browse/HDFS-9854 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Labels: encryption, supportability > Fix For: 2.8.0 > > Attachments: HADOOP-12816.001.patch > > > We've had difficulty probing the root cause of performance slowdown with > in-transit encryption using AES-NI. We finally found the root cause was the > Hadoop client did not configure encryption properties correctly, so they did > not negotiate AES cipher suite when creating an encrypted stream pair, > despite the server (a data node) supports it. Existing debug message did not > help. We saw debug message "Server using cipher suite AES/CTR/NoPadding" on > the same data node, but that refers to the communication with other data > nodes. > It would be really helpful to log a debug message if a SASL server configures > AES cipher suite, but the SASL client doesn't, or vice versa. This debug > message should also log the client address to differentiate it from other > stream pairs. > More over, the debug message "Server using cipher suite AES/CTR/NoPadding" > should also be extended to include the client's address. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-9890: -- Attachment: HDFS-9890.HDFS-8707.013.patch HDFS-9890.HDFS-8707.013.patch debug patch that set random seed to constant value and see if the previous errors still occur constantly. Also added a test where file event callback is a no-op to compare. > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-9890.HDFS-8707.000.patch, > HDFS-9890.HDFS-8707.001.patch, HDFS-9890.HDFS-8707.002.patch, > HDFS-9890.HDFS-8707.003.patch, HDFS-9890.HDFS-8707.004.patch, > HDFS-9890.HDFS-8707.005.patch, HDFS-9890.HDFS-8707.006.patch, > HDFS-9890.HDFS-8707.007.patch, HDFS-9890.HDFS-8707.008.patch, > HDFS-9890.HDFS-8707.009.patch, HDFS-9890.HDFS-8707.010.patch, > HDFS-9890.HDFS-8707.011.patch, HDFS-9890.HDFS-8707.012.patch, > HDFS-9890.HDFS-8707.012.patch, HDFS-9890.HDFS-8707.013.patch, > hs_err_pid26832.log, hs_err_pid4944.log > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359685#comment-15359685 ] James Clampffer commented on HDFS-10441: Thanks for the very detailed feedback Bob. You caught a lot of issues that slipped my mind + a few I didn't really think about. As far as bigger issues go (other points are also important to me, just want to give comments on these before the next patch): bq. Do we pass information in the data pointer for the NN failover event? If so, document it in events.h I think I accidentally left this out. Needs to be fixed, or I'll take it out of the header and add another jira for that. bq. rpc_connection.cc: HandleRpcResponse should push req back to the head of the queue; alternately, don't dequeue it if we got a standby exception. Need to take a better look at how to best do this. Will fix. bq. If HandleRpcResponse gets a kStandbyException, will CommsError be called twice (once in HandleRpcResponse and again in OnRecvComplete)? Yes, and I haven't figured out why yet, I should be able to track it down quickly now that I'm more familiar with the RPC retry code. It doesn't seem to break things but it sure makes reading the logs more confusing than they need to be. bq. rpc_engine.cc: let's use both namenodes if servers.size() >= 2 rather than just bailing out. I'm torn on this one. As far as I know you'd need to do a bit of hacking to get more than two nodes working in HA. It seems like it's best to cover the common case here. How about allowing more than 2 nodes but issuing warnings that say that your cluster probably isn't going to act as you'd expect based on the config file and that we discarded the extra nodes? bq. rpc_engine.h: IsCurrentActive/IsCurrentStandby are dangerous as designed: they're asking for race conditions as we acquire the lock, check, release the lock, then take action. Just before we take action, someone else could change the value You're totally right here. I didn't want to implement a mechanism to allow other things to lock the state of that structure for an arbitrary amount of time (seemed like it would be asking for a deadlock). I think the best fix here is to make them private to the NANamenodeTracker. bq. rpc_engine.cc: Remove RpcEngine::Start instead of deprecating it Yep, I left that in to see if someone could think of a use for a hook that got called once the IoService was known to be running. If you can't think of any I'll be happy to get rid of some dead code. bq. Don't forget to file bugs to handle more than two namenodes. Sure. I was thinking of holding off until that became widely supported but it won't hurt to plan ahead. As a general warning, I don't plan to have much internet access for the next week. Once I do I'll address the rest of the comments and get a new patch up that includes the rest of your feedback. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5086) Support RPCSEC_GSS authentication in NFSv3 gateway
[ https://issues.apache.org/jira/browse/HDFS-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359681#comment-15359681 ] John Zhuge commented on HDFS-5086: -- Hi [~jingzhao] Is your patch working to some degree? All code written from scratch? Thanks, John > Support RPCSEC_GSS authentication in NFSv3 gateway > -- > > Key: HDFS-5086 > URL: https://issues.apache.org/jira/browse/HDFS-5086 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0-alpha1 >Reporter: Brandon Li >Assignee: Jing Zhao > Attachments: HDFS-5086.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10467) Router-based HDFS federation
[
https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359625#comment-15359625
]
Hadoop QA commented on HDFS-10467:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 49s{color}
| {color:red} root generated 3 new + 706 unchanged - 2 fixed = 709 total (was
708) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 48s{color} | {color:orange} root: The patch generated 478 new + 1183
unchanged - 5 fixed = 1661 total (was 1188) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 4 new + 74 unchanged - 1 fixed =
78 total (was 75) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 36 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
7s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 37 new + 0
unchanged - 0 fixed = 37 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
1s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 4 new + 7
unchanged - 0 fixed = 11 total (was 7) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
48s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 11 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| |
org.apache.hadoop.hdfs.server.federation.locator.PathTreeNode.toString(int)
concatenates strings using + in a loop At PathTreeNode.java:in a loop At
PathTreeNode.java:[line 130] |
| | Unread field:StateStoreMetrics.java:[line 56] |
| | Synchronization performed on java.util.concurrent.CopyOnWriteArrayList in
[jira] [Commented] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[
https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359622#comment-15359622
]
Hadoop QA commented on HDFS-9890:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
26s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
23s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
7s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
1s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 40s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed CTEST tests |
test_libhdfs_threaded_hdfspp_test_shim_static |
| JDK v1.7.0_101 Failed CTEST tests |
test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815812/HDFS-9890.HDFS-8707.012.patch
|
| JIRA Issue | HDFS-9890 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 5b8977358820 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / d643d8c |
|
[jira] [Updated] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-9890: -- Attachment: HDFS-9890.HDFS-8707.012.patch > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-9890.HDFS-8707.000.patch, > HDFS-9890.HDFS-8707.001.patch, HDFS-9890.HDFS-8707.002.patch, > HDFS-9890.HDFS-8707.003.patch, HDFS-9890.HDFS-8707.004.patch, > HDFS-9890.HDFS-8707.005.patch, HDFS-9890.HDFS-8707.006.patch, > HDFS-9890.HDFS-8707.007.patch, HDFS-9890.HDFS-8707.008.patch, > HDFS-9890.HDFS-8707.009.patch, HDFS-9890.HDFS-8707.010.patch, > HDFS-9890.HDFS-8707.011.patch, HDFS-9890.HDFS-8707.012.patch, > HDFS-9890.HDFS-8707.012.patch, hs_err_pid26832.log, hs_err_pid4944.log > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[
https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359507#comment-15359507
]
Hadoop QA commented on HDFS-9890:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
6s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
19s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
22s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
44s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
47s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 44s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed CTEST tests |
test_libhdfs_threaded_hdfspp_test_shim_static |
| | test_hdfs_ext_hdfspp_test_shim_static |
| JDK v1.7.0_101 Failed CTEST tests |
test_libhdfs_mini_stress_hdfspp_test_shim_static |
| | test_hdfs_ext_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815797/HDFS-9890.HDFS-8707.012.patch
|
| JIRA Issue | HDFS-9890 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 9813efd1cb70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-10567) Improve plan command help message
[
https://issues.apache.org/jira/browse/HDFS-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359496#comment-15359496
]
Hadoop QA commented on HDFS-10567:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:85209cc |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815788/HDFS-10567-HDFS-10576.001.patch
|
| JIRA Issue | HDFS-10567 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 87c751b085e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c25021f |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15962/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15962/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15962/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve plan command help message
> -
>
[jira] [Updated] (HDFS-10467) Router-based HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10467: --- Attachment: HDFS-10467.PoC.001.patch Refactored state store and added more unit tests. > Router-based HDFS federation > > > Key: HDFS-10467 > URL: https://issues.apache.org/jira/browse/HDFS-10467 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 2.7.2 >Reporter: Inigo Goiri > Attachments: HDFS Router Federation.pdf, HDFS-10467.PoC.001.patch, > HDFS-10467.PoC.patch, HDFS-Router-Federation-Prototype.patch > > > Add a Router to provide a federated view of multiple HDFS clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaowei Zhu updated HDFS-9890: -- Attachment: HDFS-9890.HDFS-8707.012.patch HDFS-9890.HDFS-8707.012.patch silents compile warnings in hdfspp_mini_dfs.h. > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-9890.HDFS-8707.000.patch, > HDFS-9890.HDFS-8707.001.patch, HDFS-9890.HDFS-8707.002.patch, > HDFS-9890.HDFS-8707.003.patch, HDFS-9890.HDFS-8707.004.patch, > HDFS-9890.HDFS-8707.005.patch, HDFS-9890.HDFS-8707.006.patch, > HDFS-9890.HDFS-8707.007.patch, HDFS-9890.HDFS-8707.008.patch, > HDFS-9890.HDFS-8707.009.patch, HDFS-9890.HDFS-8707.010.patch, > HDFS-9890.HDFS-8707.011.patch, HDFS-9890.HDFS-8707.012.patch, > hs_err_pid26832.log, hs_err_pid4944.log > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10572) Fix TestOfflineEditsViewer#testGenerated
[
https://issues.apache.org/jira/browse/HDFS-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-10572:
-
Component/s: (was: newbie)
> Fix TestOfflineEditsViewer#testGenerated
>
>
> Key: HDFS-10572
> URL: https://issues.apache.org/jira/browse/HDFS-10572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiaoyu Yao
>Priority: Blocker
>
> The test has been failing consistently on trunk recently. This ticket is open
> to fix this test to avoid false alarm on Jenkins. Figure out which recent
> commit caused this failure can be a good start.
>
> {code}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.646 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> testGenerated(org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer)
> Time elapsed: 3.623 sec <<< FAILURE!
> java.lang.AssertionError: Generated edits and reparsed (bin to XML to bin)
> should be same
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer.testGenerated(TestOfflineEditsViewer.java:125)
> Results :
> Failed tests:
> TestOfflineEditsViewer.testGenerated:125 Generated edits and reparsed (bin
> to XML to bin) should be same
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10572) Fix TestOfflineEditsViewer#testGenerated
[
https://issues.apache.org/jira/browse/HDFS-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359347#comment-15359347
]
Akira Ajisaka commented on HDFS-10572:
--
Raise the priority to blocker because the test failure means OfflineEditsViewer
is really broken.
> Fix TestOfflineEditsViewer#testGenerated
>
>
> Key: HDFS-10572
> URL: https://issues.apache.org/jira/browse/HDFS-10572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: newbie, test
>Reporter: Xiaoyu Yao
>Priority: Blocker
>
> The test has been failing consistently on trunk recently. This ticket is open
> to fix this test to avoid false alarm on Jenkins. Figure out which recent
> commit caused this failure can be a good start.
>
> {code}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.646 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> testGenerated(org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer)
> Time elapsed: 3.623 sec <<< FAILURE!
> java.lang.AssertionError: Generated edits and reparsed (bin to XML to bin)
> should be same
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer.testGenerated(TestOfflineEditsViewer.java:125)
> Results :
> Failed tests:
> TestOfflineEditsViewer.testGenerated:125 Generated edits and reparsed (bin
> to XML to bin) should be same
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10572) Fix TestOfflineEditsViewer#testGenerated
[
https://issues.apache.org/jira/browse/HDFS-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-10572:
--
Assignee: (was: Hanisha Koneru)
> Fix TestOfflineEditsViewer#testGenerated
>
>
> Key: HDFS-10572
> URL: https://issues.apache.org/jira/browse/HDFS-10572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: newbie, test
>Reporter: Xiaoyu Yao
>Priority: Blocker
>
> The test has been failing consistently on trunk recently. This ticket is open
> to fix this test to avoid false alarm on Jenkins. Figure out which recent
> commit caused this failure can be a good start.
>
> {code}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.646 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> testGenerated(org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer)
> Time elapsed: 3.623 sec <<< FAILURE!
> java.lang.AssertionError: Generated edits and reparsed (bin to XML to bin)
> should be same
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer.testGenerated(TestOfflineEditsViewer.java:125)
> Results :
> Failed tests:
> TestOfflineEditsViewer.testGenerated:125 Generated edits and reparsed (bin
> to XML to bin) should be same
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10572) Fix TestOfflineEditsViewer#testGenerated
[
https://issues.apache.org/jira/browse/HDFS-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-10572:
-
Priority: Blocker (was: Major)
> Fix TestOfflineEditsViewer#testGenerated
>
>
> Key: HDFS-10572
> URL: https://issues.apache.org/jira/browse/HDFS-10572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: newbie, test
>Reporter: Xiaoyu Yao
>Assignee: Hanisha Koneru
>Priority: Blocker
>
> The test has been failing consistently on trunk recently. This ticket is open
> to fix this test to avoid false alarm on Jenkins. Figure out which recent
> commit caused this failure can be a good start.
>
> {code}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.646 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
> testGenerated(org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer)
> Time elapsed: 3.623 sec <<< FAILURE!
> java.lang.AssertionError: Generated edits and reparsed (bin to XML to bin)
> should be same
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer.testGenerated(TestOfflineEditsViewer.java:125)
> Results :
> Failed tests:
> TestOfflineEditsViewer.testGenerated:125 Generated edits and reparsed (bin
> to XML to bin) should be same
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10567) Improve plan command help message
[
https://issues.apache.org/jira/browse/HDFS-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359344#comment-15359344
]
Xiaobing Zhou commented on HDFS-10567:
--
Thank you [~eddyxu] for review. I posted patch v001.
1. make it clear it's local path.
2. added bandwidth unit.
3. 'disk data density' is defined in [HDFS-1312
proposal|https://issues.apache.org/jira/secure/attachment/12755226/disk-balancer-proposal.pdf].
> Improve plan command help message
> -
>
> Key: HDFS-10567
> URL: https://issues.apache.org/jira/browse/HDFS-10567
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10567-HDFS-10576.001.patch,
> HDFS-10567-HDFS-1312.000.patch
>
>
> {code}
> --bandwidth Maximum disk bandwidth to be consumed by
> diskBalancer. e.g. 10
> --maxerror Describes how many errors can be
> tolerated while copying between a pair
> of disks.
> --outFile to write output to, if not
> specified defaults will be used.
> --plan creates a plan for datanode.
> --thresholdPercentagePercentage skew that wetolerate before
> diskbalancer starts working e.g. 10
> --v Print out the summary of the plan on
> console
> {code}
> We should
> * Put the unit into {{--bandwidth}}, or its help message. Is it an integer or
> float / double number? Not clear in CLI message.
> * Give more details about {{--plan}}. It is not clear what the {{}} is
> for.
> * {{--thresholdPercentage}}, has typo {{wetolerate}} in the error message.
> Also it needs to indicated that it is the difference between space
> utilization between two disks / volumes. Is it an integer or float / double
> number?
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10512) VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
[
https://issues.apache.org/jira/browse/HDFS-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359340#comment-15359340
]
Akira Ajisaka commented on HDFS-10512:
--
Thanks [~jojochuang] for updating the patch! Would you document that
{{volumes}} must not be null in {{reportBadBlocks(ExtendedBlock block,
FsVolumeSpi volume)}}?
In addition, can we add a regression test for this issue?
> VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
> --
>
> Key: HDFS-10512
> URL: https://issues.apache.org/jira/browse/HDFS-10512
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10512.001.patch, HDFS-10512.002.patch,
> HDFS-10512.004.patch
>
>
> VolumeScanner may terminate due to unexpected NullPointerException thrown in
> {{DataNode.reportBadBlocks()}}. This is different from HDFS-8850/HDFS-9190
> I observed this bug in a production CDH 5.5.1 cluster and the same bug still
> persist in upstream trunk.
> {noformat}
> 2016-04-07 20:30:53,830 WARN
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Reporting bad
> BP-1800173197-10.204.68.5-125156296:blk_1170134484_96468685 on /dfs/dn
> 2016-04-07 20:30:53,831 ERROR
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/dfs/dn,
> DS-89b72832-2a8c-48f3-8235-48e6c5eb5ab3) exiting because of exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.reportBadBlocks(DataNode.java:1018)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner$ScanResultHandler.handle(VolumeScanner.java:287)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:443)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:547)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:621)
> 2016-04-07 20:30:53,832 INFO
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/dfs/dn,
> DS-89b72832-2a8c-48f3-8235-48e6c5eb5ab3) exiting.
> {noformat}
> I think the NPE comes from the volume variable in the following code snippet.
> Somehow the volume scanner know the volume, but the datanode can not lookup
> the volume using the block.
> {code}
> public void reportBadBlocks(ExtendedBlock block) throws IOException{
> BPOfferService bpos = getBPOSForBlock(block);
> FsVolumeSpi volume = getFSDataset().getVolume(block);
> bpos.reportBadBlocks(
> block, volume.getStorageID(), volume.getStorageType());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10567) Improve plan command help message
[
https://issues.apache.org/jira/browse/HDFS-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou updated HDFS-10567:
-
Attachment: HDFS-10567-HDFS-10576.001.patch
> Improve plan command help message
> -
>
> Key: HDFS-10567
> URL: https://issues.apache.org/jira/browse/HDFS-10567
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Lei (Eddy) Xu
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10567-HDFS-10576.001.patch,
> HDFS-10567-HDFS-1312.000.patch
>
>
> {code}
> --bandwidth Maximum disk bandwidth to be consumed by
> diskBalancer. e.g. 10
> --maxerror Describes how many errors can be
> tolerated while copying between a pair
> of disks.
> --outFile to write output to, if not
> specified defaults will be used.
> --plan creates a plan for datanode.
> --thresholdPercentagePercentage skew that wetolerate before
> diskbalancer starts working e.g. 10
> --v Print out the summary of the plan on
> console
> {code}
> We should
> * Put the unit into {{--bandwidth}}, or its help message. Is it an integer or
> float / double number? Not clear in CLI message.
> * Give more details about {{--plan}}. It is not clear what the {{}} is
> for.
> * {{--thresholdPercentage}}, has typo {{wetolerate}} in the error message.
> Also it needs to indicated that it is the difference between space
> utilization between two disks / volumes. Is it an integer or float / double
> number?
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10512) VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
[
https://issues.apache.org/jira/browse/HDFS-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359324#comment-15359324
]
Hadoop QA commented on HDFS-10512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:85209cc |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12815775/HDFS-10512.004.patch |
| JIRA Issue | HDFS-10512 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5406e35f7b21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c25021f |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15961/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15961/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15961/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
> --
>
> Key: HDFS-10512
> URL: https://issues.apache.org/jira/browse/HDFS-10512
> Project:
[jira] [Commented] (HDFS-6962) ACLs inheritance conflict with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359304#comment-15359304 ] John Zhuge commented on HDFS-6962: -- Looking into the unit test failures. They are regressions. > ACLs inheritance conflict with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10512) VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
[
https://issues.apache.org/jira/browse/HDFS-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10512:
---
Attachment: HDFS-10512.004.patch
Attached a patch based on Yiqun's original patch. Also updated the original
reportBadBlocks to invoke the overloaded reportBadBlocks.
Additionally, changed the caller of {{reportBadBlocks(ExtendedBlock block))}}
to use {{reportBadBlocks(ExtendedBlock block, FsVolumeSpi volume)}} to avoid
potential race condition.
> VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
> --
>
> Key: HDFS-10512
> URL: https://issues.apache.org/jira/browse/HDFS-10512
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10512.001.patch, HDFS-10512.002.patch,
> HDFS-10512.004.patch
>
>
> VolumeScanner may terminate due to unexpected NullPointerException thrown in
> {{DataNode.reportBadBlocks()}}. This is different from HDFS-8850/HDFS-9190
> I observed this bug in a production CDH 5.5.1 cluster and the same bug still
> persist in upstream trunk.
> {noformat}
> 2016-04-07 20:30:53,830 WARN
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Reporting bad
> BP-1800173197-10.204.68.5-125156296:blk_1170134484_96468685 on /dfs/dn
> 2016-04-07 20:30:53,831 ERROR
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/dfs/dn,
> DS-89b72832-2a8c-48f3-8235-48e6c5eb5ab3) exiting because of exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.reportBadBlocks(DataNode.java:1018)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner$ScanResultHandler.handle(VolumeScanner.java:287)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:443)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:547)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:621)
> 2016-04-07 20:30:53,832 INFO
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/dfs/dn,
> DS-89b72832-2a8c-48f3-8235-48e6c5eb5ab3) exiting.
> {noformat}
> I think the NPE comes from the volume variable in the following code snippet.
> Somehow the volume scanner know the volume, but the datanode can not lookup
> the volume using the block.
> {code}
> public void reportBadBlocks(ExtendedBlock block) throws IOException{
> BPOfferService bpos = getBPOSForBlock(block);
> FsVolumeSpi volume = getFSDataset().getVolume(block);
> bpos.reportBadBlocks(
> block, volume.getStorageID(), volume.getStorageType());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10586) Erasure Code misfunctions when 3 DataNode down
[ https://issues.apache.org/jira/browse/HDFS-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358732#comment-15358732 ] gao shan commented on HDFS-10586: - Other data nodes show a lot of Premature EOF errors too, although I have set dfs.datanode.max.transfer.threads to 8192 and added " * - nofile 655360, " in the /etc/security/limits.conf,. 2016-06-28 10:45:16,366 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-257845079-172.16.1.85-1466418599731:blk_-9223372036854736697_7811 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:204) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:522) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:923) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:846) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:171) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:105) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:289) at java.lang.Thread.run(Thread.java:745) 2016-06-28 10:45:16,366 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-257845079-172.16.1.85-1466418599731:blk_-9223372036854736697_7811, type=LAST_IN_PIPELINE: Thread is interrupted. 2016-06-28 10:45:16,366 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-257845079-172.16.1.85-1466418599731:blk_-9223372036854736697_7811, type=LAST_IN_PIPELINE terminating 2016-06-28 10:45:16,367 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-257845079-172.16.1.85-1466418599731:blk_-9223372036854736697_7811 received exception java.io.IOException: Premature EOF from inputStream 2016-06-28 10:45:16,367 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: host-172-16-1-89:9866:DataXceiver error processing WRITE_BLOCK operation src: /172.16.1.88:22616 dst: /172.16.1.89:9866 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:204) > Erasure Code misfunctions when 3 DataNode down > -- > > Key: HDFS-10586 > URL: https://issues.apache.org/jira/browse/HDFS-10586 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 > Environment: 9 DataNode and 1 NameNode,Erasured code policy is > set as "6--3", When 3 DataNode down, erasured code fails and an exception > is thrown >Reporter: gao shan > > The following is the steps to reproduce: > 1) hadoop fs -mkdir /ec > 2) set erasured code policy as "6-3" > 3) "write" data by : > time hadoop jar > /opt/hadoop/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar > TestDFSIO -D test.build.data=/ec -write -nrFiles 30 -fileSize 12288 > -bufferSize 1073741824 > 4) Manually down 3 nodes. Kill the threads of "datanode" and "nodemanager" > in 3 DataNode. > 5) By using erasured code to "read" data by: > time hadoop jar > /opt/hadoop/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar > TestDFSIO -D test.build.data=/ec -read -nrFiles 30 -fileSize 12288 > -bufferSize 1073741824 > then the failure occurs and the exception is thrown as: > INFO mapreduce.Job: Task Id : attempt_1465445965249_0008_m_34_2, Status : > FAILED > Error: java.io.IOException: 4 missing blocks, the stripe is: Offset=0, > length=8388608, fetchedChunksNum=0, missingChunksNum=4 > at > org.apache.hadoop.hdfs.DFSStripedInputStream$StripeReader.checkMissingBlocks(DFSStripedInputStream.java:614) > at > org.apache.hadoop.hdfs.DFSStripedInputStream$StripeReader.readParityChunks(DFSStripedInputStream.java:647) > at > org.apache.hadoop.hdfs.DFSStripedInputStream$StripeReader.readStripe(DFSStripedInputStream.java:762) > at > org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:316) > at > org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:450) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:941) > at java.io.DataInputStream.read(DataInputStream.java:149) > at org.apache.hadoop.fs.TestDFSIO$ReadMapper.doIO(TestDFSIO.java:531) >
[jira] [Commented] (HDFS-10586) Erasure Code misfunctions when 3 DataNode down
[
https://issues.apache.org/jira/browse/HDFS-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358715#comment-15358715
]
gao shan commented on HDFS-10586:
-
I check the log, find the following errors. All the datanodes are alive, but
what's the meaning of the WARN "Failed to find datanode " ? 172.16.1.85 is
the namenode, The other IPs are for datanodes.
2016-06-28 10:44:57,995 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:57,996 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:57,996 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073744759_7794, replicas=172.16.1.143:9866, 172.16.1.92:9866,
172.16.1.87:9866 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.jar
2016-06-28 10:44:58,233 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.jar is closed
by DFSClient_NONMAPREDUCE_1881763906_1
2016-06-28 10:44:58,239 INFO
org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication from
3 to 10 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.jar
2016-06-28 10:44:58,365 INFO
org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication from
3 to 10 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.split
2016-06-28 10:44:58,368 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:58,368 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:58,369 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073744760_7795, replicas=172.16.1.87:9866, 172.16.1.90:9866,
172.16.1.91:9866, 172.16.1.88:9866, 172.16.1.89:9866, 172.16.1.86:9866,
172.16.1.93:9866, 172.16.1.92:9866, 172.16.1.143:9866 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.split
2016-06-28 10:44:58,541 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.split is
closed by DFSClient_NONMAPREDUCE_1881763906_1
2016-06-28 10:44:58,548 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:58,549 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:58,549 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073744761_7796, replicas=172.16.1.93:9866, 172.16.1.88:9866,
172.16.1.143:9866 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.splitmetainfo
2016-06-28 10:44:58,632 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.splitmetainfo
is closed by DFSClient_NONMAPREDUCE_1881763906_1
2016-06-28 10:44:58,773 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:58,773 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:44:58,774 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073744762_7797, replicas=172.16.1.91:9866, 172.16.1.143:9866,
172.16.1.86:9866 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.xml
2016-06-28 10:44:58,857 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job.xml is closed
by DFSClient_NONMAPREDUCE_1881763906_1
2016-06-28 10:45:06,285 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:45:06,285 WARN org.apache.hadoop.net.NetworkTopology: Failed to
find datanode (scope="" excludedScope="/default-rack").
2016-06-28 10:45:06,285 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073744763_7798, replicas=172.16.1.90:9866, 172.16.1.86:9866,
172.16.1.91:9866 for
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job_1467124628054_0001_1_conf.xml
2016-06-28 10:45:06,353 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile:
/tmp/hadoop-yarn/staging/root/.staging/job_1467124628054_0001/job_1467124628054_0001_1_conf.xml
is closed by DFSClient_NONMAPREDUCE_2078921355_1
2016-06-28 10:45:12,227 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
place enough replicas, still in need of 1 to reach 9 (unavailableStorages=[],
storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK],
creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more
information, please enable DEBUG log level on
[jira] [Commented] (HDFS-10512) VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
[
https://issues.apache.org/jira/browse/HDFS-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358670#comment-15358670
]
Yiqun Lin commented on HDFS-10512:
--
Thanks Wei-Chiu for providing the patch, it also looks good to me. You can
assign this jira to yourself and do a quick fix, I don't mind that. Thanks
Wei-Chiu again for much work for this issue.
> VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
> --
>
> Key: HDFS-10512
> URL: https://issues.apache.org/jira/browse/HDFS-10512
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10512.001.patch, HDFS-10512.002.patch
>
>
> VolumeScanner may terminate due to unexpected NullPointerException thrown in
> {{DataNode.reportBadBlocks()}}. This is different from HDFS-8850/HDFS-9190
> I observed this bug in a production CDH 5.5.1 cluster and the same bug still
> persist in upstream trunk.
> {noformat}
> 2016-04-07 20:30:53,830 WARN
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Reporting bad
> BP-1800173197-10.204.68.5-125156296:blk_1170134484_96468685 on /dfs/dn
> 2016-04-07 20:30:53,831 ERROR
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/dfs/dn,
> DS-89b72832-2a8c-48f3-8235-48e6c5eb5ab3) exiting because of exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.reportBadBlocks(DataNode.java:1018)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner$ScanResultHandler.handle(VolumeScanner.java:287)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:443)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:547)
> at
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:621)
> 2016-04-07 20:30:53,832 INFO
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/dfs/dn,
> DS-89b72832-2a8c-48f3-8235-48e6c5eb5ab3) exiting.
> {noformat}
> I think the NPE comes from the volume variable in the following code snippet.
> Somehow the volume scanner know the volume, but the datanode can not lookup
> the volume using the block.
> {code}
> public void reportBadBlocks(ExtendedBlock block) throws IOException{
> BPOfferService bpos = getBPOSForBlock(block);
> FsVolumeSpi volume = getFSDataset().getVolume(block);
> bpos.reportBadBlocks(
> block, volume.getStorageID(), volume.getStorageType());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
