[jira] [Updated] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-13147:
-
Attachment: HDFS-13147.002.patch
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch, HDFS-13147.002.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-13147:
-
Status: Patch Available (was: In Progress)
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch, HDFS-13147.002.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-13147:
-
Status: In Progress (was: Patch Available)
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch, HDFS-13147.002.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363523#comment-16363523
]
genericqa commented on HDFS-13147:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 31s{color} | {color:orange} root: The patch generated 5 new + 175 unchanged
- 2 fixed = 180 total (was 177) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
46s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
47s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 51s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
51s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13147 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910497/HDFS-13147.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 77038a895f48 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 332269d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs |
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363516#comment-16363516
]
Xiao Chen commented on HDFS-13040:
--
Patch 3 ready for review.
Cleaned up Istvan's test case, and verified the test fails-before, passes-after.
> Kerberized inotify client fails despite kinit properly
> --
>
> Key: HDFS-13040
> URL: https://issues.apache.org/jira/browse/HDFS-13040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
> Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch,
> HDFS-13040.03.patch, HDFS-13040.half.test.patch,
> TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java
>
>
> This issue is similar to HDFS-10799.
> HDFS-10799 turned out to be a client side issue where client is responsible
> for renewing kerberos ticket actively.
> However we found in a slightly setup even if client has valid Kerberos
> credentials, inotify still fails.
> Suppose client uses principal h...@example.com,
> namenode 1 uses server principal hdfs/nn1.example@example.com
> namenode 2 uses server principal hdfs/nn2.example@example.com
> *After Namenodes starts for longer than kerberos ticket lifetime*, the client
> fails with the following error:
> {noformat}
> 18/01/19 11:23:02 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS)
> cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We
> encountered an error reading
> https://nn2.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3,
>
> https://nn1.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3.
> During automatic edit log failover, we noticed that all of the remaining
> edit log streams are shorter than the current one! The best remaining edit
> log ends at transaction 8683, but we thought we could read up to transaction
> 8684. If you continue, metadata will be lost forever!
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
> {noformat}
> Typically if NameNode has an expired Kerberos ticket, the error handling for
> the typical edit log tailing would let NameNode to relogin with its own
> Kerberos principal. However, when inotify uses the same code path to retrieve
> edits, since the current user is the inotify client's principal, unless
> client uses the same principal as the NameNode, NameNode can't do it on
> behalf of the client.
> Therefore, a more appropriate approach is to use proxy user so that NameNode
> can retrieving edits on behalf of the client.
> I will attach a patch to fix it. This patch has been verified to work for a
> CDH5.10.2 cluster, however it seems impossible to craft a unit test for this
> fix because the way Hadoop UGI handles Kerberos credentials (I can't have a
> single process that logins as two Kerberos principals simultaneously and let
> them establish connection)
> A possible workaround is for the inotify client to use the active NameNode's
> server principal. However, that's not going to work when there's a
[jira] [Updated] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-13040:
-
Attachment: HDFS-13040.03.patch
> Kerberized inotify client fails despite kinit properly
> --
>
> Key: HDFS-13040
> URL: https://issues.apache.org/jira/browse/HDFS-13040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
> Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch,
> HDFS-13040.03.patch, HDFS-13040.half.test.patch,
> TestDFSInotifyEventInputStreamKerberized.java, TransactionReader.java
>
>
> This issue is similar to HDFS-10799.
> HDFS-10799 turned out to be a client side issue where client is responsible
> for renewing kerberos ticket actively.
> However we found in a slightly setup even if client has valid Kerberos
> credentials, inotify still fails.
> Suppose client uses principal h...@example.com,
> namenode 1 uses server principal hdfs/nn1.example@example.com
> namenode 2 uses server principal hdfs/nn2.example@example.com
> *After Namenodes starts for longer than kerberos ticket lifetime*, the client
> fails with the following error:
> {noformat}
> 18/01/19 11:23:02 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS)
> cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We
> encountered an error reading
> https://nn2.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3,
>
> https://nn1.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3.
> During automatic edit log failover, we noticed that all of the remaining
> edit log streams are shorter than the current one! The best remaining edit
> log ends at transaction 8683, but we thought we could read up to transaction
> 8684. If you continue, metadata will be lost forever!
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1490)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
> {noformat}
> Typically if NameNode has an expired Kerberos ticket, the error handling for
> the typical edit log tailing would let NameNode to relogin with its own
> Kerberos principal. However, when inotify uses the same code path to retrieve
> edits, since the current user is the inotify client's principal, unless
> client uses the same principal as the NameNode, NameNode can't do it on
> behalf of the client.
> Therefore, a more appropriate approach is to use proxy user so that NameNode
> can retrieving edits on behalf of the client.
> I will attach a patch to fix it. This patch has been verified to work for a
> CDH5.10.2 cluster, however it seems impossible to craft a unit test for this
> fix because the way Hadoop UGI handles Kerberos credentials (I can't have a
> single process that logins as two Kerberos principals simultaneously and let
> them establish connection)
> A possible workaround is for the inotify client to use the active NameNode's
> server principal. However, that's not going to work when there's a namenode
> failover, because then the client's principal will not be consistent with the
> active NN's one, and then
[jira] [Commented] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
[ https://issues.apache.org/jira/browse/HDFS-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363496#comment-16363496 ] Abhishek Bafna commented on HDFS-13140: --- [~jojochuang] Thanks for the feedback. I will update the Jira and patch with required information. > Snapshot based replication between HDFS and cloud using DistCp > -- > > Key: HDFS-13140 > URL: https://issues.apache.org/jira/browse/HDFS-13140 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna >Priority: Major > Attachments: HDFS-13140-00.patch > > > Snapshot based replication between HDFS and cloud using DistCp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363493#comment-16363493
]
genericqa commented on HDFS-13138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 7m
6s{color} | {color:red} Docker failed to build yetus/hadoop:ea57d10. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910504/HDFS-13138.002.branch-2.7.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23058/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> webhdfs of federated namenode does not work properly
> -
>
> Key: HDFS-13138
> URL: https://issues.apache.org/jira/browse/HDFS-13138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1, 3.0.0
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: HDFS-13138.001.branch-2.7.patch, HDFS-13138.001.patch,
> HDFS-13138.002.branch-2.7.patch, HDFS-13138.002.patch
>
>
> my cluster has multiple namenodes using HDFS Federation.
> webhdfs that is not defaultFS does not work properly.
> when I uploaded to non defaultFS namenode using webhdfs.
> uploaded file was founded at defaultFS namenode.
>
> I think root cause is that
> clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462]
>
> {code:java}
> /**
>* Set the namenode address that will be used by clients to access this
>* namenode or name service. This needs to be called before the config
>* is overriden.
>*/
> public void setClientNamenodeAddress(Configuration conf) {
> String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
> if (nnAddr == null) {
> // default fs is not set.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
> URI nnUri = URI.create(nnAddr);
> String nnHost = nnUri.getHost();
> if (nnHost == null) {
> clientNamenodeAddress = null;
> return;
> }
> if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
> // host name is logical
> clientNamenodeAddress = nnHost;
> } else if (nnUri.getPort() > 0) {
> // physical address with a valid port
> clientNamenodeAddress = nnUri.getAuthority();
> } else {
> // the port is missing or 0. Figure out real bind address later.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("Clients are to use {} to access"
> + " this namenode/service.", clientNamenodeAddress );
> }
> {code}
>
> so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter
> finally file was located namenode of fs,defaultFS
>
> workaround is
> configure fs.defaultFS of each namenode to its own nameservice.
> e.g.
> hdfs://ns1 has fs.defaultFS=hdfs://ns1
> hdfs://ns2 has fs.defaultFS=hdfs://ns2
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-1686) Federation: Add more Balancer tests with federation setting
[
https://issues.apache.org/jira/browse/HDFS-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363492#comment-16363492
]
genericqa commented on HDFS-1686:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 25 unchanged - 1 fixed = 26 total (was 26) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-1686 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910490/HDFS-1686.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 387d97cf88a4 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 332269d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23056/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23056/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test
[jira] [Updated] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated HDFS-13138:
---
Attachment: HDFS-13138.002.branch-2.7.patch
> webhdfs of federated namenode does not work properly
> -
>
> Key: HDFS-13138
> URL: https://issues.apache.org/jira/browse/HDFS-13138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1, 3.0.0
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: HDFS-13138.001.branch-2.7.patch, HDFS-13138.001.patch,
> HDFS-13138.002.branch-2.7.patch, HDFS-13138.002.patch
>
>
> my cluster has multiple namenodes using HDFS Federation.
> webhdfs that is not defaultFS does not work properly.
> when I uploaded to non defaultFS namenode using webhdfs.
> uploaded file was founded at defaultFS namenode.
>
> I think root cause is that
> clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462]
>
> {code:java}
> /**
>* Set the namenode address that will be used by clients to access this
>* namenode or name service. This needs to be called before the config
>* is overriden.
>*/
> public void setClientNamenodeAddress(Configuration conf) {
> String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
> if (nnAddr == null) {
> // default fs is not set.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
> URI nnUri = URI.create(nnAddr);
> String nnHost = nnUri.getHost();
> if (nnHost == null) {
> clientNamenodeAddress = null;
> return;
> }
> if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
> // host name is logical
> clientNamenodeAddress = nnHost;
> } else if (nnUri.getPort() > 0) {
> // physical address with a valid port
> clientNamenodeAddress = nnUri.getAuthority();
> } else {
> // the port is missing or 0. Figure out real bind address later.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("Clients are to use {} to access"
> + " this namenode/service.", clientNamenodeAddress );
> }
> {code}
>
> so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter
> finally file was located namenode of fs,defaultFS
>
> workaround is
> configure fs.defaultFS of each namenode to its own nameservice.
> e.g.
> hdfs://ns1 has fs.defaultFS=hdfs://ns1
> hdfs://ns2 has fs.defaultFS=hdfs://ns2
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363487#comment-16363487
]
KWON BYUNGCHANG commented on HDFS-13138:
initial patch crashed with hadoop.hdfs.web.TestWebHdfsFileSystemContract. I
fixed.
other test crash does not related with this issue.
> webhdfs of federated namenode does not work properly
> -
>
> Key: HDFS-13138
> URL: https://issues.apache.org/jira/browse/HDFS-13138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1, 3.0.0
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: HDFS-13138.001.branch-2.7.patch, HDFS-13138.001.patch,
> HDFS-13138.002.patch
>
>
> my cluster has multiple namenodes using HDFS Federation.
> webhdfs that is not defaultFS does not work properly.
> when I uploaded to non defaultFS namenode using webhdfs.
> uploaded file was founded at defaultFS namenode.
>
> I think root cause is that
> clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462]
>
> {code:java}
> /**
>* Set the namenode address that will be used by clients to access this
>* namenode or name service. This needs to be called before the config
>* is overriden.
>*/
> public void setClientNamenodeAddress(Configuration conf) {
> String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
> if (nnAddr == null) {
> // default fs is not set.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
> URI nnUri = URI.create(nnAddr);
> String nnHost = nnUri.getHost();
> if (nnHost == null) {
> clientNamenodeAddress = null;
> return;
> }
> if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
> // host name is logical
> clientNamenodeAddress = nnHost;
> } else if (nnUri.getPort() > 0) {
> // physical address with a valid port
> clientNamenodeAddress = nnUri.getAuthority();
> } else {
> // the port is missing or 0. Figure out real bind address later.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("Clients are to use {} to access"
> + " this namenode/service.", clientNamenodeAddress );
> }
> {code}
>
> so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter
> finally file was located namenode of fs,defaultFS
>
> workaround is
> configure fs.defaultFS of each namenode to its own nameservice.
> e.g.
> hdfs://ns1 has fs.defaultFS=hdfs://ns1
> hdfs://ns2 has fs.defaultFS=hdfs://ns2
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated HDFS-13138:
---
Attachment: HDFS-13138.002.patch
> webhdfs of federated namenode does not work properly
> -
>
> Key: HDFS-13138
> URL: https://issues.apache.org/jira/browse/HDFS-13138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1, 3.0.0
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: HDFS-13138.001.branch-2.7.patch, HDFS-13138.001.patch,
> HDFS-13138.002.patch
>
>
> my cluster has multiple namenodes using HDFS Federation.
> webhdfs that is not defaultFS does not work properly.
> when I uploaded to non defaultFS namenode using webhdfs.
> uploaded file was founded at defaultFS namenode.
>
> I think root cause is that
> clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462]
>
> {code:java}
> /**
>* Set the namenode address that will be used by clients to access this
>* namenode or name service. This needs to be called before the config
>* is overriden.
>*/
> public void setClientNamenodeAddress(Configuration conf) {
> String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
> if (nnAddr == null) {
> // default fs is not set.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
> URI nnUri = URI.create(nnAddr);
> String nnHost = nnUri.getHost();
> if (nnHost == null) {
> clientNamenodeAddress = null;
> return;
> }
> if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
> // host name is logical
> clientNamenodeAddress = nnHost;
> } else if (nnUri.getPort() > 0) {
> // physical address with a valid port
> clientNamenodeAddress = nnUri.getAuthority();
> } else {
> // the port is missing or 0. Figure out real bind address later.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("Clients are to use {} to access"
> + " this namenode/service.", clientNamenodeAddress );
> }
> {code}
>
> so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter
> finally file was located namenode of fs,defaultFS
>
> workaround is
> configure fs.defaultFS of each namenode to its own nameservice.
> e.g.
> hdfs://ns1 has fs.defaultFS=hdfs://ns1
> hdfs://ns2 has fs.defaultFS=hdfs://ns2
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11310) Reduce the performance impact of the balancer (trunk port)
[ https://issues.apache.org/jira/browse/HDFS-11310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HDFS-11310: -- Target Version/s: 3.2.0 (was: 3.1.0) > Reduce the performance impact of the balancer (trunk port) > -- > > Key: HDFS-11310 > URL: https://issues.apache.org/jira/browse/HDFS-11310 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Daryn Sharp >Priority: Critical > > HDFS-7967 introduced a highly performant balancer getBlocks() query that > scales to large/dense clusters. The simple design implementation depends on > the triplets data structure. HDFS-9260 removed the triplets which > fundamentally changes the implementation. Either that patch must be reverted > or the getBlocks() patch needs reimplementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12049) Recommissioning live nodes stalls the NN
[ https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HDFS-12049: -- Target Version/s: 3.2.0 (was: 3.1.0) > Recommissioning live nodes stalls the NN > > > Key: HDFS-12049 > URL: https://issues.apache.org/jira/browse/HDFS-12049 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Priority: Critical > > A node refresh will recommission included nodes that are alive and in > decommissioning or decommissioned state. The recommission will scan all > blocks on the node, find over replicated blocks, chose an excess, queue an > invalidate. > The process is expensive and worsened by overhead of storage types (even when > not in use). It can be especially devastating because the write lock is held > for the entire node refresh. _Recommissioning 67 nodes with ~500k > blocks/node stalled rpc services for over 4 mins._ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12452) TestDataNodeVolumeFailureReporting fails in trunk Jenkins runs
[
https://issues.apache.org/jira/browse/HDFS-12452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HDFS-12452:
--
Target Version/s: 3.2.0 (was: 3.1.0)
> TestDataNodeVolumeFailureReporting fails in trunk Jenkins runs
> --
>
> Key: HDFS-12452
> URL: https://issues.apache.org/jira/browse/HDFS-12452
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Arpit Agarwal
>Priority: Critical
> Labels: flaky-test
>
> TestDataNodeVolumeFailureReporting#testSuccessiveVolumeFailures fails
> frequently in Jenkins runs but it passes locally on my dev machine.
> e.g.
> https://builds.apache.org/job/PreCommit-HDFS-Build/21134/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/
> {code}
> Error Message
> test timed out after 12 milliseconds
> Stacktrace
> java.lang.Exception: test timed out after 12 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.waitReplication(DFSTestUtil.java:761)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting.testSuccessiveVolumeFailures(TestDataNodeVolumeFailureReporting.java:189)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-13147:
-
Status: Patch Available (was: In Progress)
Change to add -c argument for HEAD and TAIL.
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-13147:
-
Attachment: HDFS-13147.001.patch
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-13147 started by Jianfei Jiang.
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-13147 started by Jianfei Jiang.
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-13147) Support -c argument for DFS command head and tail
[
https://issues.apache.org/jira/browse/HDFS-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-13147 stopped by Jianfei Jiang.
> Support -c argument for DFS command head and tail
> -
>
> Key: HDFS-13147
> URL: https://issues.apache.org/jira/browse/HDFS-13147
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Minor
> Attachments: HDFS-13147.001.patch
>
>
> The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
> to improve it that the offset can be specified by user like Linux commands.
> Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13147) Support -c argument for DFS command head and tail
Jianfei Jiang created HDFS-13147:
Summary: Support -c argument for DFS command head and tail
Key: HDFS-13147
URL: https://issues.apache.org/jira/browse/HDFS-13147
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Jianfei Jiang
Assignee: Jianfei Jiang
The offset of command {{head}} and {{tail}} is hard coded as 1024 bytes. Goal
to improve it that the offset can be specified by user like Linux commands.
Then the commands will be more flexible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12548) HDFS Jenkins build is unstable on branch-2
[
https://issues.apache.org/jira/browse/HDFS-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HDFS-12548:
--
Target Version/s: 3.2.0
> HDFS Jenkins build is unstable on branch-2
> --
>
> Key: HDFS-12548
> URL: https://issues.apache.org/jira/browse/HDFS-12548
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.9.0
>Reporter: Rushabh S Shah
>Priority: Critical
>
> Feel free move the ticket to another project (e.g. infra).
> Recently I attached branch-2 patch while working on one jira
> [HDFS-12386|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180676=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180676]
> There were at-least 100 failed and timed out tests. I am sure they are not
> related to my patch.
> Also I came across another jira which was just a javadoc related change and
> there were around 100 failed tests.
> Below are the details for pre-commits that failed in branch-2
> 1 [HDFS-12386 attempt
> 1|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180069=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180069]
> {noformat}
> Ran on slave: asf912.gq1.ygridcore.net/H12
> Failed with following error message:
> Build timed out (after 300 minutes). Marking the build as aborted.
> Build was aborted
> Performing Post build task...
> {noformat}
> 2. [HDFS-12386 attempt
> 2|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180676=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180676]
> {noformat}
> Ran on slave: asf900.gq1.ygridcore.net
> Failed with following error message:
> FATAL: command execution failed
> Command close created at
> at hudson.remoting.Command.(Command.java:60)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
> at hudson.remoting.Channel.close(Channel.java:1281)
> at hudson.remoting.Channel.close(Channel.java:1263)
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
> Caused: hudson.remoting.Channel$OrderlyShutdown
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
> at hudson.remoting.Channel$1.handle(Channel.java:527)
> at
> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
> Caused: java.io.IOException: Backing channel 'H0' is disconnected.
> at
> hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
> at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
> at com.sun.proxy.$Proxy125.isAlive(Unknown Source)
> at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
> at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
> at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
> at hudson.model.Build$BuildExecution.build(Build.java:206)
> at hudson.model.Build$BuildExecution.doRun(Build.java:163)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
> at hudson.model.Run.execute(Run.java:1735)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:405)
> {noformat}
> 3. [HDFS-12531 attempt
> 1|https://issues.apache.org/jira/browse/HDFS-12531?focusedCommentId=16176493=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16176493]
> {noformat}
> Ran on slave: asf911.gq1.ygridcore.net
> Failed with following error message:
> FATAL: command execution failed
> Command close created at
> at hudson.remoting.Command.(Command.java:60)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
> at hudson.remoting.Channel.close(Channel.java:1281)
> at hudson.remoting.Channel.close(Channel.java:1263)
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
> Caused: hudson.remoting.Channel$OrderlyShutdown
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
> at hudson.remoting.Channel$1.handle(Channel.java:527)
> at
>
[jira] [Updated] (HDFS-12548) HDFS Jenkins build is unstable on branch-2
[
https://issues.apache.org/jira/browse/HDFS-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HDFS-12548:
--
Target Version/s: (was: 3.1.0)
> HDFS Jenkins build is unstable on branch-2
> --
>
> Key: HDFS-12548
> URL: https://issues.apache.org/jira/browse/HDFS-12548
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.9.0
>Reporter: Rushabh S Shah
>Priority: Critical
>
> Feel free move the ticket to another project (e.g. infra).
> Recently I attached branch-2 patch while working on one jira
> [HDFS-12386|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180676=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180676]
> There were at-least 100 failed and timed out tests. I am sure they are not
> related to my patch.
> Also I came across another jira which was just a javadoc related change and
> there were around 100 failed tests.
> Below are the details for pre-commits that failed in branch-2
> 1 [HDFS-12386 attempt
> 1|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180069=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180069]
> {noformat}
> Ran on slave: asf912.gq1.ygridcore.net/H12
> Failed with following error message:
> Build timed out (after 300 minutes). Marking the build as aborted.
> Build was aborted
> Performing Post build task...
> {noformat}
> 2. [HDFS-12386 attempt
> 2|https://issues.apache.org/jira/browse/HDFS-12386?focusedCommentId=16180676=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16180676]
> {noformat}
> Ran on slave: asf900.gq1.ygridcore.net
> Failed with following error message:
> FATAL: command execution failed
> Command close created at
> at hudson.remoting.Command.(Command.java:60)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
> at hudson.remoting.Channel.close(Channel.java:1281)
> at hudson.remoting.Channel.close(Channel.java:1263)
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
> Caused: hudson.remoting.Channel$OrderlyShutdown
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
> at hudson.remoting.Channel$1.handle(Channel.java:527)
> at
> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
> Caused: java.io.IOException: Backing channel 'H0' is disconnected.
> at
> hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
> at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
> at com.sun.proxy.$Proxy125.isAlive(Unknown Source)
> at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
> at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
> at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
> at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
> at hudson.model.Build$BuildExecution.build(Build.java:206)
> at hudson.model.Build$BuildExecution.doRun(Build.java:163)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
> at hudson.model.Run.execute(Run.java:1735)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:405)
> {noformat}
> 3. [HDFS-12531 attempt
> 1|https://issues.apache.org/jira/browse/HDFS-12531?focusedCommentId=16176493=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16176493]
> {noformat}
> Ran on slave: asf911.gq1.ygridcore.net
> Failed with following error message:
> FATAL: command execution failed
> Command close created at
> at hudson.remoting.Command.(Command.java:60)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
> at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
> at hudson.remoting.Channel.close(Channel.java:1281)
> at hudson.remoting.Channel.close(Channel.java:1263)
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
> Caused: hudson.remoting.Channel$OrderlyShutdown
> at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
> at hudson.remoting.Channel$1.handle(Channel.java:527)
> at
>
[jira] [Updated] (HDFS-1686) Federation: Add more Balancer tests with federation setting
[ https://issues.apache.org/jira/browse/HDFS-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-1686: - Attachment: HDFS-1686.01.patch > Federation: Add more Balancer tests with federation setting > --- > > Key: HDFS-1686 > URL: https://issues.apache.org/jira/browse/HDFS-1686 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer mover, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Bharat Viswanadham >Priority: Minor > Attachments: 4358946.patch, HDFS-1686.00.patch, HDFS-1686.01.patch, > h1686_20110303.patch > > > A test with 3 Namenodes and 4 Datanodes in startup, and then adding 2 new > Datanodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363347#comment-16363347
]
Aaron T. Myers commented on HDFS-12051:
---
Thanks for the followup, [~mi...@cloudera.com]. A few responses:
bq. It won't, but I can add this functionality.
Great, thanks.
bq. As you can see, this class is not really a static singleton. Its public API
is indeed a single static put() method, but inside there is a singleton
instance of NameCache, with its instance methods. Initially I didn't have this
singleton at all, and it indeed was an instance variable of FSNamesystem. But
later I found that there are several other places in the code where duplicate
byte[] arrays are generated, and where it would be very hard to pass this
instance variable. So I ended up with this static API, which makes it easier to
use NameCache anywhere in the code. But ability to test it is not compromised.
Sorry, I shouldn't have said the class was a singleton, but I think the point
remains. Especially in the context of tests, wherein we have potentially
several HA or federated NNs running within a single process, I worry that using
a singleton instance will cause some odd behavior. Passing it around may be
difficult, but do all the places in the code where you're adding calls to
{{NameCache}} perhaps have a reference to the {{FSNamesystem}}? If so, making
the {{NameCache}} a member of the {{FSNamesystem}} may make that not so hard to
deal with.
bq. Well, I can try that, but honestly, how paranoid should we be? In my
opinion, this code is simple enough to pass with a combination of unit tests
and some runs in the cluster.
I think we need to be diligent in confirming the correctness of this change, or
any change like this, as the ramifications of a bug here are both potentially
subtle and severe.
bq. The single findbugs issue has been already explained. It's legitimate, but
we intentionally do something that wouldn't be good in general (use a volatile
field and increment it without synchronization) just to enable some information
for testing without degrading performance in production. As for unit tests -
well, every time some different unit test fails, which makes me think that they
are flaky (I had same experience in the past with my other changes in HDFS). I
looked at them but couldn't see any obvious signs that the problems are related
to my code. There are timeouts and similar things that tend to happen in flaky
tests. Here I think I really need help from someone else in the HDFS team.
I think you're probably right that the failures are flaky tests - I just wanted
to make sure you or someone had taken a look at them and confirmed that.
bq. I don't think there is any problem here. We use the same formula to get the
next slot, and it wraps around the array boundary correctly. Take a look at the
test program below that uses the same formula, and its output:
Gotcha, makes sense. This behavior would be a great thing to ensure is in a
unit test.
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051-NameCache-Rewrite.pdf, HDFS-12051.01.patch,
> HDFS-12051.02.patch, HDFS-12051.03.patch, HDFS-12051.04.patch,
> HDFS-12051.05.patch, HDFS-12051.06.patch, HDFS-12051.07.patch,
> HDFS-12051.08.patch, HDFS-12051.09.patch, HDFS-12051.10.patch,
> HDFS-12051.11.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48,
[jira] [Commented] (HDFS-6962) ACL inheritance conflicts with umaskmode
[ https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363346#comment-16363346 ] Kaidi Zhao commented on HDFS-6962: -- I looked through the comments, but don't see any update on port this patch to hadoop 2.7.x (or similar). Just curious about it status. Thanks! > ACL inheritance conflicts with umaskmode > > > Key: HDFS-6962 > URL: https://issues.apache.org/jira/browse/HDFS-6962 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.4.1 > Environment: CentOS release 6.5 (Final) >Reporter: LINTE >Assignee: John Zhuge >Priority: Critical > Labels: hadoop, security > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-6962.001.patch, HDFS-6962.002.patch, > HDFS-6962.003.patch, HDFS-6962.004.patch, HDFS-6962.005.patch, > HDFS-6962.006.patch, HDFS-6962.007.patch, HDFS-6962.008.patch, > HDFS-6962.009.patch, HDFS-6962.010.patch, HDFS-6962.1.patch, > disabled_new_client.log, disabled_old_client.log, enabled_new_client.log, > enabled_old_client.log, run_compat_tests, run_unit_tests, test_plan.md > > > In hdfs-site.xml > > dfs.umaskmode > 027 > > 1/ Create a directory as superuser > bash# hdfs dfs -mkdir /tmp/ACLS > 2/ set default ACLs on this directory rwx access for group readwrite and user > toto > bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS > bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS > 3/ check ACLs /tmp/ACLS/ > bash# hdfs dfs -getfacl /tmp/ACLS/ > # file: /tmp/ACLS > # owner: hdfs > # group: hadoop > user::rwx > group::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > user::rwx | group::r-x | other::--- matches with the umaskmode defined in > hdfs-site.xml, everything ok ! > default:group:readwrite:rwx allow readwrite group with rwx access for > inhéritance. > default:user:toto:rwx allow toto user with rwx access for inhéritance. > default:mask::rwx inhéritance mask is rwx, so no mask > 4/ Create a subdir to test inheritance of ACL > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs > 5/ check ACLs /tmp/ACLS/hdfs > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs > # file: /tmp/ACLS/hdfs > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:r-x > group::r-x > group:readwrite:rwx #effective:r-x > mask::r-x > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > Here we can see that the readwrite group has rwx ACL bu only r-x is effective > because the mask is r-x (mask::r-x) in spite of default mask for inheritance > is set to default:mask::rwx on /tmp/ACLS/ > 6/ Modifiy hdfs-site.xml et restart namenode > > dfs.umaskmode > 010 > > 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode > bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 > 8/ Check ACL on /tmp/ACLS/hdfs2 > bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 > # file: /tmp/ACLS/hdfs2 > # owner: hdfs > # group: hadoop > user::rwx > user:toto:rwx #effective:rw- > group::r-x #effective:r-- > group:readwrite:rwx #effective:rw- > mask::rw- > other::--- > default:user::rwx > default:user:toto:rwx > default:group::r-x > default:group:readwrite:rwx > default:mask::rwx > default:other::--- > So HDFS masks the ACL value (user, group and other -- exepted the POSIX > owner -- ) with the group mask of dfs.umaskmode properties when creating > directory with inherited ACL. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363313#comment-16363313
]
Ajay Kumar commented on HDFS-13055:
---
[~arpitagarwal], Created new review request
[https://reviews.apache.org/r/65643/]
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch
>
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13145) SBN crash when transition to ANN with in-progress edit tailing enabled
[
https://issues.apache.org/jira/browse/HDFS-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363287#comment-16363287
]
Chao Sun commented on HDFS-13145:
-
Sounds great [~xkrogen]! Looking forward to the design. :)
> SBN crash when transition to ANN with in-progress edit tailing enabled
> --
>
> Key: HDFS-13145
> URL: https://issues.apache.org/jira/browse/HDFS-13145
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
>
> With edit log in-progress edit log tailing enabled, {{QuorumOutputStream}}
> will send two batches to JNs, one normal edit batch followed by a dummy batch
> to update the commit ID on JNs.
> {code}
> QuorumCall qcall = loggers.sendEdits(
> segmentTxId, firstTxToFlush,
> numReadyTxns, data);
> loggers.waitForWriteQuorum(qcall, writeTimeoutMs, "sendEdits");
>
> // Since we successfully wrote this batch, let the loggers know. Any
> future
> // RPCs will thus let the loggers know of the most recent transaction,
> even
> // if a logger has fallen behind.
> loggers.setCommittedTxId(firstTxToFlush + numReadyTxns - 1);
> // If we don't have this dummy send, committed TxId might be one-batch
> // stale on the Journal Nodes
> if (updateCommittedTxId) {
> QuorumCall fakeCall = loggers.sendEdits(
> segmentTxId, firstTxToFlush,
> 0, new byte[0]);
> loggers.waitForWriteQuorum(fakeCall, writeTimeoutMs, "sendEdits");
> }
> {code}
> Between each batch, it will wait for the JNs to reach a quorum. However, if
> the ANN crashes in between, then SBN will crash while transiting to ANN:
> {code}
> java.lang.IllegalStateException: Cannot start writing at txid 24312595802
> when there is a stream available for read: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.openForWrite(FSEditLog.java:329)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1196)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1839)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1707)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1622)
> at
> org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
> at
> org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:851)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:794)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2490)
> 2018-02-13 00:43:20,728 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> {code}
> This is because without the dummy batch, the {{commitTxnId}} will lag behind
> the {{endTxId}}, which caused the check in {{openForWrite}} to fail:
> {code}
> List streams = new ArrayList();
> journalSet.selectInputStreams(streams, segmentTxId, true, false);
> if (!streams.isEmpty()) {
> String error = String.format("Cannot start writing at txid %s " +
> "when there is a stream available for read: %s",
> segmentTxId, streams.get(0));
> IOUtils.cleanupWithLogger(LOG,
> streams.toArray(new EditLogInputStream[0]));
> throw new IllegalStateException(error);
> }
> {code}
> In our environment, this can be reproduced pretty consistently, which will
> leave the cluster with no running namenodes. Even though we are using a 2.8.2
> backport, I believe the same issue also exist in 3.0.x.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HDFS-13145) SBN crash when transition to ANN with in-progress edit tailing enabled
[
https://issues.apache.org/jira/browse/HDFS-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363272#comment-16363272
]
Erik Krogen commented on HDFS-13145:
I noticed this as well and don't really think that the approach of sending a
second "dummy" batch to update commitTxnId is the right one. It also gives
additional work to the ANN which I would like to avoid.
I am about to post a design for a "fast path" for an Observer to read from JNs
which can reduce the lag time for a txn to appear on Observer after going
through on the ANN down to a few ms. I have an idea for incorporating some
logic to remove the necessity for this dummy batch. Details coming soon...
> SBN crash when transition to ANN with in-progress edit tailing enabled
> --
>
> Key: HDFS-13145
> URL: https://issues.apache.org/jira/browse/HDFS-13145
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
>
> With edit log in-progress edit log tailing enabled, {{QuorumOutputStream}}
> will send two batches to JNs, one normal edit batch followed by a dummy batch
> to update the commit ID on JNs.
> {code}
> QuorumCall qcall = loggers.sendEdits(
> segmentTxId, firstTxToFlush,
> numReadyTxns, data);
> loggers.waitForWriteQuorum(qcall, writeTimeoutMs, "sendEdits");
>
> // Since we successfully wrote this batch, let the loggers know. Any
> future
> // RPCs will thus let the loggers know of the most recent transaction,
> even
> // if a logger has fallen behind.
> loggers.setCommittedTxId(firstTxToFlush + numReadyTxns - 1);
> // If we don't have this dummy send, committed TxId might be one-batch
> // stale on the Journal Nodes
> if (updateCommittedTxId) {
> QuorumCall fakeCall = loggers.sendEdits(
> segmentTxId, firstTxToFlush,
> 0, new byte[0]);
> loggers.waitForWriteQuorum(fakeCall, writeTimeoutMs, "sendEdits");
> }
> {code}
> Between each batch, it will wait for the JNs to reach a quorum. However, if
> the ANN crashes in between, then SBN will crash while transiting to ANN:
> {code}
> java.lang.IllegalStateException: Cannot start writing at txid 24312595802
> when there is a stream available for read: ..
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.openForWrite(FSEditLog.java:329)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1196)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1839)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1707)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1622)
> at
> org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
> at
> org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:851)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:794)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2490)
> 2018-02-13 00:43:20,728 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> {code}
> This is because without the dummy batch, the {{commitTxnId}} will lag behind
> the {{endTxId}}, which caused the check in {{openForWrite}} to fail:
> {code}
> List streams = new ArrayList();
> journalSet.selectInputStreams(streams, segmentTxId, true, false);
> if (!streams.isEmpty()) {
> String error = String.format("Cannot start writing at txid %s " +
> "when there is a stream available for read: %s",
> segmentTxId, streams.get(0));
> IOUtils.cleanupWithLogger(LOG,
>
[jira] [Commented] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363250#comment-16363250
]
Wei-Chiu Chuang commented on HDFS-13114:
Looks reasonable to me. Would it be easy to add a test or modify an existing
test?
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13109) Support fully qualified hdfs path in EZ commands
[
https://issues.apache.org/jira/browse/HDFS-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363244#comment-16363244
]
genericqa commented on HDFS-13109:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
65 unchanged - 0 fixed = 68 total (was 65) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13109 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910432/HDFS-13109.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b25a4463 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 332269d |
| maven | version: Apache Maven
[jira] [Commented] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363240#comment-16363240
]
genericqa commented on HDFS-13114:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910434/HDFS-13114.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9de777b33da0 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 332269d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23054/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23054/testReport/ |
| Max. process+thread count | 2908 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-13119) RBF: Manage unavailable clusters
[
https://issues.apache.org/jira/browse/HDFS-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363216#comment-16363216
]
Íñigo Goiri commented on HDFS-13119:
The unit tests in [^HDFS-13119.002.patch] look good.
Just to summarize, the current solution has two parts:
# Limit the number of threads for parallel operations that use
{{RouterRpcClient #invokeConcurrent()}}; with this we should prevent having a
huge thread pool.
# When we fail to invoke a method in a subcluster, we check if it is
unavailable according to our membership and if it is, we try once more;
otherwise fail.
For #1, we have a separate pool for the connections. [~chris.douglas] helped
review that part. Any thoughts on this solution?
For #2, the code is kind of complicated to allow just one retry, it might be OK
to just check if the cluster is unavailable and throw the exception if so
without one retry. Otherwise, we could just do:
{code}
if (isClusterUnAvailable(nsId) && retryCount > 0) {
throw new IOException("No namenode available under nameservice " + nsId, ioe);
}
{code}
Then, the default logic takes care of the first retry.
> RBF: Manage unavailable clusters
>
>
> Key: HDFS-13119
> URL: https://issues.apache.org/jira/browse/HDFS-13119
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13119.001.patch, HDFS-13119.002.patch
>
>
> When a federated cluster has one of the subcluster down, operations that run
> in every subcluster ({{RouterRpcClient#invokeAll()}}) may take all the RPC
> connections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361857#comment-16361857
]
Xiao Chen edited comment on HDFS-13040 at 2/13/18 11:06 PM:
Worked on the test based on [~pifta]'s original.
I think one thing that worths clarifying with Istvan is that, in the test the
UGI is just the setting up the client, for doing an NN RPC. So if I understand
correctly, what you observed from tests are all RPC behavior. The bug here is
after the client authenticates with NN, when the NN is doing its part to get
the edits see below
After some debugging, it seems this needs extra efforts to make a real test
that lets NN read the edits via {{EditLogFileInputStream$URLLog}}. Currently
{{DFSInotifyEventInputStream#poll}} would end up making the NN go the
{{EditLogFileInputStream$FileLog}}, hence not triggering the authentication
path reported. On the other hand, we can't really unit test the stream, since
we do need the context of a NN RPC, and the context of NN loginuser. Will
continue working on this when I can find cycles
[~daryn] / [~kihwal],
Does the fix itself make sense to you?
was (Author: xiaochen):
Worked on the test based on [~pifta]'s original.
I think one thing that worths clarifying with Istvan is that, in the test the
UGI is just the setting up the client, for doing an NN RPC. So if I understand
correctly, what you observed from tests are all RPC behavior. The bug here is
after the client authenticates with NN, when the NN is doing its part to get
the edits see below
After some debugging, it seems this needs extra efforts to make a real test
that lets NN read the edits via {{EditLogFileInputStream$URLLog}}. Currently
{{DFSInotifyEventInputStream#poll}} would end up making the NN go the
{{EditLogFileInputStream$FileLog}}, hence not triggering the authentication
path reported. On the other hand, we can really unit test the stream, since we
do need the context of a NN RPC, and the context of NN loginuser. Will continue
working on this when I can find cycles
[~daryn] / [~kihwal],
Does the fix itself make sense to you?
> Kerberized inotify client fails despite kinit properly
> --
>
> Key: HDFS-13040
> URL: https://issues.apache.org/jira/browse/HDFS-13040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
> Environment: Kerberized, HA cluster, iNotify client, CDH5.10.2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-13040.001.patch, HDFS-13040.02.patch,
> HDFS-13040.half.test.patch, TestDFSInotifyEventInputStreamKerberized.java,
> TransactionReader.java
>
>
> This issue is similar to HDFS-10799.
> HDFS-10799 turned out to be a client side issue where client is responsible
> for renewing kerberos ticket actively.
> However we found in a slightly setup even if client has valid Kerberos
> credentials, inotify still fails.
> Suppose client uses principal h...@example.com,
> namenode 1 uses server principal hdfs/nn1.example@example.com
> namenode 2 uses server principal hdfs/nn2.example@example.com
> *After Namenodes starts for longer than kerberos ticket lifetime*, the client
> fails with the following error:
> {noformat}
> 18/01/19 11:23:02 WARN security.UserGroupInformation:
> PriviledgedActionException as:h...@gce.cloudera.com (auth:KERBEROS)
> cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): We
> encountered an error reading
> https://nn2.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3,
>
> https://nn1.example.com:8481/getJournal?jid=ns1=8662=-60%3A353531113%3A0%3Acluster3.
> During automatic edit log failover, we noticed that all of the remaining
> edit log streams are shorter than the current one! The best remaining edit
> log ends at transaction 8683, but we thought we could read up to transaction
> 8684. If you continue, metadata will be lost forever!
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.readOp(NameNodeRpcServer.java:1701)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1763)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getEditsFromTxid(AuthorizationProviderProxyClientProtocol.java:1011)
> at
>
[jira] [Commented] (HDFS-12070) Failed block recovery leaves files open indefinitely and at risk for data loss
[
https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363210#comment-16363210
]
Kihwal Lee commented on HDFS-12070:
---
{noformat}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 108.315
s - in org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
[INFO] Running org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.609 s
- in org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
124.02 s - in org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
[INFO] Running org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 s
- in org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts
[INFO] Running org.apache.hadoop.hdfs.TestMaintenanceState
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 353.62
s - in org.apache.hadoop.hdfs.TestMaintenanceState
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 87, Failures: 0, Errors: 0, Skipped: 10
{noformat}
> Failed block recovery leaves files open indefinitely and at risk for data loss
> --
>
> Key: HDFS-12070
> URL: https://issues.apache.org/jira/browse/HDFS-12070
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Kihwal Lee
>Priority: Major
> Attachments: HDFS-12070.0.patch, lease.patch
>
>
> Files will remain open indefinitely if block recovery fails which creates a
> high risk of data loss. The replication monitor will not replicate these
> blocks.
> The NN provides the primary node a list of candidate nodes for recovery which
> involves a 2-stage process. The primary node removes any candidates that
> cannot init replica recovery (essentially alive and knows about the block) to
> create a sync list. Stage 2 issues updates to the sync list – _but fails if
> any node fails_ unlike the first stage. The NN should be informed of nodes
> that did succeed.
> Manual recovery will also fail until the problematic node is temporarily
> stopped so a connection refused will induce the bad node to be pruned from
> the candidates. Recovery succeeds, the lease is released, under replication
> is fixed, and block is invalidated from the bad node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
[ https://issues.apache.org/jira/browse/HDFS-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363181#comment-16363181 ] Wei-Chiu Chuang commented on HDFS-13140: Hi [~abhishekbafna] thanks for filing the Jira & contributing the patch. Would you please: # Update Jira description to explain what does it do? I.e. in addition to an overall description, please explain how your patch does what it claims to do. # You need a unit test for the new capability added in the patch. # It seems the patch has some caveats. E.g. If the destination file system is not HDFS-based, it does not verify the destination is not changed. Is that a valid assumption? # What cloud storage system has been tested against it? # Any limitation? > Snapshot based replication between HDFS and cloud using DistCp > -- > > Key: HDFS-13140 > URL: https://issues.apache.org/jira/browse/HDFS-13140 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna >Priority: Major > Attachments: HDFS-13140-00.patch > > > Snapshot based replication between HDFS and cloud using DistCp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-10453) ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.
[
https://issues.apache.org/jira/browse/HDFS-10453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HDFS-10453:
Comment: was deleted
(was: Regarding branch-2.7:
After patch 008 an additional patch, 009 was uploaded and it seems that one was
[committed|https://github.com/apache/hadoop/commit/02f6030b35999f2f741a8c4b9363ee59f36f7e28].
The difference between the two patches is that the later one does not include
the unit test.
I do not see any comment about 009. Was committing 009 intended? Now the branch
misses the UT.)
> ReplicationMonitor thread could stuck for long time due to the race between
> replication and delete of same file in a large cluster.
> ---
>
> Key: HDFS-10453
> URL: https://issues.apache.org/jira/browse/HDFS-10453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.1, 2.5.2, 2.7.1, 2.6.4
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: HDFS-10453-branch-2.001.patch,
> HDFS-10453-branch-2.003.patch, HDFS-10453-branch-2.7.004.patch,
> HDFS-10453-branch-2.7.005.patch, HDFS-10453-branch-2.7.006.patch,
> HDFS-10453-branch-2.7.007.patch, HDFS-10453-branch-2.7.008.patch,
> HDFS-10453-branch-2.7.009.patch, HDFS-10453-branch-2.8.001.patch,
> HDFS-10453-branch-2.8.002.patch, HDFS-10453-branch-2.9.001.patch,
> HDFS-10453-branch-2.9.002.patch, HDFS-10453-branch-3.0.001.patch,
> HDFS-10453-branch-3.0.002.patch, HDFS-10453-trunk.001.patch,
> HDFS-10453-trunk.002.patch, HDFS-10453.001.patch
>
>
> ReplicationMonitor thread could stuck for long time and loss data with little
> probability. Consider the typical scenario：
> (1) create and close a file with the default replicas(3);
> (2) increase replication (to 10) of the file.
> (3) delete the file while ReplicationMonitor is scheduling blocks belong to
> that file for replications.
> if ReplicationMonitor stuck reappeared, NameNode will print log as:
> {code:xml}
> 2016-04-19 10:20:48,083 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> ..
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough
> replicas: expected size is 7 but only 0 storage types can be selected
> (replication=10, selected=[], unavailable=[DISK, ARCHIVE], removed=[DISK,
> DISK, DISK, DISK, DISK, DISK, DISK], policy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) All required storage types are unavailable:
> unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
> {code}
> This is because 2 threads (#NameNodeRpcServer and #ReplicationMonitor)
> process same block at the same moment.
> (1) ReplicationMonitor#computeReplicationWorkForBlocks get blocks to
> replicate and leave the global lock.
> (2) FSNamesystem#delete invoked to delete blocks then clear the reference in
> blocksmap, needReplications, etc. the block's NumBytes will set
> NO_ACK(Long.MAX_VALUE) which is used to indicate that the block deletion does
> not need explicit ACK from the node.
> (3) ReplicationMonitor#computeReplicationWorkForBlocks continue to
> chooseTargets for the same blocks and no node will be selected after traverse
> whole cluster because no node choice satisfy the goodness criteria
> (remaining spaces achieve required size Long.MAX_VALUE).
[jira] [Commented] (HDFS-13108) Ozone: OzoneFileSystem: Simplified url schema for Ozone File System
[
https://issues.apache.org/jira/browse/HDFS-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363170#comment-16363170
]
genericqa commented on HDFS-13108:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
30s{color} | {color:green} hadoop-ozone in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13108 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910453/HDFS-13108-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7778ffbf7912 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / f3d07ef |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23055/testReport/ |
| Max. process+thread count | 452 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-ozone U: hadoop-tools/hadoop-ozone |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23055/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: OzoneFileSystem: Simplified url schema for Ozone File System
> ---
>
>
[jira] [Commented] (HDFS-12070) Failed block recovery leaves files open indefinitely and at risk for data loss
[ https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363160#comment-16363160 ] Kihwal Lee commented on HDFS-12070: --- The failed (long) tests in precommit are all passing on my machine. > Failed block recovery leaves files open indefinitely and at risk for data loss > -- > > Key: HDFS-12070 > URL: https://issues.apache.org/jira/browse/HDFS-12070 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Kihwal Lee >Priority: Major > Attachments: HDFS-12070.0.patch, lease.patch > > > Files will remain open indefinitely if block recovery fails which creates a > high risk of data loss. The replication monitor will not replicate these > blocks. > The NN provides the primary node a list of candidate nodes for recovery which > involves a 2-stage process. The primary node removes any candidates that > cannot init replica recovery (essentially alive and knows about the block) to > create a sync list. Stage 2 issues updates to the sync list – _but fails if > any node fails_ unlike the first stage. The NN should be informed of nodes > that did succeed. > Manual recovery will also fail until the problematic node is temporarily > stopped so a connection refused will induce the bad node to be pruned from > the candidates. Recovery succeeds, the lease is released, under replication > is fixed, and block is invalidated from the bad node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13146) Ozone: Fix TestBlockDeletingService
Xiaoyu Yao created HDFS-13146:
-
Summary: Ozone: Fix TestBlockDeletingService
Key: HDFS-13146
URL: https://issues.apache.org/jira/browse/HDFS-13146
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: HDFS-7240
Reporter: Xiaoyu Yao
The unit tests in this class failed to shutdown individual ContainerManager
created in teach test, and the failure stack looks like:
{code}
org.apache.hadoop.metrics2.MetricsException:
org.apache.hadoop.metrics2.MetricsException:
Hadoop:service=OzoneDataNode,name=ContainerLocationManager already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:155)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:87)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:67)
at
org.apache.hadoop.ozone.container.common.impl.ContainerLocationManagerImpl.(ContainerLocationManagerImpl.java:74)
at
org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.init(ContainerManagerImpl.java:188)
at
org.apache.hadoop.ozone.container.common.TestBlockDeletingService.createContainerManager(TestBlockDeletingService.java:117)
at
org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletionTimeout(TestBlockDeletingService.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.apache.hadoop.metrics2.MetricsException:
Hadoop:service=OzoneDataNode,name=ContainerLocationManager already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:131)
... 33 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13052) WebHDFS: Add support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363105#comment-16363105 ] Xiaoyu Yao commented on HDFS-13052: --- [~ljain], can you fix the checkstyle issue? The patch v7 LGTM, +1 pending Jenkins and checkstyle fix. The bulk unit test failures seem to be caused by unrelated infra issues. > WebHDFS: Add support for snasphot diff > -- > > Key: HDFS-13052 > URL: https://issues.apache.org/jira/browse/HDFS-13052 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13052.001.patch, HDFS-13052.002.patch, > HDFS-13052.003.patch, HDFS-13052.004.patch, HDFS-13052.005.patch, > HDFS-13052.006.patch, HDFS-13052.007.patch > > > This Jira aims to implement snapshot diff operation for webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13108) Ozone: OzoneFileSystem: Simplified url schema for Ozone File System
[ https://issues.apache.org/jira/browse/HDFS-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13108: Attachment: HDFS-13108-HDFS-7240.003.patch > Ozone: OzoneFileSystem: Simplified url schema for Ozone File System > --- > > Key: HDFS-13108 > URL: https://issues.apache.org/jira/browse/HDFS-13108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13108-HDFS-7240.001.patch, > HDFS-13108-HDFS-7240.002.patch, HDFS-13108-HDFS-7240.003.patch > > > A. Current state > > 1. The datanode host / bucket /volume should be defined in the defaultFS (eg. > o3://datanode:9864/test/bucket1) > 2. The root file system points to the bucket (eg. 'dfs -ls /' lists all the > keys from the bucket1) > It works very well, but there are some limitations. > B. Problem one > The current code doesn't support fully qualified locations. For example 'dfs > -ls o3://datanode:9864/test/bucket1/dir1' is not working. > C.) Problem two > I tried to fix the previous problem, but it's not trivial. The biggest > problem is that there is a Path.makeQualified call which could transform > unqualified url to qualified url. This is part of the Path.java so it's > common for all the Hadoop file systems. > In the current implementations it qualifies an url with keeping the schema > (eg. o3:// ) and authority (eg: datanode: 9864) from the defaultfs and use > the relative path as the end of the qualified url. For example: > makeQualfied(defaultUri=o3://datanode:9864/test/bucket1, path=dir1/file) will > return o3://datanode:9864/dir1/file which is obviously wrong (the good would > be o3://datanode:9864/TEST/BUCKET1/dir1/file). I tried to do a workaround > with using a custom makeQualified in the Ozone code and it worked from > command line but couldn't work with Spark which use the Hadoop api and the > original makeQualified path. > D.) Solution > We should support makeQualified calls, so we can use any path in the > defaultFS. > > I propose to use a simplified schema as o3://bucket.volume/ > This is similar to the s3a format where the pattern is s3a://bucket.region/ > We don't need to set the hostname of the datanode (or ksm in case of service > discovery) but it would be configurable with additional hadoop configuraion > values such as fs.o3.bucket.buckename.volumename.address=http://datanode:9864 > (this is how the s3a works today, as I know). > We also need to define restrictions for the volume names (in our case it > should not include dot any more). > ps: some spark output > 2018-02-03 18:43:04 WARN Client:66 - Neither spark.yarn.jars nor > spark.yarn.archive is set, falling back to uploading libraries under > SPARK_HOME. > 2018-02-03 18:43:05 INFO Client:54 - Uploading resource > file:/tmp/spark-03119be0-9c3d-440c-8e9f-48c692412ab5/__spark_libs__244044896784490.zip > -> > o3://datanode:9864/user/hadoop/.sparkStaging/application_1517611085375_0001/__spark_libs__244044896784490.zip > My default fs was o3://datanode:9864/test/bucket1, but spark qualified the > name of the home directory. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
[
https://issues.apache.org/jira/browse/HDFS-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363033#comment-16363033
]
genericqa commented on HDFS-13142:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 250 unchanged - 0 fixed = 258 total (was 250) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13142 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910416/HDFS-13142.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 285a78516993 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c5e6e3d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23051/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23051/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23051/testReport/ |
| Max.
[jira] [Updated] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-13114:
--
Attachment: HDFS-13114.001.patch
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-13114:
--
Status: Patch Available (was: Open)
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13109) Support fully qualified hdfs path in EZ commands
[
https://issues.apache.org/jira/browse/HDFS-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362993#comment-16362993
]
Hanisha Koneru commented on HDFS-13109:
---
Thanks for the review, [~xyao].
Uploaded patch v02 with a unit test and fixed checkstyle.
> Support fully qualified hdfs path in EZ commands
>
>
> Key: HDFS-13109
> URL: https://issues.apache.org/jira/browse/HDFS-13109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13109.001.patch, HDFS-13109.002.patch
>
>
> When creating an Encryption Zone, if the fully qualified path is specified in
> the path argument, it throws the following error.
> {code:java}
> ~$ hdfs crypto -createZone -keyName mykey1 -path hdfs://ns1/zone1
> IllegalArgumentException: hdfs://ns1/zone1 is not the root of an encryption
> zone. Do you mean /zone1?
> ~$ hdfs crypto -createZone -keyName mykey1 -path "hdfs://namenode:9000/zone2"
> IllegalArgumentException: hdfs://namenode:9000/zone2 is not the root of an
> encryption zone. Do you mean /zone2?
> {code}
> The EZ creation succeeds as the path is resolved in
> DFS#createEncryptionZone(). But while creating the Trash directory, the path
> is not resolved and it throws the above error.
> A fully qualified path should be supported by {{crypto}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13109) Support fully qualified hdfs path in EZ commands
[
https://issues.apache.org/jira/browse/HDFS-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-13109:
--
Attachment: HDFS-13109.002.patch
> Support fully qualified hdfs path in EZ commands
>
>
> Key: HDFS-13109
> URL: https://issues.apache.org/jira/browse/HDFS-13109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13109.001.patch, HDFS-13109.002.patch
>
>
> When creating an Encryption Zone, if the fully qualified path is specified in
> the path argument, it throws the following error.
> {code:java}
> ~$ hdfs crypto -createZone -keyName mykey1 -path hdfs://ns1/zone1
> IllegalArgumentException: hdfs://ns1/zone1 is not the root of an encryption
> zone. Do you mean /zone1?
> ~$ hdfs crypto -createZone -keyName mykey1 -path "hdfs://namenode:9000/zone2"
> IllegalArgumentException: hdfs://namenode:9000/zone2 is not the root of an
> encryption zone. Do you mean /zone2?
> {code}
> The EZ creation succeeds as the path is resolved in
> DFS#createEncryptionZone(). But while creating the Trash directory, the path
> is not resolved and it throws the above error.
> A fully qualified path should be supported by {{crypto}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13109) Support fully qualified hdfs path in EZ commands
[
https://issues.apache.org/jira/browse/HDFS-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-13109:
--
Summary: Support fully qualified hdfs path in EZ commands (was: Support
fully qualified hdfs path in "crypto" commands path argument)
> Support fully qualified hdfs path in EZ commands
>
>
> Key: HDFS-13109
> URL: https://issues.apache.org/jira/browse/HDFS-13109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13109.001.patch
>
>
> When creating an Encryption Zone, if the fully qualified path is specified in
> the path argument, it throws the following error.
> {code:java}
> ~$ hdfs crypto -createZone -keyName mykey1 -path hdfs://ns1/zone1
> IllegalArgumentException: hdfs://ns1/zone1 is not the root of an encryption
> zone. Do you mean /zone1?
> ~$ hdfs crypto -createZone -keyName mykey1 -path "hdfs://namenode:9000/zone2"
> IllegalArgumentException: hdfs://namenode:9000/zone2 is not the root of an
> encryption zone. Do you mean /zone2?
> {code}
> The EZ creation succeeds as the path is resolved in
> DFS#createEncryptionZone(). But while creating the Trash directory, the path
> is not resolved and it throws the above error.
> A fully qualified path should be supported by {{crypto}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[
https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362955#comment-16362955
]
genericqa commented on HDFS-13143:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
32s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13143 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910427/HDFS-13143.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3f4a0f6dca7f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 332269d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23052/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23052/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> SnapshotDiff - snapshotDiffReport
[jira] [Commented] (HDFS-12070) Failed block recovery leaves files open indefinitely and at risk for data loss
[
https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362937#comment-16362937
]
genericqa commented on HDFS-12070:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 31 unchanged - 0 fixed = 33 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.TestMaintenanceState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12070 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910415/HDFS-12070.0.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2e41f21fedf7 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c5e6e3d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23049/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Created] (HDFS-13145) SBN crash when transition to ANN with in-progress edit tailing enabled
Chao Sun created HDFS-13145:
---
Summary: SBN crash when transition to ANN with in-progress edit
tailing enabled
Key: HDFS-13145
URL: https://issues.apache.org/jira/browse/HDFS-13145
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, namenode
Reporter: Chao Sun
Assignee: Chao Sun
With edit log in-progress edit log tailing enabled, {{QuorumOutputStream}} will
send two batches to JNs, one normal edit batch followed by a dummy batch to
update the commit ID on JNs.
{code}
QuorumCall qcall = loggers.sendEdits(
segmentTxId, firstTxToFlush,
numReadyTxns, data);
loggers.waitForWriteQuorum(qcall, writeTimeoutMs, "sendEdits");
// Since we successfully wrote this batch, let the loggers know. Any
future
// RPCs will thus let the loggers know of the most recent transaction,
even
// if a logger has fallen behind.
loggers.setCommittedTxId(firstTxToFlush + numReadyTxns - 1);
// If we don't have this dummy send, committed TxId might be one-batch
// stale on the Journal Nodes
if (updateCommittedTxId) {
QuorumCall fakeCall = loggers.sendEdits(
segmentTxId, firstTxToFlush,
0, new byte[0]);
loggers.waitForWriteQuorum(fakeCall, writeTimeoutMs, "sendEdits");
}
{code}
Between each batch, it will wait for the JNs to reach a quorum. However, if the
ANN crashes in between, then SBN will crash while transiting to ANN:
{code}
java.lang.IllegalStateException: Cannot start writing at txid 24312595802 when
there is a stream available for read: ..
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.openForWrite(FSEditLog.java:329)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1196)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1839)
at
org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1707)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1622)
at
org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
at
org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:851)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:794)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2490)
2018-02-13 00:43:20,728 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
This is because without the dummy batch, the {{commitTxnId}} will lag behind
the {{endTxId}}, which caused the check in {{openForWrite}} to fail:
{code}
List streams = new ArrayList();
journalSet.selectInputStreams(streams, segmentTxId, true, false);
if (!streams.isEmpty()) {
String error = String.format("Cannot start writing at txid %s " +
"when there is a stream available for read: %s",
segmentTxId, streams.get(0));
IOUtils.cleanupWithLogger(LOG,
streams.toArray(new EditLogInputStream[0]));
throw new IllegalStateException(error);
}
{code}
In our environment, this can be reproduced pretty consistently, which will
leave the cluster with no running namenodes. Even though we are using a 2.8.2
backport, I believe the same issue also exist in 3.0.x.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12586) EZ createZone returns IllegalArgumentException when using protocol in path
[
https://issues.apache.org/jira/browse/HDFS-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru resolved HDFS-12586.
---
Resolution: Duplicate
HDFS-13109 is a dup of this Jira. But closing this as the other one is in
progress.
Thanks for reporting this, [~shahrs87].
> EZ createZone returns IllegalArgumentException when using protocol in path
> --
>
> Key: HDFS-12586
> URL: https://issues.apache.org/jira/browse/HDFS-12586
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
>
> When trying to create an EZ and sending protocol (hdfs://) as part of the
> path, -createZone reports an IllegalArgumentException.
> IllegalArgumentException: hdfs:///tmp/fooez1 is not the
> root of an encryption zone. Do you mean /tmp/fooez1?
> Here's a sequence:
> 1. mkdir the path
> bash-4.1$ hadoop fs -mkdir /tmp/fooez1
> 2. try to make EZ using hdfs protocol, and get error
> hdfs crypto -createZone -keyName key1 -path
> hdfs:///tmp/fooez1/
> IllegalArgumentException: hdfs:///tmp/fooez1 is not the
> root of an encryption zone. Do you mean /tmp/fooez1?
> It fails while provisioning trash for ez root directory.
> The relevant chunk of code.
> {code:title=HdfsAdmin.java|borderStyle=solid}
> private void provisionEZTrash(Path path) throws IOException {
>...
>...
> String ezPath = ez.getPath();
> if (!path.toString().equals(ezPath)) {
> throw new IllegalArgumentException(path + " is not the root of an " +
> "encryption zone. Do you mean " + ez.getPath() + "?");
> }
> {code}
> It is comparing the {{supplied path}} with path component of
> {{EncryptionZone#path}} which doesn't contain scheme and authority.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[ https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362872#comment-16362872 ] Shashikant Banerjee commented on HDFS-13143: Patch v2 addresses the findBug warning. > SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff > calculation happens between a snapshot and the current tree > --- > > Key: HDFS-13143 > URL: https://issues.apache.org/jira/browse/HDFS-13143 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13143.001.patch, HDFS-13143.002.patch > > > HDFS-12594 introduced an iterative approach for computing snapshotDiffs over > multiple rpc calls. The iterative approach depends upon the startPath (path > of the directory with respect to the snapshottable root) and the size of the > createdList (0 in case the startPath refers a file) to exactly determine form > where in each iteration the calculation has to start. > > In case of the diff computation between a snapshot and current tree(if any of > the snapshot names specified in the getSnapshotDiffReport call is null or > empty), the last SnapshotDiff associated with directory/file might change > owing to changes in the current tree in between the rpc calls in the absence > of a global fsn lock. This might result in consistencies in the > snapshotDiffReport. > In case the snapshotDiffReport computation needs to be done between the > current tree and a snapshot, we should fall back to non-iterative approach to > compute snapshotDiff. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[ https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13143: --- Attachment: HDFS-13143.002.patch > SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff > calculation happens between a snapshot and the current tree > --- > > Key: HDFS-13143 > URL: https://issues.apache.org/jira/browse/HDFS-13143 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13143.001.patch, HDFS-13143.002.patch > > > HDFS-12594 introduced an iterative approach for computing snapshotDiffs over > multiple rpc calls. The iterative approach depends upon the startPath (path > of the directory with respect to the snapshottable root) and the size of the > createdList (0 in case the startPath refers a file) to exactly determine form > where in each iteration the calculation has to start. > > In case of the diff computation between a snapshot and current tree(if any of > the snapshot names specified in the getSnapshotDiffReport call is null or > empty), the last SnapshotDiff associated with directory/file might change > owing to changes in the current tree in between the rpc calls in the absence > of a global fsn lock. This might result in consistencies in the > snapshotDiffReport. > In case the snapshotDiffReport computation needs to be done between the > current tree and a snapshot, we should fall back to non-iterative approach to > compute snapshotDiff. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[
https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362869#comment-16362869
]
genericqa commented on HDFS-13143:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Nullcheck of snapshotName at line 2000 of value previously dereferenced
in org.apache.hadoop.hdfs.DistributedFileSystem.isValidSnapshotName(String) At
DistributedFileSystem.java:2000 of value previously dereferenced in
org.apache.hadoop.hdfs.DistributedFileSystem.isValidSnapshotName(String) At
DistributedFileSystem.java:[line 2000] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13143 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910422/HDFS-13143.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 29496e4496fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c5e6e3d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
|
[jira] [Created] (HDFS-13144) Ozone: DeleteContainer call in SCM does not remove the container info from the container inmemory maps.
Shashikant Banerjee created HDFS-13144: -- Summary: Ozone: DeleteContainer call in SCM does not remove the container info from the container inmemory maps. Key: HDFS-13144 URL: https://issues.apache.org/jira/browse/HDFS-13144 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee When the deleteContainer call gets executed at SCM, the container info is deleted from SCM but not from the container Maps. Even after the deleteContainer call is successful, listContainer lists the deleted containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13052) WebHDFS: Add support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13052: --- Status: Patch Available (was: Open) > WebHDFS: Add support for snasphot diff > -- > > Key: HDFS-13052 > URL: https://issues.apache.org/jira/browse/HDFS-13052 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13052.001.patch, HDFS-13052.002.patch, > HDFS-13052.003.patch, HDFS-13052.004.patch, HDFS-13052.005.patch, > HDFS-13052.006.patch, HDFS-13052.007.patch > > > This Jira aims to implement snapshot diff operation for webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13052) WebHDFS: Add support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13052: --- Status: Open (was: Patch Available) > WebHDFS: Add support for snasphot diff > -- > > Key: HDFS-13052 > URL: https://issues.apache.org/jira/browse/HDFS-13052 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13052.001.patch, HDFS-13052.002.patch, > HDFS-13052.003.patch, HDFS-13052.004.patch, HDFS-13052.005.patch, > HDFS-13052.006.patch, HDFS-13052.007.patch > > > This Jira aims to implement snapshot diff operation for webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
[ https://issues.apache.org/jira/browse/HDFS-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDFS-13140: --- Assignee: Abhishek Bafna > Snapshot based replication between HDFS and cloud using DistCp > -- > > Key: HDFS-13140 > URL: https://issues.apache.org/jira/browse/HDFS-13140 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Reporter: Abhishek Bafna >Assignee: Abhishek Bafna >Priority: Major > Attachments: HDFS-13140-00.patch > > > Snapshot based replication between HDFS and cloud using DistCp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4843) testDeleteBlockPool has a bug which makes it fail occasionally
[ https://issues.apache.org/jira/browse/HDFS-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362821#comment-16362821 ] Bharat Viswanadham commented on HDFS-4843: -- Hi [~vincent cho] Closing this one as duplicate of HDFS-5892. > testDeleteBlockPool has a bug which makes it fail occasionally > -- > > Key: HDFS-4843 > URL: https://issues.apache.org/jira/browse/HDFS-4843 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 2.0.4-alpha > Environment: normal enviroment >Reporter: vincent cho >Assignee: Bharat Viswanadham >Priority: Major > > int the test case "testDeleteBlockPool" : > first we set "DFSConfigKeys.DFS_NAMESERVICES" as > "namesServerId1,namesServerId2", but after a cluster was built, the > nameservice are set as "ns1,ns2" because the code set a default value for > them when we new one cluster. > then we refresh namenode with the nameservice set as "namesServerId2" . so we > will add a new nameservice named "namesServerId2" and remove the two old > nameservices. because "ns2" and "namesServerId2" have the same bpid, so when > the add thread run faster than the remove process, the bpByBlockPoolId in > BlockPoolManager will be empty so that it fails when create a new file or > path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4843) testDeleteBlockPool has a bug which makes it fail occasionally
[ https://issues.apache.org/jira/browse/HDFS-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDFS-4843. -- Resolution: Duplicate Assignee: Bharat Viswanadham > testDeleteBlockPool has a bug which makes it fail occasionally > -- > > Key: HDFS-4843 > URL: https://issues.apache.org/jira/browse/HDFS-4843 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 2.0.4-alpha > Environment: normal enviroment >Reporter: vincent cho >Assignee: Bharat Viswanadham >Priority: Major > > int the test case "testDeleteBlockPool" : > first we set "DFSConfigKeys.DFS_NAMESERVICES" as > "namesServerId1,namesServerId2", but after a cluster was built, the > nameservice are set as "ns1,ns2" because the code set a default value for > them when we new one cluster. > then we refresh namenode with the nameservice set as "namesServerId2" . so we > will add a new nameservice named "namesServerId2" and remove the two old > nameservices. because "ns2" and "namesServerId2" have the same bpid, so when > the add thread run faster than the remove process, the bpByBlockPoolId in > BlockPoolManager will be empty so that it fails when create a new file or > path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13052) WebHDFS: Add support for snasphot diff
[
https://issues.apache.org/jira/browse/HDFS-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362820#comment-16362820
]
genericqa commented on HDFS-13052:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-hdfs-project: The patch generated 5 new +
245 unchanged - 1 fixed = 250 total (was 246) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}175m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 10 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}246m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing |
| | hadoop.hdfs.server.namenode.TestStartup |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.blockmanagement.TestSlowDiskTracker |
| | hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.hdfs.server.namenode.TestNameNodeRecovery |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| |
[jira] [Updated] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
[ https://issues.apache.org/jira/browse/HDFS-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13142: --- Status: Patch Available (was: Open) > Define and Implement a DiifList Interface to store and manage SnapshotDiffs > --- > > Key: HDFS-13142 > URL: https://issues.apache.org/jira/browse/HDFS-13142 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13142.001.patch > > > The InodeDiffList class contains a generic List to store snapshotDiffs. The > generic List interface is bulky and to store and manage snapshotDiffs, we > need only a few specific methods. > This Jira proposes to define a new interface called DiffList interface which > will be used to store and manage snapshotDiffs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[ https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13143: --- Status: Patch Available (was: Open) > SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff > calculation happens between a snapshot and the current tree > --- > > Key: HDFS-13143 > URL: https://issues.apache.org/jira/browse/HDFS-13143 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13143.001.patch > > > HDFS-12594 introduced an iterative approach for computing snapshotDiffs over > multiple rpc calls. The iterative approach depends upon the startPath (path > of the directory with respect to the snapshottable root) and the size of the > createdList (0 in case the startPath refers a file) to exactly determine form > where in each iteration the calculation has to start. > > In case of the diff computation between a snapshot and current tree(if any of > the snapshot names specified in the getSnapshotDiffReport call is null or > empty), the last SnapshotDiff associated with directory/file might change > owing to changes in the current tree in between the rpc calls in the absence > of a global fsn lock. This might result in consistencies in the > snapshotDiffReport. > In case the snapshotDiffReport computation needs to be done between the > current tree and a snapshot, we should fall back to non-iterative approach to > compute snapshotDiff. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[ https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13143: --- Attachment: HDFS-13143.001.patch > SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff > calculation happens between a snapshot and the current tree > --- > > Key: HDFS-13143 > URL: https://issues.apache.org/jira/browse/HDFS-13143 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13143.001.patch > > > HDFS-12594 introduced an iterative approach for computing snapshotDiffs over > multiple rpc calls. The iterative approach depends upon the startPath (path > of the directory with respect to the snapshottable root) and the size of the > createdList (0 in case the startPath refers a file) to exactly determine form > where in each iteration the calculation has to start. > > In case of the diff computation between a snapshot and current tree(if any of > the snapshot names specified in the getSnapshotDiffReport call is null or > empty), the last SnapshotDiff associated with directory/file might change > owing to changes in the current tree in between the rpc calls in the absence > of a global fsn lock. This might result in consistencies in the > snapshotDiffReport. > In case the snapshotDiffReport computation needs to be done between the > current tree and a snapshot, we should fall back to non-iterative approach to > compute snapshotDiff. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
Shashikant Banerjee created HDFS-13143: -- Summary: SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree Key: HDFS-13143 URL: https://issues.apache.org/jira/browse/HDFS-13143 Project: Hadoop HDFS Issue Type: Improvement Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee HDFS-12594 introduced an iterative approach for computing snapshotDiffs over multiple rpc calls. The iterative approach depends upon the startPath (path of the directory with respect to the snapshottable root) and the size of the createdList (0 in case the startPath refers a file) to exactly determine form where in each iteration the calculation has to start. In case of the diff computation between a snapshot and current tree(if any of the snapshot names specified in the getSnapshotDiffReport call is null or empty), the last SnapshotDiff associated with directory/file might change owing to changes in the current tree in between the rpc calls in the absence of a global fsn lock. This might result in consistencies in the snapshotDiffReport. In case the snapshotDiffReport computation needs to be done between the current tree and a snapshot, we should fall back to non-iterative approach to compute snapshotDiff. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13119) RBF: Manage unavailable clusters
[
https://issues.apache.org/jira/browse/HDFS-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362719#comment-16362719
]
genericqa commented on HDFS-13119:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 433 unchanged - 0 fixed = 435 total (was 433) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.fs.contract.hdfs.TestHDFSContractConcat |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.TestAppendDifferentChecksum |
| | hadoop.fs.viewfs.TestViewFileSystemLinkFallback |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13119 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910398/HDFS-13119.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 317a3ac1fc59 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0c5d7d7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
|
[jira] [Commented] (HDFS-13111) Close recovery may incorrectly mark blocks corrupt
[
https://issues.apache.org/jira/browse/HDFS-13111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362677#comment-16362677
]
Kihwal Lee commented on HDFS-13111:
---
It looks like "trying forever" might be actually part of the problem.
{code:java}
public Replica recoverClose() throws IOException {
while (true) {
try {
try(AutoCloseableLock lock = datasetLock.acquire()) {
// check replica's state
ReplicaInfo replicaInfo = recoverCheck(b, newGS, expectedBlockLen);
// update the replica state/gs and finalize if necessary.
return replicaInfo;
}
} catch (MustStopExistingWriter e) {
e.getReplica().stopWriter(datanode.getDnConf().getXceiverStopTimeout());
}
}
}
{code}
When the I/O frees up and the original writer (normally the packet responder)
and the xceiver thread doing {{recoverClose()}} can finish in non-deterministic
order. If {{recoverClose()}} finishes last, everything is good. If the packer
responder finishes last as the example above, the replica will be marked as
corrupt until the next full block report.
> Close recovery may incorrectly mark blocks corrupt
> --
>
> Key: HDFS-13111
> URL: https://issues.apache.org/jira/browse/HDFS-13111
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Priority: Critical
>
> Close recovery can leave a block marked corrupt until the next FBR arrives
> from one of the DNs. The reason is unclear but has happened multiple times
> when a DN has io saturated disks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-12861:
---
Status: Patch Available (was: Open)
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362638#comment-16362638
]
Íñigo Goiri commented on HDFS-12861:
Thank you [~mf_borge], I assigned the JIRA to you.
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri reassigned HDFS-12861:
--
Assignee: María Fernanda Borge
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362631#comment-16362631
]
María Fernanda Borge commented on HDFS-12861:
-
I want to take this issue
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Priority: Major
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
[ https://issues.apache.org/jira/browse/HDFS-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13142: --- Attachment: HDFS-13142.001.patch > Define and Implement a DiifList Interface to store and manage SnapshotDiffs > --- > > Key: HDFS-13142 > URL: https://issues.apache.org/jira/browse/HDFS-13142 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13142.001.patch > > > The InodeDiffList class contains a generic List to store snapshotDiffs. The > generic List interface is bulky and to store and manage snapshotDiffs, we > need only a few specific methods. > This Jira proposes to define a new interface called DiffList interface which > will be used to store and manage snapshotDiffs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12070) Failed block recovery leaves files open indefinitely and at risk for data loss
[
https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362616#comment-16362616
]
Kihwal Lee commented on HDFS-12070:
---
Finally got around to add a test case. From the test output, I see a
{{commitBlockSynchronization()}} with {{close=false}} being called instead of
simply blowing up. After this, the next recovery attempt succeeds. Without the
fix, the recovery never succeeds.
> Failed block recovery leaves files open indefinitely and at risk for data loss
> --
>
> Key: HDFS-12070
> URL: https://issues.apache.org/jira/browse/HDFS-12070
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Kihwal Lee
>Priority: Major
> Attachments: HDFS-12070.0.patch, lease.patch
>
>
> Files will remain open indefinitely if block recovery fails which creates a
> high risk of data loss. The replication monitor will not replicate these
> blocks.
> The NN provides the primary node a list of candidate nodes for recovery which
> involves a 2-stage process. The primary node removes any candidates that
> cannot init replica recovery (essentially alive and knows about the block) to
> create a sync list. Stage 2 issues updates to the sync list – _but fails if
> any node fails_ unlike the first stage. The NN should be informed of nodes
> that did succeed.
> Manual recovery will also fail until the problematic node is temporarily
> stopped so a connection refused will induce the bad node to be pruned from
> the candidates. Recovery succeeds, the lease is released, under replication
> is fixed, and block is invalidated from the bad node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
[ https://issues.apache.org/jira/browse/HDFS-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362613#comment-16362613 ] Shashikant Banerjee commented on HDFS-13142: Patch v1 defines and provides an arrayList based implementation for diffList Interface to store and manage snapshotDiffs. The dependent test cases have been modified accordingly.. > Define and Implement a DiifList Interface to store and manage SnapshotDiffs > --- > > Key: HDFS-13142 > URL: https://issues.apache.org/jira/browse/HDFS-13142 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > The InodeDiffList class contains a generic List to store snapshotDiffs. The > generic List interface is bulky and to store and manage snapshotDiffs, we > need only a few specific methods. > This Jira proposes to define a new interface called DiffList interface which > will be used to store and manage snapshotDiffs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
[ https://issues.apache.org/jira/browse/HDFS-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13142: --- Attachment: (was: HDFS-13142.001.patch) > Define and Implement a DiifList Interface to store and manage SnapshotDiffs > --- > > Key: HDFS-13142 > URL: https://issues.apache.org/jira/browse/HDFS-13142 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > The InodeDiffList class contains a generic List to store snapshotDiffs. The > generic List interface is bulky and to store and manage snapshotDiffs, we > need only a few specific methods. > This Jira proposes to define a new interface called DiffList interface which > will be used to store and manage snapshotDiffs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12070) Failed block recovery leaves files open indefinitely and at risk for data loss
[ https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-12070: -- Attachment: HDFS-12070.0.patch > Failed block recovery leaves files open indefinitely and at risk for data loss > -- > > Key: HDFS-12070 > URL: https://issues.apache.org/jira/browse/HDFS-12070 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Kihwal Lee >Priority: Major > Attachments: HDFS-12070.0.patch, lease.patch > > > Files will remain open indefinitely if block recovery fails which creates a > high risk of data loss. The replication monitor will not replicate these > blocks. > The NN provides the primary node a list of candidate nodes for recovery which > involves a 2-stage process. The primary node removes any candidates that > cannot init replica recovery (essentially alive and knows about the block) to > create a sync list. Stage 2 issues updates to the sync list – _but fails if > any node fails_ unlike the first stage. The NN should be informed of nodes > that did succeed. > Manual recovery will also fail until the problematic node is temporarily > stopped so a connection refused will induce the bad node to be pruned from > the candidates. Recovery succeeds, the lease is released, under replication > is fixed, and block is invalidated from the bad node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
[ https://issues.apache.org/jira/browse/HDFS-13142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13142: --- Attachment: HDFS-13142.001.patch > Define and Implement a DiifList Interface to store and manage SnapshotDiffs > --- > > Key: HDFS-13142 > URL: https://issues.apache.org/jira/browse/HDFS-13142 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13142.001.patch > > > The InodeDiffList class contains a generic List to store snapshotDiffs. The > generic List interface is bulky and to store and manage snapshotDiffs, we > need only a few specific methods. > This Jira proposes to define a new interface called DiffList interface which > will be used to store and manage snapshotDiffs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13142) Define and Implement a DiifList Interface to store and manage SnapshotDiffs
Shashikant Banerjee created HDFS-13142: -- Summary: Define and Implement a DiifList Interface to store and manage SnapshotDiffs Key: HDFS-13142 URL: https://issues.apache.org/jira/browse/HDFS-13142 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee The InodeDiffList class contains a generic List to store snapshotDiffs. The generic List interface is bulky and to store and manage snapshotDiffs, we need only a few specific methods. This Jira proposes to define a new interface called DiffList interface which will be used to store and manage snapshotDiffs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10453) ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.
[
https://issues.apache.org/jira/browse/HDFS-10453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362473#comment-16362473
]
Andras Bokor commented on HDFS-10453:
-
Regarding branch-2.7:
After patch 008 an additional patch, 009 was uploaded and it seems that one was
[committed|https://github.com/apache/hadoop/commit/02f6030b35999f2f741a8c4b9363ee59f36f7e28].
The difference between the two patches is that the later one does not include
the unit test.
I do not see any comment about 009. Was committing 009 intended? Now the branch
misses the UT.
> ReplicationMonitor thread could stuck for long time due to the race between
> replication and delete of same file in a large cluster.
> ---
>
> Key: HDFS-10453
> URL: https://issues.apache.org/jira/browse/HDFS-10453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.1, 2.5.2, 2.7.1, 2.6.4
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: HDFS-10453-branch-2.001.patch,
> HDFS-10453-branch-2.003.patch, HDFS-10453-branch-2.7.004.patch,
> HDFS-10453-branch-2.7.005.patch, HDFS-10453-branch-2.7.006.patch,
> HDFS-10453-branch-2.7.007.patch, HDFS-10453-branch-2.7.008.patch,
> HDFS-10453-branch-2.7.009.patch, HDFS-10453-branch-2.8.001.patch,
> HDFS-10453-branch-2.8.002.patch, HDFS-10453-branch-2.9.001.patch,
> HDFS-10453-branch-2.9.002.patch, HDFS-10453-branch-3.0.001.patch,
> HDFS-10453-branch-3.0.002.patch, HDFS-10453-trunk.001.patch,
> HDFS-10453-trunk.002.patch, HDFS-10453.001.patch
>
>
> ReplicationMonitor thread could stuck for long time and loss data with little
> probability. Consider the typical scenario：
> (1) create and close a file with the default replicas(3);
> (2) increase replication (to 10) of the file.
> (3) delete the file while ReplicationMonitor is scheduling blocks belong to
> that file for replications.
> if ReplicationMonitor stuck reappeared, NameNode will print log as:
> {code:xml}
> 2016-04-19 10:20:48,083 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> ..
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough
> replicas: expected size is 7 but only 0 storage types can be selected
> (replication=10, selected=[], unavailable=[DISK, ARCHIVE], removed=[DISK,
> DISK, DISK, DISK, DISK, DISK, DISK], policy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) All required storage types are unavailable:
> unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
> {code}
> This is because 2 threads (#NameNodeRpcServer and #ReplicationMonitor)
> process same block at the same moment.
> (1) ReplicationMonitor#computeReplicationWorkForBlocks get blocks to
> replicate and leave the global lock.
> (2) FSNamesystem#delete invoked to delete blocks then clear the reference in
> blocksmap, needReplications, etc. the block's NumBytes will set
> NO_ACK(Long.MAX_VALUE) which is used to indicate that the block deletion does
> not need explicit ACK from the node.
> (3) ReplicationMonitor#computeReplicationWorkForBlocks continue to
> chooseTargets for the same blocks and no node will be selected after traverse
> whole cluster because no node choice satisfy the goodness criteria
> (remaining spaces achieve required size
[jira] [Updated] (HDFS-13119) RBF: Manage unavailable clusters
[
https://issues.apache.org/jira/browse/HDFS-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13119:
-
Attachment: HDFS-13119.002.patch
> RBF: Manage unavailable clusters
>
>
> Key: HDFS-13119
> URL: https://issues.apache.org/jira/browse/HDFS-13119
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13119.001.patch, HDFS-13119.002.patch
>
>
> When a federated cluster has one of the subcluster down, operations that run
> in every subcluster ({{RouterRpcClient#invokeAll()}}) may take all the RPC
> connections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13119) RBF: Manage unavailable clusters
[
https://issues.apache.org/jira/browse/HDFS-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362442#comment-16362442
]
Yiqun Lin commented on HDFS-13119:
--
New patch attached to address the comments.
> RBF: Manage unavailable clusters
>
>
> Key: HDFS-13119
> URL: https://issues.apache.org/jira/browse/HDFS-13119
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13119.001.patch, HDFS-13119.002.patch
>
>
> When a federated cluster has one of the subcluster down, operations that run
> in every subcluster ({{RouterRpcClient#invokeAll()}}) may take all the RPC
> connections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13141: --- Attachment: HDFS-13141.001.patch > WebHDFS: Add support for getting snasphottable directory list > - > > Key: HDFS-13141 > URL: https://issues.apache.org/jira/browse/HDFS-13141 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13141.001.patch > > > This Jira aims to implement get snapshottable directory list operation for > webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13141: --- Description: This Jira aims to implement get snapshottable directory list operation for webHdfs filesystem. > WebHDFS: Add support for getting snasphottable directory list > - > > Key: HDFS-13141 > URL: https://issues.apache.org/jira/browse/HDFS-13141 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > > This Jira aims to implement get snapshottable directory list operation for > webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
Lokesh Jain created HDFS-13141: -- Summary: WebHDFS: Add support for getting snasphottable directory list Key: HDFS-13141 URL: https://issues.apache.org/jira/browse/HDFS-13141 Project: Hadoop HDFS Issue Type: Task Reporter: Lokesh Jain Assignee: Lokesh Jain -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13052) WebHDFS: Add support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362356#comment-16362356 ] Lokesh Jain commented on HDFS-13052: v7 patch uses DFSUtilClient string to bytes[] conversion functions instead of Base64. > WebHDFS: Add support for snasphot diff > -- > > Key: HDFS-13052 > URL: https://issues.apache.org/jira/browse/HDFS-13052 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13052.001.patch, HDFS-13052.002.patch, > HDFS-13052.003.patch, HDFS-13052.004.patch, HDFS-13052.005.patch, > HDFS-13052.006.patch, HDFS-13052.007.patch > > > This Jira aims to implement snapshot diff operation for webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13052) WebHDFS: Add support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13052: --- Attachment: HDFS-13052.007.patch > WebHDFS: Add support for snasphot diff > -- > > Key: HDFS-13052 > URL: https://issues.apache.org/jira/browse/HDFS-13052 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13052.001.patch, HDFS-13052.002.patch, > HDFS-13052.003.patch, HDFS-13052.004.patch, HDFS-13052.005.patch, > HDFS-13052.006.patch, HDFS-13052.007.patch > > > This Jira aims to implement snapshot diff operation for webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13087) Fix: Snapshots On encryption zones get incorrect EZ settings when encryption zone changes
[
https://issues.apache.org/jira/browse/HDFS-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362349#comment-16362349
]
genericqa commented on HDFS-13087:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}245m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13087 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910364/HDFS-13087.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d2a1e090ea8a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision
[jira] [Commented] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362226#comment-16362226
]
genericqa commented on HDFS-13138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 46s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 391 unchanged -
0 fixed = 392 total (was 391) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 304 unchanged - 0 fixed = 306 total (was 304) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 21s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDistributedFileSystemWithECFile |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestFSOutputSummer |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910369/HDFS-13138.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e1e0d202b737 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
[
https://issues.apache.org/jira/browse/HDFS-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362151#comment-16362151
]
genericqa commented on HDFS-13140:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 41s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
49s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13140 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910366/HDFS-13140-00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9b32c44b6626 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0c5d7d7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23046/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23046/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Snapshot based replication between HDFS and cloud using DistCp
>
[jira] [Updated] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated HDFS-13138:
---
Attachment: HDFS-13138.001.patch
HDFS-13138.001.branch-2.7.patch
> webhdfs of federated namenode does not work properly
> -
>
> Key: HDFS-13138
> URL: https://issues.apache.org/jira/browse/HDFS-13138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1, 3.0.0
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: HDFS-13138.001.branch-2.7.patch, HDFS-13138.001.patch
>
>
> my cluster has multiple namenodes using HDFS Federation.
> webhdfs that is not defaultFS does not work properly.
> when I uploaded to non defaultFS namenode using webhdfs.
> uploaded file was founded at defaultFS namenode.
>
> I think root cause is that
> clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462]
>
> {code:java}
> /**
>* Set the namenode address that will be used by clients to access this
>* namenode or name service. This needs to be called before the config
>* is overriden.
>*/
> public void setClientNamenodeAddress(Configuration conf) {
> String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
> if (nnAddr == null) {
> // default fs is not set.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
> URI nnUri = URI.create(nnAddr);
> String nnHost = nnUri.getHost();
> if (nnHost == null) {
> clientNamenodeAddress = null;
> return;
> }
> if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
> // host name is logical
> clientNamenodeAddress = nnHost;
> } else if (nnUri.getPort() > 0) {
> // physical address with a valid port
> clientNamenodeAddress = nnUri.getAuthority();
> } else {
> // the port is missing or 0. Figure out real bind address later.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("Clients are to use {} to access"
> + " this namenode/service.", clientNamenodeAddress );
> }
> {code}
>
> so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter
> finally file was located namenode of fs,defaultFS
>
> workaround is
> configure fs.defaultFS of each namenode to its own nameservice.
> e.g.
> hdfs://ns1 has fs.defaultFS=hdfs://ns1
> hdfs://ns2 has fs.defaultFS=hdfs://ns2
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13138) webhdfs of federated namenode does not work properly
[
https://issues.apache.org/jira/browse/HDFS-13138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated HDFS-13138:
---
Status: Patch Available (was: Open)
I've attched patch.
> webhdfs of federated namenode does not work properly
> -
>
> Key: HDFS-13138
> URL: https://issues.apache.org/jira/browse/HDFS-13138
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0, 2.7.1
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: HDFS-13138.001.branch-2.7.patch, HDFS-13138.001.patch
>
>
> my cluster has multiple namenodes using HDFS Federation.
> webhdfs that is not defaultFS does not work properly.
> when I uploaded to non defaultFS namenode using webhdfs.
> uploaded file was founded at defaultFS namenode.
>
> I think root cause is that
> clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L462]
>
> {code:java}
> /**
>* Set the namenode address that will be used by clients to access this
>* namenode or name service. This needs to be called before the config
>* is overriden.
>*/
> public void setClientNamenodeAddress(Configuration conf) {
> String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
> if (nnAddr == null) {
> // default fs is not set.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
> URI nnUri = URI.create(nnAddr);
> String nnHost = nnUri.getHost();
> if (nnHost == null) {
> clientNamenodeAddress = null;
> return;
> }
> if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
> // host name is logical
> clientNamenodeAddress = nnHost;
> } else if (nnUri.getPort() > 0) {
> // physical address with a valid port
> clientNamenodeAddress = nnUri.getAuthority();
> } else {
> // the port is missing or 0. Figure out real bind address later.
> clientNamenodeAddress = null;
> return;
> }
> LOG.info("Clients are to use {} to access"
> + " this namenode/service.", clientNamenodeAddress );
> }
> {code}
>
> so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter
> finally file was located namenode of fs,defaultFS
>
> workaround is
> configure fs.defaultFS of each namenode to its own nameservice.
> e.g.
> hdfs://ns1 has fs.defaultFS=hdfs://ns1
> hdfs://ns2 has fs.defaultFS=hdfs://ns2
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
[ https://issues.apache.org/jira/browse/HDFS-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated HDFS-13140: -- Status: Patch Available (was: Open) > Snapshot based replication between HDFS and cloud using DistCp > -- > > Key: HDFS-13140 > URL: https://issues.apache.org/jira/browse/HDFS-13140 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Reporter: Abhishek Bafna >Priority: Major > Attachments: HDFS-13140-00.patch > > > Snapshot based replication between HDFS and cloud using DistCp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
[ https://issues.apache.org/jira/browse/HDFS-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Bafna updated HDFS-13140: -- Attachment: HDFS-13140-00.patch > Snapshot based replication between HDFS and cloud using DistCp > -- > > Key: HDFS-13140 > URL: https://issues.apache.org/jira/browse/HDFS-13140 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Reporter: Abhishek Bafna >Priority: Major > Attachments: HDFS-13140-00.patch > > > Snapshot based replication between HDFS and cloud using DistCp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13140) Snapshot based replication between HDFS and cloud using DistCp
Abhishek Bafna created HDFS-13140: - Summary: Snapshot based replication between HDFS and cloud using DistCp Key: HDFS-13140 URL: https://issues.apache.org/jira/browse/HDFS-13140 Project: Hadoop HDFS Issue Type: Improvement Components: distcp Reporter: Abhishek Bafna Snapshot based replication between HDFS and cloud using DistCp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13087) Fix: Snapshots On encryption zones get incorrect EZ settings when encryption zone changes
[
https://issues.apache.org/jira/browse/HDFS-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362061#comment-16362061
]
LiXin Ge commented on HDFS-13087:
-
For example:
1. make dir
2. take a snapshot
3. create EncryptionZone
4. {{getEncryptionZoneForPath}} on the snapshot path should return {{null}},
not {{EncryptionZone}} as in current implementation.
This patch fix that issue, the related unit test such as
{{TestEncryptionZones}}, {{TestReencryption}},
{{TestReencryptionHandler}}, {{TestReencryptionWithKMS}} succeeded in my local
server. Please kindly review it.
> Fix: Snapshots On encryption zones get incorrect EZ settings when encryption
> zone changes
> -
>
> Key: HDFS-13087
> URL: https://issues.apache.org/jira/browse/HDFS-13087
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 3.1.0
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Attachments: HDFS-13087.001.patch
>
>
> Snapshots are supposed to be immutable and read only, so the EZ settings
> which in a snapshot path shouldn't change when the origin encryption zone
> changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13087) Fix: Snapshots On encryption zones get incorrect EZ settings when encryption zone changes
[ https://issues.apache.org/jira/browse/HDFS-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HDFS-13087: Status: Patch Available (was: Open) > Fix: Snapshots On encryption zones get incorrect EZ settings when encryption > zone changes > - > > Key: HDFS-13087 > URL: https://issues.apache.org/jira/browse/HDFS-13087 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HDFS-13087.001.patch > > > Snapshots are supposed to be immutable and read only, so the EZ settings > which in a snapshot path shouldn't change when the origin encryption zone > changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13087) Fix: Snapshots On encryption zones get incorrect EZ settings when encryption zone changes
[ https://issues.apache.org/jira/browse/HDFS-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LiXin Ge updated HDFS-13087: Attachment: HDFS-13087.001.patch > Fix: Snapshots On encryption zones get incorrect EZ settings when encryption > zone changes > - > > Key: HDFS-13087 > URL: https://issues.apache.org/jira/browse/HDFS-13087 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HDFS-13087.001.patch > > > Snapshots are supposed to be immutable and read only, so the EZ settings > which in a snapshot path shouldn't change when the origin encryption zone > changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13040) Kerberized inotify client fails despite kinit properly
[
https://issues.apache.org/jira/browse/HDFS-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362045#comment-16362045
]
genericqa commented on HDFS-13040:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 18m
56s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
37s{color} | {color:red} hadoop-auth in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
40s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
35s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 49s{color}
| {color:red} root generated 1235 new + 0 unchanged - 0 fixed = 1235 total (was
0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 5s{color} | {color:orange} root: The patch generated 7 new + 141 unchanged
- 1 fixed = 148 total (was 142) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 82 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
2s{color} | {color:red} The patch 768 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} hadoop-common-project/hadoop-auth generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
32s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}221m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-auth |
| |
[jira] [Updated] (HDFS-12998) SnapshotDiff - Provide an iterator-based listing API for calculating snapshotDiff
[ https://issues.apache.org/jira/browse/HDFS-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12998: --- Status: Patch Available (was: Open) > SnapshotDiff - Provide an iterator-based listing API for calculating > snapshotDiff > - > > Key: HDFS-12998 > URL: https://issues.apache.org/jira/browse/HDFS-12998 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-12998.001.patch > > > Currently , SnapshotDiff computation happens over multiple rpc calls to > namenode depending on the no of snapshotDiff entries where each rpc call > returns at max 1000 entries by default . Each "getSnapshotDiffreportListing" > call to namenode returns a partial snapshotDiffreportList which are all > combined and processed at the client side to generate a final > snapshotDiffreport. There can be cases where SnapshotDiffReport can be huge > and in situations as such , the rpc calls to namnode should happen on demand > at the client side. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
