date:20180109

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319840#comment-16319840
 ] 

genericqa commented on HDFS-12934:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 51m  
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
46s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
47s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  0m 47s{color} | 
{color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 47s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 391 unchanged - 0 fixed = 393 total (was 391) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
47s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
21s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 46s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 29s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HDFS-12934 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905401/HDFS-12934-branch-2.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  cc  xml  |
| uname | Linux 2e4d00a7753b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 73478e3 |
| maven | version: Apache Maven 3.3.9 
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22632/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22632/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| cc | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22632/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| javac |

[jira] [Updated] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread Yiqun Lin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12934:
-
Attachment: HDFS-12934-branch-2.001.patch

Attach the patch for branch-2. No anything to be changed, just ensure this runs 
okay in branch-2.

> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934-branch-2.001.patch, HDFS-12934.001.patch, 
> HDFS-12934.002.patch, HDFS-12934.003.patch, HDFS-12934.004.patch, 
> HDFS-12934.005.patch, HDFS-12934.006.patch, HDFS-12934.007.patch, 
> HDFS-12934.008.patch, RBF support  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319745#comment-16319745
 ] 

Hudson commented on HDFS-12934:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13476 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13476/])
HDFS-12934. RBF: Federation supports global quota. Contributed by Yiqun (yqlin: 
rev d98a2e6e2383f8b66def346409b0517aa32d298d)
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUsage.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/FederationProtocol.proto
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/MountTable.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/MountTablePBImpl.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/records/TestMountTable.java
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Quota.java
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuotaManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/RouterConfigBuilder.java


> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, 
> HDFS-12934.003.patch, HDFS-12934.004.patch, HDFS-12934.005.patch, 
> HDFS-12934.006.patch, HDFS-12934.007.patch, HDFS-12934.008.patch, RBF support 
>  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319742#comment-16319742
 ] 

Jianfei Jiang commented on HDFS-12935:
--

The failure testcases in patch 006 for {{TestDFSAdminWithHA}} are related to 
environmental crash and patch 005 all passed. The difference between patch 005 
and patch 006 is only checkstyle change. The other failures are not related. 
Please review, the testcase improvement will be done in subtask jira. Thanks. 
[~brahmareddy]

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 2.9.0, 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, 
> HDFS-12935.006.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread Yiqun Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319736#comment-16319736
 ] 

Yiqun Lin commented on HDFS-12934:
--

The failed UTs are not related. Commit to trunk.

> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, 
> HDFS-12934.003.patch, HDFS-12934.004.patch, HDFS-12934.005.patch, 
> HDFS-12934.006.patch, HDFS-12934.007.patch, HDFS-12934.008.patch, RBF support 
>  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319732#comment-16319732
 ] 

genericqa commented on HDFS-12934:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
 3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 46s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 44s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}141m 34s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 24s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.namenode.ha.TestHAAppend |
|   | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12934 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905383/HDFS-12934.008.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  cc  xml  |
| uname | Linux 309aa7d48891 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 55066cc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |

[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319713#comment-16319713
 ] 

genericqa commented on HDFS-12919:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 12s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 50s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 35s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 55s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.federation.router.TestRouterRpc |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
|   | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
|   | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12919 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905380/HDFS-12919.015.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 679888079c6c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64

[jira] [Commented] (HDFS-11409) DatanodeInfo getNetworkLocation and setNetworkLocation shoud use volatile instead of synchronized

2018-01-09 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319698#comment-16319698
 ] 

Brahma Reddy Battula commented on HDFS-11409:
-

it's good candidate for {{branch-2.8}} ?

> DatanodeInfo getNetworkLocation and setNetworkLocation shoud use volatile 
> instead of synchronized
> -
>
> Key: HDFS-11409
> URL: https://issues.apache.org/jira/browse/HDFS-11409
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: performance
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11409.001.patch
>
>
> {{DatanodeInfo}} has synchronized methods {{getNetworkLocation}} and 
> {{setNetworkLocation}}. While they doing nothing more than setting and 
> getting variable {{location}}.
> Since {{location}} is not being modified based on its current value and is 
> independent from any other variables. This JIRA propose to remove 
> synchronized methods but only make {{location}} volatile. Such that threads 
> will not be blocked on get/setNetworkLocation.
> Thanks  [~szetszwo] for the offline disscussion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12986) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-00f80b4-SNAPSHOT)

2018-01-09 Thread Tsz Wo Nicholas Sze (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319671#comment-16319671
 ] 

Tsz Wo Nicholas Sze commented on HDFS-12986:


+1 the 004 patch looks good.

[~msingh], thanks for testing the patch.  I will wait for your result before 
committing this.

> Ozone: Update ozone to latest ratis snapshot build 
> (0.1.1-alpha-00f80b4-SNAPSHOT)
> -
>
> Key: HDFS-12986
> URL: https://issues.apache.org/jira/browse/HDFS-12986
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Lokesh Jain
> Fix For: HDFS-7240
>
> Attachments: HDFS-12986-HDFS-7240.001.patch, 
> HDFS-12986-HDFS-7240.002.patch, HDFS-12986-HDFS-7240.003.patch, 
> HDFS-12986-HDFS-7240.004.patch
>
>
> This jira will update ozone to latest snapshot 
> release-0.1.1-alpha-00f80b4-SNAPSHOT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319630#comment-16319630
 ] 

Hudson commented on HDFS-12802:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13475 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13475/])
HDFS-12802. RBF: Control MountTableResolver cache size. Contrubuted by 
(inigoiri: rev d9006d8a4e34eae78dfa1cf3be50eeb81c2fc11f)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java


> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>  Labels: RBF
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319628#comment-16319628
 ] 

Íñigo Goiri commented on HDFS-12934:


+1 on [^HDFS-12934.008.patch] pending Jenkins output.
I did some testing and it worked as expected, I didn't go super deep but it 
should be good enough.
Thanks [~linyiqun] for pushing this.


> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, 
> HDFS-12934.003.patch, HDFS-12934.004.patch, HDFS-12934.005.patch, 
> HDFS-12934.006.patch, HDFS-12934.007.patch, HDFS-12934.008.patch, RBF support 
>  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12896) when set replicate EC policy for a directory or file，it's EC policy cannot be querying by getPolicy command.

2018-01-09 Thread chencan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chencan resolved HDFS-12896.

Resolution: Duplicate

> when set replicate EC policy for a directory or file，it's EC policy cannot be 
> querying by getPolicy command.
> 
>
> Key: HDFS-12896
> URL: https://issues.apache.org/jira/browse/HDFS-12896
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: chencan
>
> When i set replicate EC policy for ecDir，then query it by getPolicy，it return 
> ‘The erasure coding policy of /ecDir is unspecified', as follow.
> [root@master bin]# hdfs dfs -mkdir /ecDir
> [root@master bin]# hdfs ec -setPolicy -path /ecDir -replicate
> Set erasure coding policy replication on /ecDir
> [root@master bin]# hdfs ec -getPolicy -path /ecDir
> The erasure coding policy of /ecDir is unspecified



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12802:
---
Labels: RBF  (was: )

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>  Labels: RBF
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12802:
---
Target Version/s: 3.0.0, 2.9.0

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>  Labels: RBF
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12802:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.1
   2.9.1
   2.10.0
   3.1.0
   Status: Resolved  (was: Patch Available)

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319620#comment-16319620
 ] 

Íñigo Goiri commented on HDFS-12802:


Thanks [~chris.douglas] and [~linyiqun] for the review.
Committed to trunk, branch-3.0, branch-2 and branch-2.9.

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread Yiqun Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319606#comment-16319606
 ] 

Yiqun Lin commented on HDFS-12802:
--

Also +1 from me.

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread Yiqun Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319603#comment-16319603
 ] 

Yiqun Lin commented on HDFS-12934:
--

Attach the v08 patch for addressing the comments.
Hi [~elgoiri], would you mind take a quick look if the latest patch looks good 
to you? Do you still plan to test in real cluster? If you have done the enough, 
I plan to commit this today and do the follow-up work.

> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, 
> HDFS-12934.003.patch, HDFS-12934.004.patch, HDFS-12934.005.patch, 
> HDFS-12934.006.patch, HDFS-12934.007.patch, HDFS-12934.008.patch, RBF support 
>  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread Yiqun Lin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12934:
-
Attachment: HDFS-12934.008.patch

> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, 
> HDFS-12934.003.patch, HDFS-12934.004.patch, HDFS-12934.005.patch, 
> HDFS-12934.006.patch, HDFS-12934.007.patch, HDFS-12934.008.patch, RBF support 
>  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12919:
---
Attachment: HDFS-12919.015.patch

> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
>  Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch, 
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch, 
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch, 
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch, 
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch, 
> HDFS-12919.013.patch, HDFS-12919.014.patch, HDFS-12919.015.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files. 
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
>  Operation "setErasureCodingPolicy" is not supported
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12996) DataNode Replica Trash

2018-01-09 Thread Virajith Jalaparti (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319531#comment-16319531
 ] 

Virajith Jalaparti edited comment on HDFS-12996 at 1/10/18 1:41 AM:


Thanks for posting the design document [~hanishakoneru]. A few questions - 
# In Section 13, Step 5, which component receives the _restore-replica-trash_ 
command? From my understanding this should be the Namenode. However, in step 5 
the Namenode is not running. How is this going to work?
# Have you looked at "undoing" the delete operation? I think this would need to 
handle conflicts (for example, /foo/bar was deleted but a new /foo/bar/ is 
created) but could avoid having to stop and restart the Namenode.
# Can the replica purge daemon functionality be implemented in the 
VolumeScanner?


was (Author: virajith):
Thanks for posting the design document [~hanishakoneru]. A few questions - 
# In Section 13, Step 5, which component receives the _restore-replica-trash_ 
command? From my understanding this should be the Namenode. However, in step 5 
the Namenode is not running. How is this going to work?
# Have you looked at "undoing" the delete operation? I think this would need to 
handle conflicts (for example, /foo/bar was deleted but a new /foo/bar/ is 
created) but could avoid having to stop and restart the Namenode.

> DataNode Replica Trash
> --
>
> Key: HDFS-12996
> URL: https://issues.apache.org/jira/browse/HDFS-12996
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
> Attachments: DataNode_Replica_Trash_Design_Doc.pdf
>
>
> DataNode Replica Trash will allow administrators to recover from a recent 
> delete request that resulted in catastrophic loss of user data. This is 
> achieved by placing all invalidated blocks in a replica trash on the datanode 
> before completely purging them from the system. The design doc is attached 
> here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12996) DataNode Replica Trash

2018-01-09 Thread Virajith Jalaparti (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319531#comment-16319531
 ] 

Virajith Jalaparti commented on HDFS-12996:
---

Thanks for posting the design document [~hanishakoneru]. A few questions - 
# In Section 13, Step 5, which component receives the _restore-replica-trash_ 
command? From my understanding this should be the Namenode. However, in step 5 
the Namenode is not running. How is this going to work?
# Have you looked at "undoing" the delete operation? I think this would need to 
handle conflicts (for example, /foo/bar was deleted but a new /foo/bar/ is 
created) but could avoid having to stop and restart the Namenode.

> DataNode Replica Trash
> --
>
> Key: HDFS-12996
> URL: https://issues.apache.org/jira/browse/HDFS-12996
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
> Attachments: DataNode_Replica_Trash_Design_Doc.pdf
>
>
> DataNode Replica Trash will allow administrators to recover from a recent 
> delete request that resulted in catastrophic loss of user data. This is 
> achieved by placing all invalidated blocks in a replica trash on the datanode 
> before completely purging them from the system. The design doc is attached 
> here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319518#comment-16319518
 ] 

genericqa commented on HDFS-12919:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 10s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 39s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new + 
0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 36s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
22s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 13s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 21s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
|   | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12919 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905355/HDFS-12919.014.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 0546f44fad5f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
|

[jira] [Updated] (HDFS-13005) HttpFs checks subdirectories ACL status when LISTSTATUS is used

2018-01-09 Thread Hajime Osako (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hajime Osako updated HDFS-13005:

Description: 
HttpFs LISTSTATUS call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
# NOTE: The test user "admin" has full access to /acltest
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:14000/webhdfs/v1/acltest?op=LISTSTATUS"
{"RemoteException":{"message":"Permission denied: user=admin, access=EXECUTE, 
inode=\"\/acltest\/subdir\":hdfs:test:drwxrwx---","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}

# HDFS audit log
[root@sandbox ~]# tail /var/log/hadoop/hdfs/hdfs-audit.log | grep -w admin
2018-01-09 23:09:24,362 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:KERBEROS)   ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=null  
  perm=null   proto=webhdfs
2018-01-09 23:09:31,937 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=nullperm=null   
proto=rpc
2018-01-09 23:09:31,978 INFO FSNamesystem.audit: allowed=false  ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=getAclStatussrc=/acltest/subdir dst=null
perm=null   proto=rpc
{code}

  was:
HttpFs LISTSTATUS call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname

[jira] [Comment Edited] (HDFS-12090) Handling writes from HDFS to Provided storages

2018-01-09 Thread Virajith Jalaparti (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319396#comment-16319396
 ] 

Virajith Jalaparti edited comment on HDFS-12090 at 1/9/18 11:24 PM:


Thanks for posting the patch [~ehiggs]. Here are some initial thoughts on the 
high-level design:
# As you note, the current implementation doesn't support ordered operations 
(e.g., backup of a directory hierarchy to another instance of HDFS). All the 
operations in a particular snapshot diff happen in parallel across 
(potentially) multiple datanodes. When supporting ordered operations, I think 
{{SyncServiceSatisfier}} needs to coordinate them (so that Datanodes don't 
start having additional coordination). So, the design should make sure that 
some part of it is capable of handling ordered operations. Having an abstract 
class that performs the functions handled in 
{{SyncServiceSatisfier#synchronizeBackupMount}} can be one way to solve this 
issue.
# The patch seems a completely separate path from the SPS work (HDFS-10285). 
Given that the SPS is still in a state of flux, this is OK for now. However, in 
the future (once SPS converges), it would be good to look at how this work can 
plug into/reuse parts of the SPS/refactor parts of SPS if necessary. I would 
hate to have two parallel code paths that do something very similar (satisfy 
storage policies). That said, I think that shouldn't stop progress on this JIRA.
# The data backup path is concerning. It bypasses the DN write path and one 
Datanode backs up a whole file (in {{SyncServiceSatisfierWorker#backupFile}}) 
--- it copies blocks from other datanodes in the cluster and then writes it 
back to the provides store. Compared to the SPS approach (a DN could be 
responsible for only 1 block), this approach involves 2 network transfers 
instead of 1 (the DN has to copy blocks from other DNs and then write it back 
to the provided store), and cannot benefit from the parallelism of each DN 
handling one or a few blocks for the file.
# Need for a throttling mechanism so as to limit the load on the NN. Although 
not immediate, this would be eventually required.


Some comments specific to this patch:
* In {{SyncTaskScheduler#schedule}}, why have these two separate paths?
{code}
  if (syncTask.operation == SyncTask.Operation.CREATE_FILE) {
scheduleOnFirstBlockDatanode(syncTask);
  } else {
scheduleOnRandomDatanode(syncTask);
  }
{code}

* Use a builder pattern for creating {{SyncTask}}?
* Why use the sync mount and not backup endpoint? That was the terminology used 
in the latest functional spec.
* The method names {{createSync}}, {{removeSync}}, though understandable, are 
confusing. I think {{createBackupEndPoint}}, {{removeBackupEndPoint}} etc. 
would be easier to understood (and adhere to the functional spec).





was (Author: virajith):
Thanks for posting the patch [~ehiggs]. Here are some initial thoughts on the 
high-level design:
# As you note, the current implementation doesn't support ordered operations 
(e.g., backup of a directory hierarchy to another instance of HDFS). All the 
operations in a particular snapshot diff happen in parallel across 
(potentially) multiple datanodes. When supporting ordered operations, I think 
{{SyncServiceSatisfier}} needs to coordinate them (so that Datanodes don't 
start having additional coordination). So, the design should make sure that 
some part of it is capable of handling ordered operations. Having an abstract 
class that performs the functions handled in 
{{SyncServiceSatisfier#synchronizeBackupMount}} can be one way to solve this 
issue.. 
# The data backup path is concerning. It bypasses the DN write path and one 
Datanode backs up a whole file (in {{SyncServiceSatisfierWorker#backupFile}}) 
--- it copies blocks from other datanodes in the cluster and then writes it 
back to the provides store. Compared to the SPS approach (a DN could be 
responsible for only 1 block), this approach involves 2 network transfers 
instead of 1 (the DN has to copy blocks from other DNs and then write it back 
to the provided store), and cannot benefit from the parallelism of each DN 
handling one or a few blocks for the file.
# The patch seems a completely separate path from the SPS work (HDFS-10285). 
Given that the SPS is still in a state of flux, this is OK for now. However, in 
the future (once SPS converges), it would be good to look at how this work can 
plug into/reuse parts of the SPS/refactor parts of SPS if necessary. I would 
hate to have two parallel code paths that do something very similar (satisfy 
storage policies). That said, I think that shouldn't stop progress on this JIRA.
# Need for a throttling mechanism so as to limit the load on the NN. Although 
not immediate, this would be eventually required.


Some comments specific to this patch:
* In {{SyncTaskScheduler#schedule}}, why

[jira] [Commented] (HDFS-12090) Handling writes from HDFS to Provided storages

2018-01-09 Thread Virajith Jalaparti (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319396#comment-16319396
 ] 

Virajith Jalaparti commented on HDFS-12090:
---

Thanks for posting the patch [~ehiggs]. Here are some initial thoughts on the 
high-level design:
# As you note, the current implementation doesn't support ordered operations 
(e.g., backup of a directory hierarchy to another instance of HDFS). All the 
operations in a particular snapshot diff happen in parallel across 
(potentially) multiple datanodes. When supporting ordered operations, I think 
{{SyncServiceSatisfier}} needs to coordinate them (so that Datanodes don't 
start having additional coordination). So, the design should make sure that 
some part of it is capable of handling ordered operations. Having an abstract 
class that performs the functions handled in 
{{SyncServiceSatisfier#synchronizeBackupMount}} can be one way to solve this 
issue.. 
# The data backup path is concerning. It bypasses the DN write path and one 
Datanode backs up a whole file (in {{SyncServiceSatisfierWorker#backupFile}}) 
--- it copies blocks from other datanodes in the cluster and then writes it 
back to the provides store. Compared to the SPS approach (a DN could be 
responsible for only 1 block), this approach involves 2 network transfers 
instead of 1 (the DN has to copy blocks from other DNs and then write it back 
to the provided store), and cannot benefit from the parallelism of each DN 
handling one or a few blocks for the file.
# The patch seems a completely separate path from the SPS work (HDFS-10285). 
Given that the SPS is still in a state of flux, this is OK for now. However, in 
the future (once SPS converges), it would be good to look at how this work can 
plug into/reuse parts of the SPS/refactor parts of SPS if necessary. I would 
hate to have two parallel code paths that do something very similar (satisfy 
storage policies). That said, I think that shouldn't stop progress on this JIRA.
# Need for a throttling mechanism so as to limit the load on the NN. Although 
not immediate, this would be eventually required.


Some comments specific to this patch:
* In {{SyncTaskScheduler#schedule}}, why have these two separate paths?
{code}
  if (syncTask.operation == SyncTask.Operation.CREATE_FILE) {
scheduleOnFirstBlockDatanode(syncTask);
  } else {
scheduleOnRandomDatanode(syncTask);
  }
{code}

* Use a builder pattern for creating {{SyncTask}}?
* Why use the sync mount and not backup endpoint? That was the terminology used 
in the latest functional spec.
* The method names {{createSync}}, {{removeSync}}, though understandable, are 
confusing. I think {{createBackupEndPoint}}, {{removeBackupEndPoint}} etc. 
would be easier to understood (and adhere to the functional spec).




> Handling writes from HDFS to Provided storages
> --
>
> Key: HDFS-12090
> URL: https://issues.apache.org/jira/browse/HDFS-12090
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Virajith Jalaparti
> Attachments: HDFS-12090-Functional-Specification.001.pdf, 
> HDFS-12090-Functional-Specification.002.pdf, 
> HDFS-12090-Functional-Specification.003.pdf, HDFS-12090-design.001.pdf, 
> HDFS-12090..patch
>
>
> HDFS-9806 introduces the concept of {{PROVIDED}} storage, which makes data in 
> external storage systems accessible through HDFS. However, HDFS-9806 is 
> limited to data being read through HDFS. This JIRA will deal with how data 
> can be written to such {{PROVIDED}} storages from HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-13005) HttpFs checks subdirectories ACL status when LISTSTATUS is used

2018-01-09 Thread Hajime Osako (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hajime Osako updated HDFS-13005:

Description: 
HttpFs LISTSTATUS call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:14000/webhdfs/v1/acltest?op=LISTSTATUS"
{"RemoteException":{"message":"Permission denied: user=admin, access=EXECUTE, 
inode=\"\/acltest\/subdir\":hdfs:test:drwxrwx---","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}

# HDFS audit log
[root@sandbox ~]# tail /var/log/hadoop/hdfs/hdfs-audit.log | grep -w admin
2018-01-09 23:09:24,362 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:KERBEROS)   ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=null  
  perm=null   proto=webhdfs
2018-01-09 23:09:31,937 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=nullperm=null   
proto=rpc
2018-01-09 23:09:31,978 INFO FSNamesystem.audit: allowed=false  ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=getAclStatussrc=/acltest/subdir dst=null
perm=null   proto=rpc
{code}

  was:
HttpFs call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:14000/webhdfs/v1/acltest?op=LISTSTATUS"
{"RemoteException":{"message":"Permission denied:

[jira] [Updated] (HDFS-13005) HttpFs checks subdirectories ACL status when LISTSTATUS is used

2018-01-09 Thread Hajime Osako (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hajime Osako updated HDFS-13005:

Description: 
HttpFs call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:14000/webhdfs/v1/acltest?op=LISTSTATUS"
{"RemoteException":{"message":"Permission denied: user=admin, access=EXECUTE, 
inode=\"\/acltest\/subdir\":hdfs:test:drwxrwx---","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}

# HDFS audit log
[root@sandbox ~]# tail /var/log/hadoop/hdfs/hdfs-audit.log | grep -w admin
2018-01-09 23:09:24,362 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:KERBEROS)   ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=null  
  perm=null   proto=webhdfs
2018-01-09 23:09:31,937 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=nullperm=null   
proto=rpc
2018-01-09 23:09:31,978 INFO FSNamesystem.audit: allowed=false  ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=getAclStatussrc=/acltest/subdir dst=null
perm=null   proto=rpc
{code}

  was:
HttpFs call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:14000/webhdfs/v1/acltest?op=LISTSTATUS"
{"RemoteException":{"message":"Permission denied: user=admin,

[jira] [Created] (HDFS-13005) HttpFs checks subdirectories ACL status when LISTSTATUS is used

2018-01-09 Thread Hajime Osako (JIRA)

Hajime Osako created HDFS-13005:
---

 Summary: HttpFs checks subdirectories ACL status when LISTSTATUS 
is used
 Key: HDFS-13005
 URL: https://issues.apache.org/jira/browse/HDFS-13005
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: httpfs
Affects Versions: 2.7.3
Reporter: Hajime Osako
Priority: Minor


HttpFs call fails if a subdirectory is using ACL because in 
org.apache.hadoop.fs.http.server.FSOperations.StatusPairs#StatusPairs, it gets 
the list of child objects and checks those ACL status one by one, rather than 
checking the target directory ACL.
Would like to know if this is intentional.

{code}
  /*
   * For each FileStatus, attempt to acquire an AclStatus.  If the
   * getAclStatus throws an exception, we assume that ACLs are turned
   * off entirely and abandon the attempt.
   */
  boolean useAcls = true;   // Assume ACLs work until proven otherwise
  ...
{code}

Reproduce steps:
{code}
[root@sandbox ~]# hdfs dfs -ls -R /acltest
drwxrwx---+  - hdfs test  0 2018-01-09 08:44 /acltest/subdir
-rwxrwx---   1 hdfs test647 2018-01-09 08:44 /acltest/subdir/derby.log
drwxr-xr-x   - hdfs test  0 2018-01-09 09:15 /acltest/subdir2
[root@sandbox ~]# hdfs dfs -getfacl /acltest/subdir
# file: /acltest/subdir
# owner: hdfs
# group: test
user::rwx
user:hdfs:rw-
group::r-x
mask::rwx
other::---

# WebHDFS works
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:50070/webhdfs/v1/acltest?op=LISTSTATUS"
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"aclBit":true,"blockSize":0,"childrenNum":1,"fileId":79057,"group":"test","length":0,"modificationTime":1515487493078,"owner":"hdfs","pathSuffix":"subdir","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":79059,"group":"test","length":0,"modificationTime":1515489337849,"owner":"hdfs","pathSuffix":"subdir2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

# Bat not via HttpFs
[root@sandbox ~]# sudo -u admin curl --negotiate -u : "http://`hostname 
-f`:14000/webhdfs/v1/acltest?op=LISTSTATUS"
{"RemoteException":{"message":"Permission denied: user=admin, access=EXECUTE, 
inode=\"\/acltest\/subdir\":hdfs:test:drwxrwx---","exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException"}}
[root@sandbox ~]# tail /var/log/hadoop/hdfs/hdfs-audit.log | grep -w admin
2018-01-09 23:09:24,362 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:KERBEROS)   ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=null  
  perm=null   proto=webhdfs
2018-01-09 23:09:31,937 INFO FSNamesystem.audit: allowed=true   ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=listStatus  src=/acltestdst=nullperm=null   
proto=rpc
2018-01-09 23:09:31,978 INFO FSNamesystem.audit: allowed=false  ugi=admin 
(auth:PROXY) via httpfs/sandbox.hortonworks@example.com (auth:KERBEROS)   
ip=/172.18.0.2  cmd=getAclStatussrc=/acltest/subdir dst=null
perm=null   proto=rpc
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread Chris Douglas (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319336#comment-16319336
 ] 

Chris Douglas commented on HDFS-12802:
--

+1 thanks [~elgoiri]

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12919:
---
Attachment: HDFS-12919.014.patch

> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
>  Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch, 
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch, 
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch, 
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch, 
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch, 
> HDFS-12919.013.patch, HDFS-12919.014.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files. 
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
>  Operation "setErasureCodingPolicy" is not supported
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Lei (Eddy) Xu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei (Eddy) Xu updated HDFS-12994:
-
   Resolution: Fixed
Fix Version/s: 3.0.1
   3.1.0
   Status: Resolved  (was: Patch Available)

Thanks for the quick review, [~manojg]

Committed to branch-3 and trunk

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319145#comment-16319145
 ] 

Hudson commented on HDFS-12994:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13472 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13472/])
HDFS-12994. TestReconstructStripedFile.testNNSendsErasureCodingTasks (lei: rev 
47563d86fe6ba1a2de934c9ed740d0aafbf72d4e)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java


> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319141#comment-16319141
 ] 

genericqa commented on HDFS-12919:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 58s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 35s{color} | {color:orange} hadoop-hdfs-project: The patch generated 4 new + 
0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 33s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}132m 28s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
|   | hadoop.hdfs.web.TestWebHdfsTimeouts |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
|   | hadoop.hdfs.server.federation.router.TestRouterRpc |
|   | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12919 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905313/HDFS-12919.013.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux f6925a3d9afd 4.4.0-64-generic #85-Ubuntu SMP Mon

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319100#comment-16319100
 ] 

Íñigo Goiri commented on HDFS-12802:


Yetus came back with a warning for failed unit test but the report doesn't show 
any.
Chekcing manually, the related test is there and passed as expected:
https://builds.apache.org/job/PreCommit-HDFS-Build/22628/testReport/org.apache.hadoop.hdfs.server.federation.resolver/TestMountTableResolver/

I think this is good to go; [~linyiqun], [~chris.douglas], do you mind taking 
hopefully a last check?


> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319061#comment-16319061
 ] 

genericqa commented on HDFS-12802:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  9m 
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 27s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 46s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m  2s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 23s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12802 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905314/HDFS-12802.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 480fbdedaa91 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f725b9e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22628/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22628/testReport/ |
| Max. process+thread count | 4385 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22628/console |
| Powered by | Apache

[jira] [Commented] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Manoj Govindassamy (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319031#comment-16319031
 ] 

Manoj Govindassamy commented on HDFS-12994:
---

Got it. Patch v01 looks good to me. +1, thanks for working on this.

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Manoj Govindassamy (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319031#comment-16319031
 ] 

Manoj Govindassamy edited comment on HDFS-12994 at 1/9/18 7:51 PM:
---

Got it. Patch v01 looks good to me. +1, thanks for working on this.


was (Author: manojg):
Got it. Patch v02 looks good to me. +1, thanks for working on this.

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Manoj Govindassamy (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319031#comment-16319031
 ] 

Manoj Govindassamy edited comment on HDFS-12994 at 1/9/18 7:51 PM:
---

Got it. Patch v02 looks good to me. +1, thanks for working on this.


was (Author: manojg):
Got it. Patch v01 looks good to me. +1, thanks for working on this.

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Lei (Eddy) Xu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319027#comment-16319027
 ] 

Lei (Eddy) Xu commented on HDFS-12994:
--

Yes, this patch is let DN issues being discovered within the test timeout. 
Originally, the socket timeout is {{60s}}, and the unit test is also {{60s}} 
for two configurations combined. 

bq.  And, the problem should happen always when the DN is removed right? 

This happens only when the NN schedule the recovery tasks *between* shutting 
down two DNs. If two DNs have been shutdown before scheduling the recovery 
task, none of these two DNs will be considered as valid source. 

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Lei (Eddy) Xu (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319027#comment-16319027
 ] 

Lei (Eddy) Xu edited comment on HDFS-12994 at 1/9/18 7:49 PM:
--

Yes, [~manojg]. this patch is let DN issues being discovered within the test 
timeout. Originally, the socket timeout is {{60s}}, and the unit test is also 
{{60s}} for two configurations combined. 

bq.  And, the problem should happen always when the DN is removed right? 

This happens only when the NN schedule the recovery tasks *between* shutting 
down two DNs. If two DNs have been shutdown before scheduling the recovery 
task, none of these two DNs will be considered as valid source. 


was (Author: eddyxu):
Yes, this patch is let DN issues being discovered within the test timeout. 
Originally, the socket timeout is {{60s}}, and the unit test is also {{60s}} 
for two configurations combined. 

bq.  And, the problem should happen always when the DN is removed right? 

This happens only when the NN schedule the recovery tasks *between* shutting 
down two DNs. If two DNs have been shutdown before scheduling the recovery 
task, none of these two DNs will be considered as valid source. 

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12994) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout

2018-01-09 Thread Manoj Govindassamy (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319023#comment-16319023
 ] 

Manoj Govindassamy commented on HDFS-12994:
---

[~eddyxu], 
  The intention is to let client detect DN issues quicker? And, the problem 
should happen always when the DN is removed right? Just trying to understand 
the core issue. Thanks.

> TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket 
> timeout
> 
>
> Key: HDFS-12994
> URL: https://issues.apache.org/jira/browse/HDFS-12994
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12994.00.patch, HDFS-12994.01.patch
>
>
> Occasionally, {{testNNSendsErasureCodingTasks}} fails due to socket timeout
> {code}
> 2017-12-26 20:35:19,961 [StripedBlockReconstruction-0] INFO  
> datanode.DataNode (StripedBlockReader.java:createBlockReader(132)) - 
> Exception while creating remote block reader, datanode 127.0.0.1:34145
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.newConnectedPeer(StripedBlockReader.java:148)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.createBlockReader(StripedBlockReader.java:123)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReader.(StripedBlockReader.java:83)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.createReader(StripedReader.java:169)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.initReaders(StripedReader.java:150)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.init(StripedReader.java:133)
> at 
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.run(StripedBlockReconstructor.java:56)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> while the target datanode is removed in the test:
> {code}
> 2017-12-26 20:35:18,710 [Thread-2393] INFO  net.NetworkTopology 
> (NetworkTopology.java:remove(219)) - Removing a node: 
> /default-rack/127.0.0.1:34145
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319012#comment-16319012
 ] 

genericqa commented on HDFS-13004:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 13s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 58s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 13s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 51s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestErasureCodingMultipleRacks |
|   | hadoop.hdfs.TestDFSStripedOutputStream |
|   | hadoop.hdfs.server.federation.router.TestRouterRpc |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13004 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905307/HDFS-13004.01.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux a463a4839179 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ebff4de |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22626/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22626/testReport/ |
| Max. process+thread count | 3034 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:

[jira] [Commented] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318881#comment-16318881
 ] 

genericqa commented on HDFS-12794:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  9s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
0s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 5 new + 
0 unchanged - 1 fixed = 5 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 23s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
41s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 10s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 21s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
|   | hadoop.ozone.ozShell.TestOzoneShell |
|   | hadoop.ozone.TestOzoneConfigurationFields |
|   | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | hadoop.ozone.container.common.impl.TestContainerPersistence |
|   | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
|   | hadoop.ozone.client.rpc.TestOzoneRpcClient |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   |

[jira] [Commented] (HDFS-12808) Add LOG.isDebugEnabled() guard for LOG.debug("...")

2018-01-09 Thread Bharat Viswanadham (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318838#comment-16318838
 ] 

Bharat Viswanadham commented on HDFS-12808:
---

[~MehranHassani] Thanks for info.


> Add LOG.isDebugEnabled() guard for LOG.debug("...")
> ---
>
> Key: HDFS-12808
> URL: https://issues.apache.org/jira/browse/HDFS-12808
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mehran Hassani
>Assignee: Bharat Viswanadham
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12808.00.patch, HDFS-12808.01.patch
>
>
> I am conducting research on log related bugs. I tried to make a tool to fix 
> repetitive yet simple patterns of bugs that are related to logs. In this 
> file, there is a debug level logging statement containing multiple string 
> concatenation without the if statement before them: 
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java,
>  LOG.debug("got fadvise(offset=" + offset + ", len=" + len +",flags=" + flags 
> + ")");, 82
> Would you be interested in adding the if,  to the logging statement?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread JIRA


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318792#comment-16318792
 ] 

Íñigo Goiri commented on HDFS-12934:


This looks good to me. Minor:
* Missing license in {{Quota}}
* Complete a couple javadocs that aren't complete (e.g. {{Quota}})
* Fix the two left check styles
* Remove {{isQuotaEnabled}} field from {{Router}}
* {{getQuotaUsageInternal()}} in {{Quota}} should be {{getQuotaUsage()}}, then 
the functionality that doesn't do the {{checkOperaiton}} (log the function) 
used there and by {{RouterQuotaUpdateService()}}.

> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>  Labels: RBF
> Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, 
> HDFS-12934.003.patch, HDFS-12934.004.patch, HDFS-12934.005.patch, 
> HDFS-12934.006.patch, HDFS-12934.007.patch, RBF support  global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder. 
> Currently the quota will be applied for each subcluster under the specified 
> folder via RPC call.
> It will be very useful for users that federation can support setting global 
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple 
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can 
> exceed maximun quota value.
> # We need to construct one  cache map for storing the sum  
> quota usage of these subclusters under federation folder. Every time we want 
> to do WRITE operation under specified folder, we will get its quota usage 
> from cache and verify its quota. If quota exceeded, throw exception, 
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The 
> set/unset command will be like:
> {noformat}
>  hdfs dfsrouteradmin -setQuota -ns  -ss  
>  hdfs dfsrouteradmin -clrQuota  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12802:
---
Attachment: HDFS-12802.005.patch

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch, 
> HDFS-12802.005.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12919:
---
Attachment: HDFS-12919.013.patch

> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
>  Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch, 
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch, 
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch, 
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch, 
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch, 
> HDFS-12919.013.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files. 
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
>  Operation "setErasureCodingPolicy" is not supported
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing

2018-01-09 Thread Zsolt Venczel (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zsolt Venczel updated HDFS-13004:
-
Attachment: HDFS-13004.01.patch

> TestLeaseRecoveryStriped#testLeaseRecovery is failing
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>  Labels: flaky-test
> Fix For: 3.0.1
>
> Attachments: HDFS-13004.01.patch
>
>
> {code}
> Error:
> failed testCase at i=1, 
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824> 
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError: 
> failed testCase at i=1, 
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824> 
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
>

[jira] [Updated] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing

2018-01-09 Thread Zsolt Venczel (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zsolt Venczel updated HDFS-13004:
-
Status: Patch Available  (was: In Progress)

> TestLeaseRecoveryStriped#testLeaseRecovery is failing
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>  Labels: flaky-test
> Fix For: 3.0.1
>
>
> {code}
> Error:
> failed testCase at i=1, 
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824> 
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError: 
> failed testCase at i=1, 
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824> 
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

[jira] [Created] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing

2018-01-09 Thread Zsolt Venczel (JIRA)

Zsolt Venczel created HDFS-13004:


 Summary: TestLeaseRecoveryStriped#testLeaseRecovery is failing
 Key: HDFS-13004
 URL: https://issues.apache.org/jira/browse/HDFS-13004
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Affects Versions: 3.0.0
Reporter: Zsolt Venczel
Assignee: Zsolt Venczel
 Fix For: 3.0.1


{code}
Error:
failed testCase at i=1, 
blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=

{4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
java.lang.AssertionError: File length should be the same expected:<25165824> 
but was:<18874368>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
at 
org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
at 
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
at 
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)

Stack:
java.lang.AssertionError: 
failed testCase at i=1, 
blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
,safeLength=25165824]
java.lang.AssertionError: File length should be the same expected:<25165824> 
but was:<18874368>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
at 
org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
at 
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
at 
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at

[jira] [Work started] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing

2018-01-09 Thread Zsolt Venczel (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-13004 started by Zsolt Venczel.

> TestLeaseRecoveryStriped#testLeaseRecovery is failing
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>  Labels: flaky-test
> Fix For: 3.0.1
>
>
> {code}
> Error:
> failed testCase at i=1, 
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824> 
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError: 
> failed testCase at i=1, 
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824> 
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at 
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at 
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at

[jira] [Commented] (HDFS-12808) Add LOG.isDebugEnabled() guard for LOG.debug("...")

2018-01-09 Thread Mehran Hassani (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318646#comment-16318646
 ] 

Mehran Hassani commented on HDFS-12808:
---

[~bharatviswa] It is up to you! I am not actively using a 2.x release. 

> Add LOG.isDebugEnabled() guard for LOG.debug("...")
> ---
>
> Key: HDFS-12808
> URL: https://issues.apache.org/jira/browse/HDFS-12808
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Mehran Hassani
>Assignee: Bharat Viswanadham
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12808.00.patch, HDFS-12808.01.patch
>
>
> I am conducting research on log related bugs. I tried to make a tool to fix 
> repetitive yet simple patterns of bugs that are related to logs. In this 
> file, there is a debug level logging statement containing multiple string 
> concatenation without the if statement before them: 
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java,
>  LOG.debug("got fadvise(offset=" + offset + ", len=" + len +",flags=" + flags 
> + ")");, 82
> Would you be interested in adding the if,  to the logging statement?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12090) Handling writes from HDFS to Provided storages

2018-01-09 Thread Ewan Higgs (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewan Higgs updated HDFS-12090:
--
Attachment: HDFS-12090..patch

Attached is a work in progress patch for the backup case. It's not beautiful, 
but we'd like to share this as early as possible so the direction we're taking 
this is visible.

This contains a test ( {{ TestGrandSyncServiceTask}} ) that starts a client and 
backup {{MiniDFSCluster}} and can back files up from the client to the backup. 
It's very early days, but should show the direction.

Some of the caveats:

* This patch applies onto 928964102029e96406f5482e8900802f38164501

* We pulled in the HDFS-10285-consolidated-merge-patch-03.patch from the 
StoragePolicySatisfier work. We initially used this patch with the intention of 
working with the SPS. However, as the SyncService works on the file level and 
not the block level, the APIs couldn't be used directly. Therefore we can 
rebase and remove our dependency on this patch.

* The design of the {{SyncServiceSatisfier}} and {{SyncServiceSatisfierWorker}} 
is based on the idea that we will have an object storage on the back end. This 
means that all operations could be reordered. When we run hdfs mapping to hdfs, 
however, the order of operations cannot be reordered!

* Rename and delete tests are flakey (don't work). This is because the 
success/failure reporting hasn't been implemented so the tests incorrectly 
check that files were deleted before the work was done on the backend.

* The snapshotting mechanism doesn't wait until the previous snapshot has been 
completely synchronised before looking to create more work.

* The MountManager works using paths. This will break if the local directory is 
moved. Therefore it should accept a path but apply the sync service 
functionality by holding onto an INode.

* Snapshot tests appear to have been broken.

> Handling writes from HDFS to Provided storages
> --
>
> Key: HDFS-12090
> URL: https://issues.apache.org/jira/browse/HDFS-12090
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Virajith Jalaparti
> Attachments: HDFS-12090-Functional-Specification.001.pdf, 
> HDFS-12090-Functional-Specification.002.pdf, 
> HDFS-12090-Functional-Specification.003.pdf, HDFS-12090-design.001.pdf, 
> HDFS-12090..patch
>
>
> HDFS-9806 introduces the concept of {{PROVIDED}} storage, which makes data in 
> external storage systems accessible through HDFS. However, HDFS-9806 is 
> limited to data being read through HDFS. This JIRA will deal with how data 
> can be written to such {{PROVIDED}} storages from HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12993) Add an option to Balancer for excluding the paths

2018-01-09 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318587#comment-16318587
 ] 

Brahma Reddy Battula commented on HDFS-12993:
-

[~kihwal] thanks for taking look.

bq.getBlocks() is more expensive when there are many smaller blocks. E.g. more 
blocks to go through to gather 10GB worth. Doing path exclusion in such 
clusters can have a huge performance impact.
HDFS-9412 try to address ( as per descrption) but avoided the sending the small 
blocks only.

Implementing the getblockiterator at client side should fine ( where we can 
send start block)..?

> Add an option to Balancer for excluding the paths
> -
>
> Key: HDFS-12993
> URL: https://issues.apache.org/jira/browse/HDFS-12993
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-12993.patch
>
>
> *Usecase:* Users/Customers want to keep the important/frequently used 
> blocks(where blocks are not written with favoured nodes and block pinning is 
> not enabled) when balancer is triggered.This can be useful datacentric jobs 
> which will run frequently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Status: In Progress  (was: Patch Available)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, 
> HDFS-12935.006.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Affects Version/s: 2.9.0
   Status: Patch Available  (was: In Progress)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1, 2.9.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, 
> HDFS-12935.006.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container

2018-01-09 Thread Shashikant Banerjee (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shashikant Banerjee updated HDFS-12794:
---
Attachment: HDFS-12794-HDFS-7240.008.patch

Patch v8 addresses the findbug issue, few necessary checkstyle issues and a 
test case failure due to missing config option in ozone-default.xml introduced 
with patch v7.

Please have a look.

> Ozone: Parallelize ChunkOutputSream Writes to container
> ---
>
> Key: HDFS-12794
> URL: https://issues.apache.org/jira/browse/HDFS-12794
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Fix For: HDFS-7240
>
> Attachments: HDFS-12794-HDFS-7240.001.patch, 
> HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch, 
> HDFS-12794-HDFS-7240.004.patch, HDFS-12794-HDFS-7240.005.patch, 
> HDFS-12794-HDFS-7240.006.patch, HDFS-12794-HDFS-7240.007.patch, 
> HDFS-12794-HDFS-7240.008.patch
>
>
> The chunkOutPutStream Write are sync in nature .Once one chunk of data gets 
> written, the next chunk write is blocked until the previous chunk is written 
> to the container.
> The ChunkOutputWrite Stream writes should be made async and Close on the 
> OutputStream should ensure flushing of all dirty buffers to the container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318524#comment-16318524
 ] 

genericqa commented on HDFS-12935:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 40s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 28s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 29s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.TestRollingUpgrade |
|   | hadoop.hdfs.tools.TestDFSAdminWithHA |
|   | hadoop.hdfs.server.federation.router.TestRouterRpc |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
|   | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905270/HDFS-12935.006.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 4b5c3b16b98d 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 783a01e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318480#comment-16318480
 ] 

genericqa commented on HDFS-12935:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 53s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 374 unchanged - 0 fixed = 377 total (was 374) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 59s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 45s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 33s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905268/HDFS-12935.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 476aa3c259df 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 783a01e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22622/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22622/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results |

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318364#comment-16318364
 ] 

genericqa commented on HDFS-12935:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 26s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 374 unchanged - 0 fixed = 377 total (was 374) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 34s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m  4s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileAppend2 |
|   | hadoop.hdfs.TestDecommissionWithStriped |
|   | hadoop.hdfs.web.TestWebHdfsTimeouts |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
|   | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
|   | hadoop.hdfs.TestSetrepIncreasing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905254/HDFS-12935.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 2cc06a55d8e1 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b3290c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22619/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|

[jira] [Comment Edited] (HDFS-12636) Ozone: OzoneFileSystem: Implement seek functionality for rpc client

2018-01-09 Thread Lokesh Jain (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318140#comment-16318140
 ] 

Lokesh Jain edited comment on HDFS-12636 at 1/9/18 12:38 PM:
-

In offline discussion [~nandakumar131] had pointed that ChunkGroupInputStream 
should not extend FSInputStream class. Therefore I have refactored 
ozone/OzoneInputStream and ozone/OzoneOutputStream to ozone/OzoneFSInputStream 
and ozone/OzoneFSOutputStream respectively. These classes would be used by the 
OzoneFileSystem. Currently they work with rpc client only. But later they would 
be made generic for both rest and rpc client through jira HDFS-13000.


was (Author: ljain):
In offline discussion [~nandakumar131] had pointed that ChunkGroupInputStream 
should not extend FSInputStream class. Therefore I have refactored 
ozone/OzoneInputStream and ozone/OzoneOutputStream to ozone/OzoneFSInputStream 
and ozone/OzoneFSInputStream respectively. These classes would be used by the 
OzoneFileSystem. Currently they work with rpc client only. But later they would 
be made generic for both rest and rpc client through jira HDFS-13000.

> Ozone: OzoneFileSystem: Implement seek functionality for rpc client
> ---
>
> Key: HDFS-12636
> URL: https://issues.apache.org/jira/browse/HDFS-12636
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Lokesh Jain
> Fix For: HDFS-7240
>
> Attachments: HDFS-12636-HDFS-7240.001.patch, 
> HDFS-12636-HDFS-7240.002.patch
>
>
> OzoneClient library provides a method to invoke both RPC as well as REST 
> based methods to ozone. This api will help in the improving both the 
> performance as well as the interface management in OzoneFileSystem.
> This jira will be used to convert the REST based calls to use this new 
> unified client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option

2018-01-09 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318339#comment-16318339
 ] 

Brahma Reddy Battula commented on HDFS-13003:
-

[~sree88] thanks for reporting this.

it make sense to give "0" as other fileds are zero.

Added you as HDFS contributor,from now onwards you can assign issues your self.

Please go through [How to 
contribute|https://wiki.apache.org/hadoop/HowToContribute] for more details.

> [OIV] Irrelevant access time for the dirs when run with Delimited option  
> --
>
> Key: HDFS-13003
> URL: https://issues.apache.org/jira/browse/HDFS-13003
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Sreejith MV
>Priority: Minor
>
> *#hdfs oiv -i fsimage_0018779 -p Delimited -o test*
> 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading 
> string table
> 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 
> strings
> 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading 
> inode references
> 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode 
> references
> 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 
> inode references
> 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories
> 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories in INode section.
> 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 
> directories in INode section.
> 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished 
> loading directories in 25ms
> 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading 
> INode directory section.
> 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 
> INode directories to build namespace.
> 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished 
> loading INode directory section in 3ms
> *#cat test*
> PathReplication ModificationTime*AccessTime*  
> PreferredBlockSize  BlocksCount FileSizeNSQUOTA DSQUOTA 
> Permission  UserNameGroupName
> /dirSetExpr40   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr60   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr30   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr20   2017-12-28 23:39*1970-01-01 08:00 *   0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr70   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr5/fsumm261970.txt3   2017-12-28 22:102017-12-28 
> 22:101048576 1   38  0   0   -rw-r--r--  root   
> hadoop
> /dirSetExpr6/fsumm261970.txt3   2017-12-28 22:102017-12-28 
> 22:101048576 1   38  0   0   -rw-r--r--  root   
> hadoop



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12986) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-00f80b4-SNAPSHOT)

2018-01-09 Thread Mukul Kumar Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318337#comment-16318337
 ] 

Mukul Kumar Singh commented on HDFS-12986:
--

The latest patch looks good to me. I am working on deploying this patch on 
cluster and testing it.

> Ozone: Update ozone to latest ratis snapshot build 
> (0.1.1-alpha-00f80b4-SNAPSHOT)
> -
>
> Key: HDFS-12986
> URL: https://issues.apache.org/jira/browse/HDFS-12986
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Lokesh Jain
> Fix For: HDFS-7240
>
> Attachments: HDFS-12986-HDFS-7240.001.patch, 
> HDFS-12986-HDFS-7240.002.patch, HDFS-12986-HDFS-7240.003.patch, 
> HDFS-12986-HDFS-7240.004.patch
>
>
> This jira will update ozone to latest snapshot 
> release-0.1.1-alpha-00f80b4-SNAPSHOT



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option

2018-01-09 Thread Brahma Reddy Battula (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula reassigned HDFS-13003:
---

Assignee: Sreejith MV

> [OIV] Irrelevant access time for the dirs when run with Delimited option  
> --
>
> Key: HDFS-13003
> URL: https://issues.apache.org/jira/browse/HDFS-13003
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Sreejith MV
>Assignee: Sreejith MV
>Priority: Minor
>
> *#hdfs oiv -i fsimage_0018779 -p Delimited -o test*
> 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading 
> string table
> 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 
> strings
> 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading 
> inode references
> 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode 
> references
> 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 
> inode references
> 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories
> 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories in INode section.
> 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 
> directories in INode section.
> 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished 
> loading directories in 25ms
> 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading 
> INode directory section.
> 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 
> INode directories to build namespace.
> 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished 
> loading INode directory section in 3ms
> *#cat test*
> PathReplication ModificationTime*AccessTime*  
> PreferredBlockSize  BlocksCount FileSizeNSQUOTA DSQUOTA 
> Permission  UserNameGroupName
> /dirSetExpr40   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr60   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr30   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr20   2017-12-28 23:39*1970-01-01 08:00 *   0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr70   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr5/fsumm261970.txt3   2017-12-28 22:102017-12-28 
> 22:101048576 1   38  0   0   -rw-r--r--  root   
> hadoop
> /dirSetExpr6/fsumm261970.txt3   2017-12-28 22:102017-12-28 
> 22:101048576 1   38  0   0   -rw-r--r--  root   
> hadoop



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318328#comment-16318328
 ] 

genericqa commented on HDFS-12935:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} HDFS-12935 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905275/HDFS-12935.006-branch.2.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22624/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, 
> HDFS-12935.006.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: HDFS-12935.006-branch.2.patch

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006-branch.2.patch, 
> HDFS-12935.006.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12986) Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-00f80b4-SNAPSHOT)

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318314#comment-16318314
 ] 

genericqa commented on HDFS-12986:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 10s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-client-modules/hadoop-client-runtime {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
39s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 10s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-client-modules/hadoop-client-runtime {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
17s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}155m 54s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
22s{color} | {color:green} hadoop-client-runtime in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF

[jira] [Commented] (HDFS-12999) When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318301#comment-16318301
 ] 

genericqa commented on HDFS-12999:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  9m 
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
 6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 44s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 57s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
22s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12999 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905260/HDFS-12999.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux c18949a0d3c7 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b3290c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22621/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: 
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22621/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> When reach the end of the block

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Status: Patch Available  (was: In Progress)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006.patch, 
> HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: HDFS-12935.006.patch

Fix the checkstyle.

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS-12935.006.patch, 
> HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Status: In Progress  (was: Patch Available)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12934) RBF: Federation supports global quota

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318278#comment-16318278
 ] 

genericqa commented on HDFS-12934:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  3m 
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 29s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 432 unchanged - 0 fixed = 434 total (was 432) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 59s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 15s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
23s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12934 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905243/HDFS-12934.007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  cc  xml  |
| uname | Linux 02dc8d062a77 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b3290c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22617/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit |

[jira] [Commented] (HDFS-13002) Ulimit log is overridden by the balancer sysout log in balancer out file

2018-01-09 Thread Ranith Sardar (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318273#comment-16318273
 ] 

Ranith Sardar commented on HDFS-13002:
--

This problem will come when there are no blocks to move(i.e exiting the 
balancer immediately)
I am planning to fix like below
1.Putting sleep
 {code}

if [ $command = "balancer" ]; then
sleep 20;
fi
{code}
2.Redirecting to seperate file

would like to provide the patch for same.

> Ulimit log is overridden by the balancer sysout log in balancer out file
> 
>
> Key: HDFS-13002
> URL: https://issues.apache.org/jira/browse/HDFS-13002
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Priority: Minor
>
> Whenever the balancer starts, it will redirect the sys out to the out log 
> file.  And balancer writes the system output to the log file, at the same 
> time  script also will try to append ulimit output. 
> {noformat}
>  # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
>   echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
>   # capture the ulimit info for the appropriate user
>   su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER" 
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >> 
> $log 2>&1
> else
>   echo "ulimit -a for user $USER" >> $log
>   ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
>   exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of 
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat 
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp   Iteration#  Bytes Already Moved  Bytes Left To Move  
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0  0 B 0 B  
>   0 B
> Jan 9, 2018 6:26:26 PM   Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files  (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority  (-r) 0
> stack size  (kbytes, -s) 8192
> cpu time   (seconds, -t) unlimited
> max user processes  (-u) 127350
> virtual memory  (kbytes, -v) 15992160
> file locks  (-x) unlimited
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318271#comment-16318271
 ] 

genericqa commented on HDFS-12935:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
11s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
12s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red}  0m 
31s{color} | {color:red} branch has errors when building and testing our client 
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
10s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 10s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
13s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red}  0m 
10s{color} | {color:red} patch has errors when building and testing our client 
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
11s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 10s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m  0s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12904522/HDFS-12935.004.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 1c7fe3bf7932 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 783a01e |
| maven | version: Apache Maven 3.3.9 
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22620/artifact/out/branch-mvninstall-root.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22620/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| mvnsite | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22620/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22620/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: HDFS-12935.005.patch

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option

2018-01-09 Thread Sreejith MV (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318261#comment-16318261
 ] 

Sreejith MV commented on HDFS-13003:


I think the issue is because of the below code snippet.

PBImageDelimitedTextWriter.getEntry(String, INode)
append(buffer, formatDate(0));  // Access time.


We can give "0" instead of "formatDate(0)"  to fix the issue.


Can anyone please assign this issue to me?


> [OIV] Irrelevant access time for the dirs when run with Delimited option  
> --
>
> Key: HDFS-13003
> URL: https://issues.apache.org/jira/browse/HDFS-13003
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Sreejith MV
>Priority: Minor
>
> *#hdfs oiv -i fsimage_0018779 -p Delimited -o test*
> 2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading 
> string table
> 2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 
> strings
> 2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading 
> inode references
> 2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode 
> references
> 2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 
> inode references
> 2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories
> 2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories in INode section.
> 2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 
> directories in INode section.
> 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished 
> loading directories in 25ms
> 2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading 
> INode directory section.
> 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 
> INode directories to build namespace.
> 2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished 
> loading INode directory section in 3ms
> *#cat test*
> PathReplication ModificationTime*AccessTime*  
> PreferredBlockSize  BlocksCount FileSizeNSQUOTA DSQUOTA 
> Permission  UserNameGroupName
> /dirSetExpr40   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr60   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr30   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr20   2017-12-28 23:39*1970-01-01 08:00 *   0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr70   2017-12-28 23:39*1970-01-01 08:00*0   
> 0   0   -1  -1  drwxr-xr-x  root   hadoop
> /dirSetExpr5/fsumm261970.txt3   2017-12-28 22:102017-12-28 
> 22:101048576 1   38  0   0   -rw-r--r--  root   
> hadoop
> /dirSetExpr6/fsumm261970.txt3   2017-12-28 22:102017-12-28 
> 22:101048576 1   38  0   0   -rw-r--r--  root   
> hadoop



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: (was: HDFS-12935.005.patch)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: (was: HDFS-12935.005.branch-2.patch)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-13003) [OIV] Irrelevant access time for the dirs when run with Delimited option

2018-01-09 Thread Sreejith MV (JIRA)

Sreejith MV created HDFS-13003:
--

 Summary: [OIV] Irrelevant access time for the dirs when run with 
Delimited option  
 Key: HDFS-13003
 URL: https://issues.apache.org/jira/browse/HDFS-13003
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sreejith MV
Priority: Minor


*#hdfs oiv -i fsimage_0018779 -p Delimited -o test*
2018-01-09 18:37:13,218 INFO offlineImageViewer.PBImageTextWriter: Loading 
string table
2018-01-09 18:37:13,227 INFO offlineImageViewer.FSImageHandler: Loading 2 
strings
2018-01-09 18:37:13,229 INFO offlineImageViewer.PBImageTextWriter: Loading 
inode references
2018-01-09 18:37:13,229 INFO offlineImageViewer.FSImageHandler: Loading inode 
references
2018-01-09 18:37:13,235 INFO offlineImageViewer.FSImageHandler: Loaded 0 inode 
references
2018-01-09 18:37:13,236 INFO offlineImageViewer.PBImageTextWriter: Loading 
directories
2018-01-09 18:37:13,240 INFO offlineImageViewer.PBImageTextWriter: Loading 
directories in INode section.
2018-01-09 18:37:13,263 INFO offlineImageViewer.PBImageTextWriter: Found 1 
directories in INode section.
2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Finished 
loading directories in 25ms
2018-01-09 18:37:13,264 INFO offlineImageViewer.PBImageTextWriter: Loading 
INode directory section.
2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Scanned 0 
INode directories to build namespace.
2018-01-09 18:37:13,267 INFO offlineImageViewer.PBImageTextWriter: Finished 
loading INode directory section in 3ms

*#cat test*
PathReplication ModificationTime*AccessTime*  
PreferredBlockSize  BlocksCount FileSizeNSQUOTA DSQUOTA 
Permission  UserNameGroupName
/dirSetExpr40   2017-12-28 23:39*1970-01-01 08:00*0 
  0   0   -1  -1  drwxr-xr-x  root   hadoop
/dirSetExpr60   2017-12-28 23:39*1970-01-01 08:00*0 
  0   0   -1  -1  drwxr-xr-x  root   hadoop
/dirSetExpr30   2017-12-28 23:39*1970-01-01 08:00*0 
  0   0   -1  -1  drwxr-xr-x  root   hadoop
/dirSetExpr20   2017-12-28 23:39*1970-01-01 08:00 *   0 
  0   0   -1  -1  drwxr-xr-x  root   hadoop
/dirSetExpr70   2017-12-28 23:39*1970-01-01 08:00*0 
  0   0   -1  -1  drwxr-xr-x  root   hadoop
/dirSetExpr5/fsumm261970.txt3   2017-12-28 22:102017-12-28 
22:101048576 1   38  0   0   -rw-r--r--  root   
hadoop
/dirSetExpr6/fsumm261970.txt3   2017-12-28 22:102017-12-28 
22:101048576 1   38  0   0   -rw-r--r--  root   
hadoop





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318246#comment-16318246
 ] 

genericqa commented on HDFS-12935:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 54s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 45s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 374 unchanged - 0 fixed = 377 total (was 374) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 24s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 28s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 42s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12935 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905235/HDFS-12935.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 564c26f2beb8 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b3290c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22614/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22614/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22614/testReport/ |
|

[jira] [Created] (HDFS-13002) Ulimit log is overridden by the balancer sysout log in balancer out file

2018-01-09 Thread Ranith Sardar (JIRA)

Ranith Sardar created HDFS-13002:


 Summary: Ulimit log is overridden by the balancer sysout log in 
balancer out file
 Key: HDFS-13002
 URL: https://issues.apache.org/jira/browse/HDFS-13002
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer & mover
Affects Versions: 2.7.2
Reporter: Ranith Sardar
Priority: Minor


Whenever the balancer starts, it will redirect the sys out to the out log file. 
 And balancer writes the system output to the log file, at the same time  
script also will try to append ulimit output. 
{noformat}
 # capture the ulimit output
if [ "true" = "$starting_secure_dn" ]; then
  echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
  # capture the ulimit info for the appropriate user
  su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
elif [ "true" = "$starting_privileged_nfs" ]; then
echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER" >> 
$log
su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >> $log 
2>&1
else
  echo "ulimit -a for user $USER" >> $log
  ulimit -a >> $log 2>&1
fi
sleep 3;
if ! ps -p $! > /dev/null ; then
  exit 1
fi

{noformat}
But the problem is first few lines of ulimit is overridding by the log of 
balancer.

{noformat}
vm1:/opt/install/hadoop/namenode/sbin # cat 
/opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
Time Stamp   Iteration#  Bytes Already Moved  Bytes Left To Move  
Bytes Being Moved
The cluster is balanced. Exiting...
Jan 9, 2018 6:26:26 PM0  0 B 0 B
0 B
Jan 9, 2018 6:26:26 PM   Balancing took 3.446 seconds
x memory size (kbytes, -m) 13428300
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 127350
virtual memory  (kbytes, -v) 15992160
file locks  (-x) unlimited
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: HDFS-12935.005.branch-2.patch

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.branch-2.patch, HDFS-12935.005.patch, 
> HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-13001) Testcase improvement for DFSAdmin

2018-01-09 Thread Jianfei Jiang (JIRA)

Jianfei Jiang created HDFS-13001:


 Summary: Testcase improvement for DFSAdmin
 Key: HDFS-13001
 URL: https://issues.apache.org/jira/browse/HDFS-13001
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 3.0.0, 2.9.0
Reporter: Jianfei Jiang
Assignee: Jianfei Jiang
Priority: Minor


Testcase improvement for DFSAdmin command. The commands should be tested under 
following environments:
(1) Both Namenode are up online
(2) NN1 is off offline and NN2 is up online
(3) NN1 is up online and NN2 is down offline
(4) Both Namenode are down offline
The testcases can be improved.

Testcases can be improved like code below.

{code:java}
  private void testExecuteDFSAdminCommand(int nnIndex, String[] command,
  String message) throws Exception {
setUpHaCluster(false);
switch (nnIndex) {
  case 0:
cluster.getDfsCluster().shutdownNameNode(0);
cluster.getDfsCluster().transitionToActive(1);
break;
  case 1:
cluster.getDfsCluster().shutdownNameNode(1);
cluster.getDfsCluster().transitionToActive(0);
break;
  case 2:
cluster.getDfsCluster().shutdownNameNode(0);
cluster.getDfsCluster().shutdownNameNode(1);
break;
  default:

}
int exitCode = admin.run(command);
if (nnIndex != 2) {
  assertEquals(err.toString().trim(), 0, exitCode);
  assertOutputMatches(message + newLine);
} else {
  assertNotEquals(err.toString().trim(), 0, exitCode);
  assertOutputNotMatches(message + newLine);
}
  }
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12999) When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.

2018-01-09 Thread lufei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lufei updated HDFS-12999:
-
Attachment: HDFS-12999.002.patch

> When reach the end of the block group, it may not need to flush all the data 
> packets(flushAllInternals) twice. 
> ---
>
> Key: HDFS-12999
> URL: https://issues.apache.org/jira/browse/HDFS-12999
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding, hdfs-client
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: lufei
>Assignee: lufei
> Fix For: 3.1.0
>
> Attachments: HDFS-12999.001.patch, HDFS-12999.002.patch
>
>
> In order to make the process simplification. It's no need to flush all the 
> data packets(flushAllInternals) twice,when reach the end of the block group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12802) RBF: Control MountTableResolver cache size

2018-01-09 Thread Yiqun Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318204#comment-16318204
 ] 

Yiqun Lin commented on HDFS-12802:
--

Thanks [~elgoiri] for updating patch. Overall looks good now. Just catching one 
nit: Prefix matching seems incorrect
{noformat}
+  String src = loc.getSourcePath();
+  if (src.startsWith(src)) {
+LOG.debug("Removing {}", src);
+it.remove();
{noformat}

> RBF: Control MountTableResolver cache size
> --
>
> Key: HDFS-12802
> URL: https://issues.apache.org/jira/browse/HDFS-12802
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-12802.000.patch, HDFS-12802.001.patch, 
> HDFS-12802.002.patch, HDFS-12802.003.patch, HDFS-12802.004.patch
>
>
> Currently, the {{MountTableResolver}} caches the resolutions for the 
> {{PathLocation}}. However, this cache can grow with no limits if there are a 
> lot of unique paths. Some of these cached resolutions might not be used at 
> all.
> The {{MountTableResolver}} should clean the {{locationCache}} periodically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer

2018-01-09 Thread Yiqun Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318177#comment-16318177
 ] 

Yiqun Lin commented on HDFS-12919:
--

I'm not so familiar with HDFS Erasure coding. Just some minor comments for the 
unit tests:
{{TestRouterRpc.java}}
# can we create a log instance to print detail info instead of using system out?
# If these lines are not needed, just remove them
{noformat}
   /*
+assertEquals(1, response.length);
+assertTrue(response[0].isSucceed());
+*/
{noformat}
# Following line seems incorrect.
{code}
+// Remove the policy and check the one for the test folder
+routerProtocol.removeErasureCodingPolicy(newPolicyName);
+ErasureCodingPolicy policyRouter3 =
+routerProtocol.getErasureCodingPolicy(dirPath);
+assertEquals(newPolicyName, policyRouter3.getName());
+ErasureCodingPolicy policyNamenode3 =
+routerProtocol.getErasureCodingPolicy(dirPath);   < should use 
nnProtocol to get
+assertEquals(newPolicyName, policyNamenode3.getName());
{code}

> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
>  Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch, 
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch, 
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch, 
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch, 
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files. 
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
>  Operation "setErasureCodingPolicy" is not supported
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at 
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9049) Make Datanode Netty reverse proxy port to be configurable

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318172#comment-16318172
 ] 

genericqa commented on HDFS-9049:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 31s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  0s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}126m 37s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 49s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
|   | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | hadoop.hdfs.TestRollingUpgrade |
|   | hadoop.hdfs.TestErasureCodingMultipleRacks |
|   | hadoop.hdfs.TestFileCorruption |
|   | hadoop.hdfs.TestLeaseRecoveryStriped |
|   | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-9049 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905228/HDFS-9049-04.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux bba47794d8aa 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |

[jira] [Commented] (HDFS-12999) When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318141#comment-16318141
 ] 

genericqa commented on HDFS-12999:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 13s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-client: The 
patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 36s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12999 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905247/HDFS-12999.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 64666ecaafb1 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b3290c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22618/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22618/testReport/ |
| Max. process+thread count | 441 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: 
hadoop-hdfs-project/hadoop-hdfs-client |

[jira] [Commented] (HDFS-12636) Ozone: OzoneFileSystem: Implement seek functionality for rpc client

2018-01-09 Thread Lokesh Jain (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318140#comment-16318140
 ] 

Lokesh Jain commented on HDFS-12636:


In offline discussion [~nandakumar131] had pointed that ChunkGroupInputStream 
should not extend FSInputStream class. Therefore I have refactored 
ozone/OzoneInputStream and ozone/OzoneOutputStream to ozone/OzoneFSInputStream 
and ozone/OzoneFSInputStream respectively. These classes would be used by the 
OzoneFileSystem. Currently they work with rpc client only. But later they would 
be made generic for both rest and rpc client through jira HDFS-13000.

> Ozone: OzoneFileSystem: Implement seek functionality for rpc client
> ---
>
> Key: HDFS-12636
> URL: https://issues.apache.org/jira/browse/HDFS-12636
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Lokesh Jain
> Fix For: HDFS-7240
>
> Attachments: HDFS-12636-HDFS-7240.001.patch, 
> HDFS-12636-HDFS-7240.002.patch
>
>
> OzoneClient library provides a method to invoke both RPC as well as REST 
> based methods to ozone. This api will help in the improving both the 
> performance as well as the interface management in OzoneFileSystem.
> This jira will be used to convert the REST based calls to use this new 
> unified client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-13000) Ozone: OzoneFileSystem: Implement seek functionality for rest client

2018-01-09 Thread Lokesh Jain (JIRA)

Lokesh Jain created HDFS-13000:
--

 Summary: Ozone: OzoneFileSystem: Implement seek functionality for 
rest client
 Key: HDFS-13000
 URL: https://issues.apache.org/jira/browse/HDFS-13000
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Lokesh Jain
Assignee: Lokesh Jain


This jira aims to add seekable functionality in rest client input stream.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Status: Patch Available  (was: Open)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: HDFS-12935.005.patch

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS-12935.005.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12636) Ozone: OzoneFileSystem: Implement seek functionality for rpc client

2018-01-09 Thread Lokesh Jain (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Jain updated HDFS-12636:
---
Summary: Ozone: OzoneFileSystem: Implement seek functionality for rpc 
client  (was: Ozone: OzoneFileSystem: rpc backend should be supported using 
unified OzoneClient client)

> Ozone: OzoneFileSystem: Implement seek functionality for rpc client
> ---
>
> Key: HDFS-12636
> URL: https://issues.apache.org/jira/browse/HDFS-12636
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Lokesh Jain
> Fix For: HDFS-7240
>
> Attachments: HDFS-12636-HDFS-7240.001.patch, 
> HDFS-12636-HDFS-7240.002.patch
>
>
> OzoneClient library provides a method to invoke both RPC as well as REST 
> based methods to ozone. This api will help in the improving both the 
> performance as well as the interface management in OzoneFileSystem.
> This jira will be used to convert the REST based calls to use this new 
> unified client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Status: Open  (was: Patch Available)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0, 3.0.0-beta1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up

2018-01-09 Thread Jianfei Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfei Jiang updated HDFS-12935:
-
Attachment: (was: HDFS-12935.005.patch)

> Get ambiguous result for DFSAdmin command in HA mode when only one namenode 
> is up
> -
>
> Key: HDFS-12935
> URL: https://issues.apache.org/jira/browse/HDFS-12935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-beta1, 3.0.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
> Attachments: HDFS-12935.002.patch, HDFS-12935.003.patch, 
> HDFS-12935.004.patch, HDFS_12935.001.patch
>
>
> In HA mode, if one namenode is down, most of functions can still work. When 
> considering the following two occasions:
>  (1)nn1 up and nn2 down
>  (2)nn1 down and nn2 up
> These two occasions should be equivalent. However, some of the DFSAdmin 
> commands will have ambiguous results. The commands can be send successfully 
> to the up namenode and are always functionally useful only when nn1 is up 
> regardless of exception (IOException when connecting to the down namenode 
> nn2). If only nn2 is up, the commands have no use at all and only exception 
> to connect nn1 can be found.
> See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to 
> set balancer bandwidth value for datanodes as an example. It works and all 
> the datanodes can get the setting values only when nn1 is up. If only nn2 is 
> up, the command throws exception directly and no datanode get the bandwidth 
> setting. Approximately ten DFSAdmin commands use the similar logical process 
> and may be ambiguous.
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345
> *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820*
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei02:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2
> active
> [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234
> setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to 
> jiangjianfei01:9820 failed on connection exception: 
> java.net.ConnectException: Connection refused; For more details see:  
> http://wiki.apache.org/hadoop/ConnectionRefused
> [root@jiangjianfei01 ~]# 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12462) Erasure coding policy extra options should be sorted by key value

2018-01-09 Thread liaoyuxiangqin (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318123#comment-16318123
 ] 

liaoyuxiangqin commented on HDFS-12462:
---

Thanks [~Sammi] for detailed explanation, I have read the save ec policy to 
fsimg related code and process processing, but i still have a doubt about What 
scenario needs to  compare between two fsImage or between  fsImage with 
editlog, thanks.

> Erasure coding policy extra options should be sorted by key value
> -
>
> Key: HDFS-12462
> URL: https://issues.apache.org/jira/browse/HDFS-12462
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Reporter: SammiChen
>  Labels: hdfs-ec-3.0-nice-to-have
>
> To make sure the serialized fsimage and editlog binary equal, Erasure coding 
> policy extra options should be sorted by key value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits

2018-01-09 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318074#comment-16318074
 ] 

genericqa commented on HDFS-12733:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  5s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 14s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 37s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
|   | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
|   | hadoop.hdfs.web.TestWebHdfsTimeouts |
|   | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12733 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12905222/HDFS-12733-003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 4e548b8e3c49 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b3290c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/22612/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results |

[jira] [Updated] (HDFS-12999) When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.

2018-01-09 Thread lufei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lufei updated HDFS-12999:
-
Status: Patch Available  (was: Open)

> When reach the end of the block group, it may not need to flush all the data 
> packets(flushAllInternals) twice. 
> ---
>
> Key: HDFS-12999
> URL: https://issues.apache.org/jira/browse/HDFS-12999
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding, hdfs-client
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: lufei
>Assignee: lufei
> Fix For: 3.1.0
>
> Attachments: HDFS-12999.001.patch
>
>
> In order to make the process simplification. It's no need to flush all the 
> data packets(flushAllInternals) twice,when reach the end of the block group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12999) When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.

2018-01-09 Thread lufei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-12999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lufei updated HDFS-12999:
-
Attachment: HDFS-12999.001.patch

> When reach the end of the block group, it may not need to flush all the data 
> packets(flushAllInternals) twice. 
> ---
>
> Key: HDFS-12999
> URL: https://issues.apache.org/jira/browse/HDFS-12999
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding, hdfs-client
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: lufei
>Assignee: lufei
> Fix For: 3.1.0
>
> Attachments: HDFS-12999.001.patch
>
>
> In order to make the process simplification. It's no need to flush all the 
> data packets(flushAllInternals) twice,when reach the end of the block group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

1 2 >

1 - 100 of 104 matches

Mail list logo