[jira] [Updated] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN

[star (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

star updated HDFS-14314:

Attachment: HDFS-14314.0.patch

> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.4
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314.0.patch, HDFS-14314.patch
>
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } catch(RemoteException re) {
>   
>   } // while (shouldRun())
> } // offerService{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-1156) testDelegationToken is failing in TestSecureOzoneCluster

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar reassigned HDDS-1156:


Assignee: Ajay Kumar

> testDelegationToken is failing in TestSecureOzoneCluster 
> -
>
> Key: HDDS-1156
> URL: https://issues.apache.org/jira/browse/HDDS-1156
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Ajay Kumar
>Priority: Major
>
> TestSecureOzoneCluster.testDelegationTokenRenewal is failing:
> {code:java}
> java.lang.IllegalArgumentException: Invalid argument: token is null
>   at 
> org.apache.hadoop.ozone.protocolPB.OMPBHelper.convertToTokenProto(OMPBHelper.java:139)
>   at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.renewDelegationToken(OzoneManagerProtocolClientSideTranslatorPB.java:1062)
>   at 
> org.apache.hadoop.ozone.TestSecureOzoneCluster.testDelegationToken(TestSecureOzoneCluster.java:438)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
> TestSecureOzoneCluster.testDelegationToken is failing:
> {code:java}
> java.lang.IllegalArgumentException: Invalid argument: token is null
>   at 
> org.apache.hadoop.ozone.protocolPB.OMPBHelper.convertToTokenProto(OMPBHelper.java:139)
>   at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.renewDelegationToken(OzoneManagerProtocolClientSideTranslatorPB.java:1062)
>   at 
> org.apache.hadoop.ozone.TestSecureOzoneCluster.testDelegationToken(TestSecureOzoneCluster.java:438)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1043) Enable token based authentication for S3 api

[Ajay Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776138#comment-16776138
 ] 

Ajay Kumar commented on HDDS-1043:
--

[~bharatviswa] thanks for review.
{quote}Can we use already existing Header parsers AuthorizationHeaderV4 and 
AuthorizationHeaderV2.java instead of parsing it again in new class 
AWSV4AuthParser. Same comment for V2 parser. And also can we added reference 
links, so that it will be easy to refer aws header documentation.
 And also we have AuthenticationHeaderParser which checks type V2 and V4. And 
then do required. I think we should do similar checks in 
OzoneClientProducer.java and then create token?
{quote}
Done
{quote}In OzoneDelegationTokenSecretManager.java, we call getS3Secret by 
awsaccesskeyid, but during createS3Secret we pass user login name. I think this 
logic should be modified.
{quote}
client can configure aws access id to whatever id they received during s3 
secret generation.
{code}License Header for new classes is wrongly added, it has some GPL header. 
This needs to be updated.\{code}
{quote}Thanks for catching, done.
 Can we add end to end robot test to make sure whether this header parsing and 
validation is happening correctly or not. Already we have tests which configure 
s3 robot tests.(Where we have configured, some random values, now this can be 
set using crateS3secret) Or to have a more robust testing, we can have all S3 
tests run with secure cluster. I think 2nd approach will be good to have.
{quote}
Done. Adding robot test to secure suite resulted in some overflowing changes to 
other scripts as well.

> Enable token based authentication for S3 api
> 
>
> Key: HDDS-1043
> URL: https://issues.apache.org/jira/browse/HDDS-1043
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
>  Labels: security
> Fix For: 0.4.0
>
> Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch, 
> HDDS-1043.02.patch
>
>
> Ozone has a  S3 api and mechanism to create S3 like secrets for user. This 
> jira proposes hadoop compatible token based authentication for S3 api which 
> utilizes S3 secret stored in OM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1043) Enable token based authentication for S3 api

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar updated HDDS-1043:
-
Attachment: HDDS-1043.02.patch

> Enable token based authentication for S3 api
> 
>
> Key: HDDS-1043
> URL: https://issues.apache.org/jira/browse/HDDS-1043
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
>  Labels: security
> Fix For: 0.4.0
>
> Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch, 
> HDDS-1043.02.patch
>
>
> Ozone has a  S3 api and mechanism to create S3 like secrets for user. This 
> jira proposes hadoop compatible token based authentication for S3 api which 
> utilizes S3 secret stored in OM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776135#comment-16776135
 ] 

Hadoop QA commented on HDDS-1168:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 58s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 11s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 23s{color} 
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.node.TestDeadNodeHandler |
|   | hadoop.hdds.scm.server.TestSCMClientProtocolServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1168 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959926/HDDS-1168.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 26dc31bd3cf8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c1e5b19 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2355/artifact/out/patch-unit-hadoop-hdds_server-scm.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2355/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2355/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   

[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776133#comment-16776133
 ] 

Hudson commented on HDDS-1120:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16044 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16044/])
HDDS-1120. Add a config to disable checksum verification during read. (github: 
rev 106bdc6c0474480654aa27cd7b8b50342f69e707)
* (edit) 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
* (edit) hadoop-hdds/common/src/main/resources/ozone-default.xml
* (edit) 
hadoop-ozone/objectstore-service/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java
* (edit) 
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyInputStream.java
* (edit) 
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
* (edit) 
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
* (edit) 
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java
* (edit) 
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestChunkStreams.java


> Add a config to disable checksum verification during read even though 
> checksum data is present in the persisted data
> 
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, if the checksum is computed during data write and persisted in the 
> disk, we will always end up verifying it while reading. This Jira aims to 
> selectively disable checksum verification during reads even though checksum 
> info is present in the data stored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDDS-1120:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

I have committed this to trunk.

> Add a config to disable checksum verification during read even though 
> checksum data is present in the persisted data
> 
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, if the checksum is computed during data write and persisted in the 
> disk, we will always end up verifying it while reading. This Jira aims to 
> selectively disable checksum verification during reads even though checksum 
> info is present in the data stored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776130#comment-16776130
 ] 

Bharat Viswanadham commented on HDDS-1120:
--

Thank You [~linyiqun] for the review. I have committed this to trunk. For ease 
of review I have done as 2 commits. During commit to trunk, I did squash and 
merge, it will be showing as a single commit.

 

> Add a config to disable checksum verification during read even though 
> checksum data is present in the persisted data
> 
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, if the checksum is computed during data write and persisted in the 
> disk, we will always end up verifying it while reading. This Jira aims to 
> selectively disable checksum verification during reads even though checksum 
> info is present in the data stored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1120?focusedWorklogId=203182=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-203182
 ]

ASF GitHub Bot logged work on HDDS-1120:


Author: ASF GitHub Bot
Created on: 24/Feb/19 05:37
Start Date: 24/Feb/19 05:37
Worklog Time Spent: 10m 
  Work Description: bharatviswa504 commented on pull request #513: 
HDDS-1120. Add a config to disable checksum verification during read …
URL: https://github.com/apache/hadoop/pull/513
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 203182)
Time Spent: 40m  (was: 0.5h)

> Add a config to disable checksum verification during read even though 
> checksum data is present in the persisted data
> 
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, if the checksum is computed during data write and persisted in the 
> disk, we will always end up verifying it while reading. This Jira aims to 
> selectively disable checksum verification during reads even though checksum 
> info is present in the data stored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776129#comment-16776129
 ] 

Hudson commented on HDDS-1155:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16043 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16043/])
HDDS-1155.Fix failing unit test methods of TestDeadNodeHandler. (aengineer: rev 
490206e4b4fbd4869940c9689e414bdc977aa405)
* (edit) 
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/HddsTestUtils.java
* (edit) 
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java
* (edit) 
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java
* (add) 
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestStatisticsUpdate.java


> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch, 
> HDDS-1155.003.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDDS-1155:
---
   Resolution: Fixed
Fix Version/s: 0.4.0
   Status: Resolved  (was: Patch Available)

[~nandakumar131] Thanks for the fix. [~elek] Thanks for filing the issue. I 
have committed this to the trunk.

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch, 
> HDDS-1155.003.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Attachment: HDDS-1168.002.patch

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch, HDDS-1168.001.patch, 
> HDDS-1168.002.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776125#comment-16776125
 ] 

Nanda kumar commented on HDDS-1168:
---

[~anu], thanks for the review. Updated the patch on top of latest changes in 
trunk.

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch, HDDS-1168.001.patch, 
> HDDS-1168.002.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776124#comment-16776124
 ] 

Hudson commented on HDDS-1147:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16042 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16042/])
HDDS-1147. Fix failing unit tests in TestOzoneManager. Contributed by 
(aengineer: rev c1e5b1921235316780dc439293b5fa1d3718ba3a)
* (edit) 
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java
* (edit) 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManager.java
* (edit) 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java


> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1147.000.patch, HDDS-1147.001.patch
>
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDDS-1147:
---
   Resolution: Fixed
Fix Version/s: 0.4.0
   Status: Resolved  (was: Patch Available)

[~nandakumar131] Thank you for your contribution. [~hanishakoneru] Thanks for 
filing the issue.

> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1147.000.patch, HDDS-1147.001.patch
>
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776120#comment-16776120
 ] 

Anu Engineer commented on HDDS-1168:


[~nandakumar131] I am getting a conflict. Could you please rebase this patch to 
the top of the tree. Thanks in advance.

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch, HDDS-1168.001.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDDS-1167:
---
   Resolution: Fixed
Fix Version/s: 0.4.0
   Status: Resolved  (was: Patch Available)

[~nandakumar131] Thanks for the contribution. I have committed this to the 
trunk.

> Error in hadoop-ozone/dev-support/checks/checkstyle.sh
> --
>
> Key: HDDS-1167
> URL: https://issues.apache.org/jira/browse/HDDS-1167
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1167.000.patch
>
>
> While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
> error is thrown.
> {code}
> grep: warning: recursive search of stdin
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776119#comment-16776119
 ] 

Hudson commented on HDDS-1167:
--

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16041 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16041/])
HDDS-1167. Error in hadoop-ozone/dev-support/checks/checkstyle.sh. (aengineer: 
rev fb546d18073a04418b6541c18cbdb9548205ac7f)
* (edit) hadoop-ozone/dev-support/checks/checkstyle.sh


> Error in hadoop-ozone/dev-support/checks/checkstyle.sh
> --
>
> Key: HDDS-1167
> URL: https://issues.apache.org/jira/browse/HDDS-1167
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1167.000.patch
>
>
> While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
> error is thrown.
> {code}
> grep: warning: recursive search of stdin
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1166) Fix checkstyle line length issues

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776115#comment-16776115
 ] 

Hudson commented on HDDS-1166:
--

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16040 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16040/])
HDDS-1166. Fix checkstyle line length issues. Contributed by Nandakumar. 
(aengineer: rev 014e17af783891699350b44b436071b1635868a8)
* (edit) 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/DeletedBlocksTransactionCodec.java
* (edit) 
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java
* (edit) 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CloseContainerCommandHandler.java
* (edit) 
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerCommandHandler.java
* (edit) 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java
* (edit) 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMMetadataStoreRDBImpl.java
* (edit) 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java
* (edit) 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMConfigurator.java


> Fix checkstyle line length issues
> -
>
> Key: HDDS-1166
> URL: https://issues.apache.org/jira/browse/HDDS-1166
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1166.000.patch
>
>
> Checkstyle line length issues have to be fixed in the following classes
> * BlockManagerImpl
> * DeletedBlockLogImpl
> * DeletedBlocksTransactionCodec
> * CloseContainerCommandHandler
> * SCMMetadataStoreRDBImpl
> * SCMConfigurator
> * TestUtils
> * TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776117#comment-16776117
 ] 

Anu Engineer commented on HDDS-1167:


+1, thnx for fixing this. I will commit this shortly.

> Error in hadoop-ozone/dev-support/checks/checkstyle.sh
> --
>
> Key: HDDS-1167
> URL: https://issues.apache.org/jira/browse/HDDS-1167
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1167.000.patch
>
>
> While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
> error is thrown.
> {code}
> grep: warning: recursive search of stdin
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1166) Fix checkstyle line length issues

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDDS-1166:
---
   Resolution: Fixed
Fix Version/s: 0.4.0
   Status: Resolved  (was: Patch Available)

[~nandakumar131] Thank you for your contribution. I have verified this patch on 
my local machine while committing to the trunk

> Fix checkstyle line length issues
> -
>
> Key: HDDS-1166
> URL: https://issues.apache.org/jira/browse/HDDS-1166
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1166.000.patch
>
>
> Checkstyle line length issues have to be fixed in the following classes
> * BlockManagerImpl
> * DeletedBlockLogImpl
> * DeletedBlocksTransactionCodec
> * CloseContainerCommandHandler
> * SCMMetadataStoreRDBImpl
> * SCMConfigurator
> * TestUtils
> * TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1165) Document generation in maven should be configured on execution level

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776001#comment-16776001
 ] 

Hudson commented on HDDS-1165:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16039 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16039/])
HDDS-1165. Document generation in maven should be configured on (bharat: rev 
1b87668a3b3ad0bfd6b098037c92a8fd3a8b98f2)
* (edit) hadoop-hdds/docs/pom.xml


> Document generation in maven should be configured on execution level 
> -
>
> Key: HDDS-1165
> URL: https://issues.apache.org/jira/browse/HDDS-1165
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1165.001.patch
>
>
> Documentation of Ozone/Hdds project is generated from maven with the help of 
> maven exec plugin.
> There are multiple ways to configure plugins in maven. Plugin can be 
> configured on plugin level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>   
>     
>     
>   ...
>     
>   
> {code}
> In this case not only the specific execution but all the execution will be 
> configured (even if it's triggered by mvn exec:exec)
> Or it can be configured on the execution level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>  
>  ...
>      
>   
>     
>     
>   {code}
> In this case the configuration is valid only for this specific execution 
> which is bound to a specific phase (compile in this case)
> Unfortunately it's configured in the wrong way in hadoop-hdds/docs/pom.xml: 
> the first approach should be replaced with the second with moving the 
> configuration to inside the execution.
> Without this change yetus can't detect the dependency order.
> How to test:
> The easiest way to reproduce the problem is to execute:
> {code:java}
> mvn  -fae exec:exec -Dexec.executable=pwd -Dexec.args='' -Phdds{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1165) Document generation in maven should be configured on execution level

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDDS-1165:
-
Fix Version/s: 0.4.0

> Document generation in maven should be configured on execution level 
> -
>
> Key: HDDS-1165
> URL: https://issues.apache.org/jira/browse/HDDS-1165
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1165.001.patch
>
>
> Documentation of Ozone/Hdds project is generated from maven with the help of 
> maven exec plugin.
> There are multiple ways to configure plugins in maven. Plugin can be 
> configured on plugin level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>   
>     
>     
>   ...
>     
>   
> {code}
> In this case not only the specific execution but all the execution will be 
> configured (even if it's triggered by mvn exec:exec)
> Or it can be configured on the execution level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>  
>  ...
>      
>   
>     
>     
>   {code}
> In this case the configuration is valid only for this specific execution 
> which is bound to a specific phase (compile in this case)
> Unfortunately it's configured in the wrong way in hadoop-hdds/docs/pom.xml: 
> the first approach should be replaced with the second with moving the 
> configuration to inside the execution.
> Without this change yetus can't detect the dependency order.
> How to test:
> The easiest way to reproduce the problem is to execute:
> {code:java}
> mvn  -fae exec:exec -Dexec.executable=pwd -Dexec.args='' -Phdds{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1165) Document generation in maven should be configured on execution level

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDDS-1165:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

I have committed this to trunk.

> Document generation in maven should be configured on execution level 
> -
>
> Key: HDDS-1165
> URL: https://issues.apache.org/jira/browse/HDDS-1165
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Anu Engineer
>Priority: Major
> Attachments: HDDS-1165.001.patch
>
>
> Documentation of Ozone/Hdds project is generated from maven with the help of 
> maven exec plugin.
> There are multiple ways to configure plugins in maven. Plugin can be 
> configured on plugin level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>   
>     
>     
>   ...
>     
>   
> {code}
> In this case not only the specific execution but all the execution will be 
> configured (even if it's triggered by mvn exec:exec)
> Or it can be configured on the execution level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>  
>  ...
>      
>   
>     
>     
>   {code}
> In this case the configuration is valid only for this specific execution 
> which is bound to a specific phase (compile in this case)
> Unfortunately it's configured in the wrong way in hadoop-hdds/docs/pom.xml: 
> the first approach should be replaced with the second with moving the 
> configuration to inside the execution.
> Without this change yetus can't detect the dependency order.
> How to test:
> The easiest way to reproduce the problem is to execute:
> {code:java}
> mvn  -fae exec:exec -Dexec.executable=pwd -Dexec.args='' -Phdds{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1158) TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775990#comment-16775990
 ] 

Hudson commented on HDDS-1158:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16038 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16038/])
HDDS-1158. TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis (bharat: 
rev d0e21ac16bdf064eb195ae29c4022bd8057e35d9)
* (edit) 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHA.java


> TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error
> -
>
> Key: HDDS-1158
> URL: https://issues.apache.org/jira/browse/HDDS-1158
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Hanisha Koneru
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> h3. Error Message
> {code:java}
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms){code}
> Stacktrace
> {code:java}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms) at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:586)
>  at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:230)
>  at 
> org.apache.hadoop.ozone.web.storage.DistributedStorageHandler.createVolume(DistributedStorageHandler.java:179)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testCreateVolume(TestOzoneManagerHA.java:153)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testTwoOMNodesDown(TestOzoneManagerHA.java:138)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>  at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1165) Document generation in maven should be configured on execution level

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775992#comment-16775992
 ] 

Bharat Viswanadham commented on HDDS-1165:
--

Thank You [~elek] for clear explanation of the issue and [~anu] for the fix.

I am able to see the issue when running with above command, compilation struck 
when it reach hadoop-hdds-docs.

+1 LGTM. I will commit this shortly.

> Document generation in maven should be configured on execution level 
> -
>
> Key: HDDS-1165
> URL: https://issues.apache.org/jira/browse/HDDS-1165
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Anu Engineer
>Priority: Major
> Attachments: HDDS-1165.001.patch
>
>
> Documentation of Ozone/Hdds project is generated from maven with the help of 
> maven exec plugin.
> There are multiple ways to configure plugins in maven. Plugin can be 
> configured on plugin level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>   
>     
>     
>   ...
>     
>   
> {code}
> In this case not only the specific execution but all the execution will be 
> configured (even if it's triggered by mvn exec:exec)
> Or it can be configured on the execution level:
> {code:java}
> 
>     org.codehaus.mojo
>     exec-maven-plugin
>     1.6.0
>     
>   
>     
>   exec
>     
>     compile
>  
>  ...
>      
>   
>     
>     
>   {code}
> In this case the configuration is valid only for this specific execution 
> which is bound to a specific phase (compile in this case)
> Unfortunately it's configured in the wrong way in hadoop-hdds/docs/pom.xml: 
> the first approach should be replaced with the second with moving the 
> configuration to inside the execution.
> Without this change yetus can't detect the dependency order.
> How to test:
> The easiest way to reproduce the problem is to execute:
> {code:java}
> mvn  -fae exec:exec -Dexec.executable=pwd -Dexec.args='' -Phdds{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775983#comment-16775983
 ] 

Hadoop QA commented on HDFS-14314:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} 
| {color:red} HDFS-14314 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14314 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959914/HDFS-14314.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/26312/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This message was automatically generated.



> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.4
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314.patch
>
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } 

[jira] [Updated] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN

[star (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

star updated HDFS-14314:

Attachment: HDFS-14314.patch

> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.4
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314.patch
>
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } catch(RemoteException re) {
>   
>   } // while (shouldRun())
> } // offerService{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN

[star (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775981#comment-16775981
 ] 

star commented on HDFS-14314:
-

[~jojochuang] patch available now. 

> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.4
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314.patch
>
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } catch(RemoteException re) {
>   
>   } // while (shouldRun())
> } // offerService{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN

[star (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

star updated HDFS-14314:

Fix Version/s: 2.8.4
Affects Version/s: (was: 2.8.0)
   2.8.4
   Status: Patch Available  (was: Open)

> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.4
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } catch(RemoteException re) {
>   
>   } // while (shouldRun())
> } // offerService{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1158) TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDDS-1158:
-
   Resolution: Fixed
Fix Version/s: 0.4.0
   Status: Resolved  (was: Patch Available)

Thank You [~hanishakoneru] for the contribution.

I have committed this to trunk.

> TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error
> -
>
> Key: HDDS-1158
> URL: https://issues.apache.org/jira/browse/HDDS-1158
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Hanisha Koneru
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> h3. Error Message
> {code:java}
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms){code}
> Stacktrace
> {code:java}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms) at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:586)
>  at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:230)
>  at 
> org.apache.hadoop.ozone.web.storage.DistributedStorageHandler.createVolume(DistributedStorageHandler.java:179)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testCreateVolume(TestOzoneManagerHA.java:153)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testTwoOMNodesDown(TestOzoneManagerHA.java:138)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>  at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN

[star (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

star updated HDFS-14314:

Summary: fullBlockReportLeaseId should be reset after registering to NN  
(was: fullBlockReportLeaseId should be reset after )

> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.0
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } catch(RemoteException re) {
>   
>   } // while (shouldRun())
> } // offerService{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1158) TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1158?focusedWorklogId=203063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-203063
 ]

ASF GitHub Bot logged work on HDDS-1158:


Author: ASF GitHub Bot
Created on: 23/Feb/19 18:20
Start Date: 23/Feb/19 18:20
Worklog Time Spent: 10m 
  Work Description: bharatviswa504 commented on pull request #512: 
HDDS-1158. TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error.
URL: https://github.com/apache/hadoop/pull/512
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 203063)
Time Spent: 1h 20m  (was: 1h 10m)

> TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error
> -
>
> Key: HDDS-1158
> URL: https://issues.apache.org/jira/browse/HDDS-1158
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Hanisha Koneru
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> h3. Error Message
> {code:java}
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms){code}
> Stacktrace
> {code:java}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms) at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:586)
>  at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:230)
>  at 
> org.apache.hadoop.ozone.web.storage.DistributedStorageHandler.createVolume(DistributedStorageHandler.java:179)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testCreateVolume(TestOzoneManagerHA.java:153)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testTwoOMNodesDown(TestOzoneManagerHA.java:138)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>  at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-14118:
-
   Resolution: Fixed
Fix Version/s: 3.3.0
   Status: Resolved  (was: Patch Available)

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775976#comment-16775976
 ] 

Yongjun Zhang commented on HDFS-14118:
--

HI [~fengnanli], would you please add content to the "Release Note" section of 
the jira? thanks.

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-14118:
-
Hadoop Flags: Reviewed

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775974#comment-16775974
 ] 

Yongjun Zhang commented on HDFS-14118:
--

Committed to trunk.

Thanks [~fengnanli] for the contribution and [~elgoiri] for the review!

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Fengnan Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775969#comment-16775969
 ] 

Fengnan Li commented on HDFS-14118:
---

[~elgoiri] and [~yzhangal] Thanks for the discussion and review! 

 

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775971#comment-16775971
 ] 

Hudson commented on HDFS-14118:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16037 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16037/])
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. (yzhang: 
rev f7a27cdee4e6829ebea4ea965e549a27acbf4310)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* (edit) 
hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestMockDomainNameResolver.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DomainNameResolverFactory.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DomainNameResolver.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNSDomainNameResolver.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/MockDomainNameResolver.java
* (delete) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/package.html
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/package-info.java
* (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* (edit) 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/AbstractNNFailoverProxyProvider.java


> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster

[JIRA]

[ 
https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775966#comment-16775966
 ] 

Íñigo Goiri commented on HDFS-13123:


I think [~ekanth] had some modifications on top of our internal original PoC.
[~ekanth], do you have anything that can be posted?

> RBF: Add a balancer tool to move data across subcluster 
> 
>
> Key: HDFS-13123
> URL: https://issues.apache.org/jira/browse/HDFS-13123
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Major
> Attachments: HDFS Router-Based Federation Rebalancer.pdf
>
>
> Follow the discussion in HDFS-12615. This Jira is to track effort for 
> building a rebalancer tool, used by router-based federation to move data 
> among subclusters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS

[JIRA]

[ 
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775964#comment-16775964
 ] 

Íñigo Goiri commented on HDFS-14052:


[^HDFS-14052-HDFS-13891.3.patch] LGTM.
+1

> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch, 
> HDFS-14052-HDFS-13891.1.patch, HDFS-14052-HDFS-13891.2.patch, 
> HDFS-14052-HDFS-13891.3.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(), 
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a 
> parameter and load the Router one just in case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[JIRA]

[ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775963#comment-16775963
 ] 

Íñigo Goiri commented on HDFS-14118:


[^HDFS-14118.024.patch] LGTM.
The failed unit tests are unrelated (pretty flaky in general).
+1

[~yzhangal] go ahead with the commit; thanks for the review.
Thanks [~fengnanli] for the patch!

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1158) TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1158?focusedWorklogId=203058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-203058
 ]

ASF GitHub Bot logged work on HDDS-1158:


Author: ASF GitHub Bot
Created on: 23/Feb/19 17:15
Start Date: 23/Feb/19 17:15
Worklog Time Spent: 10m 
  Work Description: bharatviswa504 commented on pull request #512: 
HDDS-1158. TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error.
URL: https://github.com/apache/hadoop/pull/512#discussion_r259585796
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHA.java
 ##
 @@ -150,17 +152,27 @@ private void testCreateVolume(boolean checkSuccess) 
throws Exception {
 createVolumeArgs.setUserName(userName);
 createVolumeArgs.setAdminName(adminName);
 
-storageHandler.createVolume(createVolumeArgs);
+try {
+  storageHandler.createVolume(createVolumeArgs);
 
-VolumeArgs getVolumeArgs = new VolumeArgs(volumeName, userArgs);
-VolumeInfo retVolumeinfo = storageHandler.getVolumeInfo(getVolumeArgs);
+  VolumeArgs getVolumeArgs = new VolumeArgs(volumeName, userArgs);
+  VolumeInfo retVolumeinfo = storageHandler.getVolumeInfo(getVolumeArgs);
 
-if (checkSuccess) {
-  Assert.assertTrue(retVolumeinfo.getVolumeName().equals(volumeName));
-  Assert.assertTrue(retVolumeinfo.getOwner().getName().equals(userName));
-} else {
-  // Verify that the request failed
-  Assert.assertTrue(retVolumeinfo.getVolumeName().isEmpty());
+  if (checkSuccess) {
+Assert.assertTrue(retVolumeinfo.getVolumeName().equals(volumeName));
+Assert.assertTrue(retVolumeinfo.getOwner().getName().equals(userName));
+  } else {
+// Verify that the request failed
+Assert.assertTrue(retVolumeinfo.getVolumeName().isEmpty());
+Assert.fail("There is no quorum. Request should have failed");
+  }
+} catch (OMException e) {
+  if (!checkSuccess) {
+GenericTestUtils.assertExceptionContains(
+"RaftRetryFailureException", e);
 
 Review comment:
   Yes, my bad. So I think now it LGTM. Thank You @hanishakoneru  for the 
clarification.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 203058)
Time Spent: 1h  (was: 50m)

> TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error
> -
>
> Key: HDDS-1158
> URL: https://issues.apache.org/jira/browse/HDDS-1158
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Hanisha Koneru
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> h3. Error Message
> {code:java}
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms){code}
> Stacktrace
> {code:java}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms) at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:586)
>  at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:230)
>  at 
> org.apache.hadoop.ozone.web.storage.DistributedStorageHandler.createVolume(DistributedStorageHandler.java:179)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testCreateVolume(TestOzoneManagerHA.java:153)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testTwoOMNodesDown(TestOzoneManagerHA.java:138)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> 

[jira] [Work logged] (HDDS-1158) TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1158?focusedWorklogId=203059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-203059
 ]

ASF GitHub Bot logged work on HDDS-1158:


Author: ASF GitHub Bot
Created on: 23/Feb/19 17:16
Start Date: 23/Feb/19 17:16
Worklog Time Spent: 10m 
  Work Description: bharatviswa504 commented on issue #512: HDDS-1158. 
TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error.
URL: https://github.com/apache/hadoop/pull/512#issuecomment-466670294
 
 
   I am +1 with the change.
   Will commit it shortly.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 203059)
Time Spent: 1h 10m  (was: 1h)

> TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error
> -
>
> Key: HDDS-1158
> URL: https://issues.apache.org/jira/browse/HDDS-1158
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Hanisha Koneru
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> h3. Error Message
> {code:java}
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms){code}
> Stacktrace
> {code:java}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: 
> org.apache.ratis.protocol.RaftRetryFailureException: Failed 
> RaftClientRequest:client-4D77D2A8F653->omNode-3@group-523986131536, cid=9, 
> seq=0 RW, 
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$373/2067504307@6afa0221
>  for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms) at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:586)
>  at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:230)
>  at 
> org.apache.hadoop.ozone.web.storage.DistributedStorageHandler.createVolume(DistributedStorageHandler.java:179)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testCreateVolume(TestOzoneManagerHA.java:153)
>  at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testTwoOMNodesDown(TestOzoneManagerHA.java:138)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>  at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775945#comment-16775945
 ] 

Hadoop QA commented on HDDS-1147:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
 6s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m 
34s{color} | {color:red} hadoop-ozone in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 42s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m 
24s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 24s{color} 
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 21s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} ozone-manager in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  9m 23s{color} 
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1147 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959904/HDDS-1147.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| 

[jira] [Updated] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-14118:
-
Description: 
In router based federation (RBF), clients will need to know about routers to 
talk to the HDFS cluster (obviously), and having routers updating 
(adding/removing) will have to make config change in every client, which is a 
painful process.

DNS can be used here to resolve the single domain name clients knows to a list 
of routers in the current config. However, DNS won't be able to consider only 
resolving to the working router based on certain health thresholds.

There are some ways about how this can be solved. One way is to have a separate 
script to regularly check the status of the router and update the DNS records 
if a router fails the health thresholds. In this way, security might be 
carefully considered for this way. Another way is to have the client do the 
normal connecting/failover after they get the list of routers, which requires 
the change of current failover proxy provider.

See the attached design document for details about the proposed solution.

  was:
Clients will need to know about routers to talk to the HDFS cluster 
(obviously), and having routers updating (adding/removing) will have to make 
every client change, which is a painful process.

DNS can be used here to resolve the single domain name clients knows to a list 
of routers in the current config. However, DNS won't be able to consider only 
resolving to the working router based on certain health thresholds.

There are some ways about how this can be solved. One way is to have a separate 
script to regularly check the status of the router and update the DNS records 
if a router fails the health thresholds. In this way, security might be 
carefully considered for this way. Another way is to have the client do the 
normal connecting/failover after they get the list of routers, which requires 
the change of current failover proxy provider.


> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> In router based federation (RBF), clients will need to know about routers to 
> talk to the HDFS cluster (obviously), and having routers updating 
> (adding/removing) will have to make config change in every client, which is a 
> painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.
> See the attached design document for details about the proposed solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-14118) Support using DNS to resolve nameservices to IP addresses

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-14118:
-
Summary: Support using DNS to resolve nameservices to IP addresses  (was: 
Use DNS to resolve Namenodes and Routers)

> Support using DNS to resolve nameservices to IP addresses
> -
>
> Key: HDFS-14118
> URL: https://issues.apache.org/jira/browse/HDFS-14118
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Major
> Attachments: DNS testing log, HDFS design doc_ Single domain name for 
> clients - Google Docs-1.pdf, HDFS design doc_ Single domain name for clients 
> - Google Docs.pdf, HDFS-14118.001.patch, HDFS-14118.002.patch, 
> HDFS-14118.003.patch, HDFS-14118.004.patch, HDFS-14118.005.patch, 
> HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch, 
> HDFS-14118.009.patch, HDFS-14118.010.patch, HDFS-14118.011.patch, 
> HDFS-14118.012.patch, HDFS-14118.013.patch, HDFS-14118.014.patch, 
> HDFS-14118.015.patch, HDFS-14118.016.patch, HDFS-14118.017.patch, 
> HDFS-14118.018.patch, HDFS-14118.019.patch, HDFS-14118.020.patch, 
> HDFS-14118.021.patch, HDFS-14118.022.patch, HDFS-14118.023.patch, 
> HDFS-14118.024.patch, HDFS-14118.patch
>
>
> Clients will need to know about routers to talk to the HDFS cluster 
> (obviously), and having routers updating (adding/removing) will have to make 
> every client change, which is a painful process.
> DNS can be used here to resolve the single domain name clients knows to a 
> list of routers in the current config. However, DNS won't be able to consider 
> only resolving to the working router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a 
> separate script to regularly check the status of the router and update the 
> DNS records if a router fails the health thresholds. In this way, security 
> might be carefully considered for this way. Another way is to have the client 
> do the normal connecting/failover after they get the list of routers, which 
> requires the change of current failover proxy provider.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775936#comment-16775936
 ] 

Hadoop QA commented on HDDS-1155:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 56s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 59s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 52s{color} 
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 21s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.block.TestBlockManager |
|   | hadoop.hdds.scm.block.TestDeletedBlockLog |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1155 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959903/HDDS-1155.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 87127482e1c6 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f19c844 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2353/artifact/out/patch-unit-hadoop-hdds_server-scm.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2353/testReport/ |
| Max. process+thread count | 380 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2353/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   

[jira] [Updated] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1147:
--
Attachment: HDDS-1147.001.patch

> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1147.000.patch, HDDS-1147.001.patch
>
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775919#comment-16775919
 ] 

Hadoop QA commented on HDDS-1147:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
 1s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m 
39s{color} | {color:red} hadoop-ozone in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 11s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m 
22s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m 22s{color} 
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} hadoop-ozone: The patch generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 12s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
37s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
34s{color} | {color:green} ozone-manager in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 31s{color} 
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 13s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ozShell.TestOzoneShell |
|   | hadoop.ozone.client.rpc.TestFailureHandlingByClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1147 |
| JIRA 

[jira] [Updated] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1147:
--
Target Version/s: 0.4.0

> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1147.000.patch
>
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Attachment: HDDS-1155.003.patch

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch, 
> HDDS-1155.003.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Status: Patch Available  (was: Open)

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch, 
> HDDS-1155.003.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1147:
--
Attachment: HDDS-1147.000.patch

> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1147.000.patch
>
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1147:
--
Status: Patch Available  (was: Open)

> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1147.000.patch
>
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775903#comment-16775903
 ] 

Hadoop QA commented on HDDS-1168:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 12s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 17s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 47s{color} 
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m  4s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.node.TestSCMNodeManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1168 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959893/HDDS-1168.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux cf1ca44c1282 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f19c844 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2351/artifact/out/patch-unit-hadoop-hdds_server-scm.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2351/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2351/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Use 

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Status: Open  (was: Patch Available)

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775898#comment-16775898
 ] 

Hadoop QA commented on HDDS-1155:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 56s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  6s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 22s{color} 
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.node.TestDeadNodeHandler |
|   | hadoop.hdds.scm.block.TestDeletedBlockLog |
|   | hadoop.hdds.scm.server.TestSCMClientProtocolServer |
|   | hadoop.hdds.scm.block.TestBlockManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1155 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959892/HDDS-1155.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 9cac0dd687a6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f19c844 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2350/artifact/out/patch-unit-hadoop-hdds_server-scm.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2350/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output | 

[jira] [Assigned] (HDDS-1147) Fix failing unit tests in TestOzoneManager

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar reassigned HDDS-1147:
-

Assignee: Nanda kumar

> Fix failing unit tests in TestOzoneManager
> --
>
> Key: HDDS-1147
> URL: https://issues.apache.org/jira/browse/HDDS-1147
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Hanisha Koneru
>Assignee: Nanda kumar
>Priority: Major
>
> The following tests are failing in TestOzoneManager
>  * testRenameKey
>  * testAccessVolume
>  * testOmInitializationFailure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14314) fullBlockReportLeaseId should be reset after

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775896#comment-16775896
 ] 

Wei-Chiu Chuang commented on HDFS-14314:


Sounds critical. Thanks for reporting the issue and offering a solution, 
[~starphin]. Would you like to contribute a patch & test as well?

> fullBlockReportLeaseId should be reset after 
> -
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.8.0
> Environment:  
>  
>  
>Reporter: star
>Priority: Critical
>
>   since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full 
> block lease id from active NN before sending full block to NN. Then DN will 
> send full block report together with lease id. If the lease id is invalid, NN 
> will reject the full block report and log "not in the pending set".
>   In a case when DN is doing full block reporting while NN is restarted. 
> It happens that DN will later send a full block report with lease id 
> ,acquired from previous NN instance, which is invalid to the new NN instance. 
> Though DN recognized the new NN instance by heartbeat and reregister itself, 
> it did not reset the lease id from previous instance.
>   The issuse may cause DNs to temporarily go dead, making it unsafe to 
> restart NN especially in hadoop cluster which has large amount of DNs. 
> HDFS-12914 reported the issue  without any clues why it occurred and remain 
> unsolved.
>    To make it clear, look at code below. We take it from method 
> offerService of class BPServiceActor. We eliminate some code to focus on 
> current issue. fullBlockReportLeaseId is a local variable to hold lease id 
> from NN. Exceptions will occur at blockReport call when NN restarting, which 
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will 
> not be set to 0. After NN restarted, DN will send full block report which 
> will be rejected by the new NN instance. DN will never send full block report 
> until the next full block report schedule, about an hour later.
>   Solution is simple, just reset fullBlockReportLeaseId to 0 after any 
> exception or after registering to NN. Thus it will ask for a valid 
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
>   long fullBlockReportLeaseId = 0;
>   //
>   // Now loop for a long time
>   //
>   while (shouldRun()) {
> try {
>   final long startTime = scheduler.monotonicNow();
>   //
>   // Every so often, send heartbeat or block-report
>   //
>   final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
>   HeartbeatResponse resp = null;
>   if (sendHeartbeat) {
>   
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
>   resp = sendHeartBeat(requestBlockReportLease);
>   assert resp != null;
>   if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
>   LOG.warn(nnAddr + " sent back a full block report lease " +
>   "ID of 0x" +
>   Long.toHexString(resp.getFullBlockReportLeaseId()) +
>   ", but we already have a lease ID of 0x" +
>   Long.toHexString(fullBlockReportLeaseId) + ". " +
>   "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
>   }
>  
> }
>   }
>
>  
>   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
>   }
>   
> } catch(RemoteException re) {
>   
>   } // while (shouldRun())
> } // offerService{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Attachment: HDDS-1168.001.patch

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch, HDDS-1168.001.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Status: Patch Available  (was: Open)

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Status: Open  (was: Patch Available)

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775886#comment-16775886
 ] 

Nanda kumar commented on HDDS-1155:
---

The test failure is reproduced in the above Jenkins run. Fixed the issue in 
patch v001.

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Attachment: HDDS-1155.001.patch

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch, HDDS-1155.001.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775883#comment-16775883
 ] 

Hadoop QA commented on HDDS-1168:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 23s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 18s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 24s{color} 
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 52s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.node.TestDeadNodeHandler |
|   | hadoop.hdds.scm.server.TestSCMClientProtocolServer |
|   | hadoop.hdds.scm.block.TestBlockManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1168 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959888/HDDS-1168.000.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 5585a59fcda4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f19c844 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2349/artifact/out/patch-unit-hadoop-hdds_server-scm.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2349/testReport/ |
| Max. process+thread count | 445 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm |
| Console output | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2349/console |
| Powered by | 

[jira] [Commented] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775882#comment-16775882
 ] 

Hadoop QA commented on HDDS-1155:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 27s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 14s{color} | {color:orange} hadoop-hdds/server-scm: The patch generated 1 
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 41s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 23s{color} 
| {color:red} server-scm in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 33s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdds.scm.block.TestDeletedBlockLog |
|   | hadoop.hdds.scm.node.TestDeadNodeHandler |
|   | hadoop.hdds.scm.block.TestBlockManager |
|   | hadoop.hdds.scm.server.TestSCMClientProtocolServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1155 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959885/HDDS-1155.000.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 3ebf187ccf3b 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f19c844 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2348/artifact/out/diff-checkstyle-hadoop-hdds_server-scm.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2348/artifact/out/patch-unit-hadoop-hdds_server-scm.txt
 |
|  Test Results | 

[jira] [Commented] (HDDS-1120) Add a config to disable checksum verification during read even though checksum data is present in the persisted data

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775879#comment-16775879
 ] 

Yiqun Lin commented on HDDS-1120:
-

LGTM, +1. By the PR way, I see there are two commits in the PR. Is this will 
still show two commit when merged to trunk? Need we combine these two commit to 
a final commit with the corresponding  message?

> Add a config to disable checksum verification during read even though 
> checksum data is present in the persisted data
> 
>
> Key: HDDS-1120
> URL: https://issues.apache.org/jira/browse/HDDS-1120
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.0
>
> Attachments: HDDS-1120.00.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, if the checksum is computed during data write and persisted in the 
> disk, we will always end up verifying it while reading. This Jira aims to 
> selectively disable checksum verification during reads even though checksum 
> info is present in the data stored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-1159) TestContainerStateManagerIntegration.testGetMatchingContainerMultipleThreads

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar reassigned HDDS-1159:
-

Assignee: Nanda kumar

> TestContainerStateManagerIntegration.testGetMatchingContainerMultipleThreads
> 
>
> Key: HDDS-1159
> URL: https://issues.apache.org/jira/browse/HDDS-1159
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
>
> Failing test: 
> TestContainerStateManagerIntegration.testGetMatchingContainerMultipleThreads
> This is an intermittent error. It's failed 7 times from 18 build:
> {code:java}
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.hadoop.hdds.scm.container.TestContainerStateManagerIntegration.testGetMatchingContainerMultipleThreads(TestContainerStateManagerIntegration.java:231)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Status: Patch Available  (was: Open)

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Attachment: HDDS-1168.000.patch

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Status: Open  (was: Patch Available)

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Attachment: (was: HDDS-1168.000.patch)

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Status: Patch Available  (was: Open)

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1168:
--
Attachment: HDDS-1168.000.patch

> Use random ports in TestBlockManager and TestDeletedBlockLog
> 
>
> Key: HDDS-1168
> URL: https://issues.apache.org/jira/browse/HDDS-1168
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1168.000.patch
>
>
> TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
> BindException when the tests are executed in parallel. We should start using 
> random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-1168) Use random ports in TestBlockManager and TestDeletedBlockLog

[Nanda kumar (JIRA)]

Nanda kumar created HDDS-1168:
-

 Summary: Use random ports in TestBlockManager and 
TestDeletedBlockLog
 Key: HDDS-1168
 URL: https://issues.apache.org/jira/browse/HDDS-1168
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: SCM
Reporter: Nanda kumar
Assignee: Nanda kumar


TestBlockManager and TestDeletedBlockLog uses default ports which are causing 
BindException when the tests are executed in parallel. We should start using 
random ports to avoid this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar reassigned HDDS-1155:
-

Assignee: Nanda kumar

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775864#comment-16775864
 ] 

Nanda kumar commented on HDDS-1155:
---

The test is failing with the below exception when parallel execution is enabled.
{code:java}
[ERROR] 
testOnMessageReplicaFailure(org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler)
  Time elapsed: 0.076 s  <<< ERROR!
java.net.BindException: Problem binding to [0.0.0.0:9861] 
java.net.BindException: Address already in use; For more details see:  
http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:736)
at org.apache.hadoop.ipc.Server.bind(Server.java:566)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:1042)
at org.apache.hadoop.ipc.Server.(Server.java:2815)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:994)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:421)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:804)
at 
org.apache.hadoop.hdds.scm.server.StorageContainerManager.startRpcServer(StorageContainerManager.java:575)
at 
org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.(SCMDatanodeProtocolServer.java:170)
at 
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:333)
at 
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:216)
at 
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:688)
at 
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:632)
at 
org.apache.hadoop.hdds.scm.HddsTestUtils.getScm(HddsTestUtils.java:90)
at 
org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.setup(TestDeadNodeHandler.java:90)
{code}
Not able to reproduce the exception mentioned in the description. Attaching 
patch which will use dynamic ports to avoid BindException when tests are 
executed in parallel.

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Priority: Major
> Attachments: HDDS-1155.000.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> 

[jira] [Updated] (HDDS-1155) Fix failing unit test methods of TestDeadNodeHandler

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1155:
--
Attachment: HDDS-1155.000.patch

> Fix failing unit test methods of TestDeadNodeHandler
> 
>
> Key: HDDS-1155
> URL: https://issues.apache.org/jira/browse/HDDS-1155
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: test
>Reporter: Elek, Marton
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1155.000.patch
>
>
> testOnMessage and {color:#00}testOnMessageReplicaFailure{color} tests are 
> constantly failing on trunk, with the same error message:
> {code:java}
> java.io.IOException: Could not allocate container. Cannot get any matching 
> pipeline for Type:STAND_ALONE, Factor:THREE, State:PipelineState.OPEN
>   at 
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.allocateContainer(ContainerStateManager.java:261)
>   at 
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.allocateContainer(SCMContainerManager.java:219)
>   at 
> org.apache.hadoop.hdds.scm.TestUtils.allocateContainer(TestUtils.java:443)
>   at 
> org.apache.hadoop.hdds.scm.node.TestDeadNodeHandler.testOnMessage(TestDeadNodeHandler.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775853#comment-16775853
 ] 

Hadoop QA commented on HDFS-14118:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 58s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 41s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  9m 22s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 41s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
49s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 11s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.http.TestHttpServer |
|   | hadoop.util.TestReadWriteDiskValidator |
|   | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
|   | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14118 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959874/HDFS-14118.024.patch |
| Optional Tests |  dupname  asflicense  compile  javac  

[jira] [Commented] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775848#comment-16775848
 ] 

Hadoop QA commented on HDDS-1167:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
51s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  1m 
47s{color} | {color:red} hadoop-ozone in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 27s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  1m 
28s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m 
15s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 58s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 29s{color} 
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1167 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12959881/HDDS-1167.000.patch |
| Optional Tests |  asflicense  mvnsite  unit  shellcheck  shelldocs  |
| uname | Linux 8ca9443d7bad 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f19c844 |
| maven | version: Apache Maven 3.3.9 |
| mvnsite | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2346/artifact/out/branch-mvnsite-hadoop-ozone.txt
 |
| shellcheck | v0.4.6 |
| mvnsite | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2346/artifact/out/patch-mvnsite-hadoop-ozone.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2346/artifact/out/patch-unit-hadoop-ozone.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2346/testReport/ |
| Max. process+thread count | 335 (vs. ulimit of 1) |
| modules | C: hadoop-ozone U: hadoop-ozone |
| Console output | 
https://builds.apache.org/job/PreCommit-HDDS-Build/2346/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Error in hadoop-ozone/dev-support/checks/checkstyle.sh
> --
>
> Key: HDDS-1167
> URL: https://issues.apache.org/jira/browse/HDDS-1167
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1167.000.patch
>
>
> While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
> error is thrown.
> {code}
> grep: warning: recursive search of stdin
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional 

[jira] [Updated] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1167:
--
Status: Patch Available  (was: Open)

> Error in hadoop-ozone/dev-support/checks/checkstyle.sh
> --
>
> Key: HDDS-1167
> URL: https://issues.apache.org/jira/browse/HDDS-1167
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1167.000.patch
>
>
> While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
> error is thrown.
> {code}
> grep: warning: recursive search of stdin
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1166) Fix checkstyle line length issues

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1166:
--
Status: Patch Available  (was: Open)

> Fix checkstyle line length issues
> -
>
> Key: HDDS-1166
> URL: https://issues.apache.org/jira/browse/HDDS-1166
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1166.000.patch
>
>
> Checkstyle line length issues have to be fixed in the following classes
> * BlockManagerImpl
> * DeletedBlockLogImpl
> * DeletedBlocksTransactionCodec
> * CloseContainerCommandHandler
> * SCMMetadataStoreRDBImpl
> * SCMConfigurator
> * TestUtils
> * TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1167:
--
Attachment: HDDS-1167.000.patch

> Error in hadoop-ozone/dev-support/checks/checkstyle.sh
> --
>
> Key: HDDS-1167
> URL: https://issues.apache.org/jira/browse/HDDS-1167
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1167.000.patch
>
>
> While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
> error is thrown.
> {code}
> grep: warning: recursive search of stdin
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1166) Fix checkstyle line length issues

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1166:
--
Attachment: HDDS-1166.000.patch

> Fix checkstyle line length issues
> -
>
> Key: HDDS-1166
> URL: https://issues.apache.org/jira/browse/HDDS-1166
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-1166.000.patch
>
>
> Checkstyle line length issues have to be fixed in the following classes
> * BlockManagerImpl
> * DeletedBlockLogImpl
> * DeletedBlocksTransactionCodec
> * CloseContainerCommandHandler
> * SCMMetadataStoreRDBImpl
> * SCMConfigurator
> * TestUtils
> * TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-1167) Error in hadoop-ozone/dev-support/checks/checkstyle.sh

[Nanda kumar (JIRA)]

Nanda kumar created HDDS-1167:
-

 Summary: Error in hadoop-ozone/dev-support/checks/checkstyle.sh
 Key: HDDS-1167
 URL: https://issues.apache.org/jira/browse/HDDS-1167
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Nanda kumar
Assignee: Nanda kumar


While running {{hadoop-ozone/dev-support/checks/checkstyle.sh}} the following 
error is thrown.
{code}
grep: warning: recursive search of stdin
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1166) Fix checkstyle line length issues

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1166:
--
Description: 
Checkstyle line length issues have to be fixed in the following classes
* BlockManagerImpl
* DeletedBlockLogImpl
* DeletedBlocksTransactionCodec
* CloseContainerCommandHandler
* SCMMetadataStoreRDBImpl
* SCMConfigurator
* TestUtils
* TestCloseContainerCommandHandler

  was:
Checkstyle line length issues have to be fixed in the following classes
* BlockManagerImpl
* DeletedBlockLogImpl
* CloseContainerCommandHandler
* TestCloseContainerCommandHandler


> Fix checkstyle line length issues
> -
>
> Key: HDDS-1166
> URL: https://issues.apache.org/jira/browse/HDDS-1166
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
>
> Checkstyle line length issues have to be fixed in the following classes
> * BlockManagerImpl
> * DeletedBlockLogImpl
> * DeletedBlocksTransactionCodec
> * CloseContainerCommandHandler
> * SCMMetadataStoreRDBImpl
> * SCMConfigurator
> * TestUtils
> * TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-1166) Fix checkstyle line length issues

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDDS-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1166:
--
Description: 
Checkstyle line length issues have to be fixed in the following classes
* BlockManagerImpl
* DeletedBlockLogImpl
* CloseContainerCommandHandler
* TestCloseContainerCommandHandler

  was:
Checkstyle line length issues have to be fixed in the following classes
* BlockManagerImpl
* CloseContainerCommandHandler
* TestCloseContainerCommandHandler


> Fix checkstyle line length issues
> -
>
> Key: HDDS-1166
> URL: https://issues.apache.org/jira/browse/HDDS-1166
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
>
> Checkstyle line length issues have to be fixed in the following classes
> * BlockManagerImpl
> * DeletedBlockLogImpl
> * CloseContainerCommandHandler
> * TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDDS-1126) Datanode is trying to qausi-close a container which is already closed

[Nanda kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDDS-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775829#comment-16775829
 ] 

Nanda kumar commented on HDDS-1126:
---

Thanks [~anu] for flagging, created HDDS-1166 to fix it.
New checkstyle version change is available from 3.2.1 and 3.3.0, In Ozone we 
depend on hadoop 3.2.0.

> Datanode is trying to qausi-close a container which is already closed
> -
>
> Key: HDDS-1126
> URL: https://issues.apache.org/jira/browse/HDDS-1126
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Datanode
>Reporter: Nilotpal Nandi
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1126.000.patch
>
>
> steps taken :
> 
>  # created 12 datanodes cluster and running workload on all the nodes
>  # running failure injection/restart on 1 datanode at a time periodically and 
> randomly.
>  
> Error seen in ozone.log :
> --
>  
> {noformat}
> 2019-02-18 06:06:32,780 [Datanode State Machine Thread - 0] DEBUG 
> (DatanodeStateMachine.java:176) - Executing cycle Number : 30
> 2019-02-18 06:06:32,784 [Command processor thread] DEBUG 
> (CloseContainerCommandHandler.java:71) - Processing Close Container command.
> 2019-02-18 06:06:32,785 [Datanode State Machine Thread - 0] DEBUG 
> (DatanodeStateMachine.java:176) - Executing cycle Number : 31
> 2019-02-18 06:06:32,785 [Command processor thread] ERROR 
> (CloseContainerCommandHandler.java:118) - Can't close container #37
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
>  Cannot quasi close container #37 while in CLOSED state.
>  at 
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.quasiCloseContainer(KeyValueHandler.java:903)
>  at 
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.quasiCloseContainer(ContainerController.java:93)
>  at 
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:110)
>  at 
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93)
>  at 
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:413)
>  at java.lang.Thread.run(Thread.java:748)
> 2019-02-18 06:06:32,785 [Command processor thread] DEBUG 
> (CloseContainerCommandHandler.java:71) - Processing Close Container command.
> 2019-02-18 06:06:32,788 [Command processor thread] DEBUG 
> (CloseContainerCommandHandler.java:71) - Processing Close Container command.
> 2019-02-18 06:06:32,788 [Datanode State Machine Thread - 0] DEBUG 
> (DatanodeStateMachine.java:176) - Executing cycle Number : 32
> 2019-02-18 06:06:34,430 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:36,608 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:38,876 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:41,084 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:43,297 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:45,469 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:47,684 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:49,958 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:52,124 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:54,344 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:56,499 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:06:58,764 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:07:00,969 [main] DEBUG (OzoneClientFactory.java:287) - Using 
> org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol.
> 2019-02-18 06:07:02,788 [Datanode State Machine Thread - 0] DEBUG 
> 

[jira] [Created] (HDDS-1166) Fix checkstyle line length issues

[Nanda kumar (JIRA)]

Nanda kumar created HDDS-1166:
-

 Summary: Fix checkstyle line length issues
 Key: HDDS-1166
 URL: https://issues.apache.org/jira/browse/HDDS-1166
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: SCM
Reporter: Nanda kumar
Assignee: Nanda kumar


Checkstyle line length issues have to be fixed in the following classes
* BlockManagerImpl
* CloseContainerCommandHandler
* TestCloseContainerCommandHandler



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org