[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1448: HDDS-4280. Document notable configurations for Recon.

2020-10-01 Thread GitBox 


adoroszlai commented on pull request #1448:
URL: https://github.com/apache/hadoop-ozone/pull/1448#issuecomment-702526133


   > Can we add a follow up JIRA to fix this inconsistent config key 
(recon.om.snapshot.task.interval.delay) ?
   
   @avijayanhwx if you mean it should be prefixed with `ozone.`, please note 
there are few other similar keys:
   
   ```
   hadoop-hdds/common/src/main/resources/ozone-default.xml
   2318:recon.om.connection.request.timeout
   2327:recon.om.connection.timeout
   2336:recon.om.socket.timeout
   2345:recon.om.snapshot.task.initial.delay
   2353:recon.om.snapshot.task.interval.delay
   2361:recon.om.snapshot.task.flush.param
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1457: HDDS-4253. Add LayoutVersion request/response for DN registration.

2020-10-01 Thread GitBox 


prashantpogde commented on a change in pull request #1457:
URL: https://github.com/apache/hadoop-ozone/pull/1457#discussion_r498600324



##
File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##
@@ -240,8 +247,19 @@ public VersionResponse getVersion(SCMVersionRequestProto 
versionRequest) {
   @Override
   public RegisteredCommand register(
   DatanodeDetails datanodeDetails, NodeReportProto nodeReport,
-  PipelineReportsProto pipelineReportsProto) {
-
+  PipelineReportsProto pipelineReportsProto,
+  LayoutVersionProto layoutInfo) {
+
+if (layoutInfo != null) {

Review comment:
   thanks @avijayanhwx  for the review. I thought about it while I wrote 
this. I wanted all registration related processing for all the passed 
parameters to be consolidated in one place and it seems like 
NodeManager.register is the right place even though we have to add another 
parameter in bunch of places. Majority of these places are unit tests.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1457: HDDS-4253. Add LayoutVersion request/response for DN registration.

2020-10-01 Thread GitBox 


prashantpogde commented on a change in pull request #1457:
URL: https://github.com/apache/hadoop-ozone/pull/1457#discussion_r498600362



##
File path: 
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java
##
@@ -170,6 +176,33 @@ public void testScmHeartbeat()
 }
   }
 
+  /**
+   * Tests that Node manager handles Layout versions correctly.
+   *
+   * @throws IOException
+   * @throws InterruptedException
+   * @throws TimeoutException
+   */
+  @Test
+  public void testScmLayoutOnRegister()
+  throws IOException, InterruptedException, AuthenticationException {
+
+try (SCMNodeManager nodeManager = createNodeManager(getConf())) {
+  LayoutVersionProto layoutInfoSuccess = LayoutVersionProto.newBuilder()
+  .setMetadataLayoutVersion(1).setSoftwareLayoutVersion(1).build();

Review comment:
   Yup, I will do this.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1457: HDDS-4253. Add LayoutVersion request/response for DN registration.

2020-10-01 Thread GitBox 


prashantpogde commented on a change in pull request #1457:
URL: https://github.com/apache/hadoop-ozone/pull/1457#discussion_r498600324



##
File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##
@@ -240,8 +247,19 @@ public VersionResponse getVersion(SCMVersionRequestProto 
versionRequest) {
   @Override
   public RegisteredCommand register(
   DatanodeDetails datanodeDetails, NodeReportProto nodeReport,
-  PipelineReportsProto pipelineReportsProto) {
-
+  PipelineReportsProto pipelineReportsProto,
+  LayoutVersionProto layoutInfo) {
+
+if (layoutInfo != null) {

Review comment:
   I thought about it while I wrote this. I wanted all registration related 
processing for all the passed parameters to be consolidated in one place and it 
seems like NodeManager.register is the right place even though we have to add 
another parameter in bunch of places. Majority of these places are unit tests.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1457: HDDS-4253. Add LayoutVersion request/response for DN registration.

2020-10-01 Thread GitBox 


prashantpogde commented on a change in pull request #1457:
URL: https://github.com/apache/hadoop-ozone/pull/1457#discussion_r498600060



##
File path: 
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
##
@@ -240,8 +247,19 @@ public VersionResponse getVersion(SCMVersionRequestProto 
versionRequest) {
   @Override
   public RegisteredCommand register(
   DatanodeDetails datanodeDetails, NodeReportProto nodeReport,
-  PipelineReportsProto pipelineReportsProto) {
-
+  PipelineReportsProto pipelineReportsProto,
+  LayoutVersionProto layoutInfo) {
+
+if (layoutInfo != null) {
+  if (layoutInfo.getSoftwareLayoutVersion() >
+  scmLayoutVersionManager.getSoftwareLayoutVersion()) {

Review comment:
   SCM can handle multiple DNs and it may make log verbose. This will be 
thrown back to the DN and it will be logged there for debugging as to why 
datanode is not able to register with SCM. 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1457: HDDS-4253. Add LayoutVersion request/response for DN registration.

2020-10-01 Thread GitBox 


prashantpogde commented on a change in pull request #1457:
URL: https://github.com/apache/hadoop-ozone/pull/1457#discussion_r498599539



##
File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/StorageContainerNodeProtocol.java
##
@@ -58,7 +60,8 @@
*/
   RegisteredCommand register(DatanodeDetails datanodeDetails,
  NodeReportProto nodeReport,
- PipelineReportsProto pipelineReport);
+ PipelineReportsProto pipelineReport,
+ LayoutVersionProto layoutVersionInfo);

Review comment:
   @linyiqun  Thanks for the review. yup I will add it.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1457: HDDS-4253. Add LayoutVersion request/response for DN registration.

2020-10-01 Thread GitBox 


prashantpogde commented on a change in pull request #1457:
URL: https://github.com/apache/hadoop-ozone/pull/1457#discussion_r498599539



##
File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/StorageContainerNodeProtocol.java
##
@@ -58,7 +60,8 @@
*/
   RegisteredCommand register(DatanodeDetails datanodeDetails,
  NodeReportProto nodeReport,
- PipelineReportsProto pipelineReport);
+ PipelineReportsProto pipelineReport,
+ LayoutVersionProto layoutVersionInfo);

Review comment:
   yup





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDDS-3966) Intermittent crash in TestOMRatisSnapshots

2020-10-01 Thread Bharat Viswanadham (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham reassigned HDDS-3966:


Assignee: Aryan Gupta

> Intermittent crash in TestOMRatisSnapshots
> --
>
> Key: HDDS-3966
> URL: https://issues.apache.org/jira/browse/HDDS-3966
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Manager
>Reporter: Attila Doroszlai
>Assignee: Aryan Gupta
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.1.0
>
>
> TestOMRatisSnapshots was recently enabled and is crashing intermittently:
> https://github.com/elek/ozone-build-results/tree/master/2020/07/14/1690/it-hdds-om
> https://github.com/elek/ozone-build-results/tree/master/2020/07/14/1710/it-hdds-om
> https://github.com/elek/ozone-build-results/tree/master/2020/07/15/1713/it-hdds-om



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-3966) Intermittent crash in TestOMRatisSnapshots

2020-10-01 Thread Bharat Viswanadham (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham resolved HDDS-3966.
--
Fix Version/s: 1.1.0
   Resolution: Fixed

> Intermittent crash in TestOMRatisSnapshots
> --
>
> Key: HDDS-3966
> URL: https://issues.apache.org/jira/browse/HDDS-3966
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: Ozone Manager
>Reporter: Attila Doroszlai
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.1.0
>
>
> TestOMRatisSnapshots was recently enabled and is crashing intermittently:
> https://github.com/elek/ozone-build-results/tree/master/2020/07/14/1690/it-hdds-om
> https://github.com/elek/ozone-build-results/tree/master/2020/07/14/1710/it-hdds-om
> https://github.com/elek/ozone-build-results/tree/master/2020/07/15/1713/it-hdds-om



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1441: HDDS-3966. Enable TestOMRatisSnapshots.

2020-10-01 Thread GitBox 


bharatviswa504 merged pull request #1441:
URL: https://github.com/apache/hadoop-ozone/pull/1441


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1441: HDDS-3966. Enable TestOMRatisSnapshots.

2020-10-01 Thread GitBox 


bharatviswa504 commented on pull request #1441:
URL: https://github.com/apache/hadoop-ozone/pull/1441#issuecomment-702469785


   Thank You @aryangupta1998 for the contribution and @hanishakoneru for the 
info.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] umamaheswararao merged pull request #1469: HDDS-4302 : Shade the org.apache.common.lang3 package as this is coming from other hadoop packages as well.

2020-10-01 Thread GitBox 


umamaheswararao merged pull request #1469:
URL: https://github.com/apache/hadoop-ozone/pull/1469


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] umamaheswararao commented on pull request #1469: HDDS-4302 : Shade the org.apache.common.lang3 package as this is coming from other hadoop packages as well.

2020-10-01 Thread GitBox 


umamaheswararao commented on pull request #1469:
URL: https://github.com/apache/hadoop-ozone/pull/1469#issuecomment-702468583


   Thanks @elek for help on reproducing the issue locally on duplicate class 
verification test.
   Thanks @bharatviswa504 for the review!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-4302) Shade the org.apache.common.lang3 package as this is coming from other hadoop packages as well.

2020-10-01 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-4302:
-
Labels: pull-request-available  (was: )

> Shade the org.apache.common.lang3 package as this is coming from other hadoop 
> packages as well.
> ---
>
> Key: HDDS-4302
> URL: https://issues.apache.org/jira/browse/HDDS-4302
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>  Labels: pull-request-available
>
> In one of our duplicate classes tests, we noticed the duplicate classes 
> because of commons-lang3. To avoid class collisions, it's good to shade the 
> common-lang3 package as well.
> java.lang.Exception: Duplicate class 
> 'org.apache.commons.lang3.arch.Processor$Arch.class' detected in 
> '/Users/umagangumalla/Work/repos/Gerrit/xxx/xxx/target/xxx-client-x2.3-dependencies/hadoop-ozone-filesystem-hadoop3-.jar',
>  class is already present in 
> '/Users/umagangumalla/Work/repos/Gerrit/xxx/xxx/target/xxx-client-xxx-dependencies/commons-lang3-3.9.jar'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] umamaheswararao opened a new pull request #1469: HDDS-4302 : Shade the org.apache.common.lang3 package as this is coming from other hadoop packages as well.

2020-10-01 Thread GitBox 


umamaheswararao opened a new pull request #1469:
URL: https://github.com/apache/hadoop-ozone/pull/1469


   ## What changes were proposed in this pull request?
   Shading the org.apache.hadoop.lang3 package.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-4302
   
   ## How was this patch tested?
   Ran the same duplicate classes tests.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-4302) Shade the org.apache.common.lang3 package as this is coming from other hadoop packages as well.

2020-10-01 Thread Uma Maheswara Rao G (Jira) 
Uma Maheswara Rao G created HDDS-4302:
-

 Summary: Shade the org.apache.common.lang3 package as this is 
coming from other hadoop packages as well.
 Key: HDDS-4302
 URL: https://issues.apache.org/jira/browse/HDDS-4302
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


In one of our duplicate classes tests, we noticed the duplicate classes because 
of commons-lang3. To avoid class collisions, it's good to shade the 
common-lang3 package as well.

java.lang.Exception: Duplicate class 
'org.apache.commons.lang3.arch.Processor$Arch.class' detected in 
'/Users/umagangumalla/Work/repos/Gerrit/xxx/xxx/target/xxx-client-x2.3-dependencies/hadoop-ozone-filesystem-hadoop3-.jar',
 class is already present in 
'/Users/umagangumalla/Work/repos/Gerrit/xxx/xxx/target/xxx-client-xxx-dependencies/commons-lang3-3.9.jar'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] vivekratnavel commented on pull request #1448: HDDS-4280. Document notable configurations for Recon.

2020-10-01 Thread GitBox 


vivekratnavel commented on pull request #1448:
URL: https://github.com/apache/hadoop-ozone/pull/1448#issuecomment-702426915


   @adoroszlai Thanks for working on this!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-4301) SCM CA certificate does not encode KeyUsage extension propertly

2020-10-01 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-4301:
-
Labels: pull-request-available  (was: )

> SCM CA certificate does not encode KeyUsage extension propertly
> ---
>
> Key: HDDS-4301
> URL: https://issues.apache.org/jira/browse/HDDS-4301
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Security
>Affects Versions: 1.0.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
>
> This could be problematic with strict security provider such as FIPS. The 
> default non-FIPS provider such as SunJCE and BC provider work fine though. 
> This ticket is opened to fix it. 
> {code:java}
> 2020-09-30 12:01:52,962 ERROR 
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer: 
> Unable to initialize CertificateServer.
> org.apache.hadoop.hdds.security.exception.SCMSecurityException: 
> java.security.cert.CertificateParsingException: cannot construct KeyUsage: 
> java.lang.IllegalArgumentException: illegal object in getInstance: 
> com.safelogic.cryptocomply.asn1.DEROctetString
> at 
> org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.getPEMEncodedString(CertificateCodec.java:105)
> at 
> org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.writeCertificate(CertificateCodec.java:182)
> at 
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateRootCertificate(DefaultCAServer.java:495)
> at 
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateSelfSignedCA(DefaultCAServer.java:303)
>   
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] xiaoyuyao opened a new pull request #1468: HDDS-4301. SCM CA certificate does not encode KeyUsage extension prop…

2020-10-01 Thread GitBox 


xiaoyuyao opened a new pull request #1468:
URL: https://github.com/apache/hadoop-ozone/pull/1468


   …ertly.
   
   ## What changes were proposed in this pull request?
   
   Fix the encoding of KeyUsage in the CSR to ensure it can be used in FIPS 
environment. 
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4301
   
   ## How was this patch tested?
   
   FIPS enabled secure ozone cluster and verify that the CA certificate can be 
properly generated and used. 
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-4301) SCM CA certificate does not encode KeyUsage extension propertly

2020-10-01 Thread Xiaoyu Yao (Jira) 
Xiaoyu Yao created HDDS-4301:


 Summary: SCM CA certificate does not encode KeyUsage extension 
propertly
 Key: HDDS-4301
 URL: https://issues.apache.org/jira/browse/HDDS-4301
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
  Components: Security
Affects Versions: 1.0.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


This could be problematic with strict security provider such as FIPS. The 
default non-FIPS provider such as SunJCE and BC provider work fine though. This 
ticket is opened to fix it. 


{code:java}
2020-09-30 12:01:52,962 ERROR 
org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer: 
Unable to initialize CertificateServer.
org.apache.hadoop.hdds.security.exception.SCMSecurityException: 
java.security.cert.CertificateParsingException: cannot construct KeyUsage: 
java.lang.IllegalArgumentException: illegal object in getInstance: 
com.safelogic.cryptocomply.asn1.DEROctetString
at 
org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.getPEMEncodedString(CertificateCodec.java:105)
at 
org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.writeCertificate(CertificateCodec.java:182)
at 
org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateRootCertificate(DefaultCAServer.java:495)
at 
org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateSelfSignedCA(DefaultCAServer.java:303)
  
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDDS-3869) Use different column families for datanode block and metadata

2020-10-01 Thread Hanisha Koneru (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hanisha Koneru resolved HDDS-3869.
--
Fix Version/s: 1.1.0
   Resolution: Fixed

> Use different column families for datanode block and metadata
> -
>
> Key: HDDS-3869
> URL: https://issues.apache.org/jira/browse/HDDS-3869
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Ozone Datanode
>Reporter: Ethan Rose
>Assignee: Ethan Rose
>Priority: Critical
>  Labels: pull-request-available, upgrade
> Fix For: 1.1.0
>
>
> Currently datanodes place all of their data under the default column family 
> in RocksDB. This differs from OM and SCM which organize their data into 
> different column families based on its type. This feature will first move the 
> datanode code off of the database utilities in the hadoop.hdds.utils package 
> (which has no column family support), and move them to the newer utilities 
> used by OM and SCM in the hadoop.hdds.utils.db package (which has column 
> family support). The datanode will divide its data into three column families:
>  # block_data: String keys (block id with optional prefix) map to BlockData 
> objects
>  # metadata: String keys (name of metadata field) map to Long objects.
>  # deleted_blocks: String keys (block id with optional prefix) map to the 
> ChunkInfo lists (lists of chunks corresponding to the block that was deleted).
> A new field, called 'schemaVersion' will be added to container files to 
> indicate whether the container was created using the original schema version 
> 1, where everything was in the default column family, or this new schema 
> version 2. Code should be able to process older schema versions for backwards 
> compatibility.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] hanishakoneru merged pull request #1298: HDDS-3869. Use different column families for datanode block and metadata

2020-10-01 Thread GitBox 


hanishakoneru merged pull request #1298:
URL: https://github.com/apache/hadoop-ozone/pull/1298


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-2949) mkdir : store directory entries in a separate table

2020-10-01 Thread Bharat Viswanadham (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDDS-2949:
-
Fix Version/s: 1.1.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> mkdir : store directory entries in a separate table
> ---
>
> Key: HDDS-2949
> URL: https://issues.apache.org/jira/browse/HDDS-2949
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: Ozone Manager
>Reporter: Supratim Deka
>Assignee: Rakesh Radhakrishnan
>Priority: Major
>  Labels: backward-incompatible, pull-request-available
> Fix For: 1.1.0
>
>
> As of HDDS-2940, all the directories from the path prefix get created as 
> entries in the key table. as per the namespace proposal attached to 
> HDDS-2939, directory entries need to be stored in a separate "directory" 
> table. Files will continue to be stored in the key table, which can be 
> thought of as the "file" table.
> The advantage of a separate directory table is to make directory lookup more 
> efficient - the entire table would fit into memory for a typical file based 
> dataset. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1404: HDDS-2949: mkdir : store directory entries in a separate table

2020-10-01 Thread GitBox 


bharatviswa504 edited a comment on pull request #1404:
URL: https://github.com/apache/hadoop-ozone/pull/1404#issuecomment-702342888


   +1 LGTM.
   Thank You @rakeshadr for multiple updates and @linyiqun for the review.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1404: HDDS-2949: mkdir : store directory entries in a separate table

2020-10-01 Thread GitBox 


bharatviswa504 commented on a change in pull request #1404:
URL: https://github.com/apache/hadoop-ozone/pull/1404#discussion_r498462050



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequestV1.java
##
@@ -0,0 +1,321 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.file;
+
+import com.google.common.base.Optional;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.hadoop.hdds.utils.db.cache.CacheKey;
+import org.apache.hadoop.hdds.utils.db.cache.CacheValue;
+import org.apache.hadoop.ozone.OzoneAcl;
+import org.apache.hadoop.ozone.audit.AuditLogger;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.helpers.*;
+import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
+import org.apache.hadoop.ozone.om.request.util.OmResponseUtil;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.om.response.file.OMDirectoryCreateResponseV1;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateDirectoryRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.CreateDirectoryResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.KeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.Status;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_ALREADY_EXISTS;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_KEY_NAME;
+import static 
org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static 
org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.*;
+
+/**
+ * Handle create directory request. It will add path components to the 
directory
+ * table and maintains file system semantics.
+ */
+public class OMDirectoryCreateRequestV1 extends OMDirectoryCreateRequest {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(OMDirectoryCreateRequestV1.class);
+
+  public OMDirectoryCreateRequestV1(OMRequest omRequest) {
+super(omRequest);
+  }
+
+  @Override
+  public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
+  long trxnLogIndex, OzoneManagerDoubleBufferHelper omDoubleBufferHelper) {
+
+CreateDirectoryRequest createDirectoryRequest = getOmRequest()
+.getCreateDirectoryRequest();
+KeyArgs keyArgs = createDirectoryRequest.getKeyArgs();
+
+String volumeName = keyArgs.getVolumeName();
+String bucketName = keyArgs.getBucketName();
+String keyName = keyArgs.getKeyName();
+int numKeysCreated = 0;
+
+OMResponse.Builder omResponse = OmResponseUtil.getOMResponseBuilder(
+getOmRequest());
+
omResponse.setCreateDirectoryResponse(CreateDirectoryResponse.newBuilder());
+OMMetrics omMetrics = ozoneManager.getMetrics();
+omMetrics.incNumCreateDirectory();
+
+AuditLogger auditLogger = ozoneManager.getAuditLogger();
+OzoneManagerProtocolProtos.UserInfo userInfo = 
getOmRequest().getUserInfo();
+
+Map auditMap = buildKeyArgsAuditMap(keyArgs);
+OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
+boolean acquiredLock = false;
+IOException

[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1404: HDDS-2949: mkdir : store directory entries in a separate table

2020-10-01 Thread GitBox 


bharatviswa504 merged pull request #1404:
URL: https://github.com/apache/hadoop-ozone/pull/1404


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1404: HDDS-2949: mkdir : store directory entries in a separate table

2020-10-01 Thread GitBox 


bharatviswa504 commented on pull request #1404:
URL: https://github.com/apache/hadoop-ozone/pull/1404#issuecomment-702342888


   +1 LGTM.
   Thank You @rakeshadr for multiple updates.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-4268) Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.

2020-10-01 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-4268:
-
Labels: pull-request-available  (was: )

> Prepare for Upgrade step should purge the log after waiting for the last txn 
> to be applied.
> ---
>
> Key: HDDS-4268
> URL: https://issues.apache.org/jira/browse/HDDS-4268
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
>  Labels: pull-request-available
>
> This is a follow up task from HDDS-4227 in which the prepare 
> upgrade/downgrade task should purge the Raft log immediately after waiting 
> for the last txn to be applied. This is to make sure that we dont "apply" 
> transactions in different versions of the code across the quorum. A lagging 
> follower will use a Ratis snapshot to bootstrap itself on restart.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1467: HDDS-4268. Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.

2020-10-01 Thread GitBox 


avijayanhwx opened a new pull request #1467:
URL: https://github.com/apache/hadoop-ozone/pull/1467


   ## What changes were proposed in this pull request?
   This is a follow up task from HDDS-4227 in which the prepare 
upgrade/downgrade task should purge the Raft log immediately after waiting for 
the last txn to be applied. This is to make sure that we dont "apply" 
transactions in different versions of the code across the quorum. A lagging 
follower will use a Ratis snapshot to bootstrap itself on restart.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-4268
   
   ## How was this patch tested?
   Manually tested.
   _Working on unit/integration tests._



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1441: HDDS-3966. Enable TestOMRatisSnapshots.

2020-10-01 Thread GitBox 


hanishakoneru commented on pull request #1441:
URL: https://github.com/apache/hadoop-ozone/pull/1441#issuecomment-702225601


   There were some changes made as part of HDDS-3741. That could have possibly 
resolved this.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1464: HDDS-4299. Display Ratis version with ozone version

2020-10-01 Thread GitBox 


adoroszlai commented on a change in pull request #1464:
URL: https://github.com/apache/hadoop-ozone/pull/1464#discussion_r498323795



##
File path: 
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/RatisVersionInfo.java
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.utils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Properties;
+
+import org.apache.hadoop.util.ThreadUtil;
+
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class returns build information about Ratis projects.
+ */
+public class RatisVersionInfo {
+
+  private static final String RATIS_VERSION_PROPERTIES =
+  "ratis-version.properties";
+
+  private final Properties info = new Properties();
+
+  public RatisVersionInfo() {
+try (InputStream is = ThreadUtil.getResourceAsStream(
+getClass().getClassLoader(),
+RATIS_VERSION_PROPERTIES)) {
+  info.load(is);
+} catch (IOException ex) {
+  LoggerFactory.getLogger(getClass()).warn("Could not read '" +
+  RATIS_VERSION_PROPERTIES, ex);

Review comment:
   ```suggestion
 LoggerFactory.getLogger(getClass()).warn("Could not read " +
 RATIS_VERSION_PROPERTIES, ex);
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-01 Thread GitBox 


linyiqun commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498312712



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;

Review comment:
   Here clientID should be thread-safe to update, can we make clientID as a 
volatile variable?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] captainzmc edited a comment on pull request #1447: HDDS-4264. Uniform naming conventions of Ozone Shell Options.

2020-10-01 Thread GitBox 


captainzmc edited a comment on pull request #1447:
URL: https://github.com/apache/hadoop-ozone/pull/1447#issuecomment-702161601


   Thanks for @adoroszlai 's reminding. I have solved the conflict. This PR can 
continue to be reviewed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1447: HDDS-4264. Uniform naming conventions of Ozone Shell Options.

2020-10-01 Thread GitBox 


captainzmc commented on pull request #1447:
URL: https://github.com/apache/hadoop-ozone/pull/1447#issuecomment-702161601


   > Thanks for working on this @captainzmc. Can you please resolve conflict 
with latest master when you get a chance?
   Thanks for @adoroszlai 's reminding. I have solved the conflict. This PR can 
continue to be reviewed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-4271) Avoid logging chunk content in Ozone Insight

2020-10-01 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-4271:
-
Labels: pull-request-available  (was: )

> Avoid logging chunk content in Ozone Insight
> 
>
> Key: HDDS-4271
> URL: https://issues.apache.org/jira/browse/HDDS-4271
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>  Labels: pull-request-available
>
> HDDS-2660 added an insight point for the datanode dispatcher.  At trace level 
> it logs all chunk content, which can be huge and contain control characters, 
> so I think we should avoid it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1466: HDDS-4271. Avoid logging chunk content in Ozone Insight

2020-10-01 Thread GitBox 


adoroszlai opened a new pull request #1466:
URL: https://github.com/apache/hadoop-ozone/pull/1466


   ## What changes were proposed in this pull request?
   
   HDDS-2660 added an insight point for the datanode dispatcher.  At trace 
level it logs all chunk content, which can be huge and contain control 
characters.  This PR removes chunk data from such log messages.
   
   https://issues.apache.org/jira/browse/HDDS-4271
   
   ## How was this patch tested?
   
   Started Ozone Insight (for all 3 datanodes in separate windows):
   
   ```
   ozone insight logs -v datanode.dispatcher -f datanode=ozone_datanode_1
   ```
   
   Ran test:
   
   ```
   ozone freon ockg -n1 -t1 -p warmup
   ozone sh key put /vol1/bucket1/passwd /etc/passwd
   ozone sh key get /vol1/bucket1/passwd /tmp/passwd
   diff -q /etc/passwd /tmp/passwd
   ```
   
   Output for `ozone freon ockg -n1 -t1 -p warmup`:
   
   ```
   [DATANODE] ... 
[TRACE|org.apache.hadoop.ozone.container.common.impl.HddsDispatcher|OzoneProtocolMessageDispatcher]
 [service=DatanodeClient] [type=WriteChunk] request is received:
   cmdType: WriteChunk
   traceID: ""
   containerID: 1
   datanodeUuid: "833184d2-c6db-409a-934e-df7699b97061"
   pipelineID: "d8e8bf6c-6eca-4941-a4a6-3b0b997fbf4f"
   writeChunk {
 blockID {
   containerID: 1
   localID: 104959326432133120
   blockCommitSequenceId: 0
 }
 chunkData {
   chunkName: "104959326432133120_chunk_1"
   offset: 0
   len: 10240
   checksumData {
 type: CRC32
 bytesPerChecksum: 1048576
 checksums: "327275221253"
   }
 }
 data: ""
   }
   
   [DATANODE] ... 
[TRACE|org.apache.hadoop.ozone.container.common.impl.HddsDispatcher|HddsDispatcher]
 Command WriteChunk, trace ID:
   [DATANODE] ... 
[TRACE|org.apache.hadoop.ozone.container.common.impl.HddsDispatcher|OzoneProtocolMessageDispatcher]
 [service=DatanodeClient] [type=WriteChunk] request is processed. Response:
   cmdType: WriteChunk
   traceID: ""
   result: SUCCESS
   message: ""
   ```
   
   Output for `ozone sh key get /vol1/bucket1/passwd /tmp/passwd`:
   
   ```
   [DATANODE] ... 
[TRACE|org.apache.hadoop.ozone.container.common.impl.HddsDispatcher|OzoneProtocolMessageDispatcher]
 [service=DatanodeClient] [type=ReadChunk] request is received:
   cmdType: ReadChunk
   traceID: ""
   containerID: 1
   datanodeUuid: "833184d2-c6db-409a-934e-df7699b97061"
   readChunk {
 blockID {
   containerID: 1
   localID: 104959326926798849
   blockCommitSequenceId: 6
 }
 chunkData {
   chunkName: "104959326926798849_chunk_1"
   offset: 0
   len: 671
   checksumData {
 type: CRC32
 bytesPerChecksum: 1048576
 checksums: "362232235315"
   }
 }
   }
   
   [DATANODE] ... 
[TRACE|org.apache.hadoop.ozone.container.common.impl.HddsDispatcher|HddsDispatcher]
 Command ReadChunk, trace ID:
   [DATANODE] ... 
[TRACE|org.apache.hadoop.ozone.container.common.impl.HddsDispatcher|OzoneProtocolMessageDispatcher]
 [service=DatanodeClient] [type=ReadChunk] request is processed. Response:
   cmdType: ReadChunk
   traceID: ""
   result: SUCCESS
   readChunk {
 blockID {
   containerID: 1
   localID: 104959326926798849
   blockCommitSequenceId: 6
 }
 chunkData {
   chunkName: "104959326926798849_chunk_1"
   offset: 0
   len: 671
   checksumData {
 type: CRC32
 bytesPerChecksum: 1048576
 checksums: "362232235315"
   }
 }
 data: ""
   }
   ```
   
   Regular CI:
   https://github.com/adoroszlai/hadoop-ozone/actions/runs/282279336



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1447: HDDS-4264. Uniform naming conventions of Ozone Shell Options.

2020-10-01 Thread GitBox 


adoroszlai commented on pull request #1447:
URL: https://github.com/apache/hadoop-ozone/pull/1447#issuecomment-702079340


   Thanks for working on this @captainzmc.  Can you please resolve conflict 
with latest master when you get a chance?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] sodonnel opened a new pull request #1465: Removed unneeded class DatanodeAdminNodeDetails

2020-10-01 Thread GitBox 


sodonnel opened a new pull request #1465:
URL: https://github.com/apache/hadoop-ozone/pull/1465


   ## What changes were proposed in this pull request?
   
   DatanodeAdminNodeDetails was added earlier in the decommission branch, to 
track metrics and, the decommission state and maintenance end time.
   
   After enhancing NodeStatus to old the Maintenance Expiry time, this class is 
no longer needed and it also duplicates information which is stored in other 
existing places.
   
   This change removes it and then metrics etc can be added later in a 
different way.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4300
   
   Please replace this section with the link to the Apache JIRA)
   
   ## How was this patch tested?
   
   Exiting tests
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-4300) Remove no longer needed class DatanodeAdminNodeDetails

2020-10-01 Thread Stephen O'Donnell (Jira) 
Stephen O'Donnell created HDDS-4300:
---

 Summary: Remove no longer needed class DatanodeAdminNodeDetails
 Key: HDDS-4300
 URL: https://issues.apache.org/jira/browse/HDDS-4300
 Project: Hadoop Distributed Data Store
  Issue Type: Sub-task
  Components: SCM
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell


DatanodeAdminNodeDetails was added earlier in the decommission branch, to track 
metrics and, the decommission state and maintenance end time. 

After enhancing NodeStatus to old the Maintenance Expiry time, this class is no 
longer needed and it also duplicates information which is stored in other 
existing places.

This change removes it and then metrics etc can be added later in a different 
way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Updated] (HDDS-4299) Display Ratis version with ozone version

2020-10-01 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDDS-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-4299:
-
Labels: pull-request-available  (was: )

> Display Ratis version with ozone version
> 
>
> Key: HDDS-4299
> URL: https://issues.apache.org/jira/browse/HDDS-4299
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
>  Labels: pull-request-available
>
> During the development Ozone uses snapshot releases from Ratis. It can be 
> useful to print out the exact version of the used Ratis as part of the output 
> of "ozone version".
> Ratis versions are part of the jar files since RATIS-1050 
> It can make the testing easier, as it's easier to check which Ratis version 
> is used. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] elek opened a new pull request #1464: HDDS-4299. Display Ratis version with ozone version

2020-10-01 Thread GitBox 


elek opened a new pull request #1464:
URL: https://github.com/apache/hadoop-ozone/pull/1464


   ## What changes were proposed in this pull request?
   
   During the development Ozone uses snapshot releases from Ratis. It can be 
useful to print out the exact version of the used Ratis as part of the output 
of "ozone version".
   
   Ratis versions are part of the jar files since RATIS-1050
   
   It can make the testing easier, as it's easier to check which Ratis version 
is used.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4299
   
   ## How was this patch tested?
   
   ```
./bin/ozone version
 //
  
    
  //  
 /    /
/   ///
   /
   / 
   /      //
 ///   /
/  /// 
 /   //  /
  //   //   /
/// 
  //  
  ///   //
 /1.1.0-SNAPSHOT(Denali)
   
   Source code repository g...@github.com:apache/hadoop-ozone.git -r 
b2c6c3b61572bfd9b765f106ab2cf501db55e5e8
   Compiled by elek on 2020-10-01T10:35Z
   Compiled with protoc 2.5.0
   From source with checksum ff92fd89808d10a6ee1dc21db8276427
   With Apache Ratis: 1.1.0-11689cd-SNAPSHOT from 
11689cd536b05bde323c9c6c6b57a8e82a60b680
   
   Using HDDS 1.1.0-SNAPSHOT
   Source code repository g...@github.com:apache/hadoop-ozone.git -r 
b2c6c3b61572bfd9b765f106ab2cf501db55e5e8
   Compiled by elek on 2020-10-01T10:34Z
   Compiled with protoc 2.5.0
   From source with checksum d0e6e752e4d52fb0f05063e16ade73d7
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[jira] [Created] (HDDS-4299) Display Ratis version with ozone version

2020-10-01 Thread Marton Elek (Jira) 
Marton Elek created HDDS-4299:
-

 Summary: Display Ratis version with ozone version
 Key: HDDS-4299
 URL: https://issues.apache.org/jira/browse/HDDS-4299
 Project: Hadoop Distributed Data Store
  Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek


During the development Ozone uses snapshot releases from Ratis. It can be 
useful to print out the exact version of the used Ratis as part of the output 
of "ozone version".

Ratis versions are part of the jar files since RATIS-1050 

It can make the testing easier, as it's easier to check which Ratis version is 
used. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-01 Thread GitBox 


linyiqun commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498051832



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;
+}
+assertClientId(upgradeClientID);
+List returningMsgs = new ArrayList<>(msgs.size()+10);
+status = isDone ? FINALIZATION_DONE : FINALIZATION_IN_PROGRESS;
+while (msgs.size() > 0) {
+  returningMsgs.add(msgs.poll());
+}
+return new StatusAndMessages(status, returningMsgs);
+  }
+
+  private void assertClientId(String id) throws OMException {
+if (!this.clientID.equals(id)) {
+  throw new OMException("Unknown client tries to get finalization 
status.\n"
+  + "The requestor is not the initiating client of the finalization,"
+  + " if you want to take over, and get unsent status messages, check"
+  + " -takeover option.", INVALID_REQUEST);
+}
+  }
+
+
+
+
+  private class Worker implements Callable {
+private OzoneManager ozoneManager;
+
+

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1456: HDDS-4172. Implement Finalize command in Ozone Manager server.

2020-10-01 Thread GitBox 


linyiqun commented on a change in pull request #1456:
URL: https://github.com/apache/hadoop-ozone/pull/1456#discussion_r498056289



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMUpgradeFinalizer.java
##
@@ -0,0 +1,303 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes;
+import org.apache.hadoop.ozone.upgrade.UpgradeFinalizer;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.PERSIST_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.REMOVE_UPGRADE_TO_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.UPDATE_LAYOUT_VERSION_FAILED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.ALREADY_FINALIZED;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_DONE;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static 
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+/**
+ * UpgradeFinalizer implementation for the Ozone Manager service.
+ */
+public class OMUpgradeFinalizer implements UpgradeFinalizer {
+
+  private Status status = ALREADY_FINALIZED;
+  private OMLayoutVersionManagerImpl versionManager;
+  private String clientID;
+
+  private Queue msgs = new ConcurrentLinkedQueue<>();
+  private boolean isDone = false;
+
+  private static final OmUpgradeAction NOOP = a -> {};
+
+  public OMUpgradeFinalizer(OMLayoutVersionManagerImpl versionManager) {
+this.versionManager = versionManager;
+if (versionManager.needsFinalization()) {
+  status = FINALIZATION_REQUIRED;
+}
+  }
+
+  @Override
+  public StatusAndMessages finalize(String upgradeClientID, OzoneManager om)
+  throws IOException {
+if (!versionManager.needsFinalization()) {
+  return FINALIZED_MSG;
+}
+clientID = upgradeClientID;
+
+// This requires some more investigation on how to do it properly while
+// requests are on the fly, and post finalize features one by one.
+// Until that is done, monitoring is not really doing anything meaningful
+// but this is a tradoff we can take for the first iteration either if needed,
+// as the finalization of the first few features should not take that long.
+// Follow up JIRA is in HDDS-4286
+//String threadName = "OzoneManager-Upgrade-Finalizer";
+//ExecutorService executor =
+//Executors.newSingleThreadExecutor(r -> new Thread(threadName));
+//executor.submit(new Worker(om));
+new Worker(om).call();
+return STARTING_MSG;
+  }
+
+  @Override
+  public StatusAndMessages reportStatus(
+  String upgradeClientID, boolean takeover
+  ) throws IOException {
+if (takeover) {
+  clientID = upgradeClientID;
+}
+assertClientId(upgradeClientID);
+List returningMsgs = new ArrayList<>(msgs.size()+10);
+status = isDone ? FINALIZATION_DONE : FINALIZATION_IN_PROGRESS;
+while (msgs.size() > 0) {
+  returningMsgs.add(msgs.poll());
+}
+return new StatusAndMessages(status, returningMsgs);
+  }
+
+  private void assertClientId(String id) throws OMException {
+if (!this.clientID.equals(id)) {
+  throw new OMException("Unknown client tries to get finalization 
status.\n"
+  + "The requestor is not the initiating client of the finalization,"
+  + " if you want to take over, and get unsent status messages, check"
+  + " -takeover option.", INVALID_REQUEST);
+}
+  }
+
+
+
+
+  private class Worker implements Callable {
+private OzoneManager ozoneManager;
+
+

[GitHub] [hadoop-ozone] elek commented on pull request #1461: Merge Master into HDDS-1880-Decom

2020-10-01 Thread GitBox 


elek commented on pull request #1461:
URL: https://github.com/apache/hadoop-ozone/pull/1461#issuecomment-701947184


   /pending This merge should be committed manually instead of using the 
"squash and merge" button.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org