[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1339: HDDS-4131. Container report should update container key count and bytes used if they differ in SCM

2020-08-26 Thread GitBox
ChenSammi commented on a change in pull request #1339: URL: https://github.com/apache/hadoop-ozone/pull/1339#discussion_r478196714 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -398,8 +398,6 @@ public voi

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1339: HDDS-4131. Container report should update container key count and bytes used if they differ in SCM

2020-08-26 Thread GitBox
ChenSammi commented on a change in pull request #1339: URL: https://github.com/apache/hadoop-ozone/pull/1339#discussion_r478196714 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -398,8 +398,6 @@ public voi

[GitHub] [hadoop-ozone] maobaolong closed pull request #1290: HDDS-4064. Show container verbose info with verbose option

2020-08-26 Thread GitBox
maobaolong closed pull request #1290: URL: https://github.com/apache/hadoop-ozone/pull/1290 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] prashantpogde commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-681631035 I have uploaded the changes after addressing all review comments. Please take a look. This is an aut

[jira] [Updated] (HDDS-4143) Implement a version factory for OM Apply Transaction that uses the implementation based on layout version.

2020-08-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4143: - Labels: pull-request-available (was: ) > Implement a version factory for OM Apply Transaction tha

[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1358: HDDS-4143. Implement a version factory for OM Apply Transaction that …

2020-08-26 Thread GitBox
avijayanhwx opened a new pull request #1358: URL: https://github.com/apache/hadoop-ozone/pull/1358 …uses the implementation based on layout version. This is an automated message from the Apache Git Service. To respond to

[jira] [Assigned] (HDDS-3881) Add current layout version to OM Ratis Request

2020-08-26 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-3881: --- Assignee: Aravindan Vijayan (was: Prashant Pogde) > Add current layout version to OM

[jira] [Assigned] (HDDS-4143) Implement a version factory for OM Apply Transaction that uses the implementation based on layout version.

2020-08-26 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4143: --- Assignee: Aravindan Vijayan (was: Prashant Pogde) > Implement a version factory for

[GitHub] [hadoop-ozone] maobaolong closed pull request #1343: HDDS-4137. Turn on the verbose mode of safe mode check on testlib

2020-08-26 Thread GitBox
maobaolong closed pull request #1343: URL: https://github.com/apache/hadoop-ozone/pull/1343 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[jira] [Created] (HDDS-4159) Rename ozone to hdds in hadoop-hdds related module

2020-08-26 Thread maobaolong (Jira)
maobaolong created HDDS-4159: Summary: Rename ozone to hdds in hadoop-hdds related module Key: HDDS-4159 URL: https://issues.apache.org/jira/browse/HDDS-4159 Project: Hadoop Distributed Data Store

[jira] [Created] (HDDS-4158) Provide a class type for Java based configuration

2020-08-26 Thread maobaolong (Jira)
maobaolong created HDDS-4158: Summary: Provide a class type for Java based configuration Key: HDDS-4158 URL: https://issues.apache.org/jira/browse/HDDS-4158 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] maobaolong closed pull request #1290: HDDS-4064. Show container verbose info with verbose option

2020-08-26 Thread GitBox
maobaolong closed pull request #1290: URL: https://github.com/apache/hadoop-ozone/pull/1290 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] maobaolong closed pull request #1343: HDDS-4137. Turn on the verbose mode of safe mode check on testlib

2020-08-26 Thread GitBox
maobaolong closed pull request #1343: URL: https://github.com/apache/hadoop-ozone/pull/1343 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1290: HDDS-4064. Show container verbose info with verbose option

2020-08-26 Thread GitBox
maobaolong commented on a change in pull request #1290: URL: https://github.com/apache/hadoop-ozone/pull/1290#discussion_r478061552 ## File path: hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/InfoSubcommand.java ## @@ -52,6 +53,10 @@ @Parameters(d

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.

2020-08-26 Thread GitBox
captainzmc commented on a change in pull request #1337: URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r478042483 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -200,9 +200,9 @@ public static Versioning getVersio

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.

2020-08-26 Thread GitBox
captainzmc commented on a change in pull request #1337: URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r478042483 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -200,9 +200,9 @@ public static Versioning getVersio

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
maobaolong commented on pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-681273117 @bharatviswa504 Thanks for your review, I reply your comments inline. In fact the performance improvement comes from https://github.com/apache/hadoop-ozone/pull/843 , I

[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
maobaolong commented on a change in pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477891301 ## File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java ## @@ -557,8 +559,14 @@ private Respon

[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
maobaolong commented on a change in pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477884943 ## File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java ## @@ -277,7 +278,8 @@ public Response

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title in doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Description: According to my understanding, the title in index.md should be the same as the directory na

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title in doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Description: According to my understanding, the title in index.md should be the same as the directory na

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Description: According to my understanding, the title in index.md should be the same as the directory. W

[jira] [Commented] (HDDS-4157) doc/content/concept does not match the title in doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185533#comment-17185533 ] Xiang Zhang commented on HDDS-4157: --- cc [~elek] > doc/content/concept does not match th

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title in doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Summary: doc/content/concept does not match the title in doc/content/concept/index.md (was: doc/content

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Description: According to my understanding, the title in index.md should be the same as the directory. W

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Component/s: documentation > doc/content/concept does not match the title doc/content/concept/index.md >

[jira] [Updated] (HDDS-4157) doc/content/concept does not match the title doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Zhang updated HDDS-4157: -- Description: According to my understanding, the title in index.md should be the same as the directory. W

[jira] [Created] (HDDS-4157) doc/content/concept does not match the title doc/content/concept/index.md

2020-08-26 Thread Xiang Zhang (Jira)
Xiang Zhang created HDDS-4157: - Summary: doc/content/concept does not match the title doc/content/concept/index.md Key: HDDS-4157 URL: https://issues.apache.org/jira/browse/HDDS-4157 Project: Hadoop Distr

[jira] [Created] (HDDS-4156) add hierarchical layout to Chinese doc

2020-08-26 Thread Xiang Zhang (Jira)
Xiang Zhang created HDDS-4156: - Summary: add hierarchical layout to Chinese doc Key: HDDS-4156 URL: https://issues.apache.org/jira/browse/HDDS-4156 Project: Hadoop Distributed Data Store Issue Ty

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477857398 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java ## @@ -0,0 +1,244 @@ +/**

[GitHub] [hadoop-ozone] prashantpogde commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-681204335 > Thanks to work on it @prashantpogde > > One related question (might be obvious): how is this servlet secured? Is it available only for admins? Server c

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477855426 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDBCheckpointServlet.java ## @@ -0,0 +1,216 @@ +/** + *

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477855935 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDBCheckpointServlet.java ## @@ -0,0 +1,216 @@ +/** + *

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477855058 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -79,6 +79,10 @@ public static final String OZ

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477851715 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java ## @@ -352,6 +352,11 @@ public static final St

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
bharatviswa504 commented on a change in pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477699663 ## File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java ## @@ -557,8 +559,14 @@ private Re

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
bharatviswa504 commented on a change in pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354#discussion_r477698606 ## File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java ## @@ -277,7 +278,8 @@ public Resp

[jira] [Updated] (HDDS-4068) Client should not retry same OM on network connection failure

2020-08-26 Thread Hanisha Koneru (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-4068: - Fix Version/s: 1.1 Target Version/s: (was: 0.7.0) > Client should not retry same OM on ne

[jira] [Resolved] (HDDS-4068) Client should not retry same OM on network connection failure

2020-08-26 Thread Hanisha Koneru (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru resolved HDDS-4068. -- Target Version/s: 0.7.0 Resolution: Fixed > Client should not retry same OM on network

[GitHub] [hadoop-ozone] hanishakoneru merged pull request #1324: HDDS-4068. Client should not retry same OM on network connection failure

2020-08-26 Thread GitBox
hanishakoneru merged pull request #1324: URL: https://github.com/apache/hadoop-ozone/pull/1324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477618462 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -79,6 +79,10 @@ public static final String OZ

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
prashantpogde commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477615492 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -79,6 +79,10 @@ public static final String OZ

[jira] [Updated] (HDDS-4155) Directory and filename can end up with same name

2020-08-26 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4155: - Description: Scenario: Create Key via S3, and Create Directory through Fs. # open key ->

[jira] [Updated] (HDDS-4155) Directory and filename can end up with same name in a path

2020-08-26 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4155: - Summary: Directory and filename can end up with same name in a path (was: Directory and f

[jira] [Updated] (HDDS-4155) Directory and filename can end up with same name

2020-08-26 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4155: - Environment: (was: Scenario: # Create key -> /a/b/c # CreateDirectory -> /a/b/c # Co

[jira] [Created] (HDDS-4155) Directory and filename can end up with same name

2020-08-26 Thread Bharat Viswanadham (Jira)
Bharat Viswanadham created HDDS-4155: Summary: Directory and filename can end up with same name Key: HDDS-4155 URL: https://issues.apache.org/jira/browse/HDDS-4155 Project: Hadoop Distributed Data

[jira] [Updated] (HDDS-4155) Directory and filename can end up with same name

2020-08-26 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4155: - Description: Scenario: # Create key -> /a/b/c # CreateDirectory -> /a/b/c # CommitKey -

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1324: HDDS-4068. Client should not retry same OM on network connection failure

2020-08-26 Thread GitBox
hanishakoneru commented on pull request #1324: URL: https://github.com/apache/hadoop-ozone/pull/1324#issuecomment-681131656 Thanks @bharatviswa504 for reviewing the patch. Checked with @vivekratnavel that it is safe to commit since the coverage CI failed only during upload. Will merge th

[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-08-26 Thread GitBox
bharatviswa504 edited a comment on pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328#issuecomment-681125960 > Thanks for the patch @bharatviswa504, I would suggest adding a related robot test, too, which would make it possible to test the AWS compatibility of the ch

[jira] [Created] (HDDS-4154) Enable S3 robot tests with ozone.om.enable.filesystem.paths enabled

2020-08-26 Thread Bharat Viswanadham (Jira)
Bharat Viswanadham created HDDS-4154: Summary: Enable S3 robot tests with ozone.om.enable.filesystem.paths enabled Key: HDDS-4154 URL: https://issues.apache.org/jira/browse/HDDS-4154 Project: Hado

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-08-26 Thread GitBox
bharatviswa504 commented on pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328#issuecomment-681125960 > Thanks for the patch @bharatviswa504, I would suggest adding a related robot test, too, which would make it possible to test the AWS compatibility of the change.

[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-08-26 Thread GitBox
bharatviswa504 edited a comment on pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328#issuecomment-681124935 > Tested and `file/` and `file` are different: > > ``` > $ aws s3api get-object --bucket=ozonetest --key=test1/file /tmp/www > { > "AcceptRa

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-08-26 Thread GitBox
bharatviswa504 commented on pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328#issuecomment-681124935 > Tested and `file/` and `file` are different: > > ``` > $ aws s3api get-object --bucket=ozonetest --key=test1/file /tmp/www > { > "AcceptRanges":

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.

2020-08-26 Thread GitBox
amaliujia commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r477534266 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java ## @@ -0,0 +1,66 @@ +package org.apach

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.

2020-08-26 Thread GitBox
amaliujia commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r477533780 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java ## @@ -230,4 +230,5 @@ Pi

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.

2020-08-26 Thread GitBox
amaliujia commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r477533235 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java ## @@ -560,6 +563,13 @@ public b

[jira] [Updated] (HDDS-4149) Implement OzoneFileStatus#toString

2020-08-26 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4149: --- Status: Patch Available (was: In Progress) > Implement OzoneFileStatus#toString > ---

[jira] [Updated] (HDDS-4152) Archive container logs for kubernetes check

2020-08-26 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4152: --- Status: Patch Available (was: In Progress) > Archive container logs for kubernetes check > --

[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #1352: HDDS-4147. Add OFS to FileSystem META-INF

2020-08-26 Thread GitBox
smengcl commented on a change in pull request #1352: URL: https://github.com/apache/hadoop-ozone/pull/1352#discussion_r477467262 ## File path: hadoop-ozone/dist/src/main/compose/ozone-csi/docker-config ## @@ -14,6 +14,8 @@ # See the License for the specific language governing

[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #1352: HDDS-4147. Add OFS to FileSystem META-INF

2020-08-26 Thread GitBox
smengcl commented on a change in pull request #1352: URL: https://github.com/apache/hadoop-ozone/pull/1352#discussion_r477467262 ## File path: hadoop-ozone/dist/src/main/compose/ozone-csi/docker-config ## @@ -14,6 +14,8 @@ # See the License for the specific language governing

[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1357: HDDS-4153. Increase default timeout in kubernetes tests

2020-08-26 Thread GitBox
codecov-commenter commented on pull request #1357: URL: https://github.com/apache/hadoop-ozone/pull/1357#issuecomment-681007741 # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1357?src=pr&el=h1) Report > Merging [#1357](https://codecov.io/gh/apache/hadoop-ozone/pull/1357?src

[jira] [Updated] (HDDS-4153) Increase default timeout in kubernetes tests

2020-08-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4153: - Labels: pull-request-available (was: ) > Increase default timeout in kubernetes tests > -

[GitHub] [hadoop-ozone] elek opened a new pull request #1357: HDDS-4153. Increase default timeout in kubernetes tests

2020-08-26 Thread GitBox
elek opened a new pull request #1357: URL: https://github.com/apache/hadoop-ozone/pull/1357 ## What changes were proposed in this pull request? Kubernetes tests are timing out sometimes. (eg. here: https://github.com/elek/ozone-build-results/tree/master/2020/08/26/2562/kubernetes)

[GitHub] [hadoop-ozone] linyiqun edited a comment on pull request #1208: HDDS-3755. Storage-class support for Ozone

2020-08-26 Thread GitBox
linyiqun edited a comment on pull request #1208: URL: https://github.com/apache/hadoop-ozone/pull/1208#issuecomment-680961442 The POC patch looks great, we can flexibly adjust the replica number/ storage type for the storage. This is very similar to the HDFS storage policy design, some hig

[jira] [Created] (HDDS-4153) Increase default timeout in kubernetes tests

2020-08-26 Thread Marton Elek (Jira)
Marton Elek created HDDS-4153: - Summary: Increase default timeout in kubernetes tests Key: HDDS-4153 URL: https://issues.apache.org/jira/browse/HDDS-4153 Project: Hadoop Distributed Data Store Is

[GitHub] [hadoop-ozone] linyiqun commented on pull request #1208: HDDS-3755. Storage-class support for Ozone

2020-08-26 Thread GitBox
linyiqun commented on pull request #1208: URL: https://github.com/apache/hadoop-ozone/pull/1208#issuecomment-680961442 The POC patch looks great, we can flexibly adjust the replica number/ storage type for the storage. This is very similar to the HDFS storage policy design, some high level

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1352: HDDS-4147. Add OFS to FileSystem META-INF

2020-08-26 Thread GitBox
adoroszlai commented on a change in pull request #1352: URL: https://github.com/apache/hadoop-ozone/pull/1352#discussion_r477384579 ## File path: hadoop-ozone/dist/src/main/compose/ozone-om-ha/docker-config ## @@ -14,8 +14,8 @@ # See the License for the specific language gover

[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op

2020-08-26 Thread Arpit Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185262#comment-17185262 ] Arpit Agarwal commented on HDDS-4097: - bq. I would prefer to keep 100% AWS compatibili

[GitHub] [hadoop-ozone] elek commented on pull request #1312: HDDS-2411. add a datanode chunk validator fo datanode chunk generator

2020-08-26 Thread GitBox
elek commented on pull request #1312: URL: https://github.com/apache/hadoop-ozone/pull/1312#issuecomment-680932560 > No, it's not related, it's happening elsewhere (other PRs and commits), too. Yes, but it doesn't seem to be a false positive. I checked and somehow the SCM couldn't c

[GitHub] [hadoop-ozone] elek commented on pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-08-26 Thread GitBox
elek commented on pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328#issuecomment-680919139 Tested and `file/` and `file` are different: ``` $ aws s3api get-object --bucket=ozonetest --key=test1/file /tmp/www { "AcceptRanges": "bytes", "Last

[jira] [Updated] (HDDS-4149) Implement OzoneFileStatus#toString

2020-08-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4149: - Labels: pull-request-available (was: ) > Implement OzoneFileStatus#toString > ---

[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1356: HDDS-4149. Implement OzoneFileStatus#toString

2020-08-26 Thread GitBox
adoroszlai opened a new pull request #1356: URL: https://github.com/apache/hadoop-ozone/pull/1356 ## What changes were proposed in this pull request? Implement `toString()` in `OzoneFileStatus` for debug purposes. https://issues.apache.org/jira/browse/HDDS-4149 ## How wa

[jira] [Updated] (HDDS-4152) Archive container logs for kubernetes check

2020-08-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4152: - Labels: pull-request-available (was: ) > Archive container logs for kubernetes check > --

[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1355: HDDS-4152. Archive container logs for kubernetes check

2020-08-26 Thread GitBox
adoroszlai opened a new pull request #1355: URL: https://github.com/apache/hadoop-ozone/pull/1355 ## What changes were proposed in this pull request? In the last few days _kubernetes_ check has been failing very frequently. This change let's it save logs from all containers to make

[GitHub] [hadoop-ozone] errose28 commented on a change in pull request #1351: HDDS-4121. Implement OmMetadataMangerImpl#getExpiredOpenKeys.

2020-08-26 Thread GitBox
errose28 commented on a change in pull request #1351: URL: https://github.com/apache/hadoop-ozone/pull/1351#discussion_r477334837 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java ## @@ -186,10 +186,11 @@ OmKeyLocationInfo alloca

[GitHub] [hadoop-ozone] elek commented on pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-08-26 Thread GitBox
elek commented on pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328#issuecomment-680898429 Thanks for the patch @bharatviswa504, I would suggest adding a related robot test, too, which would make it possible to test the AWS compatibility of the change. What d

[GitHub] [hadoop-ozone] elek commented on pull request #1208: HDDS-3755. Storage-class support for Ozone

2020-08-26 Thread GitBox
elek commented on pull request #1208: URL: https://github.com/apache/hadoop-ozone/pull/1208#issuecomment-680882568 Thanks for the help from @maobaolong , it's ready to review. The key points of the patch: 1. o3fs/ofs are not changed just ozone sh. Instead of: ```

[jira] [Created] (HDDS-4152) Archive container logs for kubernetes check

2020-08-26 Thread Attila Doroszlai (Jira)
Attila Doroszlai created HDDS-4152: -- Summary: Archive container logs for kubernetes check Key: HDDS-4152 URL: https://issues.apache.org/jira/browse/HDDS-4152 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
maobaolong commented on pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354#issuecomment-680847535 @elek Please take a look at this PR, it can solve the performance issue which involved by https://github.com/apache/hadoop-ozone/pull/1212 -

[jira] [Updated] (HDDS-4151) Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4151: - Labels: pull-request-available (was: ) > Skip the inputstream while offset larger than zero in s3

[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1354: HDDS-4151. Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread GitBox
maobaolong opened a new pull request #1354: URL: https://github.com/apache/hadoop-ozone/pull/1354 ## What changes were proposed in this pull request? Skip the inputstream while offset larger than zero in s3g. ## What is the link to the Apache JIRA HDDS-4151 ## How

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1336: HDDS-4119. Improve performance of the BufferPool management of Ozone client

2020-08-26 Thread GitBox
adoroszlai commented on a change in pull request #1336: URL: https://github.com/apache/hadoop-ozone/pull/1336#discussion_r477253394 ## File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java ## @@ -209,9 +224,16 @@ public IOExcepti

[jira] [Created] (HDDS-4151) Skip the inputstream while offset larger than zero in s3g

2020-08-26 Thread maobaolong (Jira)
maobaolong created HDDS-4151: Summary: Skip the inputstream while offset larger than zero in s3g Key: HDDS-4151 URL: https://issues.apache.org/jira/browse/HDDS-4151 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1344: HDDS-4140. Auto-close /pending pull requests after 21 days of inactivity

2020-08-26 Thread GitBox
elek commented on a change in pull request #1344: URL: https://github.com/apache/hadoop-ozone/pull/1344#discussion_r477245998 ## File path: .github/workflows/close-pending.yaml ## @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contribu

[GitHub] [hadoop-ozone] elek edited a comment on pull request #1349: HDDS-4145. Bump version to 1.1.0-SNAPSHOT on master

2020-08-26 Thread GitBox
elek edited a comment on pull request #1349: URL: https://github.com/apache/hadoop-ozone/pull/1349#issuecomment-680832534 ``` [ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.29 s <<< FAILURE! - in org.apache.hadoop.ozone.recon.api.TestEndpoints [ERROR] testG

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-08-26 Thread GitBox
elek commented on a change in pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#discussion_r477244693 ## File path: hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/datanode/DatanodeDispatcherInsight.java ## @@ -0,0 +1,93 @@ +/* + * Li

[jira] [Created] (HDDS-4150) recon.api.TestEndpoints is flaky

2020-08-26 Thread Marton Elek (Jira)
Marton Elek created HDDS-4150: - Summary: recon.api.TestEndpoints is flaky Key: HDDS-4150 URL: https://issues.apache.org/jira/browse/HDDS-4150 Project: Hadoop Distributed Data Store Issue Type: Bu

[jira] [Commented] (HDDS-4150) recon.api.TestEndpoints is flaky

2020-08-26 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185128#comment-17185128 ] Marton Elek commented on HDDS-4150: --- Might be related to HDDS-4009. > recon.api.TestEnd

[GitHub] [hadoop-ozone] elek commented on pull request #1349: HDDS-4145. Bump version to 1.1.0-SNAPSHOT on master

2020-08-26 Thread GitBox
elek commented on pull request #1349: URL: https://github.com/apache/hadoop-ozone/pull/1349#issuecomment-680832534 ``` [ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.29 s <<< FAILURE! - in org.apache.hadoop.ozone.recon.api.TestEndpoints [ERROR] testGetDatan

[GitHub] [hadoop-ozone] elek commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
elek commented on pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-680823004 Thanks to work on it @prashantpogde One related question (might be obvious): how is this servlet secured? Is it available only for admins? ---

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

2020-08-26 Thread GitBox
elek commented on a change in pull request #1353: URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477229997 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java ## @@ -352,6 +352,11 @@ public static final String HDDS

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1352: HDDS-4147. Add OFS to FileSystem META-INF

2020-08-26 Thread GitBox
elek commented on a change in pull request #1352: URL: https://github.com/apache/hadoop-ozone/pull/1352#discussion_r477227993 ## File path: hadoop-ozone/dist/src/main/compose/ozone-csi/docker-config ## @@ -14,6 +14,8 @@ # See the License for the specific language governing per

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1352: HDDS-4147. Add OFS to FileSystem META-INF

2020-08-26 Thread GitBox
elek commented on a change in pull request #1352: URL: https://github.com/apache/hadoop-ozone/pull/1352#discussion_r477226953 ## File path: hadoop-ozone/dist/src/main/compose/ozone-csi/docker-config ## @@ -14,6 +14,8 @@ # See the License for the specific language governing per

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1336: HDDS-4119. Improve performance of the BufferPool management of Ozone client

2020-08-26 Thread GitBox
bshashikant commented on a change in pull request #1336: URL: https://github.com/apache/hadoop-ozone/pull/1336#discussion_r477226741 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBuffer.java ## @@ -44,9 +45,6 @@ static ChunkBuffer allocate(

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1336: HDDS-4119. Improve performance of the BufferPool management of Ozone client

2020-08-26 Thread GitBox
elek commented on a change in pull request #1336: URL: https://github.com/apache/hadoop-ozone/pull/1336#discussion_r477224596 ## File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java ## @@ -481,7 +516,7 @@ private void handleFlus

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1336: HDDS-4119. Improve performance of the BufferPool management of Ozone client

2020-08-26 Thread GitBox
elek commented on a change in pull request #1336: URL: https://github.com/apache/hadoop-ozone/pull/1336#discussion_r477223940 ## File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java ## @@ -209,9 +224,16 @@ public IOException get

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1336: HDDS-4119. Improve performance of the BufferPool management of Ozone client

2020-08-26 Thread GitBox
elek commented on a change in pull request #1336: URL: https://github.com/apache/hadoop-ozone/pull/1336#discussion_r477222801 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBuffer.java ## @@ -44,9 +45,6 @@ static ChunkBuffer allocate(int cap

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1312: HDDS-2411. add a datanode chunk validator fo datanode chunk generator

2020-08-26 Thread GitBox
adoroszlai commented on pull request #1312: URL: https://github.com/apache/hadoop-ozone/pull/1312#issuecomment-680790911 > One check unsuccessful as Ozone Freon testing on Minikube did not passed. > Error is due to SCM not exiting safe mode when setting up the k8s cluster for minikube, h

[GitHub] [hadoop-ozone] frischHWC commented on pull request #1312: HDDS-2411. add a datanode chunk validator fo datanode chunk generator

2020-08-26 Thread GitBox
frischHWC commented on pull request #1312: URL: https://github.com/apache/hadoop-ozone/pull/1312#issuecomment-680788871 One check unsuccessful as Ozone Freon testing on Minikube did not passed. Error is due to SCM not exiting safe mode when setting up the k8s cluster for minikube, howev

[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1340: HDDS-3188 Enable SCM group with failover proxy for SCM block location.

2020-08-26 Thread GitBox
GlenGeng commented on a change in pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340#discussion_r477189151 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java ## @@ -1053,6 +1053,22 @@ public

  1   2   >