[GitHub] [hadoop-ozone] captainzmc closed pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-16 Thread GitBox
captainzmc closed pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-16 Thread GitBox
captainzmc commented on pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-693789583 Here are the freon test results: Using 100 threads to write data to different buckets under the same volume, adding quota Usage has little impact on performance.

[jira] [Updated] (HDDS-4252) Add the current layout versions to DN - SCM proto payload.

2020-09-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4252: - Labels: pull-request-available (was: ) > Add the current layout versions to DN - SCM proto

[GitHub] [hadoop-ozone] prashantpogde opened a new pull request #1432: HDDS-4252. Add the current layout versions to DN - SCM proto payload.

2020-09-16 Thread GitBox
prashantpogde opened a new pull request #1432: URL: https://github.com/apache/hadoop-ozone/pull/1432 ## What changes were proposed in this pull request? Add the current layout versions to DN - SCM proto payload. ## What is the link to the Apache JIRA

[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-09-16 Thread GitBox
rakeshadr commented on pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#issuecomment-693775540 > Thanks @rakeshadr for working on this. The patch LGTM, +1. Thanks a lot @xiaoyuyao for the reviews!

[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-09-16 Thread GitBox
rakeshadr commented on pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#issuecomment-693775383 > Thanks @rakeshadr for updating the patch. > > I have one question left: shouldn't `getFileStatus` call from `listStatus` pass `clientAddress`? > >

[jira] [Updated] (HDDS-4254) Bucket space: add usedBytes and update it when create and delete key

2020-09-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4254: - Labels: pull-request-available (was: ) > Bucket space: add usedBytes and update it when create

[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-16 Thread GitBox
captainzmc opened a new pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431 ## What changes were proposed in this pull request? In addition, the current Quota setting does not take effect. HDDS-541 gives all the work needed to perfect Quota. This PR is a

[jira] [Created] (HDDS-4254) Bucket space: add usedBytes and update it when create and delete key

2020-09-16 Thread mingchao zhao (Jira)
mingchao zhao created HDDS-4254: --- Summary: Bucket space: add usedBytes and update it when create and delete key Key: HDDS-4254 URL: https://issues.apache.org/jira/browse/HDDS-4254 Project: Hadoop

[jira] [Created] (HDDS-4253) SCM changes to process Layout Info in heartbeat request/response

2020-09-16 Thread Prashant Pogde (Jira)
Prashant Pogde created HDDS-4253: Summary: SCM changes to process Layout Info in heartbeat request/response Key: HDDS-4253 URL: https://issues.apache.org/jira/browse/HDDS-4253 Project: Hadoop

[jira] [Created] (HDDS-4252) Add the current layout versions to DN - SCM proto payload.

2020-09-16 Thread Prashant Pogde (Jira)
Prashant Pogde created HDDS-4252: Summary: Add the current layout versions to DN - SCM proto payload. Key: HDDS-4252 URL: https://issues.apache.org/jira/browse/HDDS-4252 Project: Hadoop Distributed

[GitHub] [hadoop-ozone] lamber-ken commented on a change in pull request #1426: HDDS-4247. Fixed log4j usage in some places

2020-09-16 Thread GitBox
lamber-ken commented on a change in pull request #1426: URL: https://github.com/apache/hadoop-ozone/pull/1426#discussion_r489849456 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSecretManager.java ## @@ -600,7

[GitHub] [hadoop-ozone] runzhiwang commented on pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
runzhiwang commented on pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#issuecomment-693743559 @xiaoyuyao Thanks for review. I have updated the patch. This is an automated message from the Apache

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489833136 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489833136 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489827467 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java ## @@ -102,7 +102,7 @@ public static

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489826760 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489826433 ## File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java

[jira] [Created] (HDDS-4251) Update Ratis version to latest snapshot

2020-09-16 Thread Hanisha Koneru (Jira)
Hanisha Koneru created HDDS-4251: Summary: Update Ratis version to latest snapshot Key: HDDS-4251 URL: https://issues.apache.org/jira/browse/HDDS-4251 Project: Hadoop Distributed Data Store

[jira] [Assigned] (HDDS-4251) Update Ratis version to latest snapshot

2020-09-16 Thread Hanisha Koneru (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru reassigned HDDS-4251: Assignee: Hanisha Koneru > Update Ratis version to latest snapshot >

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
xiaoyuyao commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489780674 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[jira] [Updated] (HDDS-4227) Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.

2020-09-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4227: - Labels: pull-request-available (was: ) > Implement a "prepareForUpgrade" step that applies all

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
xiaoyuyao commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489778062 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-16 Thread GitBox
avijayanhwx opened a new pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430 ## What changes were proposed in this pull request? Through HDDS-4143, we have a generic factory to handle multiple versions of apply transaction implementations based on layout

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-16 Thread GitBox
xiaoyuyao commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r489771893 ## File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-16 Thread GitBox
adoroszlai commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-693585819 Thanks @elek for updating the patch. Interestingly `log -v` stopped working, even if I execute `log` first (as mentioned previously).

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1426: HDDS-4247. Fixed log4j usage in some places

2020-09-16 Thread GitBox
adoroszlai commented on a change in pull request #1426: URL: https://github.com/apache/hadoop-ozone/pull/1426#discussion_r489623364 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSecretManager.java ## @@ -600,7

[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #1429: HDDS-4250. Fix wrong logger name

2020-09-16 Thread GitBox
xiaoyuyao merged pull request #1429: URL: https://github.com/apache/hadoop-ozone/pull/1429 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[jira] [Resolved] (HDDS-4250) Fix wrong logger name

2020-09-16 Thread Xiaoyu Yao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-4250. -- Resolution: Fixed > Fix wrong logger name > - > > Key: HDDS-4250 >

[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #1429: HDDS-4250. Fix wrong logger name

2020-09-16 Thread GitBox
xiaoyuyao commented on pull request #1429: URL: https://github.com/apache/hadoop-ozone/pull/1429#issuecomment-693504661 Thanks @lamber-ken for fixing this. LGTM, +1. This is an automated message from the Apache Git

[jira] [Resolved] (HDDS-4248) Add ratis log parser to ozone script

2020-09-16 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-4248. -- Resolution: Invalid > Add ratis log parser to ozone script >

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1427: HDDS-4248. Add ratis log parser to ozone script.

2020-09-16 Thread GitBox
bharatviswa504 commented on pull request #1427: URL: https://github.com/apache/hadoop-ozone/pull/1427#issuecomment-693445880 Thank You @bshashikant and @adoroszlai for the info. I will close this PR. This is an automated

[GitHub] [hadoop-ozone] bharatviswa504 closed pull request #1427: HDDS-4248. Add ratis log parser to ozone script.

2020-09-16 Thread GitBox
bharatviswa504 closed pull request #1427: URL: https://github.com/apache/hadoop-ozone/pull/1427 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hadoop-ozone] fapifta commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-16 Thread GitBox
fapifta commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-693386962 Hi @llemec, thank you for working on this patch, I think overall the direction is good here, though I miss a few things, and I have a stylistic problem with the

[GitHub] [hadoop-ozone] ChenSammi edited a comment on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-16 Thread GitBox
ChenSammi edited a comment on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-693378960 Thanks @sodonnel for very thoughtful suggestions. > 2. Consider if a datanode was dead for some time, and is then restarted. It will start reporting a

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-16 Thread GitBox
ChenSammi commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-693378960 Thanks @sodonnel for very thoughtful suggestions. > 2. Consider if a datanode was dead for some time, and is then restarted. It will start reporting a replica

[jira] [Updated] (HDDS-4250) Fix wrong logger name

2020-09-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4250: - Labels: pull-request-available (was: ) > Fix wrong logger name > - > >

[GitHub] [hadoop-ozone] lamber-ken opened a new pull request #1429: HDDS-4250. Fix wrong logger name

2020-09-16 Thread GitBox
lamber-ken opened a new pull request #1429: URL: https://github.com/apache/hadoop-ozone/pull/1429 ## What changes were proposed in this pull request? Fix wrong logger name, the logger name doesn't match the class name. example ``` public class OMBucketSetAclRequest

[jira] [Created] (HDDS-4250) Fix wrong logger name

2020-09-16 Thread Xie Lei (Jira)
Xie Lei created HDDS-4250: - Summary: Fix wrong logger name Key: HDDS-4250 URL: https://issues.apache.org/jira/browse/HDDS-4250 Project: Hadoop Distributed Data Store Issue Type: Improvement

[jira] [Updated] (HDDS-4192) enable SCM Raft Group based on config ozone.scm.names

2020-09-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4192: - Labels: pull-request-available (was: ) > enable SCM Raft Group based on config ozone.scm.names >

[GitHub] [hadoop-ozone] GlenGeng opened a new pull request #1428: HDDS-4192: enable SCM Raft Group based on config ozone.scm.names

2020-09-16 Thread GitBox
GlenGeng opened a new pull request #1428: URL: https://github.com/apache/hadoop-ozone/pull/1428 ## What changes were proposed in this pull request? Say ozone.scm.names is `ip1,ip2,ip3`, scm with `ip1` identifies its RaftPeerId as `scm1`, scm with `ip2` identifies its RaftPeerId as

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1423: HDDS-4244. Container deleted wrong replica cause mis-replicated.

2020-09-16 Thread GitBox
ChenSammi commented on pull request #1423: URL: https://github.com/apache/hadoop-ozone/pull/1423#issuecomment-693302200 LGTM +1. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hadoop-ozone] ChenSammi merged pull request #1423: HDDS-4244. Container deleted wrong replica cause mis-replicated.

2020-09-16 Thread GitBox
ChenSammi merged pull request #1423: URL: https://github.com/apache/hadoop-ozone/pull/1423 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-09-16 Thread GitBox
rakeshadr commented on pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#issuecomment-693298296 Thanks @adoroszlai for the reviews. Uploaded new commit addressing your comments. This is an automated

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-09-16 Thread GitBox
rakeshadr commented on a change in pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r489309204 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/fs/OzoneManagerFS.java ## @@ -49,6 +69,21 @@ OpenKeySession

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1385: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-09-16 Thread GitBox
rakeshadr commented on a change in pull request #1385: URL: https://github.com/apache/hadoop-ozone/pull/1385#discussion_r489309466 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java ## @@ -1783,7 +1802,9 @@ private

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1427: HDDS-4248. Add ratis log parser to ozone script.

2020-09-16 Thread GitBox
adoroszlai commented on pull request #1427: URL: https://github.com/apache/hadoop-ozone/pull/1427#issuecomment-693234906 > I can see RatisLogParser is available in Ozone debug cli. Yes, `ratislogparser` was moved from top-level command to sub-command of `ozone debug` in #684.

[GitHub] [hadoop-ozone] lamber-ken commented on pull request #1417: HDDS-4324. Add important comment to ListVolumes logic

2020-09-16 Thread GitBox
lamber-ken commented on pull request #1417: URL: https://github.com/apache/hadoop-ozone/pull/1417#issuecomment-693211171 hi @nandakumar131, can you have a review? refer to https://github.com/apache/hadoop-ozone/pull/385

[jira] [Created] (HDDS-4249) The usage value needs to be adjusted after client crash

2020-09-16 Thread mingchao zhao (Jira)
mingchao zhao created HDDS-4249: --- Summary: The usage value needs to be adjusted after client crash Key: HDDS-4249 URL: https://issues.apache.org/jira/browse/HDDS-4249 Project: Hadoop Distributed Data

[jira] [Resolved] (HDDS-4053) Volume space: add quotaUsageInBytes and update it when create and delete key

2020-09-16 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao resolved HDDS-4053. - Resolution: Fixed PR has been merged, close this. > Volume space: add quotaUsageInBytes and

[GitHub] [hadoop-ozone] bshashikant commented on pull request #1427: HDDS-4248. Add ratis log parser to ozone script.

2020-09-16 Thread GitBox
bshashikant commented on pull request #1427: URL: https://github.com/apache/hadoop-ozone/pull/1427#issuecomment-693197579 Thanks @bharatviswa504 . I can see RatisLogParser is available in Ozone debug cli. `sbanerjee-MBP15:bin sbanerjee$ ./ozone debug Incomplete command Usage: