[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-28 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r495775842 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -496,6 +491,28 @@ private static void

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-28 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r495775842 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -496,6 +491,28 @@ private static void

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-28 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r495775842 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -496,6 +491,28 @@ private static void

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-28 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r495797897 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/SetQuotaHandler.java ## @@ -0,0 +1,72 @@ +/* + * Licensed to

[jira] [Commented] (HDDS-3197) BackgroundPipelineCreator can only serve leader

2020-09-28 Thread Nicholas Jiang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203224#comment-17203224 ] Nicholas Jiang commented on HDDS-3197: -- [~timmylicheng], does this issue indicate that refactor

[jira] [Commented] (HDDS-3211) Make SCM HA configurable

2020-09-28 Thread Nicholas Jiang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203212#comment-17203212 ] Nicholas Jiang commented on HDDS-3211: -- [~timmylicheng], do you mean that add a switch configuration

[jira] [Commented] (HDDS-4283) Remove unsupported upgrade command in ozone cli

2020-09-28 Thread Yiqun Lin (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203145#comment-17203145 ] Yiqun Lin commented on HDDS-4283: - Thanks [~adoroszlai] for the reference, closed this JIRA. > Remove

[jira] [Resolved] (HDDS-4283) Remove unsupported upgrade command in ozone cli

2020-09-28 Thread Yiqun Lin (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin resolved HDDS-4283. - Resolution: Duplicate > Remove unsupported upgrade command in ozone cli >

[jira] [Comment Edited] (HDDS-4285) Read is slow due to the frequent usage of UGI.getCurrentUserCall()

2020-09-28 Thread Yiqun Lin (Jira)
Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Attachments: image-2020-09-28-16-19-17-581.png, > profile-20200928-161631-180518.svg &

[GitHub] [hadoop-ozone] sodonnel commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-28 Thread GitBox
sodonnel commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-700016623 This change looks almost good now. I wonder about two final things: 1. In `updateContainerStats(...)` do you think we should return if the container is DELETING or

[jira] [Updated] (HDDS-4285) Read is slow due to the frequent usage of UGI.getCurrentUserCall()

2020-09-28 Thread Marton Elek (Jira)
Assignee: Marton Elek >Priority: Major > Attachments: image-2020-09-28-16-19-17-581.png, > profile-20200928-161631-180518.svg > > > Ozone read operation turned out to be slow mainly because we do a new > UGI.getCurrentUser for block token for each

[jira] [Created] (HDDS-4285) Read is slow due to the frequent usage of UGI.getCurrentUserCall()

2020-09-28 Thread Marton Elek (Jira)
Data Store Issue Type: Improvement Reporter: Marton Elek Assignee: Marton Elek Attachments: image-2020-09-28-16-19-17-581.png, profile-20200928-161631-180518.svg Ozone read operation turned out to be slow mainly because we do a new UGI.getCurrentUser

[jira] [Updated] (HDDS-4285) Read is slow due to the frequent usage of UGI.getCurrentUserCall()

2020-09-28 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek updated HDDS-4285: -- Attachment: profile-20200928-161631-180518.svg > Read is slow due to the frequent us

[jira] [Comment Edited] (HDDS-4285) Read is slow due to the frequent usage of UGI.getCurrentUserCall()

2020-09-28 Thread Yiqun Lin (Jira)
rowse/HDDS-4285 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Attachments: image-2020-09-28-16-19-17-581.png, > profile-2020092

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-28 Thread GitBox
amaliujia commented on pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#issuecomment-700130504 @cxorm can you take another look please? Thanks! This is an automated message from the Apache Git

[jira] [Commented] (HDDS-4285) Read is slow due to the frequent usage of UGI.getCurrentUserCall()

2020-09-28 Thread Yiqun Lin (Jira)
URL: https://issues.apache.org/jira/browse/HDDS-4285 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Attachments: image-2020-09-28-16-1

[jira] [Updated] (HDDS-3698) Ozone Non-Rolling upgrades

2020-09-28 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3698: Status: Patch Available (was: Open) > Ozone Non-Rolling upgrades >

[jira] [Updated] (HDDS-3698) Ozone Non-Rolling upgrades

2020-09-28 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3698: Status: In Progress (was: Patch Available) > Ozone Non-Rolling upgrades >

[jira] [Updated] (HDDS-4227) Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.

2020-09-28 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4227: Status: Patch Available (was: In Progress) > Implement a "prepareForUpgrade" step that

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-28 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r495777131 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -334,15 +329,11 @@ public boolean

[GitHub] [hadoop-ozone] elek commented on pull request #1419: HDDS-3755. [DESIGN] Storage-class for Ozone

2020-09-28 Thread GitBox
elek commented on pull request #1419: URL: https://github.com/apache/hadoop-ozone/pull/1419#issuecomment-699905781 >> Isn't this design doc a few steps away from coding? A ton of detail is missing around how SCM will manage multiple classes of pipelines and how replication manager will

[jira] [Commented] (HDDS-4283) Remove unsupported upgrade command in ozone cli

2020-09-28 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203077#comment-17203077 ] Attila Doroszlai commented on HDDS-4283: I think this was already removed in HDDS-3992. > Remove

[jira] [Updated] (HDDS-4231) Background Service blocks on task results

2020-09-28 Thread Nanda kumar (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-4231: -- Status: Patch Available (was: Open) > Background Service blocks on task results >

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1432: HDDS-4252. Add the current layout versions to DN - SCM proto payload.

2020-09-28 Thread GitBox
avijayanhwx commented on pull request #1432: URL: https://github.com/apache/hadoop-ozone/pull/1432#issuecomment-700205929 Since failure is unrelated, and this work is going on in a branch, I am merging these changes. Thanks for the review @linyiqun.

[GitHub] [hadoop-ozone] avijayanhwx merged pull request #1432: HDDS-4252. Add the current layout versions to DN - SCM proto payload.

2020-09-28 Thread GitBox
avijayanhwx merged pull request #1432: URL: https://github.com/apache/hadoop-ozone/pull/1432 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[jira] [Updated] (HDDS-4252) Add the current layout versions to DN - SCM proto payload.

2020-09-28 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4252: Fix Version/s: 1.1.0 > Add the current layout versions to DN - SCM proto payload. >

[jira] [Resolved] (HDDS-4252) Add the current layout versions to DN - SCM proto payload.

2020-09-28 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4252. - Resolution: Fixed PR Merged. [~ppogde] Can you add some description to this JIRA? >

[GitHub] [hadoop-ozone] elek merged pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.

2020-09-28 Thread GitBox
elek merged pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hadoop-ozone] nandakumar131 commented on pull request #1414: HDDS-4231. Background Service blocks on task results.

2020-09-28 Thread GitBox
nandakumar131 commented on pull request #1414: URL: https://github.com/apache/hadoop-ozone/pull/1414#issuecomment-700200428 Overall the patch looks good to me. A minor suggestion, feel free to ignore it. Instead of having a generic reference in `BackgroundTaskQueue` (which

[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request #1378: HDDS-4133. Use new ContainerManager in SCM.

2020-09-28 Thread GitBox
nandakumar131 commented on a change in pull request #1378: URL: https://github.com/apache/hadoop-ozone/pull/1378#discussion_r496151525 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java ## @@ -242,19 +286,82 @@

[jira] [Updated] (HDDS-4102) Normalize Keypath for lookupKey

2020-09-28 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek updated HDDS-4102: -- Target Version/s: 1.1.0 Resolution: Fixed Status: Resolved (was: Patch

[jira] [Updated] (HDDS-3810) Add the logic to distribute open containers among the pipelines of a datanode

2020-09-28 Thread Nanda kumar (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-3810: -- Status: Patch Available (was: Open) > Add the logic to distribute open containers among the pipelines

[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-09-28 Thread GitBox
nandakumar131 commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r496111822 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java ## @@ -121,6 +143,19 @@ public int

[jira] [Updated] (HDDS-4182) Onboard HDDS-3869 into Layout version management

2020-09-28 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4182: Description: In HDDS-3869 (Use different column families for datanode block and metadata),

[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request #1378: HDDS-4133. Use new ContainerManager in SCM.

2020-09-28 Thread GitBox
nandakumar131 commented on a change in pull request #1378: URL: https://github.com/apache/hadoop-ozone/pull/1378#discussion_r496149425 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerID.java ## @@ -41,8 +41,8 @@ * @param id int

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1298: HDDS-3869. Use different column families for datanode block and metadata

2020-09-28 Thread GitBox
hanishakoneru commented on pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298#issuecomment-700258852 +1 pending CI. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1404: HDDS-2949: mkdir : store directory entries in a separate table

2020-09-28 Thread GitBox
bharatviswa504 commented on a change in pull request #1404: URL: https://github.com/apache/hadoop-ozone/pull/1404#discussion_r496246400 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequestV1.java ## @@ -0,0

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1404: HDDS-2949: mkdir : store directory entries in a separate table

2020-09-28 Thread GitBox
bharatviswa504 commented on a change in pull request #1404: URL: https://github.com/apache/hadoop-ozone/pull/1404#discussion_r496145730 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java ## @@ -129,6 +134,131 @@

[GitHub] [hadoop-ozone] fapifta commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-28 Thread GitBox
fapifta commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-700347698 Hi @avijayanhwx sorry for the long silence, the changes look good to me with the follow up items, +1.

[jira] [Created] (HDDS-4286) Implement one-by-one LayoutFeature finalization in OMUpgradeFinalizer

2020-09-28 Thread Jira
István Fajth created HDDS-4286: -- Summary: Implement one-by-one LayoutFeature finalization in OMUpgradeFinalizer Key: HDDS-4286 URL: https://issues.apache.org/jira/browse/HDDS-4286 Project: Hadoop

[GitHub] [hadoop-ozone] swagle commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-28 Thread GitBox
swagle commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r496341141 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisUpgradeUtils.java ## @@ -0,0 +1,96 @@ +/** + * Licensed to the

[GitHub] [hadoop-ozone] ChenSammi merged pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-28 Thread GitBox
ChenSammi merged pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-28 Thread GitBox
ChenSammi commented on pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#issuecomment-700401426 LGTM +1. This is an automated message from the Apache Git Service. To respond to the message, please

[jira] [Created] (HDDS-4287) Exclude protobuff classes from ozone-filesystem-hadoop3 jars

2020-09-28 Thread Uma Maheswara Rao G (Jira)
Uma Maheswara Rao G created HDDS-4287: - Summary: Exclude protobuff classes from ozone-filesystem-hadoop3 jars Key: HDDS-4287 URL: https://issues.apache.org/jira/browse/HDDS-4287 Project: Hadoop

[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-09-28 Thread GitBox
GlenGeng commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r496348325 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java ## @@ -40,6 +43,8 @@ private long

[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-09-28 Thread GitBox
GlenGeng commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r496361400 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -101,12 +103,23 @@ public

[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1451: Hdds 4117

2020-09-28 Thread GitBox
bharatviswa504 opened a new pull request #1451: URL: https://github.com/apache/hadoop-ozone/pull/1451 ## What changes were proposed in this pull request? Normalize Keypath for listKeys. When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the Keyname in the OM DB

[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-09-28 Thread GitBox
GlenGeng commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r496361400 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -101,12 +103,23 @@ public

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-28 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r496366104 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisUpgradeUtils.java ## @@ -0,0 +1,96 @@ +/** + * Licensed to the

[GitHub] [hadoop-ozone] bshashikant edited a comment on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-28 Thread GitBox
bshashikant edited a comment on pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700435789 I feel the CLI should be common for both OM and SCM and probably extended to Datanodes as well.

[GitHub] [hadoop-ozone] bshashikant commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-28 Thread GitBox
bshashikant commented on pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700435789 The CLI should be generic for both OM and SCM and probably extended to Datanodes as well. This is an

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-28 Thread GitBox
amaliujia commented on pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700441836 Re @bshashikant Agreed. Right now the command itself is unified (for both OM and SCM, we name this command as `roles`). Then we should unify the behavior fo both

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-28 Thread GitBox
ChenSammi commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-700409940 > > > This change looks almost good now. I wonder about two final things: > > 1. In `updateContainerStats(...)` do you think we should return if the

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-09-28 Thread GitBox
bshashikant commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r496394076 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -101,12 +103,23 @@

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-09-28 Thread GitBox
bshashikant commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r496394076 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -101,12 +103,23 @@