[GitHub] [hadoop-ozone] captainzmc commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-23 Thread GitBox
captainzmc commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-698132943 Hi all, status updates: rebase PR and resolve conflicts. This PR can be reviewed again. This is an aut

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-23 Thread GitBox
amaliujia commented on pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#issuecomment-698104826 R: @cxorm @ChenSammi @captainzmc Can you take a look please? This is an automated message from th

[jira] [Updated] (HDDS-4272) Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket

2020-09-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4272: - Labels: pull-request-available (was: ) > Volume namespace: add namespaceQuotaUsage and update it

[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-23 Thread GitBox
amaliujia opened a new pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445 ## What changes were proposed in this pull request? Volume namespace: add usedNamespace and update it when create and delete bucket ## What is the link to the Apache JIRA ht

[GitHub] [hadoop-ozone] cxorm commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
cxorm commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-698097298 The work looks great to me. A little question is about the title of the issue, I think the title is too general, could we change it to address the fix of this PR ?

[GitHub] [hadoop-ozone] cxorm commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
cxorm commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-698096286 >In a larger perspective, would it make sense to enhance checkstyle rules to warn for this kind of spacing? Yeah, thanks for the feedback, we could discuss about the id

[GitHub] [hadoop-ozone] cxorm commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-23 Thread GitBox
cxorm commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-698094498 Thanks @captainzmc for the work. Would you please rebase this PR ? I would take a look on it. This

[jira] [Updated] (HDDS-3725) Ozone sh volume client support quota option.

2020-09-23 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3725: - Fix Version/s: 1.1.0 > Ozone sh volume client support quota option. >

[jira] [Assigned] (HDDS-4272) Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket

2020-09-23 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-4272: -- Assignee: Rui Wang > Volume namespace: add namespaceQuotaUsage and update it when create and > delete

[jira] [Created] (HDDS-4272) Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket

2020-09-23 Thread Rui Wang (Jira)
Rui Wang created HDDS-4272: -- Summary: Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket Key: HDDS-4272 URL: https://issues.apache.org/jira/browse/HDDS-4272 Project: Hadoop

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia commented on a change in pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444#discussion_r493976795 ## File path: hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/helpers/OzoneAclStorage.java ## @@ -0,0 +1,63 @@ +/** + * Li

[GitHub] [hadoop-ozone] amaliujia edited a comment on pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia edited a comment on pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444#issuecomment-697997438 R @elek Only migrating `PrefixInfo` because I want to see whether my current approach makes sense (e.g. move some classes to `interface-storage` meanwhile

[jira] [Updated] (HDDS-4242) Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4242: - Labels: pull-request-available (was: ) > Copy PrefixInfo proto to new project hadoop-ozone/interf

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia commented on pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444#issuecomment-697997438 R @elek This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia opened a new pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444 ## What changes were proposed in this pull request? As a following effort, try to move `PrefixInfo` and other necessary classes to `interface-stroage` ## What is the link to the A

[jira] [Updated] (HDDS-4242) Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated HDDS-4242: --- Summary: Copy PrefixInfo proto to new project hadoop-ozone/interface-storage (was: Copy storage proto to new

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1298: HDDS-3869. Use different column families for datanode block and metadata

2020-09-23 Thread GitBox
hanishakoneru commented on pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298#issuecomment-697956338 @errose28, great work on this Jira. +1 for merge pending CI. This is an automated message from the

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread GitBox
adoroszlai commented on a change in pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443#discussion_r493694568 ## File path: dev-support/byteman/ratis-flush.btm ## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# con

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1424: HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia commented on pull request #1424: URL: https://github.com/apache/hadoop-ozone/pull/1424#issuecomment-697702757 @elek thank you for your review! This is an automated message from the Apache Git Service. To respond to

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-23 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r493728718 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java ## @@ -246,4 +246,9 @@ private OMConfigKeys() {

[GitHub] [hadoop-ozone] elek commented on pull request #1319: HDDS-4107: replace scmID with clusterID for container and volume at Datanode side

2020-09-23 Thread GitBox
elek commented on pull request #1319: URL: https://github.com/apache/hadoop-ozone/pull/1319#issuecomment-697593857 (Sorry, late answer) Why do we need to updated container files? I see no problems to keep (the valid) clusterId in the container metadata files. Or do you need t

[GitHub] [hadoop-ozone] elek commented on pull request #1083: HDDS-3814. Drop a column family through debug cli tool

2020-09-23 Thread GitBox
elek commented on pull request #1083: URL: https://github.com/apache/hadoop-ozone/pull/1083#issuecomment-697590296 @bharatviswa504 @avijayanhwx Are you fine with this approach? Command won't be visible unless you put a magic file to the right place (META-INF/services). It's a hidden

[jira] [Resolved] (HDDS-4234) Add important comment to ListVolumes logic

2020-09-23 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-4234. --- Fix Version/s: (was: 1.0.0) 1.1.0 Target Version/s: 1.1.0 (was: 0.5.0

[GitHub] [hadoop-ozone] elek merged pull request #1417: HDDS-4324. Add important comment to ListVolumes logic

2020-09-23 Thread GitBox
elek merged pull request #1417: URL: https://github.com/apache/hadoop-ozone/pull/1417 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[jira] [Resolved] (HDDS-4236) Move "Om*Codec.java" to new project hadoop-ozone/interface-storage

2020-09-23 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-4236. --- Fix Version/s: 1.1.0 Resolution: Fixed > Move "Om*Codec.java" to new project hadoop-ozone/inter

[GitHub] [hadoop-ozone] elek merged pull request #1424: HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
elek merged pull request #1424: URL: https://github.com/apache/hadoop-ozone/pull/1424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-23 Thread GitBox
ChenSammi commented on pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-697328180 LGTM, +1. This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [hadoop-ozone] ChenSammi merged pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-23 Thread GitBox
ChenSammi merged pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hadoop-ozone] adoroszlai edited a comment on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-23 Thread GitBox
adoroszlai edited a comment on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-697321399 > > I think we should avoid logging chunk content. ContainerCommandRequestMessage implements related logic to clear data. > > I checked but it's not somethi

[jira] [Created] (HDDS-4271) Avoid logging chunk content in Ozone Insight

2020-09-23 Thread Attila Doroszlai (Jira)
Attila Doroszlai created HDDS-4271: -- Summary: Avoid logging chunk content in Ozone Insight Key: HDDS-4271 URL: https://issues.apache.org/jira/browse/HDDS-4271 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-23 Thread GitBox
adoroszlai commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-697321399 > > I think we should avoid logging chunk content. ContainerCommandRequestMessage implements related logic to clear data. > > I checked but it's not something whic

[jira] [Updated] (HDDS-4270) Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4270: - Labels: pull-request-available (was: ) > Add more reusable byteman scripts to debug ofs/o3fs perf

[GitHub] [hadoop-ozone] elek opened a new pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread GitBox
elek opened a new pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443 I am using https://byteman.jboss.org to debug the performance of spark + teragen with different scripts. Some byteman scripts are already shared by HDDS-4095 or HDDS-342 but it seems to be a goo

[jira] [Updated] (HDDS-4270) Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek updated HDDS-4270: -- Description: I am using https://byteman.jboss.org to debug the performance of spark + terage with diffe

[jira] [Created] (HDDS-4270) Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread Marton Elek (Jira)
Marton Elek created HDDS-4270: - Summary: Add more reusable byteman scripts to debug ofs/o3fs performance Key: HDDS-4270 URL: https://issues.apache.org/jira/browse/HDDS-4270 Project: Hadoop Distributed Dat

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1383: HDDS-4194. Create a script to check AWS S3 compatibility

2020-09-23 Thread GitBox
elek commented on a change in pull request #1383: URL: https://github.com/apache/hadoop-ozone/pull/1383#discussion_r493498810 ## File path: hadoop-ozone/dist/src/main/smoketest/s3/s3_compatbility_check.sh ## @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Licensed to the Apache Softwa

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1383: HDDS-4194. Create a script to check AWS S3 compatibility

2020-09-23 Thread GitBox
elek commented on a change in pull request #1383: URL: https://github.com/apache/hadoop-ozone/pull/1383#discussion_r493495525 ## File path: hadoop-ozone/dist/src/main/smoketest/s3/s3_compatbility_check.sh ## @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Licensed to the Apache Softwa

[GitHub] [hadoop-ozone] elek commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-23 Thread GitBox
elek commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-697312216 > I think we should avoid logging chunk content. ContainerCommandRequestMessage implements related logic to clear data. I checked but it's not something which can be add

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-23 Thread GitBox
linyiqun commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r493463257 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java ## @@ -246,4 +246,9 @@ private OMConfigKeys() {

[GitHub] [hadoop-ozone] cku328 removed a comment on pull request #1368: HDDS-4156. add hierarchical layout to Chinese doc

2020-09-23 Thread GitBox
cku328 removed a comment on pull request #1368: URL: https://github.com/apache/hadoop-ozone/pull/1368#issuecomment-697277442 Overall LGTM. Thanks @flirmnave for working on this. This PR will make the Chinese docs more friendly. ---

[GitHub] [hadoop-ozone] cku328 commented on pull request #1368: HDDS-4156. add hierarchical layout to Chinese doc

2020-09-23 Thread GitBox
cku328 commented on pull request #1368: URL: https://github.com/apache/hadoop-ozone/pull/1368#issuecomment-697277442 Overall LGTM. Thanks @flirmnave for working on this. This PR will make the Chinese docs more friendly. Th

[GitHub] [hadoop-ozone] llemec commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
llemec commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-697250771 Hello @cxorm, Thank you for your review. I corrected the coding style as suggested. In a larger perspective, would it make sense to enhance checkstyle rules to warn fo

[GitHub] [hadoop-ozone] llemec commented on a change in pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
llemec commented on a change in pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#discussion_r493369935 ## File path: hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestInstanceHelper.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to t

[GitHub] [hadoop-ozone] llemec commented on a change in pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
llemec commented on a change in pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#discussion_r493369544 ## File path: hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestInstanceHelper.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to t

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1438: HDDS-4263. ReplicatiomManager shouldn't retain one healthy replica per origin node Id.

2020-09-23 Thread GitBox
maobaolong commented on pull request #1438: URL: https://github.com/apache/hadoop-ozone/pull/1438#issuecomment-697206664 Hi @nandakumar131 , thanks for the first review and suggestion, I agree to you that we should not remove the OriginDatanodeId handling blindly, now I add a condition for

[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
cxorm commented on a change in pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#discussion_r493277765 ## File path: hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmPrefixInfo.java ## @@ -18,39 +18,91 @@ package org.apache.

[jira] [Resolved] (HDDS-2766) security/SecuringDataNodes.md

2020-09-23 Thread YiSheng Lien (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien resolved HDDS-2766. Resolution: Fixed > security/SecuringDataNodes.md > - > >

[GitHub] [hadoop-ozone] cxorm merged pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-09-23 Thread GitBox
cxorm merged pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hadoop-ozone] cxorm commented on pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-09-23 Thread GitBox
cxorm commented on pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#issuecomment-697178787 Thanks @iamabug for the contribution. Thanks @xiaoyuyao and @smengcl for reviewing this PR, and thanks @adoroszlai for adding the label. -