[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #238: HDDS-2588. Consolidate compose environments
adoroszlai commented on a change in pull request #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238#discussion_r348926276 ## File path: hadoop-ozone/dist/src/main/compose/ozone/docker-config ## @@ -20,14 +20,22 @@ OZONE-SITE.XML_ozone.scm.names=scm OZONE-SITE.XML_ozone.scm.datanode.id.dir=/data OZONE-SITE.XML_ozone.scm.block.client.address=scm OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata -OZONE-SITE.XML_ozone.handler.type=distributed +OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon +OZONE-SITE.XML_ozone.recon.om.db.dir=/data/metadata/recon Review comment: Thanks @avijayanhwx. Then I think we can keep it as is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #238: HDDS-2588. Consolidate compose environments
avijayanhwx commented on a change in pull request #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238#discussion_r348925830 ## File path: hadoop-ozone/dist/src/main/compose/ozone/docker-config ## @@ -20,14 +20,22 @@ OZONE-SITE.XML_ozone.scm.names=scm OZONE-SITE.XML_ozone.scm.datanode.id.dir=/data OZONE-SITE.XML_ozone.scm.block.client.address=scm OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata -OZONE-SITE.XML_ozone.handler.type=distributed +OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon +OZONE-SITE.XML_ozone.recon.om.db.dir=/data/metadata/recon Review comment: @adoroszlai Yes the same directory can be used. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #238: HDDS-2588. Consolidate compose environments
adoroszlai commented on issue #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238#issuecomment-556954298 Thanks @bharatviswa504 for the review. > In a similar way, we can merge ozone-om-ha and ozone-om-ha-s3 also. Unfortunately cannot do that, since `ozone-om-ha` is very different (custom docker image with SSH, starts/stops OM manually, etc.). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #238: HDDS-2588. Consolidate compose environments
adoroszlai commented on a change in pull request #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238#discussion_r348924764 ## File path: hadoop-ozone/dist/src/main/compose/ozone/docker-config ## @@ -20,14 +20,22 @@ OZONE-SITE.XML_ozone.scm.names=scm OZONE-SITE.XML_ozone.scm.datanode.id.dir=/data OZONE-SITE.XML_ozone.scm.block.client.address=scm OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata -OZONE-SITE.XML_ozone.handler.type=distributed +OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon +OZONE-SITE.XML_ozone.recon.om.db.dir=/data/metadata/recon Review comment: Thanks @bharatviswa504 for spotting this. These config values from `ozone-recon` env as they were. @avijayanhwx @swagle can Recon use the same directory for both `ozone.recon.db.dir` and `ozone.recon.om.db.dir`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on issue #164: HDDS-426. Add field modificationTime for Volume and Bucket
cxorm commented on issue #164: HDDS-426. Add field modificationTime for Volume and Bucket URL: https://github.com/apache/hadoop-ozone/pull/164#issuecomment-556942847 > +1 LGTM. @anuengineer / @arp7 request your review as well. Thanks @dineshchitlangia for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #245: Update main.yml to skip unit tests
vivekratnavel opened a new pull request #245: Update main.yml to skip unit tests URL: https://github.com/apache/hadoop-ozone/pull/245 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #245: Update main.yml to skip unit tests
bharatviswa504 merged pull request #245: Update main.yml to skip unit tests URL: https://github.com/apache/hadoop-ozone/pull/245 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #137: HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader
smengcl commented on a change in pull request #137: HDDS-2455. Implement
MiniOzoneHAClusterImpl#getOMLeader
URL: https://github.com/apache/hadoop-ozone/pull/137#discussion_r348906923
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java
##
@@ -118,6 +118,19 @@ public OzoneManager getOzoneManager(String omNodeId) {
return this.ozoneManagerMap.get(omNodeId);
}
+ /**
+ * Get OzoneManager leader object.
+ * @return OzoneManager object, could return null if there isn't a leader.
+ */
+ public OzoneManager getOMLeader() {
+for (OzoneManager ozoneManager : this.ozoneManagers) {
+ if (ozoneManager.isLeader()) {
+return ozoneManager;
+ }
Review comment:
Indeed this could be a problem. I talked to Arpit about this. It turns out
there could be two OMs asserting themselves are both leaders in a short while,
or there could be no leader. I will update this method to handle those two
cases to some extent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #137: HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader
smengcl commented on a change in pull request #137: HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader URL: https://github.com/apache/hadoop-ozone/pull/137#discussion_r348907346 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneHACluster.java ## @@ -0,0 +1,85 @@ +package org.apache.hadoop.ozone; Review comment: Will add the license header. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #244: Add comma separated list of compiled classes
bharatviswa504 merged pull request #244: Add comma separated list of compiled classes URL: https://github.com/apache/hadoop-ozone/pull/244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #244: Add comma separated list of compiled classes
vivekratnavel opened a new pull request #244: Add comma separated list of compiled classes URL: https://github.com/apache/hadoop-ozone/pull/244 ## What changes were proposed in this pull request? Added comma separated list of classes dirs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on issue #243: HDDS-2394. Ozone S3 Gateway allows bucket name with underscore to be created
vivekratnavel commented on issue #243: HDDS-2394. Ozone S3 Gateway allows bucket name with underscore to be created URL: https://github.com/apache/hadoop-ozone/pull/243#issuecomment-556919215 @bharatviswa504 Please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #243: HDDS-2394. Ozone S3 Gateway allows bucket name with underscore to be created
vivekratnavel opened a new pull request #243: HDDS-2394. Ozone S3 Gateway allows bucket name with underscore to be created URL: https://github.com/apache/hadoop-ozone/pull/243 ## What changes were proposed in this pull request? The patch adds verification of bucket name when bucket create request is handled from s3 api. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2394 ## How was this patch tested? The patch was tested by trying to create a bucket with an invalid bucket name in ozones3 docker compose env. Also, a robot testcase is added to catch create bucket request with invalid bucket name. I verified that this newly added test passes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on issue #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor.
mukul1987 commented on issue #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor. URL: https://github.com/apache/hadoop-ozone/pull/236#issuecomment-556916520 Thanks for the review @dineshchitlangia. I have addressed the review comments in the next patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor.
mukul1987 commented on a change in pull request #236: HDDS-2284.
XceiverClientMetrics should be initialised as part of XceiverClientManager
constructor.
URL: https://github.com/apache/hadoop-ozone/pull/236#discussion_r348893132
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
##
@@ -129,6 +129,11 @@ private String getKeyName() {
return UUID.randomUUID().toString();
}
+ private XceiverClientMetrics getXceiverClientMetrics() {
+RpcClient rpc = (RpcClient)client.getObjectStore().getClientProxy();
+return rpc.getXceiverClientManager().getMetrics();
+ }
+
Review comment:
Done in the next patch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #218: HDDS-2536. Add ozone.om.internal.service.id to OM HA configuration.
bharatviswa504 commented on issue #218: HDDS-2536. Add ozone.om.internal.service.id to OM HA configuration. URL: https://github.com/apache/hadoop-ozone/pull/218#issuecomment-556912882 Thank You @anuengineer and @dineshchitlangia for the review. To make it to new style java Config opened a new Jira(https://issues.apache.org/jira/browse/HDDS-2581) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #238: HDDS-2588. Consolidate compose environments
bharatviswa504 commented on issue #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238#issuecomment-556911695 Good idea, overall LGTM. One minor comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #237: HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline.
timmylicheng commented on a change in pull request #237: HDDS-1572 Implement a
Pipeline scrubber to clean up non-OPEN pipeline.
URL: https://github.com/apache/hadoop-ozone/pull/237#discussion_r348883139
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/BackgroundPipelineCreator.java
##
@@ -96,7 +96,7 @@ private void createPipelines() {
if (scheduler.isClosed()) {
break;
}
-
+// pipelineManager.scrubPipeline(type, factor);
Review comment:
Sorry for the typo. I meant to turn on scrubber here.
Resource wise, the background creation thread is triggered every 120s by
default. I think it's a good interval for scrubber to clean up as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] abhishekaypurohit commented on issue #239: HDDS-2596. Remove unused private method "createPipeline"
abhishekaypurohit commented on issue #239: HDDS-2596. Remove unused private method "createPipeline" URL: https://github.com/apache/hadoop-ozone/pull/239#issuecomment-556904810 Ignore this one, somehow git rebase merged. I'll create another PR for this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] abhishekaypurohit closed pull request #239: HDDS-2596. Remove unused private method "createPipeline"
abhishekaypurohit closed pull request #239: HDDS-2596. Remove unused private method "createPipeline" URL: https://github.com/apache/hadoop-ozone/pull/239 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer.
dineshchitlangia commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer. URL: https://github.com/apache/hadoop-ozone/pull/205#issuecomment-556886821 > Checks @adoroszlai Thanks for the pointer. I was able to trigger a re-run from the Checks page. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #239: HDDS-2596. Remove unused private method "createPipeline"
dineshchitlangia commented on issue #239: HDDS-2596. Remove unused private method "createPipeline" URL: https://github.com/apache/hadoop-ozone/pull/239#issuecomment-556878920 @abhishekaypurohit Could you please rebase your PR with master? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #242: HDDS-2594. S3 RangeReads failing with NumberFormatException.
dineshchitlangia merged pull request #242: HDDS-2594. S3 RangeReads failing with NumberFormatException. URL: https://github.com/apache/hadoop-ozone/pull/242 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #240: HDDS-2597. Remove toString() as log calls it implicitly
dineshchitlangia merged pull request #240: HDDS-2597. Remove toString() as log calls it implicitly URL: https://github.com/apache/hadoop-ozone/pull/240 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #241: HDDS-2598. Remove unused private field "LOG"
dineshchitlangia merged pull request #241: HDDS-2598. Remove unused private field "LOG" URL: https://github.com/apache/hadoop-ozone/pull/241 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #242: HDDS-2594. S3 RangeReads failing with NumberFormatException.
bharatviswa504 opened a new pull request #242: HDDS-2594. S3 RangeReads failing with NumberFormatException. URL: https://github.com/apache/hadoop-ozone/pull/242 ## What changes were proposed in this pull request? RangerHeaderParserUtil throws NumberFormatException, because of using Integer.parseInt. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2594 ## How was this patch tested? Added UT for this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] abhishekaypurohit opened a new pull request #241: HDDS-2598. Removed unused field
abhishekaypurohit opened a new pull request #241: HDDS-2598. Removed unused field URL: https://github.com/apache/hadoop-ozone/pull/241 ## What changes were proposed in this pull request? Removed unused filed ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2598 ## How was this patch tested? mvn builds This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] abhishekaypurohit opened a new pull request #240: HDDS-2597. Removed tostring as log calls it implicitly.
abhishekaypurohit opened a new pull request #240: HDDS-2597. Removed tostring as log calls it implicitly. URL: https://github.com/apache/hadoop-ozone/pull/240 ## What changes were proposed in this pull request? Removed unnecessary toString ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2597 ## How was this patch tested? mvn builds This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] abhishekaypurohit opened a new pull request #239: HDDS-2596. Removed unused private method
abhishekaypurohit opened a new pull request #239: HDDS-2596. Removed unused private method URL: https://github.com/apache/hadoop-ozone/pull/239 ## What changes were proposed in this pull request? Removed unused method ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2596 ## How was this patch tested? mvn build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #237: HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline.
xiaoyuyao commented on a change in pull request #237: HDDS-1572 Implement a
Pipeline scrubber to clean up non-OPEN pipeline.
URL: https://github.com/apache/hadoop-ozone/pull/237#discussion_r348797902
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/BackgroundPipelineCreator.java
##
@@ -96,7 +96,7 @@ private void createPipelines() {
if (scheduler.isClosed()) {
break;
}
-
+// pipelineManager.scrubPipeline(type, factor);
Review comment:
Another thought is to run scrubber in a separate executor task so that
scrubber operation won't affecting pipeline creator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #237: HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline.
xiaoyuyao commented on a change in pull request #237: HDDS-1572 Implement a
Pipeline scrubber to clean up non-OPEN pipeline.
URL: https://github.com/apache/hadoop-ozone/pull/237#discussion_r348796012
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/BackgroundPipelineCreator.java
##
@@ -96,7 +96,7 @@ private void createPipelines() {
if (scheduler.isClosed()) {
break;
}
-
+// pipelineManager.scrubPipeline(type, factor);
Review comment:
If this line is commented, the scrub will not be activated?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #159: HDDS-2477. TableCache cleanup issue for OM non-HA.
bharatviswa504 commented on issue #159: HDDS-2477. TableCache cleanup issue for
OM non-HA.
URL: https://github.com/apache/hadoop-ozone/pull/159#issuecomment-556527944
Tested out on 3 node cluster, with this patch, I was able to upload files
with larger sizes, which is reported in HDDS-2356 MPU for larger files is
failing with CompleteMultipartUpload error. I have tried upload of files up to
10GB, and able to upload successfully.
```
[root@bh-ozone-2 ozone-0.5.0-SNAPSHOT]# cp ./10g-1.file
/mount/mpunov19/new10g.file
[root@bh-ozone-2 ozone-0.5.0-SNAPSHOT]# ls -al /mount/mpunov19/
total 30408708
drwxr-xr-x 2 root root4096 Nov 20 14:49 .
drwxr-xr-x 4 root root 39 Nov 19 12:37 ..
-rw-r--r-- 1 root root 10737418240 Nov 19 13:40 10g-1.file
-rw-r--r-- 1 root root 1073741824 Nov 19 12:48 1g-1.file
-rw-r--r-- 1 root root 3221225472 Nov 19 13:29 3g-1.file
-rw-r--r-- 1 root root 5368709120 Nov 19 13:35 5g-1.file
-rw-r--r-- 1 root root 10737418240 Nov 20 14:49 new10g.file
Get the object again, and verified data is correct or not.
[root@bh-ozone-2 ozone-0.5.0-SNAPSHOT]# ozones3api get-object --bucket
mpunov19 --key new10g.file ./ozonenew10
{
"AcceptRanges": "bytes",
"ContentType": "application/octet-stream",
"LastModified": "Wed, 20 Nov 2019 22:49:35 GMT",
"ContentLength": 10737418240,
"Expires": "Wed, 20 Nov 2019 22:53:44 GMT",
"CacheControl": "no-cache",
"Metadata": {}
}
[root@bh-ozone-2 ozone-0.5.0-SNAPSHOT]# diff 10g-1.file ozonenew10
[root@bh-ozone-2 ozone-0.5.0-SNAPSHOT]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #233: HDDS-1873 Add API to get last completed times for every Recon task.
avijayanhwx commented on a change in pull request #233: HDDS-1873 Add API to get last completed times for every Recon task. URL: https://github.com/apache/hadoop-ozone/pull/233#discussion_r348792074 ## File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java ## @@ -167,6 +167,30 @@ public OzoneManagerServiceProviderImpl( this.reconTaskStatusDao = reconTaskController.getReconTaskStatusDao(); this.ozoneManagerClient = ozoneManagerClient; this.configuration = configuration; + +registerOMDBTasks(); Review comment: There is a start() method where this can be done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #233: HDDS-1873 Add API to get last completed times for every Recon task.
swagle commented on a change in pull request #233: HDDS-1873 Add API to get last completed times for every Recon task. URL: https://github.com/apache/hadoop-ozone/pull/233#discussion_r348791800 ## File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java ## @@ -167,6 +167,30 @@ public OzoneManagerServiceProviderImpl( this.reconTaskStatusDao = reconTaskController.getReconTaskStatusDao(); this.ozoneManagerClient = ozoneManagerClient; this.configuration = configuration; + +registerOMDBTasks(); Review comment: Can we avoid calling this from a constructor? Is there a lifecycle method which can do the same thing? @avijayanhwx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #233: HDDS-1873 Add API to get last completed times for every Recon task.
swagle commented on a change in pull request #233: HDDS-1873 Add API to get
last completed times for every Recon task.
URL: https://github.com/apache/hadoop-ozone/pull/233#discussion_r348790981
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/TaskTimesService.java
##
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.MediaType;
+import java.util.List;
+
+/**
+ * Endpoint for displaying the last successful run of each Recon Task
+ */
+@Path("/task")
+@Produces(MediaType.APPLICATION_JSON)
+public class TaskTimesService {
Review comment:
Name is too generic, lets rename to TasKStatusService.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #233: HDDS-1873 Add API to get last completed times for every Recon task.
vivekratnavel commented on a change in pull request #233: HDDS-1873 Add API to
get last completed times for every Recon task.
URL: https://github.com/apache/hadoop-ozone/pull/233#discussion_r348774583
##
File path:
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/TaskTimesService.java
##
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.MediaType;
+import java.util.List;
+
+/**
+ * Endpoint for displaying the last successful run of each Recon Task
+ */
+@Path("/task")
+@Produces(MediaType.APPLICATION_JSON)
+public class TaskTimesService {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TaskTimesService.class);
+
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ @Inject
+ private Configuration sqlConfiguration;
+
+ ReconTaskStatusDao getDao() {
+return reconTaskStatusDao == null ? new
+ReconTaskStatusDao(sqlConfiguration) : reconTaskStatusDao;
+ }
+
+ /**
+ * Return the list of Recon Tasks and the last successful timestamp and
+ * sequence number.
+ * @return {@link Response}
+ */
+ @GET
+ @Path("taskStatus")
Review comment:
Can we change this path to just "status"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #207: HDDS-2522. Fix TestSecureOzoneCluster
adoroszlai commented on issue #207: HDDS-2522. Fix TestSecureOzoneCluster URL: https://github.com/apache/hadoop-ozone/pull/207#issuecomment-556393613 Thanks @xiaoyuyao and @dineshchitlangia for reviewing this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer.
adoroszlai commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer. URL: https://github.com/apache/hadoop-ozone/pull/205#issuecomment-556392618 @dineshchitlangia `/retest` doesn't work with the Github Actions CI. It should be possible to re-run somewhere on the [_Checks_](https://github.com/apache/hadoop-ozone/pull/205/checks) page here (but not sure for whom is the button/link visible -- not for me). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #207: HDDS-2522. Fix TestSecureOzoneCluster
dineshchitlangia merged pull request #207: HDDS-2522. Fix TestSecureOzoneCluster URL: https://github.com/apache/hadoop-ozone/pull/207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #232: HDDS-2523. BufferPool.releaseBuffer may release a buffer different than the head of the list
dineshchitlangia commented on issue #232: HDDS-2523. BufferPool.releaseBuffer may release a buffer different than the head of the list URL: https://github.com/apache/hadoop-ozone/pull/232#issuecomment-556375047 @adoroszlai Thanks for the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer.
dineshchitlangia commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer. URL: https://github.com/apache/hadoop-ozone/pull/205#issuecomment-556375808 Triggered tests again as HDDS-2523 is resolved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer.
dineshchitlangia commented on issue #205: HDDS-2386. Implement incremental ChunkBuffer. URL: https://github.com/apache/hadoop-ozone/pull/205#issuecomment-556375440 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #232: HDDS-2523. BufferPool.releaseBuffer may release a buffer different than the head of the list
dineshchitlangia merged pull request #232: HDDS-2523. BufferPool.releaseBuffer may release a buffer different than the head of the list URL: https://github.com/apache/hadoop-ozone/pull/232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #238: HDDS-2588. Consolidate compose environments
adoroszlai opened a new pull request #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238 ## What changes were proposed in this pull request? There are a few slightly different sample docker compose environments: `ozone`, `ozoneperf`, `ozones3`, `ozone-recon`. This change proposes to merge these 4 by minor additions to `ozoneperf`: 1. add `recon` service from `ozone-recon` 2. run GDPR and S3 tests 3. expose datanode web port (eg. for profiling) Plus: also run `ozone-shell` test (from `basic` suite), which is currently run only in `ozonesecure` I also propose to rename `ozoneperf` to `ozone` for simplicity. Consolidating these 4 environments would slightly reduce both code duplication and the time needed for acceptance tests. https://issues.apache.org/jira/browse/HDDS-2588 ## How was this patch tested? Ran acceptance test in `ozone` dir. Generated keys using freon, verified that Jaeger, Prometheus, Grafana reflect the operations. Clean CI in private branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #194: HDDS-2241. Optimize the refresh pipeline logic used by KeyManagerImpl…
bharatviswa504 commented on issue #194: HDDS-2241. Optimize the refresh pipeline logic used by KeyManagerImpl… URL: https://github.com/apache/hadoop-ozone/pull/194#issuecomment-556275911 Thank You @avijayanhwx for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #194: HDDS-2241. Optimize the refresh pipeline logic used by KeyManagerImpl…
bharatviswa504 merged pull request #194: HDDS-2241. Optimize the refresh pipeline logic used by KeyManagerImpl… URL: https://github.com/apache/hadoop-ozone/pull/194 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] anuengineer merged pull request #193: HDDS-2512 Sonar TraceAllMethod NPE Could be Thrown
anuengineer merged pull request #193: HDDS-2512 Sonar TraceAllMethod NPE Could be Thrown URL: https://github.com/apache/hadoop-ozone/pull/193 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #221: HDDS-2538. Fix issues found in DatabaseHelper.
swagle commented on a change in pull request #221: HDDS-2538. Fix issues found
in DatabaseHelper.
URL: https://github.com/apache/hadoop-ozone/pull/221#discussion_r348673936
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/common/DatabaseHelper.java
##
@@ -213,27 +223,24 @@ public static String executeTemplate(String dbName,
String template)
private static String executeStatement(String dbName, String sql)
throws SQLException {
StringBuilder result = new StringBuilder();
-ResultSet rs = null;
-Statement st = null;
-ResultSetMetaData rsm = null;
-try(Connection connection = getConnection(dbName)) {
- //loadProperties();
-
- if(connection != null){
-st = connection.createStatement();
-rs = st.executeQuery(sql);
-if(rs != null) {
- rsm = rs.getMetaData();
- int cols = rsm.getColumnCount();
- while(rs.next()){
-for(int index =1; index<=cols; index++){
- result.append(rs.getObject(index) + "\t");
+ResultSetMetaData rsm;
+try (Connection connection = getConnection(dbName)) {
+ if (connection != null){
+try (Statement st = connection.createStatement()) {
+ try (ResultSet rs = st.executeQuery(sql)) {
Review comment:
Ahh, ok I see what you mean, nm!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #221: HDDS-2538. Fix issues found in DatabaseHelper.
swagle commented on a change in pull request #221: HDDS-2538. Fix issues found
in DatabaseHelper.
URL: https://github.com/apache/hadoop-ozone/pull/221#discussion_r348672871
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/common/DatabaseHelper.java
##
@@ -213,27 +223,24 @@ public static String executeTemplate(String dbName,
String template)
private static String executeStatement(String dbName, String sql)
throws SQLException {
StringBuilder result = new StringBuilder();
-ResultSet rs = null;
-Statement st = null;
-ResultSetMetaData rsm = null;
-try(Connection connection = getConnection(dbName)) {
- //loadProperties();
-
- if(connection != null){
-st = connection.createStatement();
-rs = st.executeQuery(sql);
-if(rs != null) {
- rsm = rs.getMetaData();
- int cols = rsm.getColumnCount();
- while(rs.next()){
-for(int index =1; index<=cols; index++){
- result.append(rs.getObject(index) + "\t");
+ResultSetMetaData rsm;
+try (Connection connection = getConnection(dbName)) {
+ if (connection != null){
+try (Statement st = connection.createStatement()) {
+ try (ResultSet rs = st.executeQuery(sql)) {
Review comment:
But that would result in Sonarlint warning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #221: HDDS-2538. Fix issues found in DatabaseHelper.
swagle commented on a change in pull request #221: HDDS-2538. Fix issues found
in DatabaseHelper.
URL: https://github.com/apache/hadoop-ozone/pull/221#discussion_r348672593
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/audit/parser/common/DatabaseHelper.java
##
@@ -90,50 +100,54 @@ private static void loadProperties() {
}
private static boolean createAuditTable(String dbName) {
-
-try(Connection connection = getConnection(dbName);
-Statement st = connection.createStatement()) {
-
- st.executeUpdate(properties.get(ParserConsts.CREATE_AUDIT_TABLE));
+try (Connection connection = getConnection(dbName)) {
+ if (connection != null) {
+try (Statement st = connection.createStatement()) {
+ st.executeUpdate(properties.get(ParserConsts.CREATE_AUDIT_TABLE));
+}
+ }
} catch (SQLException e) {
LOG.error(e.getMessage());
return false;
}
return true;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #178: HDDS-2498. Fix sonar issues found in StorageContainerManager.
dineshchitlangia merged pull request #178: HDDS-2498. Fix sonar issues found in StorageContainerManager. URL: https://github.com/apache/hadoop-ozone/pull/178 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai edited a comment on issue #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler
adoroszlai edited a comment on issue #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler URL: https://github.com/apache/hadoop-ozone/pull/235#issuecomment-556069110 Thanks @dineshchitlangia for reviewing and merging this. Also for filing the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler
adoroszlai commented on issue #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler URL: https://github.com/apache/hadoop-ozone/pull/235#issuecomment-556069110 Thanks @dineshchitlangia for reviewing and merging this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler
dineshchitlangia merged pull request #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler URL: https://github.com/apache/hadoop-ozone/pull/235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler
dineshchitlangia commented on a change in pull request #235: HDDS-2580. Ensure
resources are closed in Get/PutKeyHandler
URL: https://github.com/apache/hadoop-ozone/pull/235#discussion_r348560248
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/GetKeyHandler.java
##
@@ -65,8 +65,10 @@ public Void call() throws Exception {
OzoneAddress address = new OzoneAddress(uri);
address.ensureKeyAddress();
-try (OzoneClient client =
- address.createClient(createOzoneConfiguration())) {
+
+OzoneConfiguration conf = createOzoneConfiguration();
+
+try (OzoneClient client = address.createClient(conf)) {
Review comment:
Ah! I see it now. I didn't have my coffee yet
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor. Contributed by M
dineshchitlangia commented on a change in pull request #236: HDDS-2284.
XceiverClientMetrics should be initialised as part of XceiverClientManager
constructor. Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop-ozone/pull/236#discussion_r348551260
##
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
##
@@ -212,7 +212,7 @@ public synchronized void close() {
for (ManagedChannel channel : channels.values()) {
channel.shutdownNow();
try {
-channel.awaitTermination(60, TimeUnit.MINUTES);
+channel.awaitTermination(5, TimeUnit.SECONDS);
Review comment:
Great catch!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor. Contributed by M
dineshchitlangia commented on a change in pull request #236: HDDS-2284.
XceiverClientMetrics should be initialised as part of XceiverClientManager
constructor. Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop-ozone/pull/236#discussion_r34887
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
##
@@ -129,6 +129,11 @@ private String getKeyName() {
return UUID.randomUUID().toString();
}
+ private XceiverClientMetrics getXceiverClientMetrics() {
+RpcClient rpc = (RpcClient)client.getObjectStore().getClientProxy();
+return rpc.getXceiverClientManager().getMetrics();
+ }
+
Review comment:
Wondering if we could move this method so that we don't have to duplicate
code in 4 test classes here.
Not a deal breaker but I am just tracking Sonar code duplication stats :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler
adoroszlai commented on a change in pull request #235: HDDS-2580. Ensure
resources are closed in Get/PutKeyHandler
URL: https://github.com/apache/hadoop-ozone/pull/235#discussion_r348548474
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/GetKeyHandler.java
##
@@ -65,8 +65,10 @@ public Void call() throws Exception {
OzoneAddress address = new OzoneAddress(uri);
address.ensureKeyAddress();
-try (OzoneClient client =
- address.createClient(createOzoneConfiguration())) {
+
+OzoneConfiguration conf = createOzoneConfiguration();
+
+try (OzoneClient client = address.createClient(conf)) {
Review comment:
It _is_ used to get chunk size.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #235: HDDS-2580. Ensure resources are closed in Get/PutKeyHandler
dineshchitlangia commented on a change in pull request #235: HDDS-2580. Ensure
resources are closed in Get/PutKeyHandler
URL: https://github.com/apache/hadoop-ozone/pull/235#discussion_r348546285
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/GetKeyHandler.java
##
@@ -65,8 +65,10 @@ public Void call() throws Exception {
OzoneAddress address = new OzoneAddress(uri);
address.ensureKeyAddress();
-try (OzoneClient client =
- address.createClient(createOzoneConfiguration())) {
+
+OzoneConfiguration conf = createOzoneConfiguration();
+
+try (OzoneClient client = address.createClient(conf)) {
Review comment:
@adoroszlai Can we avoid creating variable `conf` as it is only needed to
createClient and not used later?
We can stick with original code and you could make any checkstyle/formatting
changes without creating a local variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng opened a new pull request #237: HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline.
timmylicheng opened a new pull request #237: HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline. URL: https://github.com/apache/hadoop-ozone/pull/237 ## What changes were proposed in this pull request? #HDDS-1572 Implement a Pipeline scrubber to maintain healthy number of pipelines in a cluster ## What is the link to the Apache JIRA #HDDS-1572 (Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HDDS-. Fix a typo in YYY.) Please replace this section with the link to the Apache JIRA) ## How was this patch tested? UT (Please explain how this patch was tested. Ex: unit tests, manual tests) (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 opened a new pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor. Contributed by Mukul Kumar Singh.
mukul1987 opened a new pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop-ozone/pull/236 ## What changes were proposed in this pull request? This jira moves the initialization of XceiverClientMetrics to XceiverClientManager, This helps in avoiding the static class object and also ensures that the metrics are initialized as part of the client intialization. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2284 ## How was this patch tested? TestBlockOutputStream & TestBlockOutputStreamWithFailures This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #127: HDDS-2300. Publish normalized Ratis metrics via the prometheus endpoint
elek commented on a change in pull request #127: HDDS-2300. Publish normalized
Ratis metrics via the prometheus endpoint
URL: https://github.com/apache/hadoop-ozone/pull/127#discussion_r348491546
##
File path:
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/RatisNameRewriteSampleBuilder.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.server;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import io.prometheus.client.Collector.MetricFamilySamples.Sample;
+import io.prometheus.client.dropwizard.samplebuilder.DefaultSampleBuilder;
+import org.apache.logging.log4j.util.Strings;
+import static
org.apache.ratis.server.metrics.RatisMetrics.RATIS_APPLICATION_NAME_METRICS;
+
+/**
+ * Collect Dropwizard metrics and rename ratis specific metrics.
+ */
+public class RatisNameRewriteSampleBuilder extends DefaultSampleBuilder {
+
+ private List followerPatterns = new ArrayList<>();
+
+ public RatisNameRewriteSampleBuilder() {
+followerPatterns
+.add(Pattern.compile(
+"grpc_log_appender_follower_(.*)_(latency|success|inconsistency)"
++ ".*"));
+followerPatterns
+.add(Pattern.compile("follower_(.*)_lastHeartbeatElapsedTime"));
+followerPatterns.add(Pattern.compile("(.*)_peerCommitIndex"));
+
+ }
+
+ @Override
+ public Sample createSample(String dropwizardName, String nameSuffix,
+ List additionalLabelNames, List additionalLabelValues,
+ double value) {
+//this is a ratis metrics, where the second part is an instance id.
+if (dropwizardName.startsWith(RATIS_APPLICATION_NAME_METRICS)
+|| (dropwizardName.startsWith("ratis_grpc"))) {
+ List names = new ArrayList<>(additionalLabelNames);
+ List values = new ArrayList<>(additionalLabelValues);
+ String name = normalizeRatisMetric(dropwizardName, names, values);
+
+ return super
+ .createSample(name, nameSuffix,
+ names,
+ values, value);
+
+} else {
+ return super
+ .createSample(dropwizardName, nameSuffix, additionalLabelNames,
+ additionalLabelValues, value);
+}
+ }
+
+ protected String normalizeRatisMetric(String dropwizardName,
+ List names,
+ List values) {
+
+List nameParts =
+new ArrayList(Arrays.asList(dropwizardName.split("\\.")));
+//second part is id or id@group_id
+if (nameParts.size() > 2) {
+ String[] identifiers = nameParts.get(2).split("@");
+ names.add("instance");
+ values.add(identifiers[0]);
+ if (identifiers.length > 1) {
+names.add("group");
+values.add(identifiers[1]);
+ }
+ nameParts.remove(2);
+}
+
+if (nameParts.size() > 2) {
+ for (Pattern pattern : followerPatterns) {
+Matcher matcher = pattern.matcher(nameParts.get(2));
+if (matcher.matches()) {
+ names.add("follower");
+ String followerId = matcher.group(1);
+ values.add(followerId);
+ nameParts.set(2, nameParts.get(2).replace(followerId + "_", ""));
+}
+ }
+}
+return Strings.join(nameParts, '.');
+ }
+
+ private String withoutSecondPart(String[] parts) {
+StringBuilder builder = new StringBuilder();
+for (int i = 0; i < parts.length; i++) {
+ if (i != 2) {
+if (i != 0) {
+ builder.append(".");
+}
+builder.append(parts[i]);
+ }
+}
+return builder.toString();
+ }
Review comment:
thanks. deleted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[GitHub] [hadoop-ozone] adoroszlai commented on issue #204: HDDS-2485. Disable XML external entity processing
adoroszlai commented on issue #204: HDDS-2485. Disable XML external entity processing URL: https://github.com/apache/hadoop-ozone/pull/204#issuecomment-556012231 Thanks @elek for reviewing and committing this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #204: HDDS-2485. Disable XML external entity processing
elek commented on a change in pull request #204: HDDS-2485. Disable XML external entity processing URL: https://github.com/apache/hadoop-ozone/pull/204#discussion_r348488754 ## File path: hadoop-hdds/common/src/test/resources/log4j.properties ## @@ -0,0 +1,23 @@ +# Review comment: :heart: I just noticed that this file is missing from multiple projects.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #204: HDDS-2485. Disable XML external entity processing
elek closed pull request #204: HDDS-2485. Disable XML external entity processing URL: https://github.com/apache/hadoop-ozone/pull/204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
