[GitHub] [hadoop] xiaoyuyao merged pull request #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C…
xiaoyuyao merged pull request #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C… URL: https://github.com/apache/hadoop/pull/1262 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on issue #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C…
xiaoyuyao commented on issue #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C… URL: https://github.com/apache/hadoop/pull/1262#issuecomment-520118912 Thanks @bharatviswa504 and @adoroszlai for the review. I will merge this to trunk shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA.
arp7 commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1230#issuecomment-520114204 I committed this. None of the test failures looks related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 merged pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
arp7 merged pull request #1230: HDDS-1895. Support Key ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1230 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16491) Upgrade jetty version to 9.3.27
[ https://issues.apache.org/jira/browse/HADOOP-16491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904301#comment-16904301 ] Wei-Chiu Chuang commented on HADOOP-16491: -- Actually, we should start to think about migrate to Jetty 9.4. > Upgrade jetty version to 9.3.27 > --- > > Key: HADOOP-16491 > URL: https://issues.apache.org/jira/browse/HADOOP-16491 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.2.0 >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16491-001.patch > > > The current jetty version (9.3.24) has few CVEs (Ref: > [https://www.cvedetails.com/version/272598/Eclipse-Jetty-9.3.24.html]). It > would be a good idea to upgrade jetty to 9.3.27 (which is the latest version > as of today 08/05/2019). -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1264: HDFS-12125. Document the missing EC removePolicy command
hadoop-yetus commented on issue #1264: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1264#issuecomment-520111663 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 67 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1202 | trunk passed | | +1 | compile | 63 | trunk passed | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 67 | trunk passed | | +1 | shadedclient | 774 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 53 | trunk passed | | 0 | spotbugs | 179 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 177 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 62 | the patch passed | | +1 | compile | 57 | the patch passed | | +1 | javac | 57 | the patch passed | | +1 | checkstyle | 44 | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 604 unchanged - 3 fixed = 604 total (was 607) | | +1 | mvnsite | 69 | the patch passed | | -1 | whitespace | 0 | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 742 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 49 | the patch passed | | +1 | findbugs | 182 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 6953 | hadoop-hdfs in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 10690 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap | | | hadoop.hdfs.qjournal.client.TestQJMWithFaults | | | hadoop.hdfs.server.datanode.TestLargeBlockReport | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1264/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1264 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 31d14b00d52e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ce3c5a3 | | Default Java | 1.8.0_212 | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1264/1/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1264/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1264/1/testReport/ | | Max. process+thread count | 4094 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1264/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1264: HDFS-12125. Document the missing EC removePolicy command
hadoop-yetus commented on a change in pull request #1264: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1264#discussion_r312687560 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ## @@ -1385,6 +1385,54 @@ + + dfs.image.parallel.load + true + +If true, write sub-section entries to the fsimage index so it can +be loaded in parallel. Also controls whether parallel loading +will be used for an image previously created with sub-sections. +If the image contains sub-sections and this is set to false, +parallel loading will not be used. + + + + + dfs.image.parallel.target.sections + 12 + +Controls the number of sub-sections that will be written to +fsimage for each section. This should be larger than +dfs.image.parallel.threads, otherwise all threads will not be +used when loading. Ideally, have at least twice the number +of target sections as threads, so each thread must load more +than one section to avoid one long running section affecting +the load time. + + + + + dfs.image.parallel.inode.threshold + 100 + +If the image contains less inodes than this setting, then +do not write sub-sections and hence disable parallel loading. +This is because small images load very quickly in serial and +parallel loading is not needed. + + + + + dfs.image.parallel.threads + 4 + +The number of threads to use when dfs.image.parallel.load is Review comment: whitespace:end of line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
hadoop-yetus commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl. URL: https://github.com/apache/hadoop/pull/1257#issuecomment-520109012 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 73 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for branch | | +1 | mvninstall | 602 | trunk passed | | +1 | compile | 366 | trunk passed | | +1 | checkstyle | 72 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 983 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 223 | trunk passed | | 0 | spotbugs | 475 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 708 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 643 | the patch passed | | +1 | compile | 413 | the patch passed | | +1 | cc | 413 | the patch passed | | +1 | javac | 413 | the patch passed | | +1 | checkstyle | 86 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 803 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 198 | the patch passed | | +1 | findbugs | 707 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 356 | hadoop-hdds in the patch passed. | | -1 | unit | 2145 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 8671 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 5dd0fbb2f379 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ce3c5a3 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/3/testReport/ | | Max. process+thread count | 4909 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/s3gateway hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16115) [JDK 11] TestHttpServer#testJersey fails
[
https://issues.apache.org/jira/browse/HADOOP-16115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904285#comment-16904285
]
Wei-Chiu Chuang commented on HADOOP-16115:
--
Make sure to use this surefire parameter to repeat tests & rule out flakiness:
mvn -Dsurefire.rerunFailingTestsCount=2 test
https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
Looks like there's no other way than migrating to Jersey 2: HADOOP-15984
> [JDK 11] TestHttpServer#testJersey fails
>
>
> Key: HADOOP-16115
> URL: https://issues.apache.org/jira/browse/HADOOP-16115
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: mvn-test-11.0.4.log
>
>
> {noformat}
> [INFO] Running org.apache.hadoop.http.TestHttpServer
> [ERROR] Tests run: 26, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 5.954 s <<< FAILURE! - in org.apache.hadoop.http.TestHttpServer
> [ERROR] testJersey(org.apache.hadoop.http.TestHttpServer) Time elapsed:
> 0.128 s <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 500 for URL:
> http://localhost:40339/jersey/foo?op=bar
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1913)
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
> at
> org.apache.hadoop.http.HttpServerFunctionalTest.readOutput(HttpServerFunctionalTest.java:260)
> at
> org.apache.hadoop.http.TestHttpServer.testJersey(TestHttpServer.java:526)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904283#comment-16904283 ] Wei-Chiu Chuang commented on HADOOP-15565: -- HDFS-10323 looks related. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, > HADOOP-15565.0003.patch, HADOOP-15565.0004.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520105263 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 73 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 33 | Maven dependency ordering for branch | | +1 | mvninstall | 588 | trunk passed | | +1 | compile | 356 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 904 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 160 | trunk passed | | 0 | spotbugs | 417 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 610 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 544 | the patch passed | | +1 | compile | 362 | the patch passed | | +1 | javac | 362 | the patch passed | | -0 | checkstyle | 40 | hadoop-ozone: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 727 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 124 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13) | | +1 | findbugs | 722 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 355 | hadoop-hdds in the patch passed. | | -1 | unit | 2006 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 8051 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1263 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 312c1a2195d3 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ce3c5a3 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/3/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/3/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/3/testReport/ | | Max. process+thread count | 5412 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command
jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258#issuecomment-520105020 Merged the PR, thanks @smengcl and @wenxinhe ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1265: HDFS-12125. Document the missing EC removePolicy command
jojochuang commented on issue #1265: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1265#issuecomment-520104947 Merged PR #1258 so close this one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang closed pull request #1265: HDFS-12125. Document the missing EC removePolicy command
jojochuang closed pull request #1265: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1265 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #1258: HDFS-12125. Document the missing EC removePolicy command
jojochuang merged pull request #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA.
hadoop-yetus commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1230#issuecomment-520104312 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 82 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 38 | Maven dependency ordering for branch | | +1 | mvninstall | 624 | trunk passed | | +1 | compile | 378 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 963 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 173 | trunk passed | | 0 | spotbugs | 466 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 684 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 583 | the patch passed | | +1 | compile | 478 | the patch passed | | +1 | javac | 478 | the patch passed | | +1 | checkstyle | 92 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 888 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 215 | the patch passed | | +1 | findbugs | 770 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 421 | hadoop-hdds in the patch passed. | | -1 | unit | 3537 | hadoop-ozone in the patch failed. | | +1 | asflicense | 50 | The patch does not generate ASF License warnings. | | | | 10264 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.scm.TestSCMNodeManagerMXBean | | | hadoop.ozone.scm.TestSCMMXBean | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.hdds.scm.container.TestContainerStateManagerIntegration | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.web.client.TestKeysRatis | | | hadoop.ozone.client.rpc.TestKeyInputStream | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1230 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ab345089517b 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98dd7c4 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/6/testReport/ | | Max. process+thread count | 3462 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1265: HDFS-12125. Document the missing EC removePolicy command
hadoop-yetus commented on issue #1265: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1265#issuecomment-520103631 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1133 | trunk passed | | +1 | mvnsite | 72 | trunk passed | | +1 | shadedclient | 1875 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 70 | the patch passed | | +1 | mvnsite | 67 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 732 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | asflicense | 26 | The patch does not generate ASF License warnings. | | | | 2924 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1265/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1265 | | Optional Tests | dupname asflicense mvnsite | | uname | Linux 71628f114047 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ce3c5a3 | | Max. process+thread count | 445 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1265/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520103222 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 45 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 32 | Maven dependency ordering for branch | | +1 | mvninstall | 610 | trunk passed | | +1 | compile | 372 | trunk passed | | +1 | checkstyle | 81 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 884 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 168 | trunk passed | | 0 | spotbugs | 414 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 611 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 555 | the patch passed | | +1 | compile | 379 | the patch passed | | +1 | javac | 379 | the patch passed | | -0 | checkstyle | 42 | hadoop-ozone: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 677 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 93 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13) | | +1 | findbugs | 640 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 292 | hadoop-hdds in the patch passed. | | -1 | unit | 1705 | hadoop-ozone in the patch failed. | | +1 | asflicense | 54 | The patch does not generate ASF License warnings. | | | | 7564 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1263 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2dfcc121f70d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98dd7c4 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/2/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/2/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/2/testReport/ | | Max. process+thread count | 4089 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904264#comment-16904264
]
Wei-Chiu Chuang commented on HADOOP-14441:
--
[~tartina] depending on your applications,
you could apply this patch to your Hadoop installation (2.7 is EOL, I think),
or call FileSystem.addDelegationTokens() multiple times (depending on the
number of KMS instances) to get delegation tokens from all KMS.
This is actually not a problem for MapReduce, because MR unintentionally calls
FileSystem.addDelegationTokens() three times.
> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch,
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
> * The implementer of this class will take a renewer and add all
> * delegation tokens associated with the renewer to the
> * Credentials object if it is not already present,
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and
> we were puzzled why it always throws "Failed to find any Kerberos tgt"
> exceptions talking to one KMS but not the other. Turns out that client
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets
> one KMS delegation token at a time.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA.
hadoop-yetus commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1230#issuecomment-520099530 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 16 | Maven dependency ordering for branch | | +1 | mvninstall | 598 | trunk passed | | +1 | compile | 374 | trunk passed | | +1 | checkstyle | 74 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 867 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | trunk passed | | 0 | spotbugs | 423 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 619 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | +1 | mvninstall | 567 | the patch passed | | +1 | compile | 380 | the patch passed | | +1 | javac | 380 | the patch passed | | +1 | checkstyle | 82 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 681 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 172 | the patch passed | | +1 | findbugs | 641 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 297 | hadoop-hdds in the patch passed. | | -1 | unit | 1764 | hadoop-ozone in the patch failed. | | +1 | asflicense | 54 | The patch does not generate ASF License warnings. | | | | 7608 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.om.TestKeyManagerImpl | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1230 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 11ed57c004ba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98dd7c4 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/5/testReport/ | | Max. process+thread count | 5365 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1230/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang opened a new pull request #1265: HDFS-12125. Document the missing EC removePolicy command
jojochuang opened a new pull request #1265: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1265 Forked PR 1258 to trigger precommit build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command
jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258#issuecomment-520098198 Sorry. I meant PR #1265 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang closed pull request #1264: HDFS-12125. Document the missing EC removePolicy command
jojochuang closed pull request #1264: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1264 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command
jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258#issuecomment-520097591 @smengcl you are not yet a Hadoop committer so your PR doesn't trigger precommit builds. I forked your PR as #1264 to facilitate precommit build for you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang opened a new pull request #1264: HDFS-12125. Document the missing EC removePolicy command
jojochuang opened a new pull request #1264: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1264 Forked Hadoop PR 1258, to trigger precommit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
xiaoyuyao commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl. URL: https://github.com/apache/hadoop/pull/1257#issuecomment-520096638 +1, pending CI... Thanks for the quick fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
bharatviswa504 commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl. URL: https://github.com/apache/hadoop/pull/1257#issuecomment-520095907 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster
avijayanhwx commented on issue #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster URL: https://github.com/apache/hadoop/pull/1146#issuecomment-520095919 LGTM +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
bharatviswa504 commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312676316
##
File path:
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##
@@ -388,7 +388,9 @@ public void deleteVolume(String volumeName) throws
IOException {
@Override
public void createBucket(String volumeName, String bucketName)
throws IOException {
-createBucket(volumeName, bucketName, BucketArgs.newBuilder().build());
+// Set acls of current user.
+createBucket(volumeName, bucketName,
+BucketArgs.newBuilder().setAcls(getAclList()).build());
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312675211
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java
##
@@ -134,17 +132,6 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
bucketInfoBuilder.addAllMetadata(KeyValueUtil
.getFromProtobuf(bucketArgs.getMetadataList()));
- //Check ACLs to update
- if (omBucketArgs.getAddAcls() != null ||
- omBucketArgs.getRemoveAcls() != null) {
-bucketInfoBuilder.setAcls(getUpdatedAclList(oldBucketInfo.getAcls(),
-omBucketArgs.getRemoveAcls(), omBucketArgs.getAddAcls()));
-LOG.debug("Updating ACLs for bucket: {} in volume: {}",
-bucketName, volumeName);
- } else {
-bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
Review comment:
We have a similar problem here because OMBucketArgs does not have acls, we
will need to rely on the oldBucketInfo.getAcls to avoid reset existing acl on
bucket.
bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312675211
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java
##
@@ -134,17 +132,6 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
bucketInfoBuilder.addAllMetadata(KeyValueUtil
.getFromProtobuf(bucketArgs.getMetadataList()));
- //Check ACLs to update
- if (omBucketArgs.getAddAcls() != null ||
- omBucketArgs.getRemoveAcls() != null) {
-bucketInfoBuilder.setAcls(getUpdatedAclList(oldBucketInfo.getAcls(),
-omBucketArgs.getRemoveAcls(), omBucketArgs.getAddAcls()));
-LOG.debug("Updating ACLs for bucket: {} in volume: {}",
-bucketName, volumeName);
- } else {
-bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
Review comment:
We have a similar problem here because OMBucketArgs does not have acls, we
will need to rely on the oldBucketInfo.getAcls to avoid reset existing acl on
bucket.
bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command
jojochuang commented on issue #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258#issuecomment-520093660 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312674369
##
File path:
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##
@@ -388,7 +388,9 @@ public void deleteVolume(String volumeName) throws
IOException {
@Override
public void createBucket(String volumeName, String bucketName)
throws IOException {
-createBucket(volumeName, bucketName, BucketArgs.newBuilder().build());
+// Set acls of current user.
+createBucket(volumeName, bucketName,
+BucketArgs.newBuilder().setAcls(getAclList()).build());
Review comment:
This will cause creator acls added twice into the final list because the
same list will be added again when the passin bucketArgs has a non-empty acl
list.
{code}
List listOfAcls = getAclList();
//ACLs from BucketArgs
if(bucketArgs.getAcls() != null) {
listOfAcls.addAll(bucketArgs.getAcls());
}
{code}
The list of acls are not merged properly when using List#addAll, which will
be fixed in HDDS-1927.
Let's file a separate JIRA for RpcClient#createBucket issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16351) Change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR
[
https://issues.apache.org/jira/browse/HADOOP-16351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904241#comment-16904241
]
Hadoop QA commented on HADOOP-16351:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
44s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16351 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971117/HADOOP-16351.01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2128caffa82c 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 98dd7c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16465/testReport/ |
| Max. process+thread count | 607 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
|
| Console output |
[GitHub] [hadoop] hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520088237 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 45 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for branch | | +1 | mvninstall | 588 | trunk passed | | +1 | compile | 362 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 845 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 148 | trunk passed | | 0 | spotbugs | 422 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 616 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 18 | Maven dependency ordering for patch | | +1 | mvninstall | 533 | the patch passed | | +1 | compile | 355 | the patch passed | | +1 | javac | 355 | the patch passed | | -0 | checkstyle | 32 | hadoop-ozone: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 603 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 85 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13) | | +1 | findbugs | 656 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 295 | hadoop-hdds in the patch passed. | | -1 | unit | 1982 | hadoop-ozone in the patch failed. | | +1 | asflicense | 41 | The patch does not generate ASF License warnings. | | | | 7544 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.om.TestOzoneManagerHA | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1263 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2fad9f5b2750 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98dd7c4 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/1/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/1/testReport/ | | Max. process+thread count | 4981 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on a change in pull request #1258: HDFS-12125. Document the missing EC removePolicy command
smengcl commented on a change in pull request #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258#discussion_r312669126 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md ## @@ -218,6 +219,10 @@ Below are the details about each command. Enable an erasure coding policy. +* `[-removePolicy -policy ]` Review comment: Updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA.
bharatviswa504 commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1230#issuecomment-520077044 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA.
bharatviswa504 commented on issue #1230: HDDS-1895. Support Key ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1230#issuecomment-520077027 Thank You @arp7 for the offline discussion. Addressed the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
bharatviswa504 commented on a change in pull request #1230: HDDS-1895. Support
Key ACL operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1230#discussion_r312660268
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeyAddAclRequest.java
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.key.acl;
+
+import java.io.IOException;
+import java.util.List;
+
+import com.google.common.collect.Lists;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.response.key.acl.OMKeyAclResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.util.BooleanBiFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OzoneAclInfo;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.AddAclResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRequest;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMResponse;
+
+/**
+ * Handle add Acl request for bucket.
+ */
+public class OMKeyAddAclRequest extends OMKeyAclRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMKeyAddAclRequest.class);
+
+ private static BooleanBiFunction, OmKeyInfo> keyAddAclOp;
+ private String path;
+ private List ozoneAcls;
+
+ static {
+keyAddAclOp = (ozoneAcls, omKeyInfo) -> {
+ return omKeyInfo.addAcl(ozoneAcls.get(0));
Review comment:
As discussed offline, have changed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
bharatviswa504 commented on a change in pull request #1230: HDDS-1895. Support
Key ACL operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1230#discussion_r312660320
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeyAddAclRequest.java
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.key.acl;
+
+import java.io.IOException;
+import java.util.List;
+
+import com.google.common.collect.Lists;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.response.key.acl.OMKeyAclResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.util.BooleanBiFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OzoneAclInfo;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.AddAclResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRequest;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMResponse;
+
+/**
+ * Handle add Acl request for bucket.
+ */
+public class OMKeyAddAclRequest extends OMKeyAclRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMKeyAddAclRequest.class);
+
+ private static BooleanBiFunction, OmKeyInfo> keyAddAclOp;
+ private String path;
+ private List ozoneAcls;
+
+ static {
+keyAddAclOp = (ozoneAcls, omKeyInfo) -> {
+ return omKeyInfo.addAcl(ozoneAcls.get(0));
+};
+ }
+
+ public OMKeyAddAclRequest(OMRequest omRequest) {
+super(omRequest, keyAddAclOp);
+OzoneManagerProtocolProtos.AddAclRequest addAclRequest =
+getOmRequest().getAddAclRequest();
+path = addAclRequest.getObj().getPath();
+ozoneAcls = Lists.newArrayList(addAclRequest.getAcl());
+ }
+
+ @Override
+ List getAcls() {
+return ozoneAcls;
+ }
+
+ @Override
+ String getPath() {
+return path;
+ }
+
+ @Override
+ OMResponse.Builder onInit() {
+return OMResponse.newBuilder().setCmdType(
+OzoneManagerProtocolProtos.Type.AddAcl).setStatus(
+OzoneManagerProtocolProtos.Status.OK).setSuccess(true);
+
+ }
+
+ @Override
+ OMClientResponse onSuccess(OMResponse.Builder omResponse,
+ OmKeyInfo omKeyInfo, boolean operationResult) {
+omResponse.setSuccess(operationResult);
+omResponse.setAddAclResponse(AddAclResponse.newBuilder()
+.setResponse(operationResult));
+return new OMKeyAclResponse(omKeyInfo,
+omResponse.build());
+ }
+
+ @Override
+ OMClientResponse onFailure(OMResponse.Builder omResponse,
+ IOException exception) {
+return new OMKeyAclResponse(null,
+createErrorOMResponse(omResponse, exception));
+ }
+
+ @Override
+ void onComplete(boolean operationResult, IOException exception,
+ OMMetrics omMetrics) {
+if (operationResult) {
+ LOG.debug("Add acl: {} to path: {} success!", getAcls(), getPath());
+} else {
+ omMetrics.incNumBucketUpdateFails();
+ if (exception == null) {
+LOG.error("Add acl {} to path {} failed, because acl already exist",
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
bharatviswa504 commented on a change in pull request #1230: HDDS-1895. Support
Key ACL operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1230#discussion_r312660300
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
##
@@ -141,20 +144,26 @@ private static OMClientRequest getOMAclRequest(OMRequest
omRequest) {
return new OMVolumeAddAclRequest(omRequest);
} else if (ObjectType.BUCKET == type) {
return new OMBucketAddAclRequest(omRequest);
+ } else if (type == ObjectType.KEY) {
Review comment:
Updated it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16351) Change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR
[
https://issues.apache.org/jira/browse/HADOOP-16351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-16351:
-
Status: Patch Available (was: Open)
> Change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR
> ---
>
> Key: HADOOP-16351
> URL: https://issues.apache.org/jira/browse/HADOOP-16351
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.1.2
>Reporter: kevin su
>Assignee: kevin su
>Priority: Trivial
> Fix For: 3.1.2
>
> Attachments: HADOOP-16351.01.patch
>
>
> under distributedshell/Clients.java
> We should change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR, so it
> could also support Windows client
> {code}
> // add the runtime classpath needed for tests to work
> if (conf.getBoolean(YarnConfiguration.IS_MINI_YARN_CLUSTER, false)) {
> classPathEnv.append(':')
> .append(System.getProperty("java.class.path"));
> }
> {code}
> {code}
> // add the runtime classpath needed for tests to work
> if (conf.getBoolean(YarnConfiguration.IS_MINI_YARN_CLUSTER, false)) {
> classPathEnv.append(ApplicationConstants.CLASS_PATH_SEPARATOR)
> .append(System.getProperty("java.class.path"));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota
hadoop-yetus commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota URL: https://github.com/apache/hadoop/pull/1253#issuecomment-520066543 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 48 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 1068 | trunk passed | | +1 | compile | 1033 | trunk passed | | +1 | checkstyle | 161 | trunk passed | | +1 | mvnsite | 306 | trunk passed | | +1 | shadedclient | 1205 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 272 | trunk passed | | 0 | spotbugs | 77 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 549 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 208 | the patch passed | | +1 | compile | 992 | the patch passed | | +1 | javac | 992 | the patch passed | | -0 | checkstyle | 156 | root: The patch generated 11 new + 604 unchanged - 0 fixed = 615 total (was 604) | | +1 | mvnsite | 309 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 689 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 274 | the patch passed | | +1 | findbugs | 588 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 498 | hadoop-common in the patch passed. | | +1 | unit | 131 | hadoop-hdfs-client in the patch passed. | | -1 | unit | 4722 | hadoop-hdfs in the patch failed. | | +1 | unit | 314 | hadoop-hdfs-httpfs in the patch passed. | | -1 | unit | 1427 | hadoop-hdfs-rbf in the patch failed. | | +1 | asflicense | 64 | The patch does not generate ASF License warnings. | | | | 14929 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport | | | hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup | | | hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken | | | hadoop.hdfs.server.federation.router.TestRouterFaultTolerant | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1253 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e664c5c19594 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a79564f | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/5/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/5/testReport/ | | Max. process+thread count | 4229 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-rbf U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
arp7 commented on a change in pull request #1230: HDDS-1895. Support Key ACL
operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1230#discussion_r312650511
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeyAddAclRequest.java
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.key.acl;
+
+import java.io.IOException;
+import java.util.List;
+
+import com.google.common.collect.Lists;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.response.key.acl.OMKeyAclResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.util.BooleanBiFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OzoneAclInfo;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.AddAclResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRequest;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMResponse;
+
+/**
+ * Handle add Acl request for bucket.
+ */
+public class OMKeyAddAclRequest extends OMKeyAclRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMKeyAddAclRequest.class);
+
+ private static BooleanBiFunction, OmKeyInfo> keyAddAclOp;
+ private String path;
+ private List ozoneAcls;
+
+ static {
+keyAddAclOp = (ozoneAcls, omKeyInfo) -> {
+ return omKeyInfo.addAcl(ozoneAcls.get(0));
+};
+ }
+
+ public OMKeyAddAclRequest(OMRequest omRequest) {
+super(omRequest, keyAddAclOp);
+OzoneManagerProtocolProtos.AddAclRequest addAclRequest =
+getOmRequest().getAddAclRequest();
+path = addAclRequest.getObj().getPath();
+ozoneAcls = Lists.newArrayList(addAclRequest.getAcl());
+ }
+
+ @Override
+ List getAcls() {
+return ozoneAcls;
+ }
+
+ @Override
+ String getPath() {
+return path;
+ }
+
+ @Override
+ OMResponse.Builder onInit() {
+return OMResponse.newBuilder().setCmdType(
+OzoneManagerProtocolProtos.Type.AddAcl).setStatus(
+OzoneManagerProtocolProtos.Status.OK).setSuccess(true);
+
+ }
+
+ @Override
+ OMClientResponse onSuccess(OMResponse.Builder omResponse,
+ OmKeyInfo omKeyInfo, boolean operationResult) {
+omResponse.setSuccess(operationResult);
+omResponse.setAddAclResponse(AddAclResponse.newBuilder()
+.setResponse(operationResult));
+return new OMKeyAclResponse(omKeyInfo,
+omResponse.build());
+ }
+
+ @Override
+ OMClientResponse onFailure(OMResponse.Builder omResponse,
+ IOException exception) {
+return new OMKeyAclResponse(null,
+createErrorOMResponse(omResponse, exception));
+ }
+
+ @Override
+ void onComplete(boolean operationResult, IOException exception,
+ OMMetrics omMetrics) {
+if (operationResult) {
+ LOG.debug("Add acl: {} to path: {} success!", getAcls(), getPath());
+} else {
+ omMetrics.incNumBucketUpdateFails();
+ if (exception == null) {
+LOG.error("Add acl {} to path {} failed, because acl already exist",
Review comment:
Should we just make this info or even debug?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
arp7 commented on a change in pull request #1230: HDDS-1895. Support Key ACL
operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1230#discussion_r312650056
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeyAddAclRequest.java
##
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.request.key.acl;
+
+import java.io.IOException;
+import java.util.List;
+
+import com.google.common.collect.Lists;
+import org.apache.hadoop.ozone.om.OMMetrics;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.response.key.acl.OMKeyAclResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.util.BooleanBiFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OzoneAclInfo;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.AddAclResponse;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRequest;
+import
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMResponse;
+
+/**
+ * Handle add Acl request for bucket.
+ */
+public class OMKeyAddAclRequest extends OMKeyAclRequest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMKeyAddAclRequest.class);
+
+ private static BooleanBiFunction, OmKeyInfo> keyAddAclOp;
+ private String path;
+ private List ozoneAcls;
+
+ static {
+keyAddAclOp = (ozoneAcls, omKeyInfo) -> {
+ return omKeyInfo.addAcl(ozoneAcls.get(0));
Review comment:
Sorry I didn't understand this. What is it doing? Felt little odd to see a
lambda in a static block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1230: HDDS-1895. Support Key ACL operations for OM HA.
arp7 commented on a change in pull request #1230: HDDS-1895. Support Key ACL
operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1230#discussion_r312649376
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
##
@@ -141,20 +144,26 @@ private static OMClientRequest getOMAclRequest(OMRequest
omRequest) {
return new OMVolumeAddAclRequest(omRequest);
} else if (ObjectType.BUCKET == type) {
return new OMBucketAddAclRequest(omRequest);
+ } else if (type == ObjectType.KEY) {
Review comment:
The code is using reverse order for equality. Can you use the same?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #1258: HDFS-12125. Document the missing EC removePolicy command
jojochuang commented on a change in pull request #1258: HDFS-12125. Document the missing EC removePolicy command URL: https://github.com/apache/hadoop/pull/1258#discussion_r312643769 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md ## @@ -218,6 +219,10 @@ Below are the details about each command. Enable an erasure coding policy. +* `[-removePolicy -policy ]` Review comment: This is a duplicate of line 214. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] vivekratnavel commented on issue #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster
vivekratnavel commented on issue #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster URL: https://github.com/apache/hadoop/pull/1146#issuecomment-520059219 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang closed pull request #249: HDFS-12125. Document the missing -removePolicy command of ec.
jojochuang closed pull request #249: HDFS-12125. Document the missing -removePolicy command of ec. URL: https://github.com/apache/hadoop/pull/249 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #249: HDFS-12125. Document the missing -removePolicy command of ec.
jojochuang commented on issue #249: HDFS-12125. Document the missing -removePolicy command of ec. URL: https://github.com/apache/hadoop/pull/249#issuecomment-520058923 Duplicates #1258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #763: [WIP] HADOOP-15984. Update jersey from 1.19 to 2.x
jojochuang commented on a change in pull request #763: [WIP] HADOOP-15984. Update jersey from 1.19 to 2.x URL: https://github.com/apache/hadoop/pull/763#discussion_r312630346 ## File path: hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties ## @@ -34,7 +34,6 @@ log4j.additivity.kms-audit=false log4j.rootLogger=INFO, kms log4j.logger.org.apache.hadoop=INFO -log4j.logger.com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator=OFF Review comment: This is related to HADOOP-15686. If I get HADOOP-15686 in, we'll have to update here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #763: [WIP] HADOOP-15984. Update jersey from 1.19 to 2.x
jojochuang commented on a change in pull request #763: [WIP] HADOOP-15984. Update jersey from 1.19 to 2.x URL: https://github.com/apache/hadoop/pull/763#discussion_r312631751 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserProvider.java ## @@ -1,73 +1,37 @@ -/** Review comment: why was the license text removed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #763: [WIP] HADOOP-15984. Update jersey from 1.19 to 2.x
jojochuang commented on a change in pull request #763: [WIP] HADOOP-15984.
Update jersey from 1.19 to 2.x
URL: https://github.com/apache/hadoop/pull/763#discussion_r312630777
##
File path: hadoop-common-project/hadoop-kms/src/test/resources/log4j.properties
##
@@ -25,7 +25,6 @@ log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601}
%-5p %c{1} - %m%n
log4j.rootLogger=INFO, stdout
log4j.logger.org.apache.hadoop.conf=ERROR
log4j.logger.org.apache.hadoop.crytpo.key.kms.server=ALL
-log4j.logger.com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator=OFF
Review comment:
TODO: HADOOP-15686
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao opened a new pull request #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri…
xiaoyuyao opened a new pull request #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C…
hadoop-yetus commented on issue #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C… URL: https://github.com/apache/hadoop/pull/1262#issuecomment-520053537 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 598 | trunk passed | | +1 | compile | 364 | trunk passed | | +1 | checkstyle | 72 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 936 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 165 | trunk passed | | 0 | spotbugs | 429 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 625 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 544 | the patch passed | | +1 | compile | 367 | the patch passed | | +1 | javac | 367 | the patch passed | | +1 | checkstyle | 75 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 729 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 163 | the patch passed | | +1 | findbugs | 647 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 328 | hadoop-hdds in the patch passed. | | -1 | unit | 1700 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 7556 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1262/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1262 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a0f965019c36 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6ff0453 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1262/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1262/1/testReport/ | | Max. process+thread count | 4475 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1262/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
hadoop-yetus commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl. URL: https://github.com/apache/hadoop/pull/1257#issuecomment-520050278 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for branch | | +1 | mvninstall | 591 | trunk passed | | +1 | compile | 350 | trunk passed | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 794 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 152 | trunk passed | | 0 | spotbugs | 413 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 602 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 533 | the patch passed | | +1 | compile | 357 | the patch passed | | +1 | cc | 357 | the patch passed | | +1 | javac | 357 | the patch passed | | +1 | checkstyle | 68 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 648 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 148 | the patch passed | | +1 | findbugs | 678 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 288 | hadoop-hdds in the patch passed. | | -1 | unit | 1261 | hadoop-ozone in the patch failed. | | +1 | asflicense | 38 | The patch does not generate ASF License warnings. | | | | 6832 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestNodeFailure | | | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.web.TestOzoneWebAccess | | | hadoop.ozone.web.TestOzoneVolumes | | | hadoop.ozone.web.client.TestBuckets | | | hadoop.ozone.web.client.TestVolume | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerHandler | | | hadoop.ozone.web.client.TestKeysRatis | | | hadoop.ozone.web.TestOzoneRestWithMiniCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 01a93ff1f685 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 6ff0453 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/2/testReport/ | | Max. process+thread count | 3625 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/s3gateway hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1257/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] shwetayakkali commented on a change in pull request #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster
shwetayakkali commented on a change in pull request #1146: HDDS-1366. Add
ability in Recon to track the number of small files in an Ozone Cluster
URL: https://github.com/apache/hadoop/pull/1146#discussion_r312632071
##
File path:
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFileSizeCountTask.java
##
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.utils.db.TypedTable;
+import org.junit.Test;
+
+import org.junit.runner.RunWith;
+import org.powermock.core.classloader.annotations.PowerMockIgnore;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.ozone.recon.tasks.
+OMDBUpdateEvent.OMDBUpdateAction.PUT;
+import static org.junit.Assert.assertEquals;
+
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.BDDMockito.given;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.times;
+import static org.powermock.api.mockito.PowerMockito.mock;
+import static org.powermock.api.mockito.PowerMockito.when;
+
+/**
+ * Unit test for Container Key mapper task.
+ */
+@RunWith(PowerMockRunner.class)
+@PowerMockIgnore({"javax.management.*", "javax.net.ssl.*"})
+@PrepareForTest(OmKeyInfo.class)
+
+public class TestFileSizeCountTask {
+ @Test
+ public void testCalculateBinIndex() {
+FileSizeCountTask fileSizeCountTask = mock(FileSizeCountTask.class);
+
+when(fileSizeCountTask.getMaxFileSizeUpperBound()).
+thenReturn(1125899906842624L);// 1 PB
+when(fileSizeCountTask.getOneKB()).thenReturn(1024L);
+when(fileSizeCountTask.getMaxBinSize()).thenReturn(42);
+when(fileSizeCountTask.calculateBinIndex(anyLong())).thenCallRealMethod();
+when(fileSizeCountTask.nextClosestPowerIndexOfTwo(
+anyLong())).thenCallRealMethod();
+
+long fileSize = 1024L;// 1 KB
+int binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(1, binIndex);
+
+fileSize = 1023L;// 1KB - 1B
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(0, binIndex);
+
+fileSize = 562949953421312L; // 512 TB
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(40, binIndex);
+
+fileSize = 562949953421313L; // (512 TB + 1B)
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(40, binIndex);
+
+fileSize = 562949953421311L; // (512 TB - 1B)
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(39, binIndex);
+
+fileSize = 1125899906842624L; // 1 PB - last (extra) bin
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(41, binIndex);
+
+fileSize = 10L;
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(7, binIndex);
+
+fileSize = 1125899906842623L; // (1 PB - 1B)
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(40, binIndex);
+
+fileSize = 1125899906842624L * 4; // 4 PB - last extra bin
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(41, binIndex);
+
+fileSize = Long.MAX_VALUE;// extra bin
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(41, binIndex);
+ }
+
+ @Test
+ public void testFileCountBySizeReprocess() throws IOException {
Review comment:
Sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
hadoop-yetus commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy URL: https://github.com/apache/hadoop/pull/1229#discussion_r312627151 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md ## @@ -1275,3 +1323,39 @@ Please don't do that. Given that the emulated directory rename and delete operat are not atomic, even without retries, multiple S3 clients working with the same paths can interfere with each other +### Tuning S3Guard open/rename Retry Policies + +When the S3A connector attempts to open a file for which it has an entry in +its database, it will retry if the desired file is not found. This is +done if +* No file is found in S3. +* There is a file but its version or etag is not consistent with S3Guard table Review comment: whitespace:end of line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
hadoop-yetus commented on issue #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy URL: https://github.com/apache/hadoop/pull/1229#issuecomment-520043735 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for branch | | +1 | mvninstall | 1038 | trunk passed | | +1 | compile | 1049 | trunk passed | | +1 | checkstyle | 132 | trunk passed | | +1 | mvnsite | 108 | trunk passed | | +1 | shadedclient | 919 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 97 | trunk passed | | 0 | spotbugs | 59 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 174 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | +1 | mvninstall | 75 | the patch passed | | +1 | compile | 1021 | the patch passed | | +1 | javac | 1021 | the patch passed | | -0 | checkstyle | 143 | root: The patch generated 2 new + 46 unchanged - 2 fixed = 48 total (was 48) | | +1 | mvnsite | 123 | the patch passed | | -1 | whitespace | 0 | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 684 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 106 | the patch passed | | +1 | findbugs | 202 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 480 | hadoop-common in the patch failed. | | +1 | unit | 82 | hadoop-aws in the patch passed. | | +1 | asflicense | 54 | The patch does not generate ASF License warnings. | | | | 6635 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.shell.TestCopy | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1229 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 079607da25a0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a79564f | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/8/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/8/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/8/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/8/testReport/ | | Max. process+thread count | 1463 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
hadoop-yetus commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy URL: https://github.com/apache/hadoop/pull/1229#discussion_r312627144 ## File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md ## @@ -1275,3 +1323,39 @@ Please don't do that. Given that the emulated directory rename and delete operat are not atomic, even without retries, multiple S3 clients working with the same paths can interfere with each other +### Tuning S3Guard open/rename Retry Policies + +When the S3A connector attempts to open a file for which it has an entry in +its database, it will retry if the desired file is not found. This is +done if Review comment: whitespace:end of line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
avijayanhwx commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520027376 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao merged pull request #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing.
xiaoyuyao merged pull request #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing. URL: https://github.com/apache/hadoop/pull/1260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 edited a comment on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
bharatviswa504 edited a comment on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl. URL: https://github.com/apache/hadoop/pull/1257#issuecomment-520017880 Thank You @xiaoyuyao for the review. Addressed review comments, and also fixed another issue when creating bucket in one API, not setting default ACL list. To fix find bug warning, removed ACLS from OzoneBucket. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
bharatviswa504 commented on issue #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl. URL: https://github.com/apache/hadoop/pull/1257#issuecomment-520017880 Thank You @xiaoyuyao for the review. Addressed review comments, and also fixed another issue when creating bucket in one API, not setting default ACL list. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
bharatviswa504 commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312597784
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
##
@@ -272,16 +272,6 @@ public void setBucketProperty(OmBucketArgs args) throws
IOException {
.setBucketName(oldBucketInfo.getBucketName());
bucketInfoBuilder.addAllMetadata(args.getMetadata());
- //Check ACLs to update
- if (args.getAddAcls() != null || args.getRemoveAcls() != null) {
-bucketInfoBuilder.setAcls(getUpdatedAclList(oldBucketInfo.getAcls(),
-args.getRemoveAcls(), args.getAddAcls()));
-LOG.debug("Updating ACLs for bucket: {} in volume: {}",
-bucketName, volumeName);
- } else {
-bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
Review comment:
Thanks for catching. Fixed it, and also added a test to cover this scenario.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
bharatviswa504 commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312597784
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
##
@@ -272,16 +272,6 @@ public void setBucketProperty(OmBucketArgs args) throws
IOException {
.setBucketName(oldBucketInfo.getBucketName());
bucketInfoBuilder.addAllMetadata(args.getMetadata());
- //Check ACLs to update
- if (args.getAddAcls() != null || args.getRemoveAcls() != null) {
-bucketInfoBuilder.setAcls(getUpdatedAclList(oldBucketInfo.getAcls(),
-args.getRemoveAcls(), args.getAddAcls()));
-LOG.debug("Updating ACLs for bucket: {} in volume: {}",
-bucketName, volumeName);
- } else {
-bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
Review comment:
Thanks for catching fixed it, and also added a test to cover this scenario.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao opened a new pull request #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C…
xiaoyuyao opened a new pull request #1262: HDDS-1943. TestKeyManagerImpl.testLookupKeyWithLocation is failing. C… URL: https://github.com/apache/hadoop/pull/1262 …ontributed by Xiaoyu Yao. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] vivekratnavel commented on a change in pull request #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster
vivekratnavel commented on a change in pull request #1146: HDDS-1366. Add
ability in Recon to track the number of small files in an Ozone Cluster
URL: https://github.com/apache/hadoop/pull/1146#discussion_r312596657
##
File path:
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFileSizeCountTask.java
##
@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.utils.db.TypedTable;
+import org.junit.Test;
+
+import org.junit.runner.RunWith;
+import org.powermock.core.classloader.annotations.PowerMockIgnore;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.ozone.recon.tasks.
+OMDBUpdateEvent.OMDBUpdateAction.PUT;
+import static org.junit.Assert.assertEquals;
+
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.BDDMockito.given;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.times;
+import static org.powermock.api.mockito.PowerMockito.mock;
+import static org.powermock.api.mockito.PowerMockito.when;
+
+/**
+ * Unit test for Container Key mapper task.
+ */
+@RunWith(PowerMockRunner.class)
+@PowerMockIgnore({"javax.management.*", "javax.net.ssl.*"})
+@PrepareForTest(OmKeyInfo.class)
+
+public class TestFileSizeCountTask {
+ @Test
+ public void testCalculateBinIndex() {
+FileSizeCountTask fileSizeCountTask = mock(FileSizeCountTask.class);
+
+when(fileSizeCountTask.getMaxFileSizeUpperBound()).
+thenReturn(1125899906842624L);// 1 PB
+when(fileSizeCountTask.getOneKB()).thenReturn(1024L);
+when(fileSizeCountTask.getMaxBinSize()).thenReturn(42);
+when(fileSizeCountTask.calculateBinIndex(anyLong())).thenCallRealMethod();
+when(fileSizeCountTask.nextClosestPowerIndexOfTwo(
+anyLong())).thenCallRealMethod();
+
+long fileSize = 1024L;// 1 KB
+int binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(1, binIndex);
+
+fileSize = 1023L;// 1KB - 1B
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(0, binIndex);
+
+fileSize = 562949953421312L; // 512 TB
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(40, binIndex);
+
+fileSize = 562949953421313L; // (512 TB + 1B)
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(40, binIndex);
+
+fileSize = 562949953421311L; // (512 TB - 1B)
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(39, binIndex);
+
+fileSize = 1125899906842624L; // 1 PB - last (extra) bin
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(41, binIndex);
+
+fileSize = 10L;
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(7, binIndex);
+
+fileSize = 1125899906842623L; // (1 PB - 1B)
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(40, binIndex);
+
+fileSize = 1125899906842624L * 4; // 4 PB - last extra bin
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(41, binIndex);
+
+fileSize = Long.MAX_VALUE;// extra bin
+binIndex = fileSizeCountTask.calculateBinIndex(fileSize);
+assertEquals(41, binIndex);
+ }
+
+ @Test
+ public void testFileCountBySizeReprocess() throws IOException {
Review comment:
We will need another test to cover process method and `DELETE` event. But, I
am ok with adding it in another JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] vivekratnavel commented on a change in pull request #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster
vivekratnavel commented on a change in pull request #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster URL: https://github.com/apache/hadoop/pull/1146#discussion_r312595826 ## File path: hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFileSizeCountTask.java ## @@ -0,0 +1,140 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.recon.tasks; + +import org.apache.hadoop.ozone.om.OMMetadataManager; +import org.apache.hadoop.ozone.om.OmMetadataManagerImpl; +import org.apache.hadoop.ozone.om.helpers.OmKeyInfo; +import org.apache.hadoop.utils.db.TypedTable; +import org.junit.Test; + +import org.junit.runner.RunWith; +import org.powermock.core.classloader.annotations.PowerMockIgnore; +import org.powermock.core.classloader.annotations.PrepareForTest; +import org.powermock.modules.junit4.PowerMockRunner; + +import java.io.IOException; + +import static org.apache.hadoop.ozone.recon.tasks. +OMDBUpdateEvent.OMDBUpdateAction.PUT; +import static org.junit.Assert.assertEquals; + +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.BDDMockito.given; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.times; +import static org.powermock.api.mockito.PowerMockito.mock; +import static org.powermock.api.mockito.PowerMockito.when; + +/** + * Unit test for Container Key mapper task. Review comment: nit: change this to File Size Count task This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1253: HDFS-8631. WebHDFS : Support setQuota
goiri commented on a change in pull request #1253: HDFS-8631. WebHDFS : Support
setQuota
URL: https://github.com/apache/hadoop/pull/1253#discussion_r312593255
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java
##
@@ -49,7 +51,7 @@
@Test
public void testAccessTimeParam() {
final AccessTimeParam p = new AccessTimeParam(AccessTimeParam.DEFAULT);
-Assert.assertEquals(-1L, p.getValue().longValue());
+assertEquals(-1L, p.getValue().longValue());
Review comment:
To avoid so much churn, let's just do the assertEquals in the new methods
and leave this alone for the rest for now.
we can do a followup JIRA for that if so.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1253: HDFS-8631. WebHDFS : Support setQuota
goiri commented on a change in pull request #1253: HDFS-8631. WebHDFS : Support
setQuota
URL: https://github.com/apache/hadoop/pull/1253#discussion_r312592614
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
##
@@ -1788,6 +1788,34 @@ public QuotaUsage getQuotaUsage(Path f) throws
IOException {
return getContentSummary(f);
}
+ /**
+ * Set quota for the given {@link Path}.
+ *
+ * @param src the target path to set quota for
+ * @param namespaceQuota the namespace quota (i.e., # of files/directories)
to set
+ * @param storagespaceQuota the storage space quota to set
+ * @throws IOException IO failure
+ */
+ public void setQuota(Path src, final long namespaceQuota,
+ final long storagespaceQuota) throws IOException {
+throw new UnsupportedOperationException(getClass().getCanonicalName() +
Review comment:
Let's do just the new two functions in this JIRA so we introduce the
function.
Afterwards we can do a separate JIRA to replace all.
To get the method, one could do
Thread.currentThread().getStackTrace()[index].getMethodName().
The index sometimes is tricky, but doable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1234: HDDS-1917. Ignore failing test-cases in TestSecureOzoneRpcClient.
hadoop-yetus commented on issue #1234: HDDS-1917. Ignore failing test-cases in TestSecureOzoneRpcClient. URL: https://github.com/apache/hadoop/pull/1234#issuecomment-520012179 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1457 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 848 | trunk passed | | +1 | compile | 439 | trunk passed | | +1 | checkstyle | 102 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1181 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 221 | trunk passed | | 0 | spotbugs | 509 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 760 | trunk passed | | -0 | patch | 570 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 707 | the patch passed | | +1 | compile | 492 | the patch passed | | +1 | javac | 492 | the patch passed | | +1 | checkstyle | 105 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 870 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 213 | the patch passed | | +1 | findbugs | 785 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 350 | hadoop-hdds in the patch passed. | | -1 | unit | 2249 | hadoop-ozone in the patch failed. | | +1 | asflicense | 48 | The patch does not generate ASF License warnings. | | | | 11017 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1234 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7245b35ad6ba 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / e25a5c2 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/4/testReport/ | | Max. process+thread count | 5320 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota
hadoop-yetus commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota URL: https://github.com/apache/hadoop/pull/1253#issuecomment-52824 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 12 | HDFS-8631 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | JIRA Issue | HDFS-8631 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12860576/HDFS-8631-006.patch | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/6/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
hadoop-yetus commented on issue #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy URL: https://github.com/apache/hadoop/pull/1229#issuecomment-519998750 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | +1 | mvninstall | 1056 | trunk passed | | +1 | compile | 1068 | trunk passed | | +1 | checkstyle | 145 | trunk passed | | +1 | mvnsite | 128 | trunk passed | | +1 | shadedclient | 1009 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 104 | trunk passed | | 0 | spotbugs | 66 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 184 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | +1 | mvninstall | 79 | the patch passed | | +1 | compile | 1014 | the patch passed | | +1 | javac | 1014 | the patch passed | | -0 | checkstyle | 145 | root: The patch generated 4 new + 46 unchanged - 2 fixed = 50 total (was 48) | | +1 | mvnsite | 123 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 698 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 108 | the patch passed | | +1 | findbugs | 199 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 481 | hadoop-common in the patch failed. | | +1 | unit | 83 | hadoop-aws in the patch passed. | | +1 | asflicense | 53 | The patch does not generate ASF License warnings. | | | | 6797 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.shell.TestCopy | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1229 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux fd968ec4a3a0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / e25a5c2 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/7/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/7/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/7/testReport/ | | Max. process+thread count | 1470 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1229/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] leosunli commented on a change in pull request #1223: HADOOP-16112. Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
leosunli commented on a change in pull request #1223: HADOOP-16112. Delete the baseTrashPath's subDir leads to don't modify baseTrashPath URL: https://github.com/apache/hadoop/pull/1223#discussion_r312572367 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java ## @@ -171,6 +171,13 @@ public static final String FS_TRASH_INTERVAL_KEY = "fs.trash.interval"; /** Default value for FS_TRASH_INTERVAL_KEY */ public static final longFS_TRASH_INTERVAL_DEFAULT = 0; + + public static final String MOVE_TO_TRASH_FOR_TEST_KEY = "move.to.trash.for.tests"; Review comment: @jojochuang MOVE_TO_TRASH_FOR_TEST_KEY， This configuration is used as a switch. The default is off. It is only turned on when the unit is tested. It is the logic that is executed in the unit test. Other people write the same.What better advice do you have?Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on issue #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing.
xiaoyuyao commented on issue #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing. URL: https://github.com/apache/hadoop/pull/1260#issuecomment-519994319 LGTM, +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] leosunli commented on issue #1223: HADOOP-16112. Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
leosunli commented on issue #1223: HADOOP-16112. Delete the baseTrashPath's subDir leads to don't modify baseTrashPath URL: https://github.com/apache/hadoop/pull/1223#issuecomment-519994186 @jojochuang I updated this patch and uploaded it. Could you help review it? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota
sunchao commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota URL: https://github.com/apache/hadoop/pull/1253#issuecomment-519992649 Hmm. Seems something wrong with Yetus? most of the tests failed with: `java.lang.OutOfMemoryError: unable to create new native thread` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix OzoneBucket and RpcClient APIS for acl.
xiaoyuyao commented on a change in pull request #1257: HDDS-1913. Fix
OzoneBucket and RpcClient APIS for acl.
URL: https://github.com/apache/hadoop/pull/1257#discussion_r312567041
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
##
@@ -272,16 +272,6 @@ public void setBucketProperty(OmBucketArgs args) throws
IOException {
.setBucketName(oldBucketInfo.getBucketName());
bucketInfoBuilder.addAllMetadata(args.getMetadata());
- //Check ACLs to update
- if (args.getAddAcls() != null || args.getRemoveAcls() != null) {
-bucketInfoBuilder.setAcls(getUpdatedAclList(oldBucketInfo.getAcls(),
-args.getRemoveAcls(), args.getAddAcls()));
-LOG.debug("Updating ACLs for bucket: {} in volume: {}",
-bucketName, volumeName);
- } else {
-bucketInfoBuilder.setAcls(oldBucketInfo.getAcls());
Review comment:
We need to keep bucketInfoBuilder.setAcls(oldBucketInfo.getAcls()) otherwise
existing acls will be lost upon setBucketProperty. Can you add a unit test to
cover this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1209: HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
steveloughran commented on issue #1209: HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region URL: https://github.com/apache/hadoop/pull/1209#issuecomment-519986872 thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing.
hadoop-yetus commented on issue #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing. URL: https://github.com/apache/hadoop/pull/1260#issuecomment-519973075 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 578 | trunk passed | | +1 | compile | 355 | trunk passed | | +1 | checkstyle | 78 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 885 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 158 | trunk passed | | 0 | spotbugs | 497 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 718 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 578 | the patch passed | | +1 | compile | 381 | the patch passed | | +1 | javac | 381 | the patch passed | | +1 | checkstyle | 74 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 675 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 160 | the patch passed | | +1 | findbugs | 698 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 302 | hadoop-hdds in the patch passed. | | -1 | unit | 2541 | hadoop-ozone in the patch failed. | | +1 | asflicense | 51 | The patch does not generate ASF License warnings. | | | | 8423 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestKeyManagerImpl | | | hadoop.ozone.dn.scrubber.TestDataScrubber | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.om.TestOzoneManagerHA | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1260/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1260 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c2c2841ce499 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 43a91f8 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1260/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1260/1/testReport/ | | Max. process+thread count | 3609 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1260/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16481) ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
[
https://issues.apache.org/jira/browse/HADOOP-16481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903997#comment-16903997
]
Hudson commented on HADOOP-16481:
-
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17076 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17076/])
HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune
(gabor.bota: rev 189dc108841a5de146ba8faa049fcd174bb3e7bf)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardDDBRootOperations.java
> ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
> -
>
> Key: HADOOP-16481
> URL: https://issues.apache.org/jira/browse/HADOOP-16481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> The new test {{ITestS3GuardDDBRootOperations.test_300_MetastorePrune}} fails
> if you don't explicitly set the region
> {code}
> [ERROR]
> test_300_MetastorePrune(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations)
> Time elapsed: 0.845 s <<< ERROR!
> org.apache.hadoop.util.ExitUtil$ExitException: No region found from -region
> flag, config, or S3 bucket
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations.test_300_MetastorePrune(ITestS3GuardDDBRootOperations.java:186)
> {code}
> it should be picked up from the test filesystem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16481) ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
[
https://issues.apache.org/jira/browse/HADOOP-16481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903982#comment-16903982
]
Gabor Bota commented on HADOOP-16481:
-
thanks for working on this [~ste...@apache.org]; +1; committed {{GitHub Pull
Request #1209}} to trunk.
> ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
> -
>
> Key: HADOOP-16481
> URL: https://issues.apache.org/jira/browse/HADOOP-16481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> The new test {{ITestS3GuardDDBRootOperations.test_300_MetastorePrune}} fails
> if you don't explicitly set the region
> {code}
> [ERROR]
> test_300_MetastorePrune(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations)
> Time elapsed: 0.845 s <<< ERROR!
> org.apache.hadoop.util.ExitUtil$ExitException: No region found from -region
> flag, config, or S3 bucket
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations.test_300_MetastorePrune(ITestS3GuardDDBRootOperations.java:186)
> {code}
> it should be picked up from the test filesystem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16481) ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
[
https://issues.apache.org/jira/browse/HADOOP-16481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16481.
-
Resolution: Fixed
Fix Version/s: 3.3.0
Target Version/s: 3.3.0
> ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
> -
>
> Key: HADOOP-16481
> URL: https://issues.apache.org/jira/browse/HADOOP-16481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> The new test {{ITestS3GuardDDBRootOperations.test_300_MetastorePrune}} fails
> if you don't explicitly set the region
> {code}
> [ERROR]
> test_300_MetastorePrune(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations)
> Time elapsed: 0.845 s <<< ERROR!
> org.apache.hadoop.util.ExitUtil$ExitException: No region found from -region
> flag, config, or S3 bucket
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations.test_300_MetastorePrune(ITestS3GuardDDBRootOperations.java:186)
> {code}
> it should be picked up from the test filesystem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg merged pull request #1209: HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
bgaborg merged pull request #1209: HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region URL: https://github.com/apache/hadoop/pull/1209 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16499) S3A retry policy to be exponential
[ https://issues.apache.org/jira/browse/HADOOP-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16499. - Resolution: Fixed Fix Version/s: 3.3.0 Target Version/s: 3.3.0 > S3A retry policy to be exponential > -- > > Key: HADOOP-16499 > URL: https://issues.apache.org/jira/browse/HADOOP-16499 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Fix For: 3.3.0 > > > the fixed s3a retry policy doesnt leave big enough gaps for cached 404s to > expire; we cant recover from this > HADOOP-16490 is a full fix for this, but one we can backport is moving from > fixed to exponential retries -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1209: HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region
bgaborg commented on issue #1209: HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region URL: https://github.com/apache/hadoop/pull/1209#issuecomment-519963174 +1; will commit now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13430) Optimize getFileStatus in S3A
[ https://issues.apache.org/jira/browse/HADOOP-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903962#comment-16903962 ] Steve Loughran commented on HADOOP-13430: - # for create file we can skip the GET object when overwrite=true # for list operations we should attempt the list and only fall back to the HEAD calls if that fails item #2 should make a big difference to treewalks > Optimize getFileStatus in S3A > - > > Key: HADOOP-13430 > URL: https://issues.apache.org/jira/browse/HADOOP-13430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steven K. Wong >Assignee: Steven K. Wong >Priority: Minor > Attachments: HADOOP-13430.001.WIP.patch > > > Currently, S3AFileSystem.getFileStatus(Path f) sends up to 3 requests to S3 > when pathToKey(f) = key = "foo/bar" is a directory: > 1. HEAD key=foo/bar \[continue if not found] > 2. HEAD key=foo/bar/ \[continue if not found] > 3. LIST prefix=foo/bar/ delimiter=/ max-keys=1 > My experience (and generally true, I reckon) is that almost all directories > are nonempty directories without a "fake directory" file (e.g. "foo/bar/"). > Under this condition, request #2 is mostly unhelpful; it only slows down > getFileStatus. Therefore, I propose swapping the order of requests #2 and #3. > The swapped HEAD request will be skipped in practically all cases. > Furthermore, when key = "foo/bar" is a nonempty directory that contains a > "fake directory" file (in addition to actual files), getFileStatus currently > returns an S3AFileStatus with isEmptyDirectory=true, which is wrong. Swapping > will fix this. The swapped LIST request will use max-keys=2 to determine > isEmptyDirectory correctly. (Removing the delimiter from the LIST request > should make the logic a little simpler than otherwise.) > Note that key = "foo/bar/" has the same problem with isEmptyDirectory. To fix > it, I propose skipping request #1 when key ends with "/". The price is this > will, for an empty directory, replace a HEAD request with a LIST request > that's generally more taxing on S3. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on a change in pull request #1234: HDDS-1917. Ignore failing test-cases in TestSecureOzoneRpcClient.
nandakumar131 commented on a change in pull request #1234: HDDS-1917. Ignore
failing test-cases in TestSecureOzoneRpcClient.
URL: https://github.com/apache/hadoop/pull/1234#discussion_r312508108
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##
@@ -2280,6 +2280,7 @@ public void testNativeAclsForVolume() throws Exception {
validateOzoneAccessAcl(ozObj);
}
+ @Ignore("This will be fixed when HA support is added to acl operations")
Review comment:
Thanks @bharatviswa504 and @xiaoyuyao for the review and suggestions.
All the tests which are extending `TestOzoneRpcClientAbstract` is failing
with the same error, but the failure is seen only in the pre-commit jobs (not
able to repro them in local).
Ref:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1211/5/artifact/out/patch-unit-hadoop-ozone.txt
I will modify the test-cases to get more info whenever they fail instead of
Ignoring them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16499) S3A retry policy to be exponential
[ https://issues.apache.org/jira/browse/HADOOP-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903925#comment-16903925 ] Hudson commented on HADOOP-16499: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17075 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17075/]) HADOOP-16499. S3A retry policy to be exponential (#1246). Contributed by (gabor.bota: rev e25a5c2eab2ac5bfaf795d4f211a24e8b73448ae) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AInconsistency.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ADelayedFNF.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/TestStagingCommitter.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java > S3A retry policy to be exponential > -- > > Key: HADOOP-16499 > URL: https://issues.apache.org/jira/browse/HADOOP-16499 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > the fixed s3a retry policy doesnt leave big enough gaps for cached 404s to > expire; we cant recover from this > HADOOP-16490 is a full fix for this, but one we can backport is moving from > fixed to exponential retries -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16499) S3A retry policy to be exponential
[ https://issues.apache.org/jira/browse/HADOOP-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903905#comment-16903905 ] Gabor Bota commented on HADOOP-16499: - +1 on GitHub Pull Request #1246; committing. > S3A retry policy to be exponential > -- > > Key: HADOOP-16499 > URL: https://issues.apache.org/jira/browse/HADOOP-16499 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > the fixed s3a retry policy doesnt leave big enough gaps for cached 404s to > expire; we cant recover from this > HADOOP-16490 is a full fix for this, but one we can backport is moving from > fixed to exponential retries -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg merged pull request #1246: HADOOP-16499. S3A retry policy to be exponential
bgaborg merged pull request #1246: HADOOP-16499. S3A retry policy to be exponential URL: https://github.com/apache/hadoop/pull/1246 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adamantal opened a new pull request #1261: YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl…
adamantal opened a new pull request #1261: YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl… URL: https://github.com/apache/hadoop/pull/1261 … and connected classes Also removed cleanupOldLogTimes and unused methods. Looks to me that `AggregatedLogFormat` and `LogAggregationFileController` has proper logging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 opened a new pull request #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing.
nandakumar131 opened a new pull request #1260: HDDS-1906. TestScmSafeMode#testSCMSafeModeRestrictedOp is failing. URL: https://github.com/apache/hadoop/pull/1260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg edited a comment on issue #1246: HADOOP-16499. S3A retry policy to be exponential
bgaborg edited a comment on issue #1246: HADOOP-16499. S3A retry policy to be exponential URL: https://github.com/apache/hadoop/pull/1246#issuecomment-519913905 seems good to me, +1; will commit soon. (test failure is a separate issue) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1246: HADOOP-16499. S3A retry policy to be exponential
bgaborg commented on issue #1246: HADOOP-16499. S3A retry policy to be exponential URL: https://github.com/apache/hadoop/pull/1246#issuecomment-519913905 seems good to me, +1; will commit soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on issue #1218: HDDS-1891. Ozone fs shell command should work with default port when port number is not specified
smengcl commented on issue #1218: HDDS-1891. Ozone fs shell command should work with default port when port number is not specified URL: https://github.com/apache/hadoop/pull/1218#issuecomment-519896335 Thanks for the comment @adoroszlai ! Just posted a new commit to address both checkstyle issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on a change in pull request #1218: HDDS-1891. Ozone fs shell command should work with default port when port number is not specified
smengcl commented on a change in pull request #1218: HDDS-1891. Ozone fs shell command should work with default port when port number is not specified URL: https://github.com/apache/hadoop/pull/1218#discussion_r312451506 ## File path: hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java ## @@ -54,6 +55,8 @@ import static org.apache.hadoop.fs.ozone.Constants.OZONE_USER_DIR; import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_DELIMITER; import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_SCHEME; +import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_PORT_DEFAULT; Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on a change in pull request #1218: HDDS-1891. Ozone fs shell command should work with default port when port number is not specified
smengcl commented on a change in pull request #1218: HDDS-1891. Ozone fs shell
command should work with default port when port number is not specified
URL: https://github.com/apache/hadoop/pull/1218#discussion_r312451547
##
File path:
hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemWithMocks.java
##
@@ -78,6 +79,42 @@ public void testFSUriWithHostPortOverrides() throws
Exception {
OzoneClientFactory.getRpcClient("local.host", 5899, conf);
}
+ @Test
+ public void testFSUriWithHostPortUnspecified() throws Exception {
+Configuration conf = new OzoneConfiguration();
+final int OM_PORT_DEFAULT = OmUtils.getOmRpcPort(conf);
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16315) ABFS: transform full UPN for named user in AclStatus
[ https://issues.apache.org/jira/browse/HADOOP-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903836#comment-16903836 ] Hudson commented on HADOOP-16315: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17074/]) HADOOP-16315. ABFS: transform full UPN for named user in AclStatus (stevel: rev 43a91f820a5fce75ea69f78a62331bdc58e09a37) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java > ABFS: transform full UPN for named user in AclStatus > > > Key: HADOOP-16315 > URL: https://issues.apache.org/jira/browse/HADOOP-16315 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.3.0 > > > When converting the identity in AclStatus, only "owner" and "owning group" > are transformed. We need to add the conversion for the AclEntry in AclStatus > too. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
