[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941538#comment-16941538
]
Vinayakumar B commented on HADOOP-13363:
If you are specifically pointing to above mentioned Spark/Hive's case, I could
compile the spark from its root, by passing {{-Dprotobuf.version=3.7.1}}.
In Spark's root pom.xml protobuf-2.5.0 is explicitly added as dependency to
avoid problem in maven auto resolution to different version. As per comments
this dependency is only for hadoop/yarn.
{quote} If we are passing protobuf types around, either as inputs or outputs.
Then we cannot shade the artifacts. And so: we cannot update protobuf
"transparently".
{color:#172b4d}For Token, if that toProbuf method is considered private, well,
we could move the operation out into its own class and invoke passing in a
Token instance, but it is probably a symptom of a bigger problem -protobuf
types in public APIs{color}
{quote}
IMO, Token's API change will not be problematic in normal cases.
{{TokenProto}} is a generated class, whose package will not change, but its
super class changes to {{GeneratedMessageV3}} from {{GeneratedMessage}}.
Direct usages of {{TokenProto}} or its {{Builder}} will not be a problem. It
can be problematic if application is embedding tokenproto in some other proto
message, for which code is generated using 2.5.0, or using common
{{GeneratedMessage}} as type for {{TokenProto}}. But this is unlikely as this
feature is only provided in Hadoop-3.0.0 onwards.
Otherwise, I think it won't be a problem.
> Upgrade protobuf from 2.5.0 to something newer
> --
>
> Key: HADOOP-13363
> URL: https://issues.apache.org/jira/browse/HADOOP-13363
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Anu Engineer
>Assignee: Vinayakumar B
>Priority: Major
> Labels: security
> Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch,
> HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch
>
>
> Standard protobuf 2.5.0 does not work properly on many platforms. (See, for
> example, https://gist.github.com/BennettSmith/7111094 ). In order for us to
> avoid crazy work arounds in the build environment and the fact that 2.5.0 is
> starting to slowly disappear as a standard install-able package for even
> Linux/x86, we need to either upgrade or self bundle or something else.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a
hadoop-yetus commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a URL: https://github.com/apache/hadoop/pull/1516#issuecomment-536853958 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2097 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1223 | trunk passed | | +1 | compile | 32 | trunk passed | | +1 | checkstyle | 25 | trunk passed | | +1 | mvnsite | 35 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | trunk passed | | 0 | spotbugs | 58 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 56 | trunk passed | | -0 | patch | 79 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 32 | the patch passed | | +1 | compile | 26 | the patch passed | | +1 | javac | 26 | the patch passed | | -0 | checkstyle | 18 | hadoop-tools/hadoop-aws: The patch generated 11 new + 10 unchanged - 2 fixed = 21 total (was 12) | | +1 | mvnsite | 31 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 850 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | +1 | findbugs | 62 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 70 | hadoop-aws in the patch passed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 5582 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1516 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 03eee379ce5d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/3/testReport/ | | Max. process+thread count | 357 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1517: HDDS-2169
hadoop-yetus commented on issue #1517: HDDS-2169 URL: https://github.com/apache/hadoop/pull/1517#issuecomment-536845581 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 96 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 34 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 59 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 971 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 18 | hadoop-hdds in trunk failed. | | -1 | javadoc | 16 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1058 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 31 | hadoop-hdds in trunk failed. | | -1 | findbugs | 17 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for patch | | -1 | mvninstall | 33 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 36 | hadoop-ozone in the patch failed. | | -1 | compile | 21 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 21 | hadoop-hdds in the patch failed. | | -1 | javac | 17 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 54 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 779 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 28 | hadoop-hdds in the patch failed. | | -1 | findbugs | 18 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 24 | hadoop-hdds in the patch failed. | | -1 | unit | 23 | hadoop-ozone in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 2550 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1517 | | JIRA Issue | HDDS-2169 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f83947a622e3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/7/artifact/out/patch-compile-hadoop-hdds.txt | | javac |
[GitHub] [hadoop] hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API.
hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-536837796 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 68 | Maven dependency ordering for branch | | -1 | mvninstall | 44 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 39 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 14 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 955 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 38 | hadoop-hdds in trunk failed. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | -1 | mvninstall | 36 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 40 | hadoop-ozone in the patch failed. | | -1 | compile | 26 | hadoop-hdds in the patch failed. | | -1 | compile | 18 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-hdds in the patch failed. | | -1 | javac | 18 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 28 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 715 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 24 | hadoop-hdds in the patch failed. | | -1 | javadoc | 18 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 29 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 2467 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 08929fca86df 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac |
[GitHub] [hadoop] hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API.
hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-536837875 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 36 | hadoop-ozone in trunk failed. | | -1 | compile | 20 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 818 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 25 | hadoop-hdds in trunk failed. | | -1 | javadoc | 16 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 919 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 34 | hadoop-hdds in trunk failed. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | -1 | mvninstall | 37 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 39 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 16 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 25 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 707 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 30 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2360 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 883182c3dde4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac |
[GitHub] [hadoop] sidseth commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a
sidseth commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a URL: https://github.com/apache/hadoop/pull/1516#issuecomment-536837025 Tests run against a bucket in us-east-2 `mvn -T 1C clean verify -Dparallel-tests -DtestsThreadCount=12 -Ds3guard -Dauth -Ddynamo` Passed, except for the usual flaky tests (ITestDynamoDBMetadataStore.setUp:159 » IllegalArgument Table s3guard-sseth-in-...) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a
hadoop-yetus commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a URL: https://github.com/apache/hadoop/pull/1516#issuecomment-536836978 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1788 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1062 | trunk passed | | +1 | compile | 35 | trunk passed | | +1 | checkstyle | 27 | trunk passed | | +1 | mvnsite | 41 | trunk passed | | +1 | shadedclient | 793 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 31 | trunk passed | | 0 | spotbugs | 59 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 57 | trunk passed | | -0 | patch | 84 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 29 | the patch passed | | +1 | javac | 29 | the patch passed | | -0 | checkstyle | 20 | hadoop-tools/hadoop-aws: The patch generated 11 new + 10 unchanged - 2 fixed = 21 total (was 12) | | +1 | mvnsite | 33 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 777 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | the patch passed | | +1 | findbugs | 62 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 71 | hadoop-aws in the patch passed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 5020 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1516 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d6ba1c17f3d8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/2/testReport/ | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1516/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szetszwo commented on issue #1517: HDDS-2169
szetszwo commented on issue #1517: HDDS-2169 URL: https://github.com/apache/hadoop/pull/1517#issuecomment-536836755 > Thanks @szetszwo for working on this. With the patch, while running the tests in TestDataValidateWithUnsafeByteOperations, the below issue is observed. > > `2019-09-30 21:58:02,745 [grpc-default-executor-2] ERROR segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:(449)) - e4ab8454-30fe-420c-a1cf-40d223cb4898@group-D0335C23E8DA-SegmentedRaftLogWorker: writeStateMachineData failed for index 1, entry=(t:1, i:1), STATEMACHINELOGENTRY, client-6C45A0D09519, cid=8 java.lang.IndexOutOfBoundsException: End index: 135008824 >= 207 at org.apache.ratis.thirdparty.com.google.protobuf.ByteString.checkRange(ByteString.java:1233) at org.apache.ratis.thirdparty.com.google.protobuf.ByteString$LiteralByteString.substring(ByteString.java:1288) at org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage.toProto(ContainerCommandRequestMessage.java:66) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:375) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(ContainerStateMachine.java:494) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.(SegmentedRaftLogWorker.java:447) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.writeLogEntry(SegmentedRaftLogWorker.java:397) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:411) at org.apache.ratis.server.raftlog.RaftLog.lambda$appendEntry$10(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:77) at org.apache.ratis.server.raftlog.RaftLog.appendEntry(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLog.appendImpl(RaftLog.java:183) at org.apache.ratis.server.raftlog.RaftLog.lambda$append$2(RaftLog.java:159) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:68) at org.apache.ratis.server.raftlog.RaftLog.append(RaftLog.java:159) at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:282) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:505) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:576) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)` > > We need to change to code in
[GitHub] [hadoop] bharatviswa504 opened a new pull request #1555: HDDS-1984. Fix listBucket API.
bharatviswa504 opened a new pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sidseth commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a
sidseth commented on issue #1516: HADOOP-16599. Allow a SignerInitializer to be specified along with a URL: https://github.com/apache/hadoop/pull/1516#issuecomment-536819145 Updated PR based on the review comments, except for what is called out further up. Also adds unit tests and an integration test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-16531:
-
Fix Version/s: 3.2.2
3.1.4
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941426#comment-16941426
]
Wei-Chiu Chuang commented on HADOOP-16531:
--
Commit applies cleanly in branch-3.1 and branch-3.2.
I'm going to cherry pick the commit into these two branches.
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method
[
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-15865:
-
Fix Version/s: 3.2.2
3.1.4
> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of
> ConcurrentModificationException since we use iteration over Properties
> object.
> {code}
> private void overlay(Properties to, Properties from) {
> for (Entry entry: from.entrySet()) {
> to.put(entry.getKey(), entry.getValue());
> }
> }
> {code}
> Properties class is thread-safe but iterator is not. We should manually
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService:
> Service ResourceManager failed in state STARTED; cause:
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
> at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
> at
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
> at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
> at
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
> at
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
> removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
> Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on
> 8032
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1370: HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.
jojochuang commented on issue #1370: HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. URL: https://github.com/apache/hadoop/pull/1370#issuecomment-536806323 Checkstyle was addressed. @bshashikant i believe this is good. Test failures look unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method
[
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941425#comment-16941425
]
Wei-Chiu Chuang commented on HADOOP-15865:
--
Patch applies cleanly in branch-3.1 and branch-3.2.
Cherry picking the commit into these two branches.
> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of
> ConcurrentModificationException since we use iteration over Properties
> object.
> {code}
> private void overlay(Properties to, Properties from) {
> for (Entry entry: from.entrySet()) {
> to.put(entry.getKey(), entry.getValue());
> }
> }
> {code}
> Properties class is thread-safe but iterator is not. We should manually
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService:
> Service ResourceManager failed in state STARTED; cause:
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
> at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
> at
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
> at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
> at
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
> at
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
> removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
> Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on
> 8032
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager.
anuengineer commented on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager. URL: https://github.com/apache/hadoop/pull/1538#issuecomment-536804802 Can you please add a test case that proves that RocksDB actually produces logs that we can see. There is a log listener class in the Hadoop. You can use them, or create a test and then grep for some of the log statements. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15014) KMS should log the IP address of the clients
[ https://issues.apache.org/jira/browse/HADOOP-15014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-15014: - Fix Version/s: 3.2.2 3.1.4 > KMS should log the IP address of the clients > > > Key: HADOOP-15014 > URL: https://issues.apache.org/jira/browse/HADOOP-15014 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Affects Versions: 2.8.1 >Reporter: Zsombor Gegesy >Assignee: Zsombor Gegesy >Priority: Major > Labels: kms, log > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HADOOP-15014.patch > > > Currently KMSMDCFilter only captures http request url and method, but not the > remote address of the client. > Storing this information in a thread local variable would help external > authorizer plugins to do more thorough checks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer edited a comment on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager.
anuengineer edited a comment on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager. URL: https://github.com/apache/hadoop/pull/1538#issuecomment-536804802 Can you please add a test case that proves that RocksDB actually produces logs that we can see. There is a log listener class in the Hadoop. You can use them, or create a test and then grep for some of the log statements. Otherwise the change looks quite good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15014) KMS should log the IP address of the clients
[ https://issues.apache.org/jira/browse/HADOOP-15014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941424#comment-16941424 ] Wei-Chiu Chuang commented on HADOOP-15014: -- Commits apply cleanly in branch-3.2 and branch-3.1. I'm going to cherry pick them into these two branches. > KMS should log the IP address of the clients > > > Key: HADOOP-15014 > URL: https://issues.apache.org/jira/browse/HADOOP-15014 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Affects Versions: 2.8.1 >Reporter: Zsombor Gegesy >Assignee: Zsombor Gegesy >Priority: Major > Labels: kms, log > Fix For: 3.3.0 > > Attachments: HADOOP-15014.patch > > > Currently KMSMDCFilter only captures http request url and method, but not the > remote address of the client. > Storing this information in a thread local variable would help external > authorizer plugins to do more thorough checks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock
[
https://issues.apache.org/jira/browse/HADOOP-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-16461:
-
Fix Version/s: 3.2.2
3.1.4
> Regression: FileSystem cache lock parses XML within the lock
>
>
> Key: HADOOP-16461
> URL: https://issues.apache.org/jira/browse/HADOOP-16461
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.3.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
>
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3388
> {code}
> fs = createFileSystem(uri, conf);
> synchronized (this) { // refetch the lock again
> FileSystem oldfs = map.get(key);
> if (oldfs != null) { // a file system is created while lock is
> releasing
> fs.close(); // close the new file system
> return oldfs; // return the old file system
> }
> // now insert the new file system into the map
> if (map.isEmpty()
> && !ShutdownHookManager.get().isShutdownInProgress()) {
> ShutdownHookManager.get().addShutdownHook(clientFinalizer,
> SHUTDOWN_HOOK_PRIORITY);
> }
> fs.key = key;
> map.put(key, fs);
> if (conf.getBoolean(
> FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
> toAutoClose.add(key);
> }
> return fs;
> }
> {code}
> The lock now has a ShutdownHook creation, which ends up doing
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L205
> {code}
> HookEntry(Runnable hook, int priority) {
> this(hook, priority,
> getShutdownTimeout(new Configuration()),
> TIME_UNIT_DEFAULT);
> }
> {code}
> which ends up doing a "new Configuration()" within the locked section.
> This indirectly hurts the cache hit scenarios as well, since if the lock on
> this is held, then the other section cannot be entered either.
> https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/TezSpillRecord.java#L65
> {code}
> I/O Setup 0 State: BLOCKED CPU usage on sample: 6ms
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(URI, Configuration,
> FileSystem$Cache$Key) FileSystem.java:3345
> org.apache.hadoop.fs.FileSystem$Cache.get(URI, Configuration)
> FileSystem.java:3320
> org.apache.hadoop.fs.FileSystem.get(URI, Configuration) FileSystem.java:479
> org.apache.hadoop.fs.FileSystem.getLocal(Configuration) FileSystem.java:435
> {code}
> slowing down the RawLocalFileSystem when there are other threads creating
> HDFS FileSystem objects at the same time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock
[
https://issues.apache.org/jira/browse/HADOOP-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941416#comment-16941416
]
Wei-Chiu Chuang commented on HADOOP-16461:
--
Commit applies to branch-3.2 and branch-3.1.
I'll cherry pick the commit into these two branches.
> Regression: FileSystem cache lock parses XML within the lock
>
>
> Key: HADOOP-16461
> URL: https://issues.apache.org/jira/browse/HADOOP-16461
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.3.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Fix For: 3.3.0
>
>
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3388
> {code}
> fs = createFileSystem(uri, conf);
> synchronized (this) { // refetch the lock again
> FileSystem oldfs = map.get(key);
> if (oldfs != null) { // a file system is created while lock is
> releasing
> fs.close(); // close the new file system
> return oldfs; // return the old file system
> }
> // now insert the new file system into the map
> if (map.isEmpty()
> && !ShutdownHookManager.get().isShutdownInProgress()) {
> ShutdownHookManager.get().addShutdownHook(clientFinalizer,
> SHUTDOWN_HOOK_PRIORITY);
> }
> fs.key = key;
> map.put(key, fs);
> if (conf.getBoolean(
> FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
> toAutoClose.add(key);
> }
> return fs;
> }
> {code}
> The lock now has a ShutdownHook creation, which ends up doing
> https://github.com/apache/hadoop/blob/2546e6ece240924af2188bb39b3954a4896e4a4f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java#L205
> {code}
> HookEntry(Runnable hook, int priority) {
> this(hook, priority,
> getShutdownTimeout(new Configuration()),
> TIME_UNIT_DEFAULT);
> }
> {code}
> which ends up doing a "new Configuration()" within the locked section.
> This indirectly hurts the cache hit scenarios as well, since if the lock on
> this is held, then the other section cannot be entered either.
> https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/sort/impl/TezSpillRecord.java#L65
> {code}
> I/O Setup 0 State: BLOCKED CPU usage on sample: 6ms
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(URI, Configuration,
> FileSystem$Cache$Key) FileSystem.java:3345
> org.apache.hadoop.fs.FileSystem$Cache.get(URI, Configuration)
> FileSystem.java:3320
> org.apache.hadoop.fs.FileSystem.get(URI, Configuration) FileSystem.java:479
> org.apache.hadoop.fs.FileSystem.getLocal(Configuration) FileSystem.java:435
> {code}
> slowing down the RawLocalFileSystem when there are other threads creating
> HDFS FileSystem objects at the same time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager.
anuengineer commented on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager. URL: https://github.com/apache/hadoop/pull/1538#issuecomment-536796152 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1547: HDDS-1615. ManagedChannel references are being leaked in ReplicationS…
anuengineer commented on issue #1547: HDDS-1615. ManagedChannel references are being leaked in ReplicationS… URL: https://github.com/apache/hadoop/pull/1547#issuecomment-536795997 +1. LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer closed pull request #1548: HDDS-2205. checkstyle.sh reports wrong failure count
anuengineer closed pull request #1548: HDDS-2205. checkstyle.sh reports wrong failure count URL: https://github.com/apache/hadoop/pull/1548 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on issue #1548: HDDS-2205. checkstyle.sh reports wrong failure count
anuengineer commented on issue #1548: HDDS-2205. checkstyle.sh reports wrong failure count URL: https://github.com/apache/hadoop/pull/1548#issuecomment-536795581 Thank you for the contribution. I have committed this patch to the trunk. @dineshchitlangia Thank you for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-11245) Update NFS gateway to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-11245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng reassigned HADOOP-11245: --- Assignee: Siyao Meng > Update NFS gateway to use Netty4 > > > Key: HADOOP-11245 > URL: https://issues.apache.org/jira/browse/HADOOP-11245 > Project: Hadoop Common > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Siyao Meng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] shanyu commented on a change in pull request #1446: YARN-9834. Allow using a pool of local users to run Yarn Secure Conta…
shanyu commented on a change in pull request #1446: YARN-9834. Allow using a
pool of local users to run Yarn Secure Conta…
URL: https://github.com/apache/hadoop/pull/1446#discussion_r329827155
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
##
@@ -212,6 +213,15 @@ public ResourceLocalizationService(Dispatcher dispatcher,
this.delService = delService;
this.dirsHandler = dirsHandler;
+this.disablePrivateVis = UserGroupInformation.isSecurityEnabled() &&
+context.getConf().getBoolean(
+YarnConfiguration.NM_SECURE_MODE_USE_POOL_USER,
+YarnConfiguration.DEFAULT_NM_SECURE_MODE_USE_POOL_USER);
+if (this.disablePrivateVis) {
+ LOG.info("When " + YarnConfiguration.NM_SECURE_MODE_USE_POOL_USER +
Review comment:
Sure, will add comments. Basically if a real user application require
private resources, when local user pooling is enabled, we will treat the
private resources as application resources. This means for each application we
will download (localize) the resources to application folder, and will delete
it after the application is finished.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16588) Update commons-beanutils version to 1.9.4 in branch-2
[
https://issues.apache.org/jira/browse/HADOOP-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941380#comment-16941380
]
Jonathan Hung commented on HADOOP-16588:
[~iwasakims]'s suggestion makes sense to me. commons-beanutils-core was removed
as part of HADOOP-13660 which is incompatible, so we can just exclude
commons-beanutils-core.
> Update commons-beanutils version to 1.9.4 in branch-2
> -
>
> Key: HADOOP-16588
> URL: https://issues.apache.org/jira/browse/HADOOP-16588
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Critical
> Labels: release-blocker
> Attachments: HADOOP-16588.branch-2.001.patch
>
>
> Similar to HADOOP-16542 but we need to do it differently.
> In branch-2, we pull in commons-beanutils through commons-configuration 1.6
> --> commons-digester 1.8
> {noformat}
> [INFO] +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] | +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> {noformat}
> I have a patch to update version of the transitive dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1554: HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10
hadoop-yetus commented on issue #1554: HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10 URL: https://github.com/apache/hadoop/pull/1554#issuecomment-536781398 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 78 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1199 | trunk passed | | +1 | compile | 16 | trunk passed | | +1 | mvnsite | 20 | trunk passed | | +1 | shadedclient | 2032 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 18 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 12 | the patch passed | | +1 | compile | 12 | the patch passed | | +1 | javac | 12 | the patch passed | | +1 | mvnsite | 14 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 866 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 15 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 14 | hadoop-project in the patch passed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 3231 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1554/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1554 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux 7eeec70ba3d5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4d3c580 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1554/1/testReport/ | | Max. process+thread count | 338 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1554/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15849) Upgrade netty version to 3.10.6
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941374#comment-16941374 ] Masatake Iwasaki commented on HADOOP-15849: --- cherry-picked to branch-3.2 and branch-3.1. > Upgrade netty version to 3.10.6 > > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15849) Upgrade netty version to 3.10.6
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-15849: -- Fix Version/s: 3.2.2 3.1.4 > Upgrade netty version to 3.10.6 > > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941357#comment-16941357 ] Wei-Chiu Chuang commented on HADOOP-16579: -- bq. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. So we don't have to update org.apache.zookeeper. Good. That said, it would be a good idea to update zookeeper to 3.5.x. It gets rid of Netty 3, and we can gradually remove Netty 3 code in Hadoop: HADOOP-11219. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-11245) Update NFS gateway to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-11245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HADOOP-11245: --- Assignee: (was: Bharat Viswanadham) > Update NFS gateway to use Netty4 > > > Key: HADOOP-11245 > URL: https://issues.apache.org/jira/browse/HADOOP-11245 > Project: Hadoop Common > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HADOOP-15327: --- Assignee: (was: Bharat Viswanadham) > Upgrade MR ShuffleHandler to use Netty4 > --- > > Key: HADOOP-15327 > URL: https://issues.apache.org/jira/browse/HADOOP-15327 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Priority: Major > > This way, we can remove the dependencies on the netty3 (jboss.netty) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1554: HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10
jojochuang commented on issue #1554: HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10 URL: https://github.com/apache/hadoop/pull/1554#issuecomment-536759827 +1 pending Jenkins This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl opened a new pull request #1554: HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10
smengcl opened a new pull request #1554: HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10 URL: https://github.com/apache/hadoop/pull/1554 Compiled locally on my Mac. Confirmed that it uses 2.9.10 jars for `jackson-core`, `jackson-annotations`, `jackson-databind`, etc.: ``` $ cd hadoop-dist/target/hadoop-3.3.0-SNAPSHOT $ find . -name "*jackson*" ./share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.9.10.jar ./share/hadoop/yarn/lib/jackson-jaxrs-base-2.9.10.jar ./share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.9.10.jar ./share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar ./share/hadoop/common/lib/jackson-xc-1.9.13.jar ./share/hadoop/common/lib/jackson-core-2.9.10.jar ./share/hadoop/common/lib/jackson-core-asl-1.9.13.jar ./share/hadoop/common/lib/jackson-annotations-2.9.10.jar ./share/hadoop/common/lib/jackson-databind-2.9.10.jar ./share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar ./share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar ./share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar ./share/hadoop/hdfs/lib/jackson-core-2.9.10.jar ./share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar ./share/hadoop/hdfs/lib/jackson-annotations-2.9.10.jar ./share/hadoop/hdfs/lib/jackson-databind-2.9.10.jar ./share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16619) Upgrade jackson and jackson-databind to 2.9.10
[ https://issues.apache.org/jira/browse/HADOOP-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16619: Summary: Upgrade jackson and jackson-databind to 2.9.10 (was: Update jackson and jackson-databind to 2.9.10) > Upgrade jackson and jackson-databind to 2.9.10 > -- > > Key: HADOOP-16619 > URL: https://issues.apache.org/jira/browse/HADOOP-16619 > Project: Hadoop Common > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Two more CVEs (CVE-2019-16335 and CVE-2019-14540) are addressed in > jackson-databind 2.9.10. > For details see Jackson Release 2.9.10 [release > notes|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16619) Update jackson and jackson-databind to 2.9.10
[ https://issues.apache.org/jira/browse/HADOOP-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16619: Summary: Update jackson and jackson-databind to 2.9.10 (was: Update jackson-* and jackson-databind to 2.9.10) > Update jackson and jackson-databind to 2.9.10 > - > > Key: HADOOP-16619 > URL: https://issues.apache.org/jira/browse/HADOOP-16619 > Project: Hadoop Common > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Two more CVEs (CVE-2019-16335 and CVE-2019-14540) are addressed in > jackson-databind 2.9.10. > For details see Jackson Release 2.9.10 [release > notes|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11219) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/HADOOP-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941332#comment-16941332 ] Wei-Chiu Chuang commented on HADOOP-11219: -- Netty 3 is EOL and the last Netty 3 was released on 6/2016, more than 3 years old. We can't afford if Netty is found to have security vulnerability one day. Time to prioritize this work. > Upgrade to netty 4 > -- > > Key: HADOOP-11219 > URL: https://issues.apache.org/jira/browse/HADOOP-11219 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Major > > This is an umbrella jira to track the effort of upgrading to Netty 4. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16619) Update jackson-* and jackson-databind to 2.9.10
Siyao Meng created HADOOP-16619: --- Summary: Update jackson-* and jackson-databind to 2.9.10 Key: HADOOP-16619 URL: https://issues.apache.org/jira/browse/HADOOP-16619 Project: Hadoop Common Issue Type: Bug Reporter: Siyao Meng Assignee: Siyao Meng Two more CVEs (CVE-2019-16335 and CVE-2019-14540) are addressed in jackson-databind 2.9.10. For details see Jackson Release 2.9.10 [release notes|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1115: HADOOP-16207 testMR failures
hadoop-yetus commented on issue #1115: HADOOP-16207 testMR failures URL: https://github.com/apache/hadoop/pull/1115#issuecomment-536750811 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 79 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 13 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1248 | trunk passed | | +1 | compile | 38 | trunk passed | | +1 | checkstyle | 27 | trunk passed | | +1 | mvnsite | 43 | trunk passed | | +1 | shadedclient | 949 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 30 | trunk passed | | 0 | spotbugs | 66 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 64 | trunk passed | | -0 | patch | 88 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 34 | the patch passed | | -1 | javac | 34 | hadoop-tools_hadoop-aws generated 1 new + 15 unchanged - 1 fixed = 16 total (was 16) | | -0 | checkstyle | 19 | hadoop-tools/hadoop-aws: The patch generated 8 new + 10 unchanged - 4 fixed = 18 total (was 14) | | +1 | mvnsite | 30 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 933 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 33 | the patch passed | | +1 | findbugs | 71 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 78 | hadoop-aws in the patch passed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 3843 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/18/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1115 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 2ce16fddaa25 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4d3c580 | | Default Java | 1.8.0_222 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/18/artifact/out/diff-compile-javac-hadoop-tools_hadoop-aws.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/18/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/18/testReport/ | | Max. process+thread count | 330 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/18/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941321#comment-16941321 ] Wei-Chiu Chuang commented on HADOOP-16579: -- The changes looks good to me. Let me know once you have additional test results. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15911) Over-eager allocation in ByteBufferUtil.fallbackRead
[
https://issues.apache.org/jira/browse/HADOOP-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941313#comment-16941313
]
Jacques Nadeau commented on HADOOP-15911:
-
I believe this occurs when Orc is being read off of S3 using zero copy path.
> Over-eager allocation in ByteBufferUtil.fallbackRead
>
>
> Key: HADOOP-15911
> URL: https://issues.apache.org/jira/browse/HADOOP-15911
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Vanco Buca
>Priority: Major
>
> The heap-memory path of ByteBufferUtil.fallbackRead ([see master branch code
> here|[https://github.com/apache/hadoop/blob/a0da1ec01051108b77f86799dd5e97563b2a3962/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferUtil.java#L95])]
> massively overallocates memory when the underlying input stream returns data
> in smaller chunks. This happens on a regular basis when using the S3 input
> stream as input.
> The behavior is an O(N^2)-ish. In a recent debug session, we were trying to
> read 6MB, but getting 16K at a time. The code would:
> * allocate 16M, use the first 16K
> * allocate 16M - 16K, use the first 16K of that
> * allocate 16M - 32K, use the first 16K of that
> * (etc)
> The patch is simple. Here's the text version of the patch:
> {code}
> @@ -88,10 +88,17 @@ public final class ByteBufferUtil {
> buffer.flip();
>} else {
> buffer.clear();
> -int nRead = stream.read(buffer.array(),
> - buffer.arrayOffset(), maxLength);
> -if (nRead >= 0) {
> - buffer.limit(nRead);
> +int totalRead = 0;
> +while (totalRead < maxLength) {
> + final int nRead = stream.read(buffer.array(),
> +buffer.arrayOffset() + totalRead, maxLength - totalRead);
> + if (nRead <= 0) {
> +break;
> + }
> + totalRead += nRead;
> +}
> +if (totalRead >= 0) {
> + buffer.limit(totalRead);
>success = true;
> }
>}
> {code}
> so, essentially, do the same thing that the code in the direct memory path is
> doing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1552: HDDS-2210. ContainerStateMachine should not be marked unhealthy if ap…
hadoop-yetus commented on a change in pull request #1552: HDDS-2210.
ContainerStateMachine should not be marked unhealthy if ap…
URL: https://github.com/apache/hadoop/pull/1552#discussion_r329774113
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
##
@@ -270,12 +270,16 @@ public void persistContainerSet(OutputStream out) throws
IOException {
// container happens outside of Ratis.
IOUtils.write(builder.build().toByteArray(), out);
}
+
Review comment:
whitespace:end of line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1552: HDDS-2210. ContainerStateMachine should not be marked unhealthy if ap…
hadoop-yetus commented on issue #1552: HDDS-2210. ContainerStateMachine should not be marked unhealthy if ap… URL: https://github.com/apache/hadoop/pull/1552#issuecomment-536740211 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1773 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | -1 | mvninstall | 44 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 38 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-hdds in trunk failed. | | -1 | compile | 12 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 58 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 937 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in trunk failed. | | -1 | javadoc | 17 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1027 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 33 | hadoop-hdds in trunk failed. | | -1 | findbugs | 17 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for patch | | -1 | mvninstall | 33 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 35 | hadoop-ozone in the patch failed. | | -1 | compile | 21 | hadoop-hdds in the patch failed. | | -1 | compile | 15 | hadoop-ozone in the patch failed. | | -1 | javac | 21 | hadoop-hdds in the patch failed. | | -1 | javac | 15 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 53 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | -1 | whitespace | 0 | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | shadedclient | 800 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 30 | hadoop-hdds in the patch failed. | | -1 | findbugs | 17 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 24 | hadoop-hdds in the patch failed. | | -1 | unit | 24 | hadoop-ozone in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 4251 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1552 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1dedef292033 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4d3c580 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1552/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac |
[GitHub] [hadoop] hadoop-yetus commented on issue #1553: HDDS-2211. Collect docker logs if env fails to start
hadoop-yetus commented on issue #1553: HDDS-2211. Collect docker logs if env fails to start URL: https://github.com/apache/hadoop/pull/1553#issuecomment-536739509 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 86 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | shelldocs | 1 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 56 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 43 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 891 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 815 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 2182 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1553 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs | | uname | Linux 30685cfa644f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4d3c580 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/testReport/ | | Max. process+thread count | 307 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/dist U: hadoop-ozone/dist | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1553/1/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16544) update io.netty in branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-16544: -- Fix Version/s: 2.10.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thanks. I committed this to branch-2. > update io.netty in branch-2 > --- > > Key: HADOOP-16544 > URL: https://issues.apache.org/jira/browse/HADOOP-16544 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Masatake Iwasaki >Priority: Major > Labels: release-blocker > Fix For: 2.10.0 > > Attachments: HADOOP-16544-branch-2.001.patch, > HADOOP-16544-branch-2.002.patch, HADOOP-16544-branch-2.003.patch, > HADOOP-16544-branch-2.004.patch > > > branch-2 pulls in io.netty 3.6.2.Final which is more than 5 years old. > The latest is 3.10.6Final. I know updating netty is sensitive but it deserves > some attention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
hadoop-yetus commented on issue #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows URL: https://github.com/apache/hadoop/pull/1481#issuecomment-536736409 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 47 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1327 | trunk passed | | +1 | compile | 34 | trunk passed | | +1 | checkstyle | 21 | trunk passed | | +1 | mvnsite | 35 | trunk passed | | +1 | shadedclient | 972 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | trunk passed | | 0 | spotbugs | 58 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 54 | trunk passed | | -0 | patch | 80 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 30 | the patch passed | | +1 | compile | 25 | the patch passed | | +1 | javac | 25 | the patch passed | | +1 | checkstyle | 18 | the patch passed | | +1 | mvnsite | 28 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 792 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | +1 | findbugs | 57 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 82 | hadoop-azure in the patch passed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 3741 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1481 | | JIRA Issue | HADOOP-16587 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d223bbc06f31 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4d3c580 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/3/testReport/ | | Max. process+thread count | 463 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[
https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941306#comment-16941306
]
Hadoop QA commented on HADOOP-16587:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
58s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:orange}-0{color} | {color:orange} patch {color} | {color:orange} 1m
20s{color} | {color:orange} Used diff version of patch file. Binary files and
potentially other changes not applied. Please rebase and squash commits if
necessary. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1481/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1481 |
| JIRA Issue | HADOOP-16587 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d223bbc06f31 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 4d3c580 |
| Default Java |
[jira] [Commented] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[
https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941300#comment-16941300
]
Hadoop QA commented on HADOOP-16587:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | HADOOP-16587 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981827/HADOOP-16587.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 20af344010e3 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4d3c580 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16561/testReport/ |
| Max. process+thread count | 318 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16561/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make AAD endpoint configurable on all Auth flows
>
[GitHub] [hadoop] hadoop-yetus commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host
hadoop-yetus commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host URL: https://github.com/apache/hadoop/pull/1551#issuecomment-536728763 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 50 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 843 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in trunk failed. | | -1 | javadoc | 21 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 943 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 33 | hadoop-hdds in trunk failed. | | -1 | findbugs | 20 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 36 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 35 | hadoop-ozone in the patch failed. | | -1 | compile | 23 | hadoop-hdds in the patch failed. | | -1 | compile | 18 | hadoop-ozone in the patch failed. | | -1 | javac | 23 | hadoop-hdds in the patch failed. | | -1 | javac | 18 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 703 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 31 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 29 | hadoop-hdds in the patch failed. | | -1 | unit | 27 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2320 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1551 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 59d2934bbdb9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4d3c580 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/2/artifact/out/patch-compile-hadoop-ozone.txt
[GitHub] [hadoop] adoroszlai commented on issue #1553: HDDS-2211. Collect docker logs if env fails to start
adoroszlai commented on issue #1553: HDDS-2211. Collect docker logs if env fails to start URL: https://github.com/apache/hadoop/pull/1553#issuecomment-536723744 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai opened a new pull request #1553: HDDS-2211. Collect docker logs if env fails to start
adoroszlai opened a new pull request #1553: HDDS-2211. Collect docker logs if env fails to start URL: https://github.com/apache/hadoop/pull/1553 ## What changes were proposed in this pull request? 1. Collect docker logs if environment fails to start up (previously it was collected only after actual test run) 2. Copy docker logs to aggregate result directory 3. Fail fast if datanodes cannot be started 4. Avoid [ANSI codes in output](https://github.com/elek/ozone-ci-q4/blob/f700ebb2254527527960593496f15008245094d6/trunk/trunk-nightly-extra-20190930-74rp4/acceptance/output.log#L3774) 5. Sort test cases for consistent run ordering (Robot summary table is already sorted) https://issues.apache.org/jira/browse/HDDS-2211 ## How was this patch tested? Ran `acceptance.sh` locally. Also simulated failure during datanode startup, verified that docker log is saved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set DtService of token in S3Gateway for OM HA.
xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set
DtService of token in S3Gateway for OM HA.
URL: https://github.com/apache/hadoop/pull/1489#discussion_r329748308
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneServiceProvider.java
##
@@ -20,33 +20,75 @@
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.ozone.OmUtils;
+import org.apache.hadoop.ozone.s3.util.OzoneS3Util;
import org.apache.hadoop.security.SecurityUtil;
import javax.annotation.PostConstruct;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.inject.Produces;
import javax.inject.Inject;
+
+import java.util.Arrays;
+import java.util.Collection;
+
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_NODES_KEY;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY;
+
/**
* This class creates the OM service .
*/
@ApplicationScoped
public class OzoneServiceProvider {
- private Text omServiceAdd;
+ private Text omServiceAddr;
+
+ private String omserviceID;
@Inject
private OzoneConfiguration conf;
@PostConstruct
public void init() {
-omServiceAdd = SecurityUtil.buildTokenService(OmUtils.
-getOmAddressForClients(conf));
+Collection serviceIdList =
+conf.getTrimmedStringCollection(OZONE_OM_SERVICE_IDS_KEY);
+if (serviceIdList.size() == 0) {
+ // Non-HA cluster
+ omServiceAddr = SecurityUtil.buildTokenService(OmUtils.
+ getOmAddressForClients(conf));
+} else {
+ // HA cluster.
+ //For now if multiple service id's are configured we throw exception.
Review comment:
This is very similar to what HDFS HA does. Since OM support service
discovery, can we get the internal service id via service discovery?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set DtService of token in S3Gateway for OM HA.
xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set
DtService of token in S3Gateway for OM HA.
URL: https://github.com/apache/hadoop/pull/1489#discussion_r329746422
##
File path:
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestOzoneS3Util.java
##
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.s3.util;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.OmUtils;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Collection;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP;
+import static org.junit.Assert.fail;
+
+/**
+ * Class used to test OzoneS3Util.
+ */
+public class TestOzoneS3Util {
+
+
+ private OzoneConfiguration configuration;
+
+ @Before
+ public void setConf() {
+configuration = new OzoneConfiguration();
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+ }
+
+ @Test
+ public void testBuildServiceNameForToken() {
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+
+Collection nodeIDList = configuration.getStringCollection(
+OMConfigKeys.OZONE_OM_NODE_ID_KEY);
+
+String expectedOmServiceAddress = buildOMNodeAddresses(nodeIDList,
+serviceID);
+
+SecurityUtil.setConfiguration(configuration);
+String omserviceAddr = OzoneS3Util.buildServiceNameForToken(configuration,
+serviceID, nodeIDList);
+
+Assert.assertEquals(expectedOmServiceAddress, omserviceAddr);
+ }
+
+ @Test
+ public void testBuildServiceNameForTokenIncorrectConfig() {
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+
+Collection nodeIDList = configuration.getStringCollection(
+OMConfigKeys.OZONE_OM_NODE_ID_KEY);
+
+// Don't set om3 node rpc address.
+configuration.set(OmUtils.addKeySuffixes(OMConfigKeys.OZONE_OM_ADDRESS_KEY,
+serviceID, "om1"), "om1:9862");
+configuration.set(OmUtils.addKeySuffixes(OMConfigKeys.OZONE_OM_ADDRESS_KEY,
+serviceID, "om2"), "om2:9862");
+
+
+SecurityUtil.setConfiguration(configuration);
+
+try {
+ OzoneS3Util.buildServiceNameForToken(configuration,
+ serviceID, nodeIDList);
+ fail("testBuildServiceNameForTokenIncorrectConfig failed");
+} catch (IllegalArgumentException ex) {
+ GenericTestUtils.assertExceptionContains("Could not find rpcAddress " +
+ "for", ex);
+}
+
+
+ }
+
+
+ private String buildOMNodeAddresses(Collection nodeIDList,
+ String serviceID) {
+StringBuilder omServiceAddrBuilder = new StringBuilder();
+int port = 9862;
+int nodesLength = nodeIDList.size();
+int counter = 0;
+for (String nodeID : nodeIDList) {
+ counter++;
+ String addr = nodeID + ":" + port++;
+ configuration.set(OmUtils.addKeySuffixes(
+ OMConfigKeys.OZONE_OM_ADDRESS_KEY, serviceID, nodeID), addr);
Review comment:
NIT: This helper not only makes and returns the address, but also changes
the configuration implicitly. Suggest add some comments to help understand the
difference between the two test cases.
This is an automated message from the Apache Git Service.
To respond to the
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set DtService of token in S3Gateway for OM HA.
xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set
DtService of token in S3Gateway for OM HA.
URL: https://github.com/apache/hadoop/pull/1489#discussion_r329746422
##
File path:
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestOzoneS3Util.java
##
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.s3.util;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.OmUtils;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Collection;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP;
+import static org.junit.Assert.fail;
+
+/**
+ * Class used to test OzoneS3Util.
+ */
+public class TestOzoneS3Util {
+
+
+ private OzoneConfiguration configuration;
+
+ @Before
+ public void setConf() {
+configuration = new OzoneConfiguration();
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+ }
+
+ @Test
+ public void testBuildServiceNameForToken() {
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+
+Collection nodeIDList = configuration.getStringCollection(
+OMConfigKeys.OZONE_OM_NODE_ID_KEY);
+
+String expectedOmServiceAddress = buildOMNodeAddresses(nodeIDList,
+serviceID);
+
+SecurityUtil.setConfiguration(configuration);
+String omserviceAddr = OzoneS3Util.buildServiceNameForToken(configuration,
+serviceID, nodeIDList);
+
+Assert.assertEquals(expectedOmServiceAddress, omserviceAddr);
+ }
+
+ @Test
+ public void testBuildServiceNameForTokenIncorrectConfig() {
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+
+Collection nodeIDList = configuration.getStringCollection(
+OMConfigKeys.OZONE_OM_NODE_ID_KEY);
+
+// Don't set om3 node rpc address.
+configuration.set(OmUtils.addKeySuffixes(OMConfigKeys.OZONE_OM_ADDRESS_KEY,
+serviceID, "om1"), "om1:9862");
+configuration.set(OmUtils.addKeySuffixes(OMConfigKeys.OZONE_OM_ADDRESS_KEY,
+serviceID, "om2"), "om2:9862");
+
+
+SecurityUtil.setConfiguration(configuration);
+
+try {
+ OzoneS3Util.buildServiceNameForToken(configuration,
+ serviceID, nodeIDList);
+ fail("testBuildServiceNameForTokenIncorrectConfig failed");
+} catch (IllegalArgumentException ex) {
+ GenericTestUtils.assertExceptionContains("Could not find rpcAddress " +
+ "for", ex);
+}
+
+
+ }
+
+
+ private String buildOMNodeAddresses(Collection nodeIDList,
+ String serviceID) {
+StringBuilder omServiceAddrBuilder = new StringBuilder();
+int port = 9862;
+int nodesLength = nodeIDList.size();
+int counter = 0;
+for (String nodeID : nodeIDList) {
+ counter++;
+ String addr = nodeID + ":" + port++;
+ configuration.set(OmUtils.addKeySuffixes(
+ OMConfigKeys.OZONE_OM_ADDRESS_KEY, serviceID, nodeID), addr);
Review comment:
NIT: This helper not only make the address, but also change the
configuration explicitly. Suggest add some comments to help understand the
difference between the two test cases.
This is an automated message from the Apache Git Service.
To respond to the message, please log
[GitHub] [hadoop] bshashikant opened a new pull request #1552: HDDS-2210. ContainerStateMachine should not be marked unhealthy if ap…
bshashikant opened a new pull request #1552: HDDS-2210. ContainerStateMachine should not be marked unhealthy if ap… URL: https://github.com/apache/hadoop/pull/1552 …plyTransaction fails with closed container exception. ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[ https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16587: -- Attachment: HADOOP-16587.003.patch > Make AAD endpoint configurable on all Auth flows > > > Key: HADOOP-16587 > URL: https://issues.apache.org/jira/browse/HADOOP-16587 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Attachments: HADOOP-16587.001.patch, HADOOP-16587.002.patch, > HADOOP-16587.003.patch > > > Make AAD endpoint configurable on all Auth flows. Currently auth endpoint is > hard coded for refreshtoken flow and MSI flow. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set DtService of token in S3Gateway for OM HA.
xiaoyuyao commented on a change in pull request #1489: HDDS-2019. Handle Set
DtService of token in S3Gateway for OM HA.
URL: https://github.com/apache/hadoop/pull/1489#discussion_r329740302
##
File path:
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestOzoneS3Util.java
##
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.s3.util;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.OmUtils;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Collection;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP;
+import static org.junit.Assert.fail;
+
+/**
+ * Class used to test OzoneS3Util.
+ */
+public class TestOzoneS3Util {
+
+
+ private OzoneConfiguration configuration;
+
+ @Before
+ public void setConf() {
+configuration = new OzoneConfiguration();
+String serviceID = "omService";
+String nodeIDs = "om1,om2,om3";
+configuration.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
+configuration.set(OMConfigKeys.OZONE_OM_NODE_ID_KEY, nodeIDs);
+configuration.setBoolean(HADOOP_SECURITY_TOKEN_SERVICE_USE_IP, false);
+ }
+
+ @Test
+ public void testBuildServiceNameForToken() {
+String serviceID = "omService";
Review comment:
Line 55-59 is redundant since you have them in setConf before each test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16544) update io.netty in branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941250#comment-16941250 ] Wei-Chiu Chuang commented on HADOOP-16544: -- We should. > update io.netty in branch-2 > --- > > Key: HADOOP-16544 > URL: https://issues.apache.org/jira/browse/HADOOP-16544 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Masatake Iwasaki >Priority: Major > Labels: release-blocker > Attachments: HADOOP-16544-branch-2.001.patch, > HADOOP-16544-branch-2.002.patch, HADOOP-16544-branch-2.003.patch, > HADOOP-16544-branch-2.004.patch > > > branch-2 pulls in io.netty 3.6.2.Final which is more than 5 years old. > The latest is 3.10.6Final. I know updating netty is sensitive but it deserves > some attention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15684) triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens.
[ https://issues.apache.org/jira/browse/HADOOP-15684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-15684: - Labels: multi-sbnn (was: ) > triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException > happens. > > > Key: HADOOP-15684 > URL: https://issues.apache.org/jira/browse/HADOOP-15684 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 3.0.0-alpha1 >Reporter: Rong Tang >Assignee: Rong Tang >Priority: Critical > Labels: multi-sbnn > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: > 0001-RollEditLog-try-next-NN-when-exception-happens.patch, > HADOOP-15684.000.patch, HADOOP-15684.001.patch, HADOOP-15684.002.patch, > HADOOP-15684.003.patch, HADOOP-15684.004.patch, HADOOP-15684.005.patch, > hadoop--rollingUpgrade-SourceMachine001.log > > > When name node call triggerActiveLogRoll, and the cachedActiveProxy is a dead > name node, it will throws a ConnectTimeoutException, expected behavior is to > try next NN, but current logic doesn't do so, instead, it keeps trying the > dead, mistakenly take it as active. > > 2018-08-17 10:02:12,001 WARN [Edit log tailer] > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a > roll of the active NN > org.apache.hadoop.net.ConnectTimeoutException: Call From > SourceMachine001/SourceIP to001 TargetMachine001.ap.gbl:8020 failed on socket > timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 2 > millis timeout > org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:298) > > C:\Users\rotang>ping TargetMachine001 > Pinging TargetMachine001[TargetIP001] with 32 bytes of data: > Request timed out. > Request timed out. > Request timed out. > Request timed out. > Attachment is a log file saying how it repeatedly retries a dead name node, > and a fix patch. > I replaced the actual machine name/ip as SourceMachine001/SourceIP001 and > TargetMachine001/TargetIP001. > > How to Repro: > In a good running NNs, take down the active NN (don't let it come back during > test), and then the stand by NNs will keep trying dead (old active) NN, > because it is the cached one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs
adoroszlai commented on issue #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs URL: https://github.com/apache/hadoop/pull/1537#issuecomment-536689992 > datanodes in ozonesecure-mr are not started AFAIK > > https://github.com/elek/ozone-ci-q4/blob/master/pr/pr-hdds-2187-2nl4x/acceptance/output.log I think it's `ozone-recon`, not `ozonesecure-mr`: https://github.com/elek/ozone-ci-q4/blob/c4e58095c9584185c75d6c6af4721c33edba854a/pr/pr-hdds-2187-2nl4x/acceptance/output.log#L1453 I've seen this problem intermittently in earlier runs, eg: https://github.com/elek/ozone-ci-q4/blob/f700ebb2254527527960593496f15008245094d6/trunk/trunk-nightly-extra-20190930-74rp4/acceptance/output.log#L3765-L3768 Working on a change to collect the docker logs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941228#comment-16941228 ] Michael Stack commented on HADOOP-13363: bq. If we are passing protobuf types around, either as inputs or outputs. Then we cannot shade the artifacts. And so: we cannot update protobuf "transparently". bq. but it is probably a symptom of a bigger problem -protobuf types in public APIs If it helps, in hbase, we've been working to remove all mention of pb in API. A few instances remain still (though they'll be gone soon). For this small set of remainders, we use a different (unshaded) version of pb from that which we use internally. The internal usage relies on the relocated/shaded protobuf. In a few cases, we even have convertion that moves data/invocation-description from old pb to new (shaded) pb. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Anu Engineer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1115: HADOOP-16207 testMR failures
hadoop-yetus commented on issue #1115: HADOOP-16207 testMR failures URL: https://github.com/apache/hadoop/pull/1115#issuecomment-536681137 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2104 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 13 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1117 | trunk passed | | +1 | compile | 35 | trunk passed | | +1 | checkstyle | 25 | trunk passed | | +1 | mvnsite | 39 | trunk passed | | +1 | shadedclient | 752 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 28 | trunk passed | | 0 | spotbugs | 60 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 58 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 30 | the patch passed | | +1 | javac | 30 | the patch passed | | -0 | checkstyle | 21 | hadoop-tools/hadoop-aws: The patch generated 7 new + 13 unchanged - 1 fixed = 20 total (was 14) | | +1 | mvnsite | 33 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 758 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 25 | the patch passed | | +1 | findbugs | 65 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 84 | hadoop-aws in the patch passed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 5322 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/17/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1115 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 4f7a2f76c87f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98ca07e | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/17/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/17/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/17/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] elek commented on issue #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs
elek commented on issue #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs URL: https://github.com/apache/hadoop/pull/1537#issuecomment-536677464 Thanks the update @adoroszlai Looks good. Good go have `ozone fs` fixed One problem: datanodes in ozonesecure-mr are not started AFAIK https://github.com/elek/ozone-ci-q4/blob/master/pr/pr-hdds-2187-2nl4x/acceptance/output.log This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16544) update io.netty in branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941196#comment-16941196 ] Jonathan Hung commented on HADOOP-16544: Thanks [~iwasakims] for working on this. TestNameNodeHttpServerXFrame passes locally for me. +1 for the latest patch. Once this is committed, should we also commit HADOOP-15849 to branch-3.2 and branch-3.1? > update io.netty in branch-2 > --- > > Key: HADOOP-16544 > URL: https://issues.apache.org/jira/browse/HADOOP-16544 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Masatake Iwasaki >Priority: Major > Labels: release-blocker > Attachments: HADOOP-16544-branch-2.001.patch, > HADOOP-16544-branch-2.002.patch, HADOOP-16544-branch-2.003.patch, > HADOOP-16544-branch-2.004.patch > > > branch-2 pulls in io.netty 3.6.2.Final which is more than 5 years old. > The latest is 3.10.6Final. I know updating netty is sensitive but it deserves > some attention. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager.
avijayanhwx commented on issue #1538: HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager. URL: https://github.com/apache/hadoop/pull/1538#issuecomment-536674819 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1517: HDDS-2169
hadoop-yetus commented on issue #1517: HDDS-2169 URL: https://github.com/apache/hadoop/pull/1517#issuecomment-536668387 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2244 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 36 | Maven dependency ordering for branch | | -1 | mvninstall | 35 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 38 | hadoop-ozone in trunk failed. | | -1 | compile | 19 | hadoop-hdds in trunk failed. | | -1 | compile | 13 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 59 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 940 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in trunk failed. | | -1 | javadoc | 17 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 1028 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 30 | hadoop-hdds in trunk failed. | | -1 | findbugs | 17 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 15 | Maven dependency ordering for patch | | -1 | mvninstall | 32 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | -1 | compile | 22 | hadoop-hdds in the patch failed. | | -1 | compile | 15 | hadoop-ozone in the patch failed. | | -1 | javac | 22 | hadoop-hdds in the patch failed. | | -1 | javac | 15 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 25 | hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 789 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 29 | hadoop-hdds in the patch failed. | | -1 | findbugs | 17 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 22 | hadoop-ozone in the patch failed. | | +1 | asflicense | 30 | The patch does not generate ASF License warnings. | | | | 4686 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/Dockerfile | | JIRA Issue | HDDS-2169 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12981115/o2169_20190923.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 11b7ced0c5c4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98ca07e | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1517/6/artifact/out/patch-compile-hadoop-hdds.txt | | compile |
[jira] [Commented] (HADOOP-16617) ITestGetNameSpaceEnabled#testFailedRequestWhenFSNotExist fails with ns disabled account
[ https://issues.apache.org/jira/browse/HADOOP-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941145#comment-16941145 ] Da Zhou commented on HADOOP-16617: -- There is a PR out with fix to this test failure : [https://github.com/apache/hadoop/pull/1498] > ITestGetNameSpaceEnabled#testFailedRequestWhenFSNotExist fails with ns > disabled account > --- > > Key: HADOOP-16617 > URL: https://issues.apache.org/jira/browse/HADOOP-16617 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test >Affects Versions: 3.2.1 >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > > AzureBlobFileSystemStore#getIsNamespaceEnabled() gets ACL status of root path > to decide whether the account is XNS or not. If not it returns with 400 as > error code which means the account is a non-XNS acc. > The problem is that we get 400 and the getIsNamespaceEnabled return false > even if the filesystem does not exist which seems ok but according to the > test we should get 404. So it seems the expected behavior is to return 404. > At this point I am not sure how to fix it. Should we insist to the expected > behavior and fix it on server side or we just adjust the test to expect false > in case of non XNS account? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 closed pull request #1497: HDDS-2001. Update Ratis version to 0.4.0.
nandakumar131 closed pull request #1497: HDDS-2001. Update Ratis version to 0.4.0. URL: https://github.com/apache/hadoop/pull/1497 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on issue #1497: HDDS-2001. Update Ratis version to 0.4.0.
nandakumar131 commented on issue #1497: HDDS-2001. Update Ratis version to 0.4.0. URL: https://github.com/apache/hadoop/pull/1497#issuecomment-536657469 Fixed as part of #1524 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on issue #1497: HDDS-2001. Update Ratis version to 0.4.0.
nandakumar131 commented on issue #1497: HDDS-2001. Update Ratis version to 0.4.0. URL: https://github.com/apache/hadoop/pull/1497#issuecomment-536654104 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1486: HDDS-2158. Fixing Json Injection Issue in JsonUtils.
hadoop-yetus commented on issue #1486: HDDS-2158. Fixing Json Injection Issue in JsonUtils. URL: https://github.com/apache/hadoop/pull/1486#issuecomment-536652675 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 11 | https://github.com/apache/hadoop/pull/1486 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1486 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1486/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[
https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941123#comment-16941123
]
Hadoop QA commented on HADOOP-16587:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HADOOP-16587 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16587 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981816/HADOOP-16587.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16560/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make AAD endpoint configurable on all Auth flows
>
>
> Key: HADOOP-16587
> URL: https://issues.apache.org/jira/browse/HADOOP-16587
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Bilahari T H
>Assignee: Bilahari T H
>Priority: Minor
> Attachments: HADOOP-16587.001.patch, HADOOP-16587.002.patch
>
>
> Make AAD endpoint configurable on all Auth flows. Currently auth endpoint is
> hard coded for refreshtoken flow and MSI flow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329675515
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##
@@ -635,4 +641,24 @@ void setWriteBufferSize(int bufferSize) {
void setEnableFlush(boolean enableFlush) {
this.enableFlush = enableFlush;
}
+
+ private String getMsiAuthEndpoint() throws IOException {
+String authEndpoint = getPasswordString(
+FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT);
+if (Utils.isEmpty(authEndpoint)) {
+ authEndpoint =
+ AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT;
+}
+return authEndpoint;
+ }
+
+ private String getRefreshTokenAuthEndpoint() throws IOException {
+String authEndpoint = getPasswordString(
+FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN_ENDPOINT);
+if (Utils.isEmpty(authEndpoint)) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329675288
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java
##
@@ -109,15 +112,12 @@ public static AzureADToken
getTokenUsingClientCreds(String authEndpoint,
* @return {@link AzureADToken} obtained using the creds
* @throws IOException throws IOException if there is a failure in obtaining
the token
*/
- public static AzureADToken getTokenFromMsi(String tenantGuid, String
clientId,
+ public static AzureADToken getTokenFromMsi(final String authEndpoint, String
tenantGuid, String clientId,
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329675339
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java
##
@@ -141,14 +141,16 @@ public static AzureADToken getTokenFromMsi(String
tenantGuid, String clientId,
/**
* Gets Azure Active Directory token using refresh token.
*
+ * @param authEndpoint the OAuth 2.0 token endpoint associated
+ * with the user's directory (obtain from
+ * Active Directory configuration)
* @param clientId the client ID (GUID) of the client web app obtained from
Azure Active Directory configuration
* @param refreshToken the refresh token
* @return {@link AzureADToken} obtained using the refresh token
* @throws IOException throws IOException if there is a failure in
connecting to Azure AD
*/
- public static AzureADToken getTokenUsingRefreshToken(String clientId,
+ public static AzureADToken getTokenUsingRefreshToken(final String
authEndpoint, String clientId,
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329675048
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java
##
@@ -109,15 +112,12 @@ public static AzureADToken
getTokenUsingClientCreds(String authEndpoint,
* @return {@link AzureADToken} obtained using the creds
* @throws IOException throws IOException if there is a failure in obtaining
the token
*/
- public static AzureADToken getTokenFromMsi(String tenantGuid, String
clientId,
+ public static AzureADToken getTokenFromMsi(final String authEndpoint, String
tenantGuid, String clientId,
boolean bypassCache) throws
IOException {
-String authEndpoint =
"http://169.254.169.254/metadata/identity/oauth2/token;;
-
QueryParams qp = new QueryParams();
qp.add("api-version", "2018-02-01");
qp.add("resource", RESOURCE_NAME);
-
if (tenantGuid != null && tenantGuid.length() > 0) {
String authority = "https://login.microsoftonline.com/; + tenantGuid;
Review comment:
value for authority also made configurable
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329674769
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##
@@ -635,4 +641,24 @@ void setWriteBufferSize(int bufferSize) {
void setEnableFlush(boolean enableFlush) {
this.enableFlush = enableFlush;
}
+
+ private String getMsiAuthEndpoint() throws IOException {
+String authEndpoint = getPasswordString(
+FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT);
+if (Utils.isEmpty(authEndpoint)) {
+ authEndpoint =
+ AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT;
+}
+return authEndpoint;
+ }
+
+ private String getRefreshTokenAuthEndpoint() throws IOException {
+String authEndpoint = getPasswordString(
+FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN_ENDPOINT);
+if (Utils.isEmpty(authEndpoint)) {
+ authEndpoint =
+
AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN_ENDPOINT;
+}
+return authEndpoint;
Review comment:
Currently the values are hard coded without space
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329674367
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/Utils.java
##
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+public final class Utils {
+
+ private Utils() {
+ }
+
+ public static boolean isEmpty(String str) {
Review comment:
Used apache commons-language utility method
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[ https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16587: -- Attachment: HADOOP-16587.002.patch > Make AAD endpoint configurable on all Auth flows > > > Key: HADOOP-16587 > URL: https://issues.apache.org/jira/browse/HADOOP-16587 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Attachments: HADOOP-16587.001.patch, HADOOP-16587.002.patch > > > Make AAD endpoint configurable on all Auth flows. Currently auth endpoint is > hard coded for refreshtoken flow and MSI flow. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth endpoints configurable for MSI and refresh token flows
bilaharith commented on a change in pull request #1481: HADOOP-16587: Made auth
endpoints configurable for MSI and refresh token flows
URL: https://github.com/apache/hadoop/pull/1481#discussion_r329673753
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AuthConfigurations.java
##
@@ -0,0 +1,40 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Responsible to keep all the Azure Blob File System auth related
+ * configurations.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public final class AuthConfigurations {
+
+ public static final String DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT =
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant edited a comment on issue #1517: HDDS-2169
bshashikant edited a comment on issue #1517: HDDS-2169 URL: https://github.com/apache/hadoop/pull/1517#issuecomment-536642924 Thanks @szetszwo for working on this. With the patch, while running the tests in TestDataValidateWithUnsafeByteOperations, the below issue is observed. `2019-09-30 21:58:02,745 [grpc-default-executor-2] ERROR segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:(449)) - e4ab8454-30fe-420c-a1cf-40d223cb4898@group-D0335C23E8DA-SegmentedRaftLogWorker: writeStateMachineData failed for index 1, entry=(t:1, i:1), STATEMACHINELOGENTRY, client-6C45A0D09519, cid=8 java.lang.IndexOutOfBoundsException: End index: 135008824 >= 207 at org.apache.ratis.thirdparty.com.google.protobuf.ByteString.checkRange(ByteString.java:1233) at org.apache.ratis.thirdparty.com.google.protobuf.ByteString$LiteralByteString.substring(ByteString.java:1288) at org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage.toProto(ContainerCommandRequestMessage.java:66) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:375) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(ContainerStateMachine.java:494) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.(SegmentedRaftLogWorker.java:447) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.writeLogEntry(SegmentedRaftLogWorker.java:397) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:411) at org.apache.ratis.server.raftlog.RaftLog.lambda$appendEntry$10(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:77) at org.apache.ratis.server.raftlog.RaftLog.appendEntry(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLog.appendImpl(RaftLog.java:183) at org.apache.ratis.server.raftlog.RaftLog.lambda$append$2(RaftLog.java:159) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:68) at org.apache.ratis.server.raftlog.RaftLog.append(RaftLog.java:159) at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:282) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:505) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:576) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at
[GitHub] [hadoop] bshashikant commented on issue #1517: HDDS-2169
bshashikant commented on issue #1517: HDDS-2169 URL: https://github.com/apache/hadoop/pull/1517#issuecomment-536642924 Thanks @szetszwo for working on this. With the patch, while running the tests in TestDataValidateWithUnsafeByteOperations, the below issue is observed. `2019-09-30 21:58:02,745 [grpc-default-executor-2] ERROR segmented.SegmentedRaftLogWorker (SegmentedRaftLogWorker.java:(449)) - e4ab8454-30fe-420c-a1cf-40d223cb4898@group-D0335C23E8DA-SegmentedRaftLogWorker: writeStateMachineData failed for index 1, entry=(t:1, i:1), STATEMACHINELOGENTRY, client-6C45A0D09519, cid=8 java.lang.IndexOutOfBoundsException: End index: 135008824 >= 207 at org.apache.ratis.thirdparty.com.google.protobuf.ByteString.checkRange(ByteString.java:1233) at org.apache.ratis.thirdparty.com.google.protobuf.ByteString$LiteralByteString.substring(ByteString.java:1288) at org.apache.hadoop.hdds.ratis.ContainerCommandRequestMessage.toProto(ContainerCommandRequestMessage.java:66) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.getContainerCommandRequestProto(ContainerStateMachine.java:375) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.writeStateMachineData(ContainerStateMachine.java:494) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.(SegmentedRaftLogWorker.java:447) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.writeLogEntry(SegmentedRaftLogWorker.java:397) at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:411) at org.apache.ratis.server.raftlog.RaftLog.lambda$appendEntry$10(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:77) at org.apache.ratis.server.raftlog.RaftLog.appendEntry(RaftLog.java:359) at org.apache.ratis.server.raftlog.RaftLog.appendImpl(RaftLog.java:183) at org.apache.ratis.server.raftlog.RaftLog.lambda$append$2(RaftLog.java:159) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:68) at org.apache.ratis.server.raftlog.RaftLog.append(RaftLog.java:159) at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:282) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:505) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:576) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at
[jira] [Commented] (HADOOP-16587) Make AAD endpoint configurable on all Auth flows
[ https://issues.apache.org/jira/browse/HADOOP-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941112#comment-16941112 ] Bilahari T H commented on HADOOP-16587: --- [~ste...@apache.org] yes this is related to HADOOP-16498 > Make AAD endpoint configurable on all Auth flows > > > Key: HADOOP-16587 > URL: https://issues.apache.org/jira/browse/HADOOP-16587 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Attachments: HADOOP-16587.001.patch > > > Make AAD endpoint configurable on all Auth flows. Currently auth endpoint is > hard coded for refreshtoken flow and MSI flow. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16612) Track Azure Blob File System client-perceived latency
[ https://issues.apache.org/jira/browse/HADOOP-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941104#comment-16941104 ] Da Zhou commented on HADOOP-16612: -- Could you open a GitHub PR for this JIRA? that will make it easy for others to review too. > Track Azure Blob File System client-perceived latency > - > > Key: HADOOP-16612 > URL: https://issues.apache.org/jira/browse/HADOOP-16612 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, hdfs-client >Reporter: Jeetesh Mangwani >Assignee: Jeetesh Mangwani >Priority: Major > > Track the end-to-end performance of ADLS Gen 2 REST APIs by measuring > latencies in the Hadoop ABFS driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host
hadoop-yetus commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host URL: https://github.com/apache/hadoop/pull/1551#issuecomment-536626155 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 48 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 44 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 866 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 24 | hadoop-hdds in trunk failed. | | -1 | javadoc | 19 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 978 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 46 | hadoop-hdds in trunk failed. | | -1 | findbugs | 18 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 32 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | -1 | compile | 24 | hadoop-hdds in the patch failed. | | -1 | compile | 17 | hadoop-ozone in the patch failed. | | -1 | javac | 24 | hadoop-hdds in the patch failed. | | -1 | javac | 17 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 30 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 713 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in the patch failed. | | -1 | javadoc | 21 | hadoop-ozone in the patch failed. | | -1 | findbugs | 30 | hadoop-hdds in the patch failed. | | -1 | findbugs | 22 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 29 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 2401 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1551 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e049700d492e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 98ca07e | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1551/1/artifact/out/patch-compile-hadoop-hdds.txt | | javac |
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1115: HADOOP-16207 testMR failures
hadoop-yetus removed a comment on issue #1115: HADOOP-16207 testMR failures URL: https://github.com/apache/hadoop/pull/1115#issuecomment-527275589 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 11 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1069 | trunk passed | | +1 | compile | 35 | trunk passed | | +1 | checkstyle | 26 | trunk passed | | +1 | mvnsite | 38 | trunk passed | | +1 | shadedclient | 766 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | trunk passed | | 0 | spotbugs | 60 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 57 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 35 | the patch passed | | +1 | compile | 29 | the patch passed | | +1 | javac | 29 | the patch passed | | -0 | checkstyle | 19 | hadoop-tools/hadoop-aws: The patch generated 9 new + 5 unchanged - 1 fixed = 14 total (was 6) | | +1 | mvnsite | 32 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 736 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 25 | the patch passed | | +1 | findbugs | 62 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 82 | hadoop-aws in the patch passed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 3202 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/15/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1115 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux cd6d31a485eb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 915cbc9 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/15/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/15/testReport/ | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1115/15/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16617) ITestGetNameSpaceEnabled#testFailedRequestWhenFSNotExist fails with ns disabled account
[ https://issues.apache.org/jira/browse/HADOOP-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor updated HADOOP-16617: -- Affects Version/s: 3.2.1 > ITestGetNameSpaceEnabled#testFailedRequestWhenFSNotExist fails with ns > disabled account > --- > > Key: HADOOP-16617 > URL: https://issues.apache.org/jira/browse/HADOOP-16617 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test >Affects Versions: 3.2.1 >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > > AzureBlobFileSystemStore#getIsNamespaceEnabled() gets ACL status of root path > to decide whether the account is XNS or not. If not it returns with 400 as > error code which means the account is a non-XNS acc. > The problem is that we get 400 and the getIsNamespaceEnabled return false > even if the filesystem does not exist which seems ok but according to the > test we should get 404. So it seems the expected behavior is to return 404. > At this point I am not sure how to fix it. Should we insist to the expected > behavior and fix it on server side or we just adjust the test to expect false > in case of non XNS account? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16617) ITestGetNameSpaceEnabled#testFailedRequestWhenFSNotExist fails with ns disabled account
[ https://issues.apache.org/jira/browse/HADOOP-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor updated HADOOP-16617: -- Component/s: test fs/azure > ITestGetNameSpaceEnabled#testFailedRequestWhenFSNotExist fails with ns > disabled account > --- > > Key: HADOOP-16617 > URL: https://issues.apache.org/jira/browse/HADOOP-16617 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > > AzureBlobFileSystemStore#getIsNamespaceEnabled() gets ACL status of root path > to decide whether the account is XNS or not. If not it returns with 400 as > error code which means the account is a non-XNS acc. > The problem is that we get 400 and the getIsNamespaceEnabled return false > even if the filesystem does not exist which seems ok but according to the > test we should get 404. So it seems the expected behavior is to return 404. > At this point I am not sure how to fix it. Should we insist to the expected > behavior and fix it on server side or we just adjust the test to expect false > in case of non XNS account? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16616) ITestAzureFileSystemInstrumentation#testMetricsOnBigFileCreateRead fails
[
https://issues.apache.org/jira/browse/HADOOP-16616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-16616:
--
Affects Version/s: 3.2.1
> ITestAzureFileSystemInstrumentation#testMetricsOnBigFileCreateRead fails
>
>
> Key: HADOOP-16616
> URL: https://issues.apache.org/jira/browse/HADOOP-16616
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.2.1
>Reporter: Andras Bokor
>Priority: Major
>
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 238.687 s <<< FAILURE! - in
> org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation
> [ERROR]
> testMetricsOnBigFileCreateRead(org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation)
> Time elapsed: 238.5 s <<< FAILURE!
> java.lang.AssertionError: The download latency 0 should be greater than zero
> now that I've just downloaded a file.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation.testMetricsOnBigFileCreateRead(ITestAzureFileSystemInstrumentation.java:303)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16616) ITestAzureFileSystemInstrumentation#testMetricsOnBigFileCreateRead fails
[
https://issues.apache.org/jira/browse/HADOOP-16616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-16616:
--
Component/s: fs/azure
> ITestAzureFileSystemInstrumentation#testMetricsOnBigFileCreateRead fails
>
>
> Key: HADOOP-16616
> URL: https://issues.apache.org/jira/browse/HADOOP-16616
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Reporter: Andras Bokor
>Priority: Major
>
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 238.687 s <<< FAILURE! - in
> org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation
> [ERROR]
> testMetricsOnBigFileCreateRead(org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation)
> Time elapsed: 238.5 s <<< FAILURE!
> java.lang.AssertionError: The download latency 0 should be greater than zero
> now that I've just downloaded a file.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation.testMetricsOnBigFileCreateRead(ITestAzureFileSystemInstrumentation.java:303)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16618) increase the default number of threads and http connections in S3A
Steve Loughran created HADOOP-16618: --- Summary: increase the default number of threads and http connections in S3A Key: HADOOP-16618 URL: https://issues.apache.org/jira/browse/HADOOP-16618 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.2.1 Reporter: Steve Loughran Enable bigger thread and http pools in the S3A connector, especially now that the transfer manager is doing parallel block transfer, as is rename() We can make a lot more with parallelism in a single thread, and for applications with multiple worker threads, we need bigger defaults -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues
hadoop-yetus commented on issue #1442: HADOOP-16570. S3A committers encounter scale issues URL: https://github.com/apache/hadoop/pull/1442#issuecomment-536612827 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1177 | trunk passed | | +1 | compile | 31 | trunk passed | | +1 | checkstyle | 23 | trunk passed | | +1 | mvnsite | 35 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 24 | trunk passed | | 0 | spotbugs | 57 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 55 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 32 | the patch passed | | +1 | compile | 26 | the patch passed | | +1 | javac | 26 | the patch passed | | -0 | checkstyle | 19 | hadoop-tools/hadoop-aws: The patch generated 6 new + 46 unchanged - 0 fixed = 52 total (was 46) | | +1 | mvnsite | 30 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 867 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 22 | the patch passed | | +1 | findbugs | 61 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 84 | hadoop-aws in the patch passed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 3489 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1442 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux eaa2f1e2a04a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b46d823 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/9/testReport/ | | Max. process+thread count | 340 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1442/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941048#comment-16941048
]
Adam Antal commented on HADOOP-16503:
-
Thanks to the patch [~kmarton]. Could you please explain what does this patch
fix? You mentioned that YARN-9784 fixed the issue seen above, so I am unsure
what do we aim with the patch.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: HADOOP-16503.001.patch, HADOOP-16503.002.patch,
> HADOOP-16503.003.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs
hadoop-yetus commented on issue #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs URL: https://github.com/apache/hadoop/pull/1537#issuecomment-536609986 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | yamllint | 0 | yamllint was not available. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 16 | Maven dependency ordering for branch | | -1 | mvninstall | 37 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 40 | hadoop-ozone in trunk failed. | | -1 | compile | 23 | hadoop-hdds in trunk failed. | | -1 | compile | 20 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 817 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 25 | hadoop-hdds in trunk failed. | | -1 | javadoc | 24 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for patch | | -1 | mvninstall | 40 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 41 | hadoop-ozone in the patch failed. | | -1 | compile | 30 | hadoop-hdds in the patch failed. | | -1 | compile | 24 | hadoop-ozone in the patch failed. | | -1 | javac | 30 | hadoop-hdds in the patch failed. | | -1 | javac | 24 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 761 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 27 | hadoop-hdds in the patch failed. | | -1 | javadoc | 23 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 30 | hadoop-hdds in the patch failed. | | -1 | unit | 33 | hadoop-ozone in the patch failed. | | +1 | asflicense | 39 | The patch does not generate ASF License warnings. | | | | 2291 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1537 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs | | uname | Linux 96c3e820363b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b46d823 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-javadoc-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-javadoc-hadoop-ozone.txt |
[GitHub] [hadoop] mukul1987 merged pull request #1550: HDDS-2207. Update Ratis to latest snapshot.
mukul1987 merged pull request #1550: HDDS-2207. Update Ratis to latest snapshot. URL: https://github.com/apache/hadoop/pull/1550 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lokeshj1703 commented on issue #1550: HDDS-2207. Update Ratis to latest snapshot.
lokeshj1703 commented on issue #1550: HDDS-2207. Update Ratis to latest snapshot. URL: https://github.com/apache/hadoop/pull/1550#issuecomment-536601731 @bshashikant Thanks for working on this! The changes look good to me. +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lokeshj1703 edited a comment on issue #1469: HDDS-2034. Async RATIS pipeline creation and destroy through heartbea…
lokeshj1703 edited a comment on issue #1469: HDDS-2034. Async RATIS pipeline creation and destroy through heartbea… URL: https://github.com/apache/hadoop/pull/1469#issuecomment-536592541 > I think the purpose of safemode is to guarantee that Ozone cluster is ready to provide service to Ozone client once safemode is exited. @ChenSammi I agree with that. I think the problem occurs with OneReplicaPipelineSafeModeRule. This rule makes sure that atleast one datanode in the old pipeline is reported so that reads for OPEN containers can go through. Here I think that old pipelines need to be tracked separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host
sodonnel commented on issue #1551: HDDS-2199 In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host URL: https://github.com/apache/hadoop/pull/1551#issuecomment-536599549 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
