[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to the latest version
[
https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716207#comment-16716207
]
Hadoop QA commented on HADOOP-15994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-15994 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15994 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951314/HADOOP-15994-002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15631/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Upgrade Jackson2 to the latest version
> --
>
> Key: HADOOP-15994
> URL: https://issues.apache.org/jira/browse/HADOOP-15994
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Akira Ajisaka
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15994-001.patch, HADOOP-15994-002.patch
>
>
> Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's
> upgrade to 2.9.6 or upper.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Attachment: HADOOP-15994-002.patch > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch, HADOOP-15994-002.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-15994: > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14930) Upgrade Jetty to 9.4 version
[
https://issues.apache.org/jira/browse/HADOOP-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716134#comment-16716134
]
Akira Ajisaka commented on HADOOP-14930:
bq. Can we upgrade to 9.4.x in Hadoop 3.3.0 or later to fix CVE-2017-9735?
This is my mistake. Thanks [~kihwal] for the information.
> Upgrade Jetty to 9.4 version
>
>
> Key: HADOOP-14930
> URL: https://issues.apache.org/jira/browse/HADOOP-14930
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-14930.00.patch
>
>
> Currently 9.3.19.v20170502 is used.
> In hbase 2.0+, 9.4.6.v20170531 is used.
> When starting mini dfs cluster in hbase unit tests, we get the following:
> {code}
> java.lang.NoSuchMethodError:
> org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
> at
> org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:548)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:529)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:119)
> at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:415)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:887)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:723)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:949)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:928)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1277)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1046)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:921)
> {code}
> This issue is to upgrade Jetty to 9.4 version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716127#comment-16716127 ] Akira Ajisaka commented on HADOOP-15994: The latest version is 2.9.7. Can we try the latest version? > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15994: --- Status: Patch Available (was: Reopened) > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716065#comment-16716065
]
Íñigo Goiri commented on HADOOP-15995:
--
Thanks [~lukmajercak] for the patch.
Minor comments:
* In the unit test you can do: {{char[] bindpass = "bindpass".toCharArray();}}.
* Is there a way to use the credential to get he password instead of the
deprecated method?
> Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when
> using multiple providers through CompositeGroupsMapping
> -
>
> Key: HADOOP-15995
> URL: https://issues.apache.org/jira/browse/HADOOP-15995
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Major
> Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch,
> HADOOP-15995.003.patch, HADOOP-15995.004.patch
>
>
> Currently, the property name hadoop.security.group.mapping.ldap.bind.password
> is used as an alias to get password from CredentialProviders. This has a big
> issue, which is that when we configure multiple LdapGroupsMapping providers
> through CompositeGroupsMapping, they will all have the same alias, and won't
> be able to be distinguished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715891#comment-16715891
]
Hadoop QA commented on HADOOP-15995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 19s{color}
| {color:red} root generated 2 new + 1490 unchanged - 0 fixed = 1492 total (was
1490) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 28 unchanged - 1 fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
36s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951278/HADOOP-15995.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 6402b9d2f6f5 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 80e59e7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15630/artifact/out/diff-compile-javac-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15630/testReport/ |
| Max. process+thread count | 1461 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715877#comment-16715877
]
Hadoop QA commented on HADOOP-15995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 35m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 21s{color}
| {color:red} root generated 2 new + 1490 unchanged - 0 fixed = 1492 total (was
1490) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 28 unchanged - 1 fixed = 29 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 18s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951277/HADOOP-15995.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux a22b5c1f70a7 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 80e59e7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15629/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
[jira] [Commented] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[
https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715829#comment-16715829
]
Hudson commented on HADOOP-15428:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15585 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15585/])
HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set
(mackrorysd: rev 3ff8580f2289664a54f369b500ad77f31a07c326)
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
* (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
> s3guard bucket-info will create s3guard table if FS is set to do this
> automatically
> ---
>
> Key: HADOOP-15428
> URL: https://issues.apache.org/jira/browse/HADOOP-15428
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15428.001.patch
>
>
> If you call hadoop s3guard bucket-info on a bucket where the fs is set to
> create a s3guard table on demand, then the DDB table is automatically
> created. As a result
> the {{bucket-info -unguarded}} option cannot be used, and the call has
> significant side effects (i.e. it can run up bills)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15988) Should be able to set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative directory listings
[
https://issues.apache.org/jira/browse/HADOOP-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715805#comment-16715805
]
Sean Mackrory commented on HADOOP-15988:
Good stuff. A few nits:
* Can you clean up the checkstyle issues that were raised?
* Most of the code between the 2 tests is shared. Can we refactor that into a
single test that just tests the same sequence with a different auth value and
outcome? If that turns out to be messy for some reason it's not a deal breaker,
but worth a couple of minutes if that's all it takes.
+1 otherwise.
> Should be able to set empty directory flag to TRUE in
> DynamoDBMetadataStore#innerGet when using authoritative directory listings
>
>
> Key: HADOOP-15988
> URL: https://issues.apache.org/jira/browse/HADOOP-15988
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15988.001.patch
>
>
> We have the following comment and implementation in DynamoDBMetadataStore:
> {noformat}
> // When this class has support for authoritative
> // (fully-cached) directory listings, we may also be able to answer
> // TRUE here. Until then, we don't know if we have full listing or
> // not, thus the UNKNOWN here:
> meta.setIsEmptyDirectory(
> hasChildren ? Tristate.FALSE : Tristate.UNKNOWN);
> {noformat}
> We have authoritative listings now in dynamo since HADOOP-15621, so we should
> resolve this comment, implement the solution and test it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15987) ITestDynamoDBMetadataStore should check if test ddb table set properly before initializing the test
[
https://issues.apache.org/jira/browse/HADOOP-15987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715814#comment-16715814
]
Sean Mackrory commented on HADOOP-15987:
As discussed offline today - I don't think there's much to change - I'm just
not a fan of referring to the original config as "production". That is how you
configure it in production, but the distinction is just regular functionality
tests vs. tests that impact the table lifecycle. I just want to be clearer in
the error message and / or docs. But I couldn't think of a better term than
"production". I'll wait until tomorrow to see what you come up with, but it
neither of us has a much more helpful idea, I'm good to just commit as-is.
> ITestDynamoDBMetadataStore should check if test ddb table set properly before
> initializing the test
> ---
>
> Key: HADOOP-15987
> URL: https://issues.apache.org/jira/browse/HADOOP-15987
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15987.001.patch
>
>
> The jira covers the following:
> * We should assert that the table name is configured when
> DynamoDBMetadataStore is used for testing, so the test should fail if it's
> not configured.
> * We should assert that the test table is not the same as the production
> table, as the test table could be modified and destroyed multiple times
> during the test.
> * This behavior should be added to the testing docs.
> [Assume from junit
> doc|http://junit.sourceforge.net/javadoc/org/junit/Assume.html]:
> {noformat}
> A set of methods useful for stating assumptions about the conditions in which
> a test is meaningful.
> A failed assumption does not mean the code is broken, but that the test
> provides no useful information.
> The default JUnit runner treats tests with failing assumptions as ignored.
> {noformat}
> A failed assert will cause test failure instead of just skipping the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715787#comment-16715787
]
Hadoop QA commented on HADOOP-15995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 19m 2s{color}
| {color:red} root generated 2 new + 1490 unchanged - 0 fixed = 1492 total (was
1490) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951269/HADOOP-15995.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 5792687f21b8 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 80e59e7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15628/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15628/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Updated] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[
https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-15428:
---
Fix Version/s: 3.3.0
> s3guard bucket-info will create s3guard table if FS is set to do this
> automatically
> ---
>
> Key: HADOOP-15428
> URL: https://issues.apache.org/jira/browse/HADOOP-15428
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15428.001.patch
>
>
> If you call hadoop s3guard bucket-info on a bucket where the fs is set to
> create a s3guard table on demand, then the DDB table is automatically
> created. As a result
> the {{bucket-info -unguarded}} option cannot be used, and the call has
> significant side effects (i.e. it can run up bills)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15428) s3guard bucket-info will create s3guard table if FS is set to do this automatically
[
https://issues.apache.org/jira/browse/HADOOP-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-15428:
---
Release Note: The -unguarded flag, passed to `hadoop s3guard bucket-info`
will no proceed with S3Guard disabled instead of failing if S3Guard is not
already disabled.
Committed to trunk. We should relnote this, IMO. Any thoughts on the wording
[~gabor.bota] or [~ste...@apache.org]?
> s3guard bucket-info will create s3guard table if FS is set to do this
> automatically
> ---
>
> Key: HADOOP-15428
> URL: https://issues.apache.org/jira/browse/HADOOP-15428
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15428.001.patch
>
>
> If you call hadoop s3guard bucket-info on a bucket where the fs is set to
> create a s3guard table on demand, then the DDB table is automatically
> created. As a result
> the {{bucket-info -unguarded}} option cannot be used, and the call has
> significant side effects (i.e. it can run up bills)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715795#comment-16715795 ] Lukas Majercak commented on HADOOP-15995: - For some reason, yetus still picked patch002. > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch, HADOOP-15995.004.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Attachment: HADOOP-15995.004.patch > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch, HADOOP-15995.004.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Status: Patch Available (was: Open) > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch, HADOOP-15995.004.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Status: Open (was: Patch Available) > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch, HADOOP-15995.004.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715772#comment-16715772 ] Lukas Majercak commented on HADOOP-15995: - Another checkstyle issue fixed in patch004. > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch, HADOOP-15995.004.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715771#comment-16715771
]
Hadoop QA commented on HADOOP-15995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 49s{color}
| {color:red} root generated 2 new + 1490 unchanged - 0 fixed = 1492 total (was
1490) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
30s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951268/HADOOP-15995.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux af15dcd2bd95 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 80e59e7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15627/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15627/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Assigned] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HADOOP-15967: Assignee: George Huang > KMS Benchmark Tool > -- > > Key: HADOOP-15967 > URL: https://issues.apache.org/jira/browse/HADOOP-15967 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: George Huang >Priority: Major > > We've been working on several pieces of KMS improvement work. One thing > that's missing so far is a good benchmark tool for KMS, similar to > NNThroughputBenchmark. > Some requirements I have in mind: > # it should be a standalone benchmark tool, requiring only KMS and a > benchmark client. No NameNode or DataNode should be involved. > # specify the type of KMS request sent by client. E.g., generate_eek, > decrypt_eek, reencrypt_eek > # optionally specify number of threads sending KMS requests. > File this jira to gather more requirements. Thoughts? [~knanasi] [~xyao] > [~daryn] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Attachment: HADOOP-15995.003.patch > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715707#comment-16715707 ] Lukas Majercak commented on HADOOP-15995: - Patch 003 to fix the checkstyle issue. > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch, > HADOOP-15995.003.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715687#comment-16715687
]
Hadoop QA commented on HADOOP-15995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 28 unchanged - 1 fixed = 29 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 2s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.ssl.TestSSLFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951259/HADOOP-15995.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux ced090886930 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ac578c0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15626/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Created] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop
Eric Yang created HADOOP-15996: -- Summary: Plugin interface to support more complex usernames in Hadoop Key: HADOOP-15996 URL: https://issues.apache.org/jira/browse/HADOOP-15996 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Eric Yang Hadoop does not allow support of @ character in username in recent security mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must match to authorize user to login to Hadoop cluster. This design does not work well in multi-realm environment where identical username between two realms do not map to the same user. There is also possibility that lossy regex can incorrect map users. In the interest of supporting multi-realms, it maybe preferred to pass principal name without rewrite to uniquely distinguish users. This jira is to revisit if Hadoop can support full principal names without rewrite and provide a plugin to override Hadoop's default implementation of auth_to_local for multi-realm use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HADOOP-15996: --- Description: Hadoop does not allow support of @ character in username in recent security mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must match to authorize user to login to Hadoop cluster. This design does not work well in multi-realm environment where identical username between two realms do not map to the same user. There is also possibility that lossy regex can incorrectly map users. In the interest of supporting multi-realms, it maybe preferred to pass principal name without rewrite to uniquely distinguish users. This jira is to revisit if Hadoop can support full principal names without rewrite and provide a plugin to override Hadoop's default implementation of auth_to_local for multi-realm use case. (was: Hadoop does not allow support of @ character in username in recent security mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must match to authorize user to login to Hadoop cluster. This design does not work well in multi-realm environment where identical username between two realms do not map to the same user. There is also possibility that lossy regex can incorrect map users. In the interest of supporting multi-realms, it maybe preferred to pass principal name without rewrite to uniquely distinguish users. This jira is to revisit if Hadoop can support full principal names without rewrite and provide a plugin to override Hadoop's default implementation of auth_to_local for multi-realm use case.) > Plugin interface to support more complex usernames in Hadoop > > > Key: HADOOP-15996 > URL: https://issues.apache.org/jira/browse/HADOOP-15996 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Eric Yang >Priority: Major > > Hadoop does not allow support of @ character in username in recent security > mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must > match to authorize user to login to Hadoop cluster. This design does not > work well in multi-realm environment where identical username between two > realms do not map to the same user. There is also possibility that lossy > regex can incorrectly map users. In the interest of supporting multi-realms, > it maybe preferred to pass principal name without rewrite to uniquely > distinguish users. This jira is to revisit if Hadoop can support full > principal names without rewrite and provide a plugin to override Hadoop's > default implementation of auth_to_local for multi-realm use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Attachment: HADOOP-15995.002.patch > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Attachment: HADOOP-15995.002.patch > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Attachment: (was: HADOOP-15995.002.patch) > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Description: Currently, the property name hadoop.security.group.mapping.ldap.bind.password is used as an alias to get password from CredentialProviders. This has a big issue, which is that when we configure multiple LdapGroupsMapping providers through CompositeGroupsMapping, they will all have the same alias, and won't be able to be distinguished. (was: Currently, the property name hadoop.security.group.mapping.ldap.bind.password is used as an alias to get password from CredentialProviders. This has a big issue, which is that when we configure multiple LdapGroupsMapping providers through CompositeGroupsMapping, they will all have the same alias, and won't be able to be distinguished. The proposal is to use the value of the property instead, which would fix this issue.) > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Summary: Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when using multiple providers through CompositeGroupsMapping (was: LdapGroupsMapping should use the bind.password config value as credential alias) > Add ldap.bind.password.alias in LdapGroupsMapping to distinguish aliases when > using multiple providers through CompositeGroupsMapping > - > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715614#comment-16715614 ] Lukas Majercak commented on HADOOP-15995: - Added .ldap.bind.password.alias in patch002. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch, HADOOP-15995.002.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715577#comment-16715577 ] Larry McCay commented on HADOOP-15995: -- +1 - that sounds better. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715548#comment-16715548 ] Larry McCay commented on HADOOP-15995: -- Hi [~lukmajercak] - can you explain why the current implementation which is aligned with other previously configured passwords doesn't meet your needs? Changing the behavior to use the value of the property rather than the key of the property as the alias definitely makes it different from other properties and it doesn't seem to even preserve the ability to use the key. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715570#comment-16715570 ] Lukas Majercak commented on HADOOP-15995: - I guess we could add a "ldap.bind.password.alias" configuration, and try that first, if we don't find anything we just fallback to the current version? > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715569#comment-16715569 ] Lukas Majercak commented on HADOOP-15995: - Mm, not sure about adding a new property, as the password management is already quite convoluted in the ldapgroupsmapping. For your second suggestion, we would need to change the logic in CompositeGroupsMapping, as it currently creates a copy of the config and populates the needed configuration keys and stripping the provider name. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Assignee: Lukas Majercak Status: Patch Available (was: Open) > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715560#comment-16715560 ] Larry McCay commented on HADOOP-15995: -- I see [~lukmajercak], so can we make a change where we add the notion of the provider qualified properties? Where the caller will need to know which provider they are accessing and make a call for properties more closely resembling: hadoop.security.group.mapping.provider.a.ldap.bind.password hadoop.security.group.mapping.provider.b.ldap.bind.password That would leave the use of a single LdapGroupsMapping and existing deployments whole while accommodating the extended semantics that you need. Note: I have not dug into who would need that context and how it would be acquired yet. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715548#comment-16715548 ] Larry McCay edited comment on HADOOP-15995 at 12/10/18 9:03 PM: Hi [~lukmajercak] - can you explain why the current implementation which is aligned with other previously configured passwords doesn't meet your needs? Changing the behavior to use the value of the property rather than the key of the property as the alias definitely makes it different from other properties and it doesn't seem to even preserve the ability to use the key. I see now that you have described it better - or I somehow misread it before. :) My concern is that this is not backward compatible and that previously, not only was the value not used but the property was not required to be configured at all. Now, you would require it to be set AND the password be provisioned with a different alias. If this is only a Composite group lookup issue, maybe a new property would make sense? was (Author: lmccay): Hi [~lukmajercak] - can you explain why the current implementation which is aligned with other previously configured passwords doesn't meet your needs? Changing the behavior to use the value of the property rather than the key of the property as the alias definitely makes it different from other properties and it doesn't seem to even preserve the ability to use the key. I see now that you have described it better - or I some how misread it before. :) My concern is that this is not backward compatible and that previously, not only was the value not used but the property was not required to be configured at all. Now, you would require it to be set AND the password be provisioned with a different alias. If this is only a Composite group lookup issue, maybe a new property would make sense? > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715548#comment-16715548 ] Larry McCay edited comment on HADOOP-15995 at 12/10/18 9:01 PM: Hi [~lukmajercak] - can you explain why the current implementation which is aligned with other previously configured passwords doesn't meet your needs? Changing the behavior to use the value of the property rather than the key of the property as the alias definitely makes it different from other properties and it doesn't seem to even preserve the ability to use the key. I see now that you have described it better - or I some how misread it before. :) My concern is that this is not backward compatible and that previously, not only was the value not used but the property was not required to be configured at all. Now, you would require it to be set AND the password be provisioned with a different alias. If this is only a Composite group lookup issue, maybe a new property would make sense? was (Author: lmccay): Hi [~lukmajercak] - can you explain why the current implementation which is aligned with other previously configured passwords doesn't meet your needs? Changing the behavior to use the value of the property rather than the key of the property as the alias definitely makes it different from other properties and it doesn't seem to even preserve the ability to use the key. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715554#comment-16715554 ] Lukas Majercak commented on HADOOP-15995: - Thanks for the quick comment [~lmccay]. Say i have two providers: hadoop.security.group.mapping=org.apache.hadoop.security.CompositeGroupsMapping hadoop.security.group.mapping.providers=a,b hadoop.security.group.mapping.provider.a=org.apache.hadoop.security.LdapGroupsMapping hadoop.security.group.mapping.provider.b=org.apache.hadoop.security.LdapGroupsMapping hadoop.security.group.mapping.provider.a.ldap.bind.password=foo hadoop.security.group.mapping.provider.b.ldap.bind.password=bar Both providers will use "hadoop.security.group.mapping.provider.ldap.bind.password" as the alias to get password from config. i.e. they won't be distinguishable. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Description: Currently, the property name hadoop.security.group.mapping.ldap.bind.password is used as an alias to get password from CredentialProviders. This has a big issue, which is that when we configure multiple LdapGroupsMapping providers through CompositeGroupsMapping, they will all have the same alias, and won't be able to be distinguished. The proposal is to use the value of the property instead, which would fix this issue. > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > > Currently, the property name hadoop.security.group.mapping.ldap.bind.password > is used as an alias to get password from CredentialProviders. This has a big > issue, which is that when we configure multiple LdapGroupsMapping providers > through CompositeGroupsMapping, they will all have the same alias, and won't > be able to be distinguished. The proposal is to use the value of the property > instead, which would fix this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
[ https://issues.apache.org/jira/browse/HADOOP-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-15995: Attachment: HADOOP-15995.001.patch > LdapGroupsMapping should use the bind.password config value as credential > alias > --- > > Key: HADOOP-15995 > URL: https://issues.apache.org/jira/browse/HADOOP-15995 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Lukas Majercak >Priority: Major > Attachments: HADOOP-15995.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15995) LdapGroupsMapping should use the bind.password config value as credential alias
Lukas Majercak created HADOOP-15995: --- Summary: LdapGroupsMapping should use the bind.password config value as credential alias Key: HADOOP-15995 URL: https://issues.apache.org/jira/browse/HADOOP-15995 Project: Hadoop Common Issue Type: Bug Components: common Reporter: Lukas Majercak -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15987) ITestDynamoDBMetadataStore should check if test ddb table set properly before initializing the test
[
https://issues.apache.org/jira/browse/HADOOP-15987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715234#comment-16715234
]
Gabor Bota commented on HADOOP-15987:
-
I'll correct that an upload a new patch soon.
> ITestDynamoDBMetadataStore should check if test ddb table set properly before
> initializing the test
> ---
>
> Key: HADOOP-15987
> URL: https://issues.apache.org/jira/browse/HADOOP-15987
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15987.001.patch
>
>
> The jira covers the following:
> * We should assert that the table name is configured when
> DynamoDBMetadataStore is used for testing, so the test should fail if it's
> not configured.
> * We should assert that the test table is not the same as the production
> table, as the test table could be modified and destroyed multiple times
> during the test.
> * This behavior should be added to the testing docs.
> [Assume from junit
> doc|http://junit.sourceforge.net/javadoc/org/junit/Assume.html]:
> {noformat}
> A set of methods useful for stating assumptions about the conditions in which
> a test is meaningful.
> A failed assumption does not mean the code is broken, but that the test
> provides no useful information.
> The default JUnit runner treats tests with failing assumptions as ignored.
> {noformat}
> A failed assert will cause test failure instead of just skipping the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15808) Harden Token service loader use
[ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715232#comment-16715232 ] Larry McCay commented on HADOOP-15808: -- [~ste...@apache.org] - what is the difference between no class found and not being able to be instantiated - they are separate exceptions but wondering whether the javadoc is correct in... + * @return the token identifier, or null if there was no class found for it + * @throws IOException failure to unmarshall the data + * @throws RuntimeException if the token class could not be instantiated. is it literally distinguishing between the two - so that a class that is found but cannot be loaded gets a runtime exception and a class not found gets a null returned? I assume once this gets in a patch for all of the consumers that will need to accommodate null will created? > Harden Token service loader use > --- > > Key: HADOOP-15808 > URL: https://issues.apache.org/jira/browse/HADOOP-15808 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.9.1, 3.1.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, > HADOOP-15808-003.patch > > > The Hadoop token service loading (identifiers, renewers...) works provided > there's no problems loading any registered implementation. If there's a > classloading or classcasting problem, the exception raised will stop all > token support working; possibly the application not starting. > This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on > the classpath. It probably lurks in the wasb/abfs support too, but things > have worked there because the installations with DT support there have always > had correctly set up classpaths. > Fix: do what we did for the FS service loader. Catch failures to instantiate > a service provider impl and skip it -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14930) Upgrade Jetty to 9.4 version
[
https://issues.apache.org/jira/browse/HADOOP-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714801#comment-16714801
]
Kihwal Lee commented on HADOOP-14930:
-
To clarify, jetty-9.3.20.v20170531 addressed CVE-2017-9735, so the current
jetty version in Hadoop 3.x is okay.
> Upgrade Jetty to 9.4 version
>
>
> Key: HADOOP-14930
> URL: https://issues.apache.org/jira/browse/HADOOP-14930
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-14930.00.patch
>
>
> Currently 9.3.19.v20170502 is used.
> In hbase 2.0+, 9.4.6.v20170531 is used.
> When starting mini dfs cluster in hbase unit tests, we get the following:
> {code}
> java.lang.NoSuchMethodError:
> org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
> at
> org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:548)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:529)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:119)
> at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:415)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:887)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:723)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:949)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:928)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1277)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1046)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:921)
> {code}
> This issue is to upgrade Jetty to 9.4 version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14930) Upgrade Jetty to 9.4 version
[
https://issues.apache.org/jira/browse/HADOOP-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714801#comment-16714801
]
Kihwal Lee edited comment on HADOOP-14930 at 12/10/18 2:42 PM:
---
To clarify, jetty-9.3.20.v20170531 addressed CVE-2017-9735, so the current
jetty version (9.3.24.v20180605) in Hadoop 3.x is okay.
was (Author: kihwal):
To clarify, jetty-9.3.20.v20170531 addressed CVE-2017-9735, so the current
jetty version in Hadoop 3.x is okay.
> Upgrade Jetty to 9.4 version
>
>
> Key: HADOOP-14930
> URL: https://issues.apache.org/jira/browse/HADOOP-14930
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-14930.00.patch
>
>
> Currently 9.3.19.v20170502 is used.
> In hbase 2.0+, 9.4.6.v20170531 is used.
> When starting mini dfs cluster in hbase unit tests, we get the following:
> {code}
> java.lang.NoSuchMethodError:
> org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
> at
> org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:548)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:529)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:119)
> at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:415)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:887)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:723)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:949)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:928)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1277)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1046)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:921)
> {code}
> This issue is to upgrade Jetty to 9.4 version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15988) Should be able to set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative directory listings
[
https://issues.apache.org/jira/browse/HADOOP-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714605#comment-16714605
]
Hadoop QA commented on HADOOP-15988:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 4
new + 6 unchanged - 0 fixed = 10 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
29s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15988 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951178/HADOOP-15988.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4b15e5b1ff84 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 17a8708 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15625/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15625/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15625/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to the latest version
[
https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714594#comment-16714594
]
Hadoop QA commented on HADOOP-15994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
33m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
13s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15994 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951176/HADOOP-15994-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux c1e9ece0ae3a 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 17a8708 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15624/testReport/ |
| Max. process+thread count | 339 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15624/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Upgrade Jackson2 to the latest version
> --
>
> Key: HADOOP-15994
> URL: https://issues.apache.org/jira/browse/HADOOP-15994
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Akira Ajisaka
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15994-001.patch
>
>
> Now Jackson 2.9.5 is used and it is
[jira] [Updated] (HADOOP-15563) S3guard init and set-capacity to support DDB autoscaling
[ https://issues.apache.org/jira/browse/HADOOP-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-15563: Priority: Major (was: Minor) > S3guard init and set-capacity to support DDB autoscaling > > > Key: HADOOP-15563 > URL: https://issues.apache.org/jira/browse/HADOOP-15563 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > > To keep costs down on DDB, autoscaling is a key feature: you set the max > values and when idle, you don't get billed, *at the cost of delayed scale > time and risk of not getting the max value when AWS is busy* > It can be done from the AWS web UI, but not in the s3guard init and > set-capacity calls > It can be done [through the > API|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.HowTo.SDK.html] > Usual issues then: wiring up, CLI params, testing. It'll be hard to test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15563) S3guard init and set-capacity to support DDB autoscaling
[ https://issues.apache.org/jira/browse/HADOOP-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-15563: --- Assignee: Gabor Bota > S3guard init and set-capacity to support DDB autoscaling > > > Key: HADOOP-15563 > URL: https://issues.apache.org/jira/browse/HADOOP-15563 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Minor > > To keep costs down on DDB, autoscaling is a key feature: you set the max > values and when idle, you don't get billed, *at the cost of delayed scale > time and risk of not getting the max value when AWS is busy* > It can be done from the AWS web UI, but not in the s3guard init and > set-capacity calls > It can be done [through the > API|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.HowTo.SDK.html] > Usual issues then: wiring up, CLI params, testing. It'll be hard to test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Attachment: HADOOP-15994-001.patch > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15988) Should be able to set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative directory listings
[
https://issues.apache.org/jira/browse/HADOOP-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15988:
Status: Patch Available (was: In Progress)
submitted patch v001, tested against eu-west-1.
Issues:
* org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster#testWithMiniCluster
(known)
* ITestS3GuardToolDynamoDB#testPruneCommandCLI (added to [S3 flakyness
collecting
spreadsheet|https://docs.google.com/spreadsheets/d/1Z9dkg5yC7Hu7VQ5G2Wz40hG1pb0DhLuLjJ3PZe7WL3Q/edit?usp=sharing])
- timeout, succesful after rerun
These issues are not related.
> Should be able to set empty directory flag to TRUE in
> DynamoDBMetadataStore#innerGet when using authoritative directory listings
>
>
> Key: HADOOP-15988
> URL: https://issues.apache.org/jira/browse/HADOOP-15988
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15988.001.patch
>
>
> We have the following comment and implementation in DynamoDBMetadataStore:
> {noformat}
> // When this class has support for authoritative
> // (fully-cached) directory listings, we may also be able to answer
> // TRUE here. Until then, we don't know if we have full listing or
> // not, thus the UNKNOWN here:
> meta.setIsEmptyDirectory(
> hasChildren ? Tristate.FALSE : Tristate.UNKNOWN);
> {noformat}
> We have authoritative listings now in dynamo since HADOOP-15621, so we should
> resolve this comment, implement the solution and test it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15988) Should be able to set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative directory listings
[
https://issues.apache.org/jira/browse/HADOOP-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15988:
Attachment: HADOOP-15988.001.patch
> Should be able to set empty directory flag to TRUE in
> DynamoDBMetadataStore#innerGet when using authoritative directory listings
>
>
> Key: HADOOP-15988
> URL: https://issues.apache.org/jira/browse/HADOOP-15988
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15988.001.patch
>
>
> We have the following comment and implementation in DynamoDBMetadataStore:
> {noformat}
> // When this class has support for authoritative
> // (fully-cached) directory listings, we may also be able to answer
> // TRUE here. Until then, we don't know if we have full listing or
> // not, thus the UNKNOWN here:
> meta.setIsEmptyDirectory(
> hasChildren ? Tristate.FALSE : Tristate.UNKNOWN);
> {noformat}
> We have authoritative listings now in dynamo since HADOOP-15621, so we should
> resolve this comment, implement the solution and test it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Status: Patch Available (was: Open) > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15994-001.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15994) Upgrade Jackson2 to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee reassigned HADOOP-15994: -- Assignee: lqjacklee > Upgrade Jackson2 to the latest version > -- > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15994) Upgrade Jackson2 to the latest version
Akira Ajisaka created HADOOP-15994: -- Summary: Upgrade Jackson2 to the latest version Key: HADOOP-15994 URL: https://issues.apache.org/jira/browse/HADOOP-15994 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Akira Ajisaka Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's upgrade to 2.9.6 or upper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14930) Upgrade Jetty to 9.4 version
[
https://issues.apache.org/jira/browse/HADOOP-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reopened HADOOP-14930:
Can we upgrade to 9.4.x in Hadoop 3.3.0 or later to fix CVE-2017-9735?
> Upgrade Jetty to 9.4 version
>
>
> Key: HADOOP-14930
> URL: https://issues.apache.org/jira/browse/HADOOP-14930
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-14930.00.patch
>
>
> Currently 9.3.19.v20170502 is used.
> In hbase 2.0+, 9.4.6.v20170531 is used.
> When starting mini dfs cluster in hbase unit tests, we get the following:
> {code}
> java.lang.NoSuchMethodError:
> org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
> at
> org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:548)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:529)
> at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:119)
> at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:415)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:887)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:723)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:949)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:928)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1277)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1046)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:921)
> {code}
> This issue is to upgrade Jetty to 9.4 version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15566) Remove HTrace support
[
https://issues.apache.org/jira/browse/HADOOP-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714435#comment-16714435
]
Elek, Marton commented on HADOOP-15566:
---
Thanks [~cmccabe], I agree with your points about the importance of the
compatibility and to keep the htrace support.
My proposal is:
1.) Create a lightweight Hadoop API for the tracing where multiple
implementation can be plugged in
2.) Provide a default implementation which uses the existing htrace code.
Implementation details:
a) Add a new optional bytes field for the RpcHeader. Different tracing
libraries could require different size of serialized context:
{code:java}
diff --git a/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto
b/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto
index aa146162896..e42f64eb631 100644
--- a/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto
+++ b/hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto
@@ -61,9 +61,9 @@ enum RpcKindProto {
* what span caused the new span we will create when this message is received.
*/
message RPCTraceInfoProto {
optional int64 traceId = 1; // parentIdHigh
optional int64 parentId = 2; // parentIdLow
+optional bytes tracingContext = 3; //generic tracingInformation
}
{code}
This is a a backward-compatible change.
b) In the rpc Server.java a (htrace) TraceScope is initialized based on the rpc
header and propagated as part of the RpcCall:
{code:java}
RpcCall call = new RpcCall(this, header.getCallId(),
header.getRetryCount(), rpcRequest,
ProtoUtil.convert(header.getRpcKind()),
header.getClientId().toByteArray(), traceScope, callerContext);
{code}
I propose to replace this traceScope with a hadoop specific TraceScope marker
interface. The default implementation could be a simple class which contains
the htrace implementation.
c. We can create a simple Tracing singleton (similar to the
DefaultMetricsSystem):
Example call:
{code:java}
try (TracingSpan context =
HadoopTracing.INSTANCE.newContext(call.tracingSpan, "RpcServerCall")) {
if (remoteUser != null) {
remoteUser.doAs(call);
} else {
call.run();
}
}
{code}
d. HadoopTracing could be something like this:
{code:java}
package org.apache.hadoop.tracing;
public enum HadoopTracing {
INSTANCE;
private TracingProvider provider;
public TracingSpan importContext(byte[] data) {
return provider.importContext(data);
}
public byte[] exportContext() {
return provider.exportContext();
}
public TracingSpan newContext(String name) {
return provider.newContext(name);
}
public TracingSpan newContext(TracingSpan parentSpan, String name) {
return null;
}
}
{code}
e. We can add multiple TracingProvider (and provide one for Htrace for
compatibility reason.)
+1. Personally I prefer to use some utility which adds trace support to
specific methods which are annotated. It could simplify the usage of the
tracing but requires java proxy. But this is an independent question.
> Remove HTrace support
> -
>
> Key: HADOOP-15566
> URL: https://issues.apache.org/jira/browse/HADOOP-15566
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 3.1.0
>Reporter: Todd Lipcon
>Priority: Major
> Labels: security
> Attachments: Screen Shot 2018-06-29 at 11.59.16 AM.png,
> ss-trace-s3a.png
>
>
> The HTrace incubator project has voted to retire itself and won't be making
> further releases. The Hadoop project currently has various hooks with HTrace.
> It seems in some cases (eg HDFS-13702) these hooks have had measurable
> performance overhead. Given these two factors, I think we should consider
> removing the HTrace integration. If there is someone willing to do the work,
> replacing it with OpenTracing might be a better choice since there is an
> active community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15993) Upgrade Kafka version in hadoop-kafka module
Akira Ajisaka created HADOOP-15993: -- Summary: Upgrade Kafka version in hadoop-kafka module Key: HADOOP-15993 URL: https://issues.apache.org/jira/browse/HADOOP-15993 Project: Hadoop Common Issue Type: Improvement Components: build, security Reporter: Akira Ajisaka Now the version is 0.8.2.1 and it has net.jpountz.lz4:lz4:1.2.0 dependency, which is vulnerable. (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-4611) Let's upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
