[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710904#comment-16710904
]
Weiwei Yang commented on YARN-9057:
---
Thanks [~eyang]!
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: YARN-9057.001.patch, YARN-9057.002.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710642#comment-16710642
]
Hudson commented on YARN-9057:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15562 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15562/])
YARN-9057. Removed third party class bundle from CSI jar file.
(eyang: rev 1dabb31cdf907cbee418c469368a59393fd52844)
* (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
* (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: YARN-9057.001.patch, YARN-9057.002.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710623#comment-16710623
]
Eric Yang commented on YARN-9057:
-
Thank you for the patch [~cheersyang].
+1 for patch 002. Will commit shortly.
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9057.001.patch, YARN-9057.002.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709684#comment-16709684
]
Hadoop QA commented on YARN-9057:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
52m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-csi in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9057 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950647/YARN-9057.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux f34a21f8b6ff 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 228156c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22781/testReport/ |
| Max. process+thread count | 332 (vs. ulimit of 1) |
| modules | C: hadoop-assemblies
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22781/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> CSI jar file should not
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709637#comment-16709637
]
Weiwei Yang commented on YARN-9057:
---
Hi [~eyang], using provided scope for hadoop-yarn-api/hadoop-yarn-common
resolves this problem. Please help to review. Thanks
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9057.001.patch, YARN-9057.002.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709465#comment-16709465
]
Weiwei Yang commented on YARN-9057:
---
Thanks [~eyang], indeed that is not expected. Not sure why copy dependencies
would ever move existing jars. Let me check, thanks!
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9057.001.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709320#comment-16709320
]
Eric Yang commented on YARN-9057:
-
[~cheersyang] Thank you for the patch.
HADOOP_HOME/share/hadoop/yarn/csi/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar has the
right content.
However, I get error message when launching application:
{code}
$ ./bin/yarn app -status abc
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/conf/YarnConfiguration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at
sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.conf.YarnConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
{code}
When I look in HADOOP_HOME/share/hadoop/yarn, hadoop-yarn-api-*.jar file is
missing. It copied hadoop-yarn-api-*.jar into:
{code}
$ tar tfvz hadoop-3.3.0-SNAPSHOT.tar.gz |grep yarn-api
-rw-rw-r-- eyang/eyang 3369775 2018-12-04 16:46
hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/csi/lib/hadoop-yarn-api-3.3.0-SNAPSHOT.jar
{code}
Seems like an unexpected behavior.
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9057.001.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708658#comment-16708658
]
Hadoop QA commented on YARN-9057:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
50m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-csi in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9057 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950539/YARN-9057.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux eba0bb6a87be 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / de42555 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22777/testReport/ |
| Max. process+thread count | 339 (vs. ulimit of 1) |
| modules | C: hadoop-assemblies
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22777/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> CSI jar file should not
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708581#comment-16708581
]
Weiwei Yang commented on YARN-9057:
---
Attached patch to remove the shading code, and yarn-csi copies its dependencies
to share/hadoop/yarn/csi/lib, so this is self-contained. This will run with its
own classpath, I have tried to use AUX service to launch the service, it works
fine.
[~sunilg], [~ste...@apache.org], [~eyang], pls help to review.
Thanks.
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9057.001.patch
>
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708496#comment-16708496
]
Steve Loughran commented on YARN-9057:
--
bq. it it seems to create more problems than the ones it fixed.
Afraid so.
General practise in hadoop-*: unshaded in all our cross references, moving to
shaded for public artifacts (which we still need to do for the object stores).
And we dream of a java9-only world...
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708109#comment-16708109
]
Weiwei Yang commented on YARN-9057:
---
Thanks [~eyang], [~ste...@apache.org]. Let me revisit this shading again, I was
trying to use such shading to avoid dependency conflicts, but it seems to
create more problems than the ones it fixed. In CSI deployment, we can also
wrap up them as individual containers and launch as a simple YRAN service app.
In that case, the shading is no longer needed. Will try and upload a patch
today.
Thanks.
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707513#comment-16707513
]
Steve Loughran commented on YARN-9057:
--
FWIW I'm not rebasing any of my ongoing work onto trunk until shading is
optional. I'm building multiple times an hour, either the fill binary
distribution or a set of JARs for use downstream. This adds minutes to the
round trip time. Uprating to blocker
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Priority: Major
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9057) CSI jar file should not bundle third party dependencies
[
https://issues.apache.org/jira/browse/YARN-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707508#comment-16707508
]
Steve Loughran commented on YARN-9057:
--
I agree. The shaded JAR must be a secondary -shaded artifact which is not
created when the -DskipShade option is passed to the build
> CSI jar file should not bundle third party dependencies
> ---
>
> Key: YARN-9057
> URL: https://issues.apache.org/jira/browse/YARN-9057
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Priority: Major
>
> hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
> shaded jar instead of CSI only classes. This is generating error messages
> for YARN cli:
> {code}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
