Mark Grover created SPARK-20756:
-----------------------------------
Summary: yarn-shuffle jar has references to unshaded guava and
contains scala classes
Key: SPARK-20756
URL: https://issues.apache.org/jira/browse/SPARK-20756
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 2.2.0
Reporter: Mark Grover
There are 2 problems with yarn's shuffle jar currently:
1. It contains shaded guava but it contains references to unshaded classes.
{code}
# Guava is correctly relocated
>jar -tf common/network-yarn/target/scala-2.11/spark*yarn-shuffle.jar | grep
>guava | head
META-INF/maven/com.google.guava/
META-INF/maven/com.google.guava/guava/
META-INF/maven/com.google.guava/guava/pom.properties
META-INF/maven/com.google.guava/guava/pom.xml
org/spark_project/guava/
org/spark_project/guava/annotations/
org/spark_project/guava/annotations/Beta.class
org/spark_project/guava/annotations/GwtCompatible.class
org/spark_project/guava/annotations/GwtIncompatible.class
org/spark_project/guava/annotations/VisibleForTesting.class
# But, there are still references to unshaded guava
>javap -cp common/network-yarn/target/scala-2.11/spark*yarn-shuffle.jar -c
>org/apache/spark/network/yarn/YarnShuffleService | grep google
57: invokestatic #139 // Method
com/google/common/collect/Lists.newArrayList:()Ljava/util/ArrayList;
{code}
2. There are references to scala classes in the uber jar:
{code}
jar -tf
/opt/src/spark/common/network-yarn/target/scala-2.11/spark-*yarn-shuffle.jar |
grep "^scala"
scala/AnyVal.class
{code}
We should fix this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
