[
https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell resolved SPARK-5172.
------------------------------------
Resolution: Fixed
Fix Version/s: 1.3.0
Assignee: Sean Owen
> spark-examples-***.jar shades a wrong Hadoop distribution
> ---------------------------------------------------------
>
> Key: SPARK-5172
> URL: https://issues.apache.org/jira/browse/SPARK-5172
> Project: Spark
> Issue Type: Bug
> Components: Build
> Reporter: Shixiong Zhu
> Assignee: Sean Owen
> Priority: Minor
> Fix For: 1.3.0
>
>
> Steps to check it:
> 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from
> http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
> 2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`.
> 3. There is a file called `org/apache/hadoop/package-info.class` in the jar.
> It doesn't exist in hadoop 2.4.
> 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info"
> {code}
> Compiled from "package-info.java"
> interface org.apache.hadoop.package-info
> SourceFile: "package-info.java"
> RuntimeVisibleAnnotations: length = 0x24
> 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
> 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
> 11 73 00 12
> minor version: 0
> major version: 50
> Constant pool:
> const #1 = Asciz org/apache/hadoop/package-info;
> const #2 = class #1; // "org/apache/hadoop/package-info"
> const #3 = Asciz java/lang/Object;
> const #4 = class #3; // java/lang/Object
> const #5 = Asciz package-info.java;
> const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;;
> const #7 = Asciz version;
> const #8 = Asciz 1.2.1;
> const #9 = Asciz revision;
> const #10 = Asciz 1503152;
> const #11 = Asciz user;
> const #12 = Asciz mattf;
> const #13 = Asciz date;
> const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013;
> const #15 = Asciz url;
> const #16 = Asciz
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
> const #17 = Asciz srcChecksum;
> const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a;
> const #19 = Asciz SourceFile;
> const #20 = Asciz RuntimeVisibleAnnotations;
> {
> }
> {code}
> The version is {{1.2.1}}
> It comes because a wrong hbase version settings in examples project. Here is
> a part of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.4.0 -pl examples dependency:tree"
> {noformat}
> [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
> [INFO] | +-
> org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | +-
> org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile
> [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile
> [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
> [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile
> [INFO] | | \- asm:asm:jar:3.3.1:test
> [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
> [INFO] | +-
> org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
> [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] | | \- commons-el:commons-el:jar:1.0:compile
> [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
> [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
> [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
> [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
> [INFO] | +-
> com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] | \- junit:junit:jar:4.10:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test
> {noformat}
> If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am
> dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
