[
https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated SPARK-5172:
--------------------------------
Description:
Steps to check it:
1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from
http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`.
3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It
doesn't exist in hadoop 2.4.
4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info"
{code}
Compiled from "package-info.java"
interface org.apache.hadoop.package-info
SourceFile: "package-info.java"
RuntimeVisibleAnnotations: length = 0x24
00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
11 73 00 12
minor version: 0
major version: 50
Constant pool:
const #1 = Asciz org/apache/hadoop/package-info;
const #2 = class #1; // "org/apache/hadoop/package-info"
const #3 = Asciz java/lang/Object;
const #4 = class #3; // java/lang/Object
const #5 = Asciz package-info.java;
const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;;
const #7 = Asciz version;
const #8 = Asciz 1.2.1;
const #9 = Asciz revision;
const #10 = Asciz 1503152;
const #11 = Asciz user;
const #12 = Asciz mattf;
const #13 = Asciz date;
const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013;
const #15 = Asciz url;
const #16 = Asciz
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
const #17 = Asciz srcChecksum;
const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a;
const #19 = Asciz SourceFile;
const #20 = Asciz RuntimeVisibleAnnotations;
{
}
{code}
The version is {{1.2.1}}
It comes because a wrong hbase version settings in examples project. Here is a
port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4
-Dhadoop.version=2.4.0 -pl examples dependency:tree"
{noformat}
[INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
[INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile
[INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile
[INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
[INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile
[INFO] | | \- asm:asm:jar:3.3.1:test
[INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
[INFO] | +-
org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
[INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | | \- commons-el:commons-el:jar:1.0:compile
[INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
[INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
[INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
[INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
[INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
[INFO] | +-
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] | \- junit:junit:jar:4.10:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test
{noformat}
If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am
dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right.
was:
Steps to check it:
1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from
http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
2. unzip `spark-assembly-1.2.0-hadoop2.4.0.jar`.
3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It
doesn't exist in hadoop 2.4.
4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info"
{code}
Compiled from "package-info.java"
interface org.apache.hadoop.package-info
SourceFile: "package-info.java"
RuntimeVisibleAnnotations: length = 0x24
00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
11 73 00 12
minor version: 0
major version: 50
Constant pool:
const #1 = Asciz org/apache/hadoop/package-info;
const #2 = class #1; // "org/apache/hadoop/package-info"
const #3 = Asciz java/lang/Object;
const #4 = class #3; // java/lang/Object
const #5 = Asciz package-info.java;
const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;;
const #7 = Asciz version;
const #8 = Asciz 1.2.1;
const #9 = Asciz revision;
const #10 = Asciz 1503152;
const #11 = Asciz user;
const #12 = Asciz mattf;
const #13 = Asciz date;
const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013;
const #15 = Asciz url;
const #16 = Asciz
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
const #17 = Asciz srcChecksum;
const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a;
const #19 = Asciz SourceFile;
const #20 = Asciz RuntimeVisibleAnnotations;
{
}
{code}
The version is {{1.2.1}}
It comes because a wrong hbase version settings in examples project. Here is a
port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4
-Dhadoop.version=2.4.0 -pl examples dependency:tree"
{noformat}
[INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
[INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile
[INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile
[INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
[INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile
[INFO] | | \- asm:asm:jar:3.3.1:test
[INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
[INFO] | +-
org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
[INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | | \- commons-el:commons-el:jar:1.0:compile
[INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
[INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
[INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
[INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
[INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
[INFO] | +-
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] | \- junit:junit:jar:4.10:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test
{noformat}
If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am
dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right.
> spark-examples-***.jar shades a wrong Hadoop distribution
> ---------------------------------------------------------
>
> Key: SPARK-5172
> URL: https://issues.apache.org/jira/browse/SPARK-5172
> Project: Spark
> Issue Type: Bug
> Components: Build
> Reporter: Shixiong Zhu
> Priority: Minor
>
> Steps to check it:
> 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from
> http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
> 2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`.
> 3. There is a file called `org/apache/hadoop/package-info.class` in the jar.
> It doesn't exist in hadoop 2.4.
> 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info"
> {code}
> Compiled from "package-info.java"
> interface org.apache.hadoop.package-info
> SourceFile: "package-info.java"
> RuntimeVisibleAnnotations: length = 0x24
> 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
> 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
> 11 73 00 12
> minor version: 0
> major version: 50
> Constant pool:
> const #1 = Asciz org/apache/hadoop/package-info;
> const #2 = class #1; // "org/apache/hadoop/package-info"
> const #3 = Asciz java/lang/Object;
> const #4 = class #3; // java/lang/Object
> const #5 = Asciz package-info.java;
> const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;;
> const #7 = Asciz version;
> const #8 = Asciz 1.2.1;
> const #9 = Asciz revision;
> const #10 = Asciz 1503152;
> const #11 = Asciz user;
> const #12 = Asciz mattf;
> const #13 = Asciz date;
> const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013;
> const #15 = Asciz url;
> const #16 = Asciz
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
> const #17 = Asciz srcChecksum;
> const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a;
> const #19 = Asciz SourceFile;
> const #20 = Asciz RuntimeVisibleAnnotations;
> {
> }
> {code}
> The version is {{1.2.1}}
> It comes because a wrong hbase version settings in examples project. Here is
> a port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.4.0 -pl examples dependency:tree"
> {noformat}
> [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
> [INFO] | +-
> org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | +-
> org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile
> [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile
> [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
> [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile
> [INFO] | | \- asm:asm:jar:3.3.1:test
> [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
> [INFO] | +-
> org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
> [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] | | \- commons-el:commons-el:jar:1.0:compile
> [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
> [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
> [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
> [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
> [INFO] | +-
> com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] | \- junit:junit:jar:4.10:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test
> {noformat}
> If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am
> dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
