[
https://issues.apache.org/jira/browse/SPARK-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001366#comment-14001366
]
Patrick Wendell commented on SPARK-1875:
----------------------------------------
The issue here is that somehow the commons-lang exclusion from the hive project
is being respected when building an assembly for Hadoop 1. So it's excluded
from hadoop-client even though hadoop-client 1.0.4 depends on it.
{code}
mvn -Phive install
mvn -pl assembly -Phive dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Assembly 1.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ spark-assembly_2.10
---
[INFO] org.apache.spark:spark-assembly_2.10:pom:1.0.1-SNAPSHOT
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.0.1-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-client:jar:1.0.4:compile
[INFO] | | \- org.apache.hadoop:hadoop-core:jar:1.0.4:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- org.apache.commons:commons-math:jar:2.1:compile
[INFO] | | +- commons-el:commons-el:jar:1.0:compile
[INFO] | | +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] | | \- oro:oro:jar:2.0.8:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:runtime
[INFO] | | +- commons-codec:commons-codec:jar:1.3:compile
[INFO] | | \- commons-httpclient:commons-httpclient:jar:3.1:runtime
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] | | | \- org.apache.curator:curator-client:jar:2.4.0:compile
[INFO] | | \- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] | +- org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031:compile
[INFO] | | +-
org.eclipse.jetty.orbit:javax.transaction:jar:1.1.1.v201105210645:compile
[INFO] | | +- org.eclipse.jetty:jetty-webapp:jar:8.1.14.v20131031:compile
[INFO] | | | +- org.eclipse.jetty:jetty-xml:jar:8.1.14.v20131031:compile
[INFO] | | | \- org.eclipse.jetty:jetty-servlet:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty:jetty-jndi:jar:8.1.14.v20131031:compile
[INFO] | | \-
org.eclipse.jetty.orbit:javax.mail.glassfish:jar:1.4.1.v201005082020:compile
[INFO] | | \-
org.eclipse.jetty.orbit:javax.activation:jar:1.1.0.v201105071233:compile
[INFO] | +- org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:compile
[INFO] | +- org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:compile
[INFO] | +- org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:compile
[INFO] | | +-
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
[INFO] | | +-
org.eclipse.jetty:jetty-continuation:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty:jetty-http:jar:8.1.14.v20131031:compile
[INFO] | | \- org.eclipse.jetty:jetty-io:jar:8.1.14.v20131031:compile
{code}
If you run
{code}
mvn -pl assembly dependency:tree
{code}
it includes commons-lang correctly.
> NoClassDefFoundError: StringUtils when building against Hadoop 1
> ----------------------------------------------------------------
>
> Key: SPARK-1875
> URL: https://issues.apache.org/jira/browse/SPARK-1875
> Project: Spark
> Issue Type: Bug
> Reporter: Matei Zaharia
> Assignee: Guoqiang Li
> Priority: Blocker
> Fix For: 1.0.0
>
>
> Maybe I missed something, but after building an assembly with Hadoop 1.2.1
> and Hive enabled, if I go into it and run spark-shell, I get this:
> {code}
> java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
> at
> org.apache.hadoop.metrics2.lib.MetricMutableStat.<init>(MetricMutableStat.java:59)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.<init>(MetricsSystemImpl.java:75)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.<init>(MetricsSystemImpl.java:120)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:216)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:184)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:236)
> at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:209)
> at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:226)
> at
> org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:36)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:109)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:228)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
