Re: Spark 1.0: slf4j version conflicts with pig
Remark, just including the jar built by sbt will produce the same
error. i,.e this pig script will fail:
REGISTER
/usr/share/osi1/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar;
edgeList0 = LOAD
'/user/rfcompton/twitter-mention-networks/bidirectional-network-current/part-r-1'
USING PigStorage() AS (id1:long, id2:long, weight:int);
ttt = LIMIT edgeList0 10;
DUMP ttt;
On Wed, May 28, 2014 at 12:55 PM, Ryan Compton compton.r...@gmail.com wrote:
It appears to be Spark 1.0 related. I made a pom.xml with a single
dependency on Spark, registering the resulting jar created the error.
Spark 1.0 was compiled via $ SPARK_HADOOP_VERSION=0.20.2-cdh3u4 sbt/sbt
assembly
The pom.xml, as well as some other information, is below. The only
thing that should not be standard is the inclusion of my in-house
repository (it's where I host the spark jar I compiled above).
project xmlns=http://maven.apache.org/POM/4.0.0;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
modelVersion4.0.0/modelVersion
groupIdcom.mycompany.app/groupId
artifactIdmy-app/artifactId
version1.0-SNAPSHOT/version
packagingjar/packaging
namemy-app/name
urlhttp://maven.apache.org/url
properties
maven.compiler.source1.6/maven.compiler.source
maven.compiler.target1.6/maven.compiler.target
encodingUTF-8/encoding
scala.version2.10.4/scala.version
/properties
build
pluginManagement
plugins
plugin
groupIdnet.alchim31.maven/groupId
artifactIdscala-maven-plugin/artifactId
version3.1.5/version
/plugin
plugin
groupIdorg.apache.maven.plugins/groupId
artifactIdmaven-compiler-plugin/artifactId
version2.0.2/version
/plugin
/plugins
/pluginManagement
plugins
plugin
groupIdnet.alchim31.maven/groupId
artifactIdscala-maven-plugin/artifactId
executions
execution
idscala-compile-first/id
phaseprocess-resources/phase
goals
goaladd-source/goal
goalcompile/goal
/goals
/execution
execution
idscala-test-compile/id
phaseprocess-test-resources/phase
goals
goaltestCompile/goal
/goals
/execution
/executions
/plugin
!-- Plugin to create a single jar that includes all
dependencies --
plugin
artifactIdmaven-assembly-plugin/artifactId
version2.4/version
configuration
descriptorRefs
descriptorRefjar-with-dependencies/descriptorRef
/descriptorRefs
/configuration
executions
execution
idmake-assembly/id
phasepackage/phase
goals
goalsingle/goal
/goals
/execution
/executions
/plugin
/plugins
/build
repositories
!-- needed for cdh build of Spark --
repository
idreleases/id
url10.10.1.29:8081/nexus/content/repositories/releases/url
/repository
repository
idcloudera/id
urlhttps://repository.cloudera.com/artifactory/cloudera-repos/url
/repository
/repositories
dependencies
dependency
groupIdorg.scala-lang/groupId
artifactIdscala-library/artifactId
version${scala.version}/version
/dependency
!--on node29--
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-assembly/artifactId
version1.0.0-cdh3u4/version
classifiercdh3u4/classifier
/dependency
!--spark docs says I need hadoop-client, cdh3u3 repo no
longer exists--
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-client/artifactId
version0.20.2-cdh3u4/version
/dependency
/dependencies
/project
Here's what I get in the dependency tree:
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ my-app ---
[INFO]
Re: Spark 1.0: slf4j version conflicts with pig
posted a JIRA https://issues.apache.org/jira/browse/SPARK-1952
On Wed, May 28, 2014 at 1:14 PM, Ryan Compton compton.r...@gmail.com wrote:
Remark, just including the jar built by sbt will produce the same
error. i,.e this pig script will fail:
REGISTER
/usr/share/osi1/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar;
edgeList0 = LOAD
'/user/rfcompton/twitter-mention-networks/bidirectional-network-current/part-r-1'
USING PigStorage() AS (id1:long, id2:long, weight:int);
ttt = LIMIT edgeList0 10;
DUMP ttt;
On Wed, May 28, 2014 at 12:55 PM, Ryan Compton compton.r...@gmail.com wrote:
It appears to be Spark 1.0 related. I made a pom.xml with a single
dependency on Spark, registering the resulting jar created the error.
Spark 1.0 was compiled via $ SPARK_HADOOP_VERSION=0.20.2-cdh3u4 sbt/sbt
assembly
The pom.xml, as well as some other information, is below. The only
thing that should not be standard is the inclusion of my in-house
repository (it's where I host the spark jar I compiled above).
project xmlns=http://maven.apache.org/POM/4.0.0;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
modelVersion4.0.0/modelVersion
groupIdcom.mycompany.app/groupId
artifactIdmy-app/artifactId
version1.0-SNAPSHOT/version
packagingjar/packaging
namemy-app/name
urlhttp://maven.apache.org/url
properties
maven.compiler.source1.6/maven.compiler.source
maven.compiler.target1.6/maven.compiler.target
encodingUTF-8/encoding
scala.version2.10.4/scala.version
/properties
build
pluginManagement
plugins
plugin
groupIdnet.alchim31.maven/groupId
artifactIdscala-maven-plugin/artifactId
version3.1.5/version
/plugin
plugin
groupIdorg.apache.maven.plugins/groupId
artifactIdmaven-compiler-plugin/artifactId
version2.0.2/version
/plugin
/plugins
/pluginManagement
plugins
plugin
groupIdnet.alchim31.maven/groupId
artifactIdscala-maven-plugin/artifactId
executions
execution
idscala-compile-first/id
phaseprocess-resources/phase
goals
goaladd-source/goal
goalcompile/goal
/goals
/execution
execution
idscala-test-compile/id
phaseprocess-test-resources/phase
goals
goaltestCompile/goal
/goals
/execution
/executions
/plugin
!-- Plugin to create a single jar that includes all
dependencies --
plugin
artifactIdmaven-assembly-plugin/artifactId
version2.4/version
configuration
descriptorRefs
descriptorRefjar-with-dependencies/descriptorRef
/descriptorRefs
/configuration
executions
execution
idmake-assembly/id
phasepackage/phase
goals
goalsingle/goal
/goals
/execution
/executions
/plugin
/plugins
/build
repositories
!-- needed for cdh build of Spark --
repository
idreleases/id
url10.10.1.29:8081/nexus/content/repositories/releases/url
/repository
repository
idcloudera/id
urlhttps://repository.cloudera.com/artifactory/cloudera-repos/url
/repository
/repositories
dependencies
dependency
groupIdorg.scala-lang/groupId
artifactIdscala-library/artifactId
version${scala.version}/version
/dependency
!--on node29--
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-assembly/artifactId
version1.0.0-cdh3u4/version
classifiercdh3u4/classifier
/dependency
!--spark docs says I need hadoop-client, cdh3u3 repo no
longer exists--
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-client/artifactId
version0.20.2-cdh3u4/version
/dependency
/dependencies
/project
Here's what
Spark 1.0: slf4j version conflicts with pig
I use both Pig and Spark. All my code is built with Maven into a giant *-jar-with-dependencies.jar. I recently upgraded to Spark 1.0 and now all my pig scripts fail with: Caused by: java.lang.RuntimeException: Could not resolve error that occured when launching map reduce job: java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:598) at java.lang.Thread.dispatchUncaughtException(Thread.java:1874) Did Spark 1.0 change the version of slf4j? I can't seem to find it via mvn dependency:tree
Re: Spark 1.0: slf4j version conflicts with pig
Spark uses 1.7.5, and you should probably see 1.7.{4,5} in use through
Hadoop. But those are compatible.
That method appears to have been around since 1.3. What version does Pig want?
I usually do mvn -Dverbose dependency:tree to see both what the
final dependencies are, and what got overwritten, to diagnose things
like this.
My hunch is that something is depending on an old slf4j in your build
and it's overwriting Spark et al.
On Tue, May 27, 2014 at 10:45 PM, Ryan Compton compton.r...@gmail.com wrote:
I use both Pig and Spark. All my code is built with Maven into a giant
*-jar-with-dependencies.jar. I recently upgraded to Spark 1.0 and now
all my pig scripts fail with:
Caused by: java.lang.RuntimeException: Could not resolve error that
occured when launching map reduce job: java.lang.NoSuchMethodError:
org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:598)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1874)
Did Spark 1.0 change the version of slf4j? I can't seem to find it via
mvn dependency:tree
