When building off of master it looks like the wrong version of the jar is
added to the assembly jar. The fix for SPARK-3039 added the "hadoop2"
classifier at compile time when using the hadoop2 profile, but in the
assembly stage I get:
"[INFO] Including org.apache.avro:avro-mapred:jar:1.7.6 in the shaded
jar."
It's not clear if there are other dependencies that have classifiers being
added to the assembly. But when running with the current version I end up
with a common exception about TaskAttemptContext. After manually unjar-ing
the assembly and replacing the correct version of avro-mapred it works
correctly.
If I knew more about maven I'd try to help out with a PR to fix this.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/avro-mapred-hadoop2-not-in-assembly-jar-tp8790.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
