and to be clear, this is not in the REPL or with Hive (both well known situations in which these errors arise)
On Mon, Jan 16, 2017 at 11:51 PM, Koert Kuipers <ko...@tresata.com> wrote: > i am experiencing a ScalaReflectionException exception when doing an > aggregation on a spark-sql DataFrame. the error looks like this: > > Exception in thread "main" scala.ScalaReflectionException: class > <myclass> in JavaMirror with sun.misc.Launcher$AppClassLoader@28d93b30 of > type class sun.misc.Launcher$AppClassLoader with classpath [<classpath>] > not found. > at scala.reflect.internal.Mirrors$RootsBase.staticClass( > Mirrors.scala:123) > at scala.reflect.internal.Mirrors$RootsBase.staticClass( > Mirrors.scala:22) > at <my.package> > at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$ > lzycompute(TypeTags.scala:232) > at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) > at org.apache.spark.sql.SQLImplicits$$typecreator9$1. > apply(SQLImplicits.scala:127) > at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$ > lzycompute(TypeTags.scala:232) > at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) > at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$. > apply(ExpressionEncoder.scala:49) > at org.apache.spark.sql.SQLImplicits.newProductSeqEncoder( > SQLImplicits.scala:127) > at <my.package> > > > some things to note: > * <claspath> contains driver-class-path as indicated by me using > spark-submit, and all the jars that spark added. but it does not contain my > own assembly jar which contains <myclass> > * the class that is missing is a simple case class that is only used in > the aggregators on the executors, never driver-side > * i am running spark 2.1.0 with java 8 on yarn, but i can reproduce the > same error in local mode > > what is this classloader that excludes my jar? > the error looks somewhat like SPARK-8470, but i am not using hive, and > spark was not build with hive support. > > i can fix the error by adding my assembly jar to driver-classpath, but > that feels like a hack. > > thanks, > koert > >
