Interesting idea. Tried that, didn't work. Here is my new SBT file:
name := """testMain"""
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.1" % "provided",
"org.apache.spark" %% "spark-sql" % "1.4.1" % "provided",
"org.scala-lang" % "scala-reflect" % "2.11.6"
)
On Mon, Sep 7, 2015 at 9:55 PM, Jonathan Coveney <jcove...@gmail.com> wrote:
> Try adding the following to your build.sbt
>
> libraryDependencies += "org.scala-lang" % "scala-reflect" % "2.11.6"
>
>
> I believe that spark shades the scala library, and this is a library that it
> looks like you need in an unshaded way.
>
>
> 2015-09-07 16:48 GMT-04:00 Gheorghe Postelnicu <
> gheorghe.posteln...@gmail.com>:
>
>> Hi,
>>
>> The following code fails when compiled from SBT:
>>
>> package main.scala
>>
>> import org.apache.spark.SparkContext
>> import org.apache.spark.sql.SQLContext
>>
>> object TestMain {
>> def main(args: Array[String]): Unit = {
>> implicit val sparkContext = new SparkContext()
>> val sqlContext = new SQLContext(sparkContext)
>> import sqlContext.implicits._
>> sparkContext.parallelize(1 to 10).map(i => (i,
>> i.toString)).toDF("intCol", "strCol")
>> }
>> }
>>
>> with the following error:
>>
>> 15/09/07 21:39:21 INFO BlockManagerMaster: Registered BlockManager
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
>> at main.scala.Bof$.main(Bof.scala:14)
>> at main.scala.Bof.main(Bof.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> 15/09/07 21:39:22 INFO SparkContext: Invoking stop() from shutdown hook
>>
>> whereas the code above works in a spark shell.
>>
>> The code is compiled using Scala 2.11.6 and precompiled Spark 1.4.1
>>
>> Any suggestion on how to fix this would be much appreciated.
>>
>> Best,
>> Gheorghe
>>
>>
>
