H SaiSai!
Thanks for your reply.
About how we use livy:

   - We have a service done in scala running in spark which provide us
   semantic search features. This is call Recommender.
   - We need to provide these results externally.
   - Our point is to create a REST API (java) that request Livy who will
   send the job to spark as well as will gather the results, so:
      - REST HTTP(user) --> CUSTOM REST API (java) --> Livy --> Spark

Livy Configurations:

   - We using Livy 0.5. Spark 2.2.1.
   - We are using standalone spark, so in the config we
   have: livy.spark.master = spark://XX.XXX.XXX.XX:7077
   - Usage: POST /sessions with body {"kind": "spark"}
   - Fat jar included in rsc-jars/, size 150MB. Scala jar compiled in
   2.11.12 (although we have tried to lower up to 2.11.0 with same results).
   Here some of the family dependencies of this fat jar:
      - com.fasterxml.jackson.core (2.6.5) | org.elasticsearch (6.0.1)
      | org.apache.spark (2.2.1) | JohnSnowLabs-spark-nlp (1.2.3)
      | org.apache.tika-tika-parsers (1.17) | net.ruippeixotog (2.0.0)
      | com.typesafe.akka (2.5.9) | om.github.romix.akka (0.5.0)
      | com.github.fommil.netlib (1.1.2) | com.vitorsvieira (0.1.2)
      | com.carrotsearch (1.0.0) | org.apache.jena (3.6.0)

The jar is working fine using spark-submit or even being used by Zeppelin.
As is a big file, I tried using Livi's addJar (in the create session req)
so it takes the jar located in the master, but as I'm not working on YARN
mode, I think that my approach is wrong.

Complete error Stack:

NFO SparkEntries: Spark context finished initialization in
1642ms","18/02/22 16:02:52 INFO SparkEntries: Created Spark
session.","Exception in thread \"main\"
scala.reflect.internal.FatalError: object Predef does not have a
member classOf","\tat
scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)","\tat
scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)","\tat
scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)","\tat
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)","\tat
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)","\tat
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)","\tat
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)","\tat
scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)","\tat
scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)","\tat
scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)","\tat
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)","\tat
scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)","\tat
scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)","\tat
scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)","\tat
scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)","\tat
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)","\tat
scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)","\tat
scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)","\tat
scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)","\tat
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)","\tat
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:93)","\tat
scala.collection.Iterator$class.foreach(Iterator.scala:893)","\tat
scala.collection.AbstractIterator.foreach(Iterator.scala:1336)","\tat
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.run(Analyzer.scala:93)","\tat
scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1501)","\tat
scala.tools.nsc.Global$Run.compileUnits(Global.scala:1486)","\tat
scala.tools.nsc.Global$Run.compileSources(Global.scala:1481)","\tat
scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:435)","\tat
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:855)","\tat
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:813)","\tat
scala.tools.nsc.interpreter.IMain.bind(IMain.scala:675)","\tat
org.apache.livy.repl.SparkInterpreter$$anonfun$bind$1.apply(SparkInterpreter.scala:146)","\tat
org.apache.livy.repl.SparkInterpreter$$anonfun$bind$1.apply(SparkInterpreter.scala:146)","\tat
scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)","\tat
org.apache.livy.repl.SparkInterpreter.bind(SparkInterpreter.scala:145)","\tat
org.apache.livy.repl.AbstractSparkInterpreter.postStart(AbstractSparkInterpreter.scala:72)","\tat
org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:95)","\tat
org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:70)","\tat
org.apache.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:70)","\tat
org.apache.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:340)","\tat
org.apache.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:70)","\tat
org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:128)","\tat
org.apache.livy.repl.Session$$anonfun$1.apply(Session.scala:122)","\tat
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)","\tat
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)","\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)","\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)","\tat
java.lang.Thread.run(Thread.java:745)","18/02/22 16:02:52 WARN
SingleThreadEventExecutor: An event executor terminated with non-empty
task queue (1)"]}


Some help would be very grateful!
David

2018-02-23 7:34 GMT+01:00 Saisai Shao <sai.sai.s...@gmail.com>:

> This seems like a Scala version issue (as far as I can see from the
> stack). Would you please describe the details about how you use Livy and
> what configurations you set?
>
> Thanks
> Jerry
>
> 2018-02-23 6:30 GMT+08:00 David Espinosa <espi...@gmail.com>:
>
>> Hi all,
>>
>> I'm a new user in Livy. I've created a scala app that runs well with a
>> Spark 2.2.1. The version of scala used is 2.11.12..
>> I have tried to create a session in Livy 0.5.0 using this jar, and once
>> the Spark Session is created an error raises and the session goes dead.
>>
>> This is a part of the error Stak:
>>
>> "18/02/22 17:00:23 INFO SparkEntries: Spark context finished
>> initialization in 2282ms",
>>         "18/02/22 17:00:24 INFO SparkEntries: Created Spark session.",
>>         "Exception in thread \"main\" scala.reflect.internal.FatalError:
>> object Predef does not have a member classOf",
>>         "\tat scala.reflect.internal.Definit
>> ions$DefinitionsClass.scala$reflect$internal$Definitions$Def
>> initionsClass$$fatalMissingSymbol(Definitions.scala:1186)",
>>         "\tat scala.reflect.internal.Definit
>> ions$DefinitionsClass.getMember(Definitions.scala:1203)",
>>         "\tat scala.reflect.internal.Definit
>> ions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)",
>>
>> Has somebody found and fixed a problem like this?
>>
>> Thanks in advance,
>> David
>>
>>
>>
>

Reply via email to