It turns out I need to prefix with file:// in either "jars" or "spark.jars".
Thanks, Q On Thu, Feb 1, 2018 at 2:10 PM, Kyunam Kim <[email protected]> wrote: > Here are my findings. > If you could explain why I'm seeing these behaviors, that'll be great. > > A. I have to specify BOTH jars and spark.jars > I have to have jars locally in the Livy server AND same jars in HDFS. > For example, {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf": > {"spark.jars": "/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":" > true"}} > > Why do I need to specify both "jars" and "spark.jars"? Is this as designed? > > B. error: not found: value Source > 'myUber.jar' contains 'Source' class. When {"jars": ["/tmp/myUber.jar"], > "conf": {"spark.jars": "/tmp/myUber.jar","spark.files.overwrite":"true"}} > is used, it doesn't see 'Source' class. Notice only 'myUber.jar' is > included. > > I had to supply the original library jar that contains 'Source' class in > addition to 'myUber.jar' as follows. > {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf": {"spark.jars": > "/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}} > So, 'Source' class exists in 'myLib0.jar' and 'myUber.jar'. > > I also tried {"jars": ["/tmp/myLib0.jar,/tmp/myApp.jar"], "conf": > {"spark.jars": "/tmp/myLib0.jar,/tmp/myApp.jar","spark.files.overwrite":" > true"}} > In this case, 'Source' only exists in 'myLib0.jar'. But the same error > pops up. > > Why 'Source' has be to be repeated through multiple jars? > > Thanks, > Q > -- Thanks, Q
