Here are my findings.
If you could explain why I'm seeing these behaviors, that'll be great.
A. I have to specify BOTH jars and spark.jars
I have to have jars locally in the Livy server AND same jars in HDFS.
For example, {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf":
{"spark.jars":
"/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}}
Why do I need to specify both "jars" and "spark.jars"? Is this as designed?
B. error: not found: value Source
'myUber.jar' contains 'Source' class. When {"jars": ["/tmp/myUber.jar"],
"conf": {"spark.jars": "/tmp/myUber.jar","spark.files.overwrite":"true"}}
is used, it doesn't see 'Source' class. Notice only 'myUber.jar' is
included.
I had to supply the original library jar that contains 'Source' class in
addition to 'myUber.jar' as follows.
{"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf": {"spark.jars":
"/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}}
So, 'Source' class exists in 'myLib0.jar' and 'myUber.jar'.
I also tried {"jars": ["/tmp/myLib0.jar,/tmp/myApp.jar"], "conf":
{"spark.jars":
"/tmp/myLib0.jar,/tmp/myApp.jar","spark.files.overwrite":"true"}}
In this case, 'Source' only exists in 'myLib0.jar'. But the same error pops
up.
Why 'Source' has be to be repeated through multiple jars?
Thanks,
Q
