Re: Not understanding "jars" & "conf": {"spark.jars":" options in POST /sessions

[Kyunam Kim]

It turns out I need to prefix with file:// in either "jars" or "spark.jars".

Thanks,
Q

On Thu, Feb 1, 2018 at 2:10 PM, Kyunam Kim  wrote:

> Here are my findings.
> If you could explain why I'm seeing these behaviors, that'll be great.
>
> A. I have to specify BOTH jars and spark.jars
> I have to have jars locally in the Livy server AND same jars in HDFS.
> For example, {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf":
> {"spark.jars": "/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"
> true"}}
>
> Why do I need to specify both "jars" and "spark.jars"? Is this as designed?
>
> B. error: not found: value Source
> 'myUber.jar' contains 'Source' class. When {"jars": ["/tmp/myUber.jar"],
> "conf": {"spark.jars": "/tmp/myUber.jar","spark.files.overwrite":"true"}}
> is used, it doesn't see 'Source' class. Notice only 'myUber.jar' is
> included.
>
> I had to supply the original library jar that contains 'Source' class in
> addition to 'myUber.jar' as follows.
> {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf": {"spark.jars":
> "/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}}
> So, 'Source' class exists in 'myLib0.jar' and 'myUber.jar'.
>
> I also tried {"jars": ["/tmp/myLib0.jar,/tmp/myApp.jar"], "conf":
> {"spark.jars": "/tmp/myLib0.jar,/tmp/myApp.jar","spark.files.overwrite":"
> true"}}
> In this case, 'Source' only exists in 'myLib0.jar'. But the same error
> pops up.
>
> Why 'Source' has be to be repeated through multiple jars?
>
> Thanks,
> Q
>



-- 

Thanks,
Q

Not understanding "jars" & "conf": {"spark.jars":" options in POST /sessions

[Kyunam Kim]

Here are my findings.
If you could explain why I'm seeing these behaviors, that'll be great.

A. I have to specify BOTH jars and spark.jars
I have to have jars locally in the Livy server AND same jars in HDFS.
For example, {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf":
{"spark.jars":
"/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}}

Why do I need to specify both "jars" and "spark.jars"? Is this as designed?

B. error: not found: value Source
'myUber.jar' contains 'Source' class. When {"jars": ["/tmp/myUber.jar"],
"conf": {"spark.jars": "/tmp/myUber.jar","spark.files.overwrite":"true"}}
is used, it doesn't see 'Source' class. Notice only 'myUber.jar' is
included.

I had to supply the original library jar that contains 'Source' class in
addition to 'myUber.jar' as follows.
{"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf": {"spark.jars":
"/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}}
So, 'Source' class exists in 'myLib0.jar' and 'myUber.jar'.

I also tried {"jars": ["/tmp/myLib0.jar,/tmp/myApp.jar"], "conf":
{"spark.jars":
"/tmp/myLib0.jar,/tmp/myApp.jar","spark.files.overwrite":"true"}}
In this case, 'Source' only exists in 'myLib0.jar'. But the same error pops
up.

Why 'Source' has be to be repeated through multiple jars?

Thanks,
Q