Re: Can not deserialize instance of scala.collection.immutable.List out of VALUE_STRING (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest)
I am now so embarrased...
Thanks Marcelo !!!
On Mon, Jan 29, 2018 at 3:37 PM, Marcelo Vanzin wrote:
> "jars" is a list so you should probably try:
>
> "jars": [ "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar" ]
>
> On Mon, Jan 29, 2018 at 3:35 PM, Kyunam Kim wrote:
> > I get the following when I try "jars" in POST /sessions.
> >
> > I tried:
> > - "jars": "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"
> > - "jars": "/user/root/1x/my-lib-SNAPSHOT.jar"
> >
> > livy.spark.deploy-mode = cluster (client is also tried)
> >
> > Basically, I just want to load my custom library into LIVY.
> >
> > By the way, even if this works, do I need to let Spark know about my
> custom
> > library by using "conf:
> > {"spark.jars":"hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"}"
> ??
> >
> >
> > curl -k -v -H 'Content-Type: application/json' -X POST -d
> > '{"kind":"spark","driverMemory":"1000M","executorCores":2,"jars":
> > "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"}'
> > "http://myserver:8998/sessions";
> > Note: Unnecessary use of -X or --request, POST is already inferred.
> > * Trying 10.49.51.117...
> > * TCP_NODELAY set
> > * Connected to ... port 8998 (#0)
> >> POST /sessions HTTP/1.1
> >> Host: ...:8998
> >> User-Agent: curl/7.55.1
> >> Accept: */*
> >> Content-Type: application/json
> >> Content-Length: 139
> >>
> > * upload completely sent off: 139 out of 139 bytes
> > < HTTP/1.1 400 Bad Request
> > < Date: Mon, 29 Jan 2018 23:21:04 GMT
> > < Content-Type: application/json; charset=UTF-8
> > < Content-Length: 251
> > < Server: Jetty(9.2.16.v20160414)
> > <
> > * Connection #0 to host myserver left intact
> > "Can not deserialize instance of scala.collection.immutable.List out of
> > VALUE_STRING token\n at [Source: HttpInputOverHTTP@1f8a778f; line: 1,
> > column: 57] (through reference chain:
> > org.apache.livy.server.interactive.CreateInteractiveRequest[\"jars\"])"
> >
> > Thanks,
> > Q
>
>
>
> --
> Marcelo
>
--
Thanks,
Q
Re: In POST /sessions, what's the difference between "jars" and "spark.jar" in conf ?
Thanks Jerry. When I tried "jars", I get Can not deserialize instance of scala.collection.immutable.List out of VALUE_STRING token\n at [Source: HttpInputOverHTTP@1f8a778f; line: 1, column: 57] (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest[\"jars\"]). I just started a new thread on this topic. Thanks, Q On Sun, Jan 28, 2018 at 10:09 PM, Saisai Shao wrote: > I think they are basically the same, you can use either. Why don't you > take a try. > > Thanks > Jerry > > 2018-01-29 12:37 GMT+08:00 Kyunam Kim : > >> Basically, which option is equivalent to "spark-submit --jar >> myLib0,jar,myLib1.jar" ? >> >> Thanks, >> Q >> > > -- Thanks, Q
Re: Can not deserialize instance of scala.collection.immutable.List out of VALUE_STRING (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest)
"jars" is a list so you should probably try:
"jars": [ "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar" ]
On Mon, Jan 29, 2018 at 3:35 PM, Kyunam Kim wrote:
> I get the following when I try "jars" in POST /sessions.
>
> I tried:
> - "jars": "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"
> - "jars": "/user/root/1x/my-lib-SNAPSHOT.jar"
>
> livy.spark.deploy-mode = cluster (client is also tried)
>
> Basically, I just want to load my custom library into LIVY.
>
> By the way, even if this works, do I need to let Spark know about my custom
> library by using "conf:
> {"spark.jars":"hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"}" ??
>
>
> curl -k -v -H 'Content-Type: application/json' -X POST -d
> '{"kind":"spark","driverMemory":"1000M","executorCores":2,"jars":
> "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"}'
> "http://myserver:8998/sessions";
> Note: Unnecessary use of -X or --request, POST is already inferred.
> * Trying 10.49.51.117...
> * TCP_NODELAY set
> * Connected to ... port 8998 (#0)
>> POST /sessions HTTP/1.1
>> Host: ...:8998
>> User-Agent: curl/7.55.1
>> Accept: */*
>> Content-Type: application/json
>> Content-Length: 139
>>
> * upload completely sent off: 139 out of 139 bytes
> < HTTP/1.1 400 Bad Request
> < Date: Mon, 29 Jan 2018 23:21:04 GMT
> < Content-Type: application/json; charset=UTF-8
> < Content-Length: 251
> < Server: Jetty(9.2.16.v20160414)
> <
> * Connection #0 to host myserver left intact
> "Can not deserialize instance of scala.collection.immutable.List out of
> VALUE_STRING token\n at [Source: HttpInputOverHTTP@1f8a778f; line: 1,
> column: 57] (through reference chain:
> org.apache.livy.server.interactive.CreateInteractiveRequest[\"jars\"])"
>
> Thanks,
> Q
--
Marcelo
Can not deserialize instance of scala.collection.immutable.List out of VALUE_STRING (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest)
I get the following when I try "jars" in POST /sessions.
I tried:
- "jars": "hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"
- "jars": "/user/root/1x/my-lib-SNAPSHOT.jar"
livy.spark.deploy-mode = cluster (client is also tried)
Basically, I just want to load my custom library into LIVY.
By the way, even if this works, do I need to let Spark know about my custom
library by using "conf: {"spark.jars":
"hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"}" ??
curl -k -v -H 'Content-Type: application/json' -X POST -d
'{"kind":"spark","driverMemory":"1000M","executorCores":2,"jars":
"hdfs://myserver:8020/user/root/1x/my-lib-SNAPSHOT.jar"}' "
http://myserver:8998/sessions";
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 10.49.51.117...
* TCP_NODELAY set
* Connected to ... port 8998 (#0)
> POST /sessions HTTP/1.1
> Host: ...:8998
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 139
>
* upload completely sent off: 139 out of 139 bytes
< HTTP/1.1 400 Bad Request
< Date: Mon, 29 Jan 2018 23:21:04 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 251
< Server: Jetty(9.2.16.v20160414)
<
* Connection #0 to host myserver left intact
"Can not deserialize instance of scala.collection.immutable.List out of
VALUE_STRING token\n at [Source: HttpInputOverHTTP@1f8a778f; line: 1,
column: 57] (through reference chain:
org.apache.livy.server.interactive.CreateInteractiveRequest[\"jars\"])"
Thanks,
Q
