Re: Livy with Spark package
Thanks and agreed :) On Thu, Aug 24, 2017 at 12:01 PM, Saisai Shao wrote: > You could set "spark.jars.packages" in `conf` field of session post API ( > https://github.com/apache/incubator-livy/blob/master/ > docs/rest-api.md#post-sessions). This is equal to --package in > spark-submit. > > BTW you'd better ask livy question in u...@livy.incubator.apache.org. > > Thanks > Jerry > > On Thu, Aug 24, 2017 at 8:11 AM, ayan guha wrote: > >> Hi >> >> I have a python program which I invoke as >> >> spark-submit --packages com.databricks:spark-avro_2.11:3.2.0 >> somefile.py "2017-08-23 02:00:00" and it works >> >> Now I want to submit this file using Livy. I could work out most of the >> stuff (like putting files to HDFS etc) but not able to understand how/where >> to configure the "packages" switch...Any help? >> -- >> Best Regards, >> Ayan Guha >> > > -- Best Regards, Ayan Guha
Re: Livy with Spark package
You could set "spark.jars.packages" in `conf` field of session post API ( https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md#post-sessions). This is equal to --package in spark-submit. BTW you'd better ask livy question in u...@livy.incubator.apache.org. Thanks Jerry On Thu, Aug 24, 2017 at 8:11 AM, ayan guha wrote: > Hi > > I have a python program which I invoke as > > spark-submit --packages com.databricks:spark-avro_2.11:3.2.0 somefile.py > "2017-08-23 02:00:00" and it works > > Now I want to submit this file using Livy. I could work out most of the > stuff (like putting files to HDFS etc) but not able to understand how/where > to configure the "packages" switch...Any help? > -- > Best Regards, > Ayan Guha >
Livy with Spark package
Hi I have a python program which I invoke as spark-submit --packages com.databricks:spark-avro_2.11:3.2.0 somefile.py "2017-08-23 02:00:00" and it works Now I want to submit this file using Livy. I could work out most of the stuff (like putting files to HDFS etc) but not able to understand how/where to configure the "packages" switch...Any help? -- Best Regards, Ayan Guha
