Re: [ANNOUNCE] Apache Livy 0.6.0-incubating released
Looks like the maven artifacts are not published yet. https://mvnrepository.com/artifact/org.apache.livy/livy-api On Wed, Apr 3, 2019 at 9:54 AM Marcelo Vanzin wrote: > I closed the bug and will fix the website link. Thanks for pointing it out! > > On Wed, Apr 3, 2019 at 4:55 AM Alexey Romanenko > wrote: > > > > Great, thanks to all who worked on this! > > > > Just a couple of minor notes I noticed: > > - Page http://livy.incubator.apache.org/history/ , wrong link > "0.6.0-incubating”, it leads to 0.5.0 tag instead of 0.6.0 > > - LIVY-489 (umbrella jira for Hive-compatible JDBC / ODBC server > feature) is not resolved and not linked to 0.6.0 release. Do we need to > properly close this jira? > > > > Alexey > > > > On 3 Apr 2019, at 07:05, Jeff Zhang wrote: > > > > Thanks Marcelo > > > > > > 于2019年4月3日周三 下午12:15写道： > >> > >> Nice!! > >> > >> Sent from my iPhone > >> > >> On Apr 2, 2019, at 8:21 PM, Ruslan Dautkhanov > wrote: > >> > >> Thanks a lot Marcelo ! > >> > >> Ruslan > >> > >> > >> > >> On Tue, Apr 2, 2019 at 12:24 PM Marcelo Vanzin > wrote: > >>> > >>> The Apache Livy team is proud to announce the release of Apache Livy > >>> 0.6.0-incubating. > >>> > >>> Livy is web service that exposes a REST interface for managing long > >>> running Apache Spark contexts in your cluster. Livy enables > >>> programmatic, fault-tolerant, multi-tenant submission of Spark jobs > >>> from web/mobile apps (no Spark client needed). So, multiple users can > >>> interact with your Spark cluster concurrently and reliably. > >>> > >>> Download Apache Livy 0.6.0-incubating: > >>> http://livy.incubator.apache.org/download/ > >>> > >>> Release Notes: > >>> http://livy.incubator.apache.org/history/ > >>> > >>> For more about Livy check our website: > >>> http://livy.incubator.apache.org/ > >>> > >>> We would like to thank the contributors that made the release possible! > >>> > >>> > >>> -- > >>> Marcelo > >> > >> -- > >> > >> -- > >> Ruslan Dautkhanov > > > > > > > > -- > > Best Regards > > > > Jeff Zhang > > > > > > > -- > Marcelo >
Re: Livy-0.6 release?
Any rough timeline on 0.6? IfLivyy doesn't allow to choose a higher spark version I guess that will be a blocker fora lot of people who want to leverage new features from spark. Any good solution to fix this? On Mon, Feb 11, 2019 at 3:46 PM Ruslan Dautkhanov wrote: > Got it. Thanks Marcelo. > > I see LIVY-551 is now part of the master. Hope to see Livy 0.6 perhaps > soon. > > > Thank you! > Ruslan Dautkhanov > > > On Tue, Feb 5, 2019 at 12:38 PM Marcelo Vanzin > wrote: > >> I think LIVY-551 is the current blocker. Unfortunately I don't think >> we're really tracking things in jira that well, as far as releases go. >> At least I'm not. >> >> On Mon, Feb 4, 2019 at 6:32 PM Ruslan Dautkhanov >> wrote: >> > >> > +1 for 0.6 release so folks can upgrade to Spark 2.4.. >> > >> > Marcelo, what particular patches are blocking Livy 0.6 release? >> > >> > I see 3 jiras with 0.6 as Fix Version - not sure if that's correct way >> to find blockers. >> > https://goo.gl/9axfsw >> > >> > >> > Thank you! >> > Ruslan Dautkhanov >> > >> > >> > On Mon, Jan 28, 2019 at 2:24 PM Marcelo Vanzin >> wrote: >> >> >> >> There are a couple of patches under review that are currently blocking >> >> the release. >> >> >> >> Once those are done, we can work on releasing 0.6. >> >> >> >> On Mon, Jan 28, 2019 at 11:18 AM Roger Liu >> wrote: >> >> > >> >> > Hey there, >> >> > >> >> > >> >> > >> >> > I’m wondering if we have a timeline for releasing Livy-0.6? Its been >> a year since the last release and there are features like Spark-2.4 support >> that are not incorporated in the livy-0.5 package. >> >> > >> >> > >> >> > >> >> > Thanks, >> >> > >> >> > Roger Liu >> >> >> >> >> >> >> >> -- >> >> Marcelo >> >> >> >> -- >> Marcelo >> >
How to submit a batch application using Programmatic way?
Hi All, How to submit a batch application using Programmatic way but not through REST or Curl? such that it is similar to /batches. Thanks!
Does livy on yarn be able to restart the failed jobs by itself if so, how?
Hi All, Let say I am running livy on yarn in recovery mode. I can see that it is recovering session when livy crashes but the jobs that were crashed when livy crashed are not getting restarted. Can someone please explain how this works? This info is not available anywhere in the docs. I just want to make sure how can the jobs recover and continue making progress in the event livy crash. What should I do besides running livy in a recovery mode? Thanks!
can someone explain what this is livy.rsc.launcher.port ?
Hi All, Can someone explain what this is livy.rsc.launcher.port.range = 1~10110 Sometimes I have all the 10 ports open I am not sure why? How do I make sure this doesn't happen? Thanks!
unable to pass Jackson Avro Object to a Job
Hi All, I wanted to pass org.apache.avro.Schema object to a Job via constructor however since the schema object is not serializable I used jackson-dataformat-avro to pass com.fasterxml.jackson.dataformat.avro.AvroSchema but now I get the error below. Any suggestions? org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: alues 18/03/26 01:26:05 INFO LineBufferedStream: stdout: Serialization trace: 18/03/26 01:26:05 INFO LineBufferedStream: stdout: fieldMap (org.apache.avro.Schema$RecordSchema) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: _writerSchema (com.fasterxml.jackson.dataformat.avro.AvroSchema) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.MapSerializer.read(MapSerializer.java:139) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.MapSerializer.read(MapSerializer.java:17) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.Kryo.readObject(Kryo.java:694) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.ObjectField.read(ObjectField.java:106) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.Kryo.readObject(Kryo.java:694) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.ObjectField.read(ObjectField.java:106) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.Kryo.readObject(Kryo.java:694) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.ObjectField.read(ObjectField.java:106) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:776) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.lang.Thread.run(Thread.java:745) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: Caused by: java.lang.ClassNotFoundException: alues 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.lang.Class.forName0(Native Method) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at java.lang.Class.forName(Class.java:348) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: at org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136) 18/03/26 01:26:05 INFO LineBufferedStream: stdout: ... 25 more Thanks!
Re: What happens if Livy server crashes ? All the spark jobs are gone?
I have the following but I am unable to successfully submit a job though livy in cluster mode. here are my settings # spark-defaults.conf spark.master yarn #livy.conf livy.spark.master=yarn livy.spark.deploy-mode = cluster livy.server.recovery.mode = recovery livy.server.recovery.state-store = zookeeper livy.server.recovery.state-store.url = localhost:2181 Anything wrong with this conf? Thanks! On Tue, Mar 20, 2018 at 5:38 PM, kant kodali <kanth...@gmail.com> wrote: > got it! is it livy.spark.deploy-mode=yarn-cluster or livy.spark.deploy-mode > = cluster ? Sorry to ask this question. I couldn't find it in docs or the > comments in livy.conf and I am using livy 0.4.0 > > On Tue, Mar 20, 2018 at 5:01 PM, Meisam Fathi <meisam.fa...@gmail.com> > wrote: > >> If you are running on cluster mode, the application should keep running >> on YRAN. >> >> On Tue, Mar 20, 2018 at 3:34 PM kant kodali <kanth...@gmail.com> wrote: >> >>> @Meisam Fathi I am running with yarn and zookeeper as a state store. I >>> spawned a job via livy that reads from kafka and writes to Kafka >>> but the moment I kill the livy server the job also is getting killed. >>> not sure why? I believe once the livy server crashes the spark context also >>> get's killed so do I need to need to set the livy.spark.deploy.mode ? if >>> so, what value should I set it to? >>> >>> >>> On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <meisam.fa...@gmail.com> >>> wrote: >>> >>>> On YARN, your application keeps running even if the launcher fails. So >>>> after recovery, Livy reconnects to the application. On Spark standalone, I >>>> am not sure what happens to the application of the launcher fails. >>>> >>>> Thanks, >>>> Meisam >>>> >>>> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <kanth...@gmail.com> >>>> wrote: >>>> >>>>> can someone please explain how YARN helps here? And why not spark >>>>> master? >>>>> >>>>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto < >>>>> m.durighe...@miriade.it> wrote: >>>>> >>>>>> >>>>>> >>>>>> 2018-03-12 9:58 GMT+01:00 kant kodali <kanth...@gmail.com>: >>>>>> >>>>>>> Sorry I see there is a recovery mode and also I can set state store >>>>>>> to zookeeper but looks like I need YARN? because I get the error message >>>>>>> below >>>>>>> >>>>>>> "requirement failed: Session recovery requires YARN" >>>>>>> >>>>>>> >>>>>>> I am using spark standalone and I don't use YARN anywhere in my >>>>>>> cluster. is there any other option for recovery in this case? >>>>>>> >>>>>>> >>>>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> When my live server crashes it looks like all my spark jobs are >>>>>>>> gone. I am trying to see how I can make it more resilient? other >>>>>>>> words, I >>>>>>>> would like spark jobs that were spawned by Livy to be running even if >>>>>>>> my >>>>>>>> Livy server crashes because in theory Livy server can crash anytime and >>>>>>>> Spark Jobs should run for weeks or months in my case. How can I achieve >>>>>>>> this? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>> Hello, >>>>>> to enable recovery in Livy you need Spark on YARN >>>>>> >>>>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html ) >>>>>> >>>>>> >>>>>> >>>>>> Kind Regards >>>>>> >>>>> >>>>> >>> >
Re: What happens if Livy server crashes ? All the spark jobs are gone?
got it! is it livy.spark.deploy-mode=yarn-cluster or livy.spark.deploy-mode = cluster ? Sorry to ask this question. I couldn't find it in docs or the comments in livy.conf and I am using livy 0.4.0 On Tue, Mar 20, 2018 at 5:01 PM, Meisam Fathi <meisam.fa...@gmail.com> wrote: > If you are running on cluster mode, the application should keep running on > YRAN. > > On Tue, Mar 20, 2018 at 3:34 PM kant kodali <kanth...@gmail.com> wrote: > >> @Meisam Fathi I am running with yarn and zookeeper as a state store. I >> spawned a job via livy that reads from kafka and writes to Kafka >> but the moment I kill the livy server the job also is getting killed. not >> sure why? I believe once the livy server crashes the spark context also >> get's killed so do I need to need to set the livy.spark.deploy.mode ? if >> so, what value should I set it to? >> >> >> On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <meisam.fa...@gmail.com> >> wrote: >> >>> On YARN, your application keeps running even if the launcher fails. So >>> after recovery, Livy reconnects to the application. On Spark standalone, I >>> am not sure what happens to the application of the launcher fails. >>> >>> Thanks, >>> Meisam >>> >>> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <kanth...@gmail.com> wrote: >>> >>>> can someone please explain how YARN helps here? And why not spark >>>> master? >>>> >>>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto < >>>> m.durighe...@miriade.it> wrote: >>>> >>>>> >>>>> >>>>> 2018-03-12 9:58 GMT+01:00 kant kodali <kanth...@gmail.com>: >>>>> >>>>>> Sorry I see there is a recovery mode and also I can set state store >>>>>> to zookeeper but looks like I need YARN? because I get the error message >>>>>> below >>>>>> >>>>>> "requirement failed: Session recovery requires YARN" >>>>>> >>>>>> >>>>>> I am using spark standalone and I don't use YARN anywhere in my >>>>>> cluster. is there any other option for recovery in this case? >>>>>> >>>>>> >>>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> When my live server crashes it looks like all my spark jobs are >>>>>>> gone. I am trying to see how I can make it more resilient? other words, >>>>>>> I >>>>>>> would like spark jobs that were spawned by Livy to be running even if my >>>>>>> Livy server crashes because in theory Livy server can crash anytime and >>>>>>> Spark Jobs should run for weeks or months in my case. How can I achieve >>>>>>> this? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>> Hello, >>>>> to enable recovery in Livy you need Spark on YARN >>>>> >>>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html ) >>>>> >>>>> >>>>> >>>>> Kind Regards >>>>> >>>> >>>> >>
Re: What happens if Livy server crashes ? All the spark jobs are gone?
@Meisam Fathi I am running with yarn and zookeeper as a state store. I spawned a job via livy that reads from kafka and writes to Kafka but the moment I kill the livy server the job also is getting killed. not sure why? I believe once the livy server crashes the spark context also get's killed so do I need to need to set the livy.spark.deploy.mode ? if so, what value should I set it to? On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <meisam.fa...@gmail.com> wrote: > On YARN, your application keeps running even if the launcher fails. So > after recovery, Livy reconnects to the application. On Spark standalone, I > am not sure what happens to the application of the launcher fails. > > Thanks, > Meisam > > On Mon, Mar 12, 2018 at 10:34 AM kant kodali <kanth...@gmail.com> wrote: > >> can someone please explain how YARN helps here? And why not spark master? >> >> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto < >> m.durighe...@miriade.it> wrote: >> >>> >>> >>> 2018-03-12 9:58 GMT+01:00 kant kodali <kanth...@gmail.com>: >>> >>>> Sorry I see there is a recovery mode and also I can set state store to >>>> zookeeper but looks like I need YARN? because I get the error message below >>>> >>>> "requirement failed: Session recovery requires YARN" >>>> >>>> >>>> I am using spark standalone and I don't use YARN anywhere in my >>>> cluster. is there any other option for recovery in this case? >>>> >>>> >>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth...@gmail.com> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> When my live server crashes it looks like all my spark jobs are gone. >>>>> I am trying to see how I can make it more resilient? other words, I would >>>>> like spark jobs that were spawned by Livy to be running even if my Livy >>>>> server crashes because in theory Livy server can crash anytime and Spark >>>>> Jobs should run for weeks or months in my case. How can I achieve this? >>>>> >>>>> Thanks! >>>>> >>>>> >>>> Hello, >>> to enable recovery in Livy you need Spark on YARN >>> >>> ( https://spark.apache.org/docs/latest/running-on-yarn.html ) >>> >>> >>> >>> Kind Regards >>> >> >>
Livy unable to recover spark session using yarn. using livy 0.4.0 & spark 2.3.0 & hadoop 2.7.2
Hi All, I am using Yarn and had set the following setting in livy.conf but Livy was unable to recover the session moreover the spark job spawned by Livy got killed when it is supposed to be running. Below is the stack trace. I am not sure what to deduce from it? The following are the settings I set in livy.conf that are needed for recovery. I did not set anything for "livy.spark.deploy-mode" (It's actually commented out) livy.spark.master = yarn livy.server.recovery.mode = recovery livy.server.recovery.state-store = zookeeper livy.server.recovery.state-store.url = localhost:2181 Here is my stacktrace 18/03/20 09:15:57 INFO ZooKeeper: Client environment:user.dir=/home/ubuntu/livy-0.4.0-incubating-bin/bin 18/03/20 09:15:57 INFO ZooKeeper: Initiating client connection, connectString=x.x.x.x:2181 sessionTimeout=6 watcher=org.apache.curator.ConnectionState@4ae9cfc1 18/03/20 09:15:57 INFO StateStore$: Using ZooKeeperStateStore for recovery. 18/03/20 09:15:57 INFO ClientCnxn: Opening socket connection to server x.x.x.x/x.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) 18/03/20 09:15:57 INFO ClientCnxn: Socket connection established to x.x.x.x/x.x.x.x:2181, initiating session 18/03/20 09:15:57 INFO ClientCnxn: Session establishment complete on server x.x.x.x/x.x.x.x:2181, sessionid = 0x162426cb5710007, negotiated timeout = 6 18/03/20 09:15:57 INFO ConnectionStateManager: State change: CONNECTED 18/03/20 09:15:57 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0 18/03/20 09:15:58 ERROR RSCClient: Failed to connect to context. java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) 18/03/20 09:15:58 INFO InteractiveSessionManager: Recovered 1 interactive sessions. Next session id: 4 18/03/20 09:15:58 INFO InteractiveSessionManager: Registering new session 3 18/03/20 09:15:58 INFO InteractiveSessionManager: Heartbeat watchdog thread started. 18/03/20 09:15:58 ERROR RSCClient: RPC error. java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) 18/03/20 09:15:58 INFO WebServer: Starting server on http://x.x.x.x.us-west-2.compute.internal:8998 18/03/20 09:18:32 ERROR SessionServlet$: internal error java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.livy.server.interactive.InteractiveSession.statements(InteractiveSession.scala:471) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:69) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:67) at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionServlet.scala:205) at
Re: What happens if Livy server crashes ? All the spark jobs are gone?
can someone please explain how YARN helps here? And why not spark master? On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <m.durighe...@miriade.it> wrote: > > > 2018-03-12 9:58 GMT+01:00 kant kodali <kanth...@gmail.com>: > >> Sorry I see there is a recovery mode and also I can set state store to >> zookeeper but looks like I need YARN? because I get the error message below >> >> "requirement failed: Session recovery requires YARN" >> >> >> I am using spark standalone and I don't use YARN anywhere in my cluster. >> is there any other option for recovery in this case? >> >> >> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth...@gmail.com> wrote: >> >>> Hi All, >>> >>> When my live server crashes it looks like all my spark jobs are gone. I >>> am trying to see how I can make it more resilient? other words, I would >>> like spark jobs that were spawned by Livy to be running even if my Livy >>> server crashes because in theory Livy server can crash anytime and Spark >>> Jobs should run for weeks or months in my case. How can I achieve this? >>> >>> Thanks! >>> >>> >> Hello, > to enable recovery in Livy you need Spark on YARN > > ( https://spark.apache.org/docs/latest/running-on-yarn.html ) > > > > Kind Regards >
Re: What happens if Livy server crashes ? All the spark jobs are gone?
Sorry I see there is a recovery mode and also I can set state store to zookeeper but looks like I need YARN? because I get the error message below "requirement failed: Session recovery requires YARN" I am using spark standalone and I don't use YARN anywhere in my cluster. is there any other option for recovery in this case? On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > When my live server crashes it looks like all my spark jobs are gone. I am > trying to see how I can make it more resilient? other words, I would like > spark jobs that were spawned by Livy to be running even if my Livy server > crashes because in theory Livy server can crash anytime and Spark Jobs > should run for weeks or months in my case. How can I achieve this? > > Thanks! > >
Re: When I submit a livy job am I running in a client mode or cluster mode ?
livy.spark.master yarn ?? Do I need to have yarn ? can I use spark standalone master? On Mon, Feb 12, 2018 at 1:19 AM, Jeff Zhang <zjf...@gmail.com> wrote: > livy.spark.master yarn > livy.spark.deploy-mode cluster > > > kant kodali <kanth...@gmail.com>于2018年2月12日周一 下午4:36写道： > >> I checked my livy.conf and livy.spark.deploy-mode is not set at all. so >> I wonder which mode it runs by default? >> >> On Sun, Feb 11, 2018 at 11:50 PM, Jeff Zhang <zjf...@gmail.com> wrote: >> >>> >>> Via livy.spark.master & livy.spark.deploy-mode in livy.conf >>> >>> >>> >>> kant kodali <kanth...@gmail.com>于2018年2月12日周一 下午3:49写道： >>> >>>> Hi All, >>>> >>>> When I submit a livy job to livy server am I running in a client mode >>>> or cluster mode ? How can I switch from one mode to another? >>>> >>>> Thanks! >>>> >>>> >>
Re: When I submit a livy job am I running in a client mode or cluster mode ?
I checked my livy.conf and livy.spark.deploy-mode is not set at all. so I wonder which mode it runs by default? On Sun, Feb 11, 2018 at 11:50 PM, Jeff Zhang <zjf...@gmail.com> wrote: > > Via livy.spark.master & livy.spark.deploy-mode in livy.conf > > > > kant kodali <kanth...@gmail.com>于2018年2月12日周一 下午3:49写道： > >> Hi All, >> >> When I submit a livy job to livy server am I running in a client mode or >> cluster mode ? How can I switch from one mode to another? >> >> Thanks! >> >>
When I submit a livy job am I running in a client mode or cluster mode ?
Hi All, When I submit a livy job to livy server am I running in a client mode or cluster mode ? How can I switch from one mode to another? Thanks!
Re: How do I set spark conf parameters?
Sorry. I found an answer online.It should be something like this
new LivyClientBuilder().setConf("spark.es.index.auto.create",
"true").setConf("spark.cassandra.connection.host",
"127.0.0.01").build();
On Wed, Jan 31, 2018 at 1:16 AM, kant kodali <kanth...@gmail.com> wrote:
> Hi All,
>
>
> How do I set Spark Conf Parameters ? The below doesnt seem to get picked
> up? If so, how can I change my program such that it can pick it up. I am
> not seeing a way if sparkcontext is already created?
>
> public String call(JobContext ctx) throws Exception {
> ctx.sc().setLogLevel("INFO");
> ctx.sc().getConf()
> .set("spark.cassandra.connection.host",
> config.getString("cassandra.host"))
> .set("spark.cassandra.auth.username",
> config.getString("cassandra.user"))
> .set("spark.cassandra.auth.password",
> config.getString("cassandra.pass"))
> .set("es.index.auto.create", "true");
>
> }
>
>
> Thanks!
>
>
How do I set spark conf parameters?
Hi All,
How do I set Spark Conf Parameters ? The below doesnt seem to get picked
up? If so, how can I change my program such that it can pick it up. I am
not seeing a way if sparkcontext is already created?
public String call(JobContext ctx) throws Exception {
ctx.sc().setLogLevel("INFO");
ctx.sc().getConf()
.set("spark.cassandra.connection.host",
config.getString("cassandra.host"))
.set("spark.cassandra.auth.username",
config.getString("cassandra.user"))
.set("spark.cassandra.auth.password",
config.getString("cassandra.pass"))
.set("es.index.auto.create", "true");
}
Thanks!
Problem serializing list ?
Hi All,
I get a serialization error when I try submit a job to livy. I am not sure
what I am doing wrong?
public class MyJob implements Job {
private final String *name*;
private List cols;
private List *tags*;
public MyJob(String name, List cols, List tags) {
this.name = name;
this.cols = cols;
this.tags = tags;
}
}
and then I do something like this
livy.submit(new MyJob("cat", Arrays.asList("hello"), new ArrayList<>())).get();
I get the following Exception
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.livy.shaded.kryo.kryo.KryoException:
java.lang.NullPointerException
Serialization trace:
cols (com.hello.sample.jobs.MyJob)
org.apache.livy.shaded.kryo.kryo.serializers.ObjectField.read(ObjectField.java:125)
org.apache.livy.shaded.kryo.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
Thanks!
Re: How to cancel the running streaming job using livy?
Sure I guess I can do that..Is that the only way? Is there any REST call I
can make to SPARK maybe to cancel any of the Streaming Query? Sorry if this
is too naive.
On Wed, Jan 24, 2018 at 2:46 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> Then that has nothing to do with Livy.
>
> You need to store a reference to your StreamingQuery (returned by start())
> somewhere, and if you want to stop it, call its "stop()" method by
> submitting a new Livy job that does it.
>
> On Wed, Jan 24, 2018 at 2:42 PM, kant kodali <kanth...@gmail.com> wrote:
>
>> Ok Let me paste some code to try and avoid the confusion. In the below
>> code I am running two streaming queries. Now here are my two simple
>> questions.
>>
>> 1) Does each Streaming Query below spawn one job or multiple jobs?
>> 2) What should I do if I need to kill everything related to streaming
>> query1 but not streaming query2?
>>
>>
>> public Void call(JobContext ctx) throws Exception {
>>SparkSession sparkSession = ctx.sparkSession();
>>Dataset df = sparkSession.readStream().format("kafka").load();
>>df.createOrReplaceTempView("table");
>>
>>Dataset resultSet1 = sparkSession.sql("select * from table");
>>
>>resultSet1.writeStream().format("console").start(); //Streaming query1
>> started
>>
>>Dataset resultSet2 = sparkSession.sql("select count(*) from table");
>> //Streaming query2 started
>>
>>sparkSession.streams().awaitAnyTermination();
>>
>>return null;
>>
>> }
>>
>>
>> Thanks!
>>
>>
>> On Wed, Jan 24, 2018 at 1:47 PM, Marcelo Vanzin <van...@cloudera.com>
>> wrote:
>>
>>> I'm a little confused about what is meant as a job here, after all this
>>> discussion...
>>>
>>> For "interactive sessions", stopping a session means stopping the
>>> SparkContext. So the final state of any running jobs in that session should
>>> be the same as if you stopped the SparkContext without explicitly stopping
>>> the jobs in a normal, non-Livy application.
>>>
>>> For batches, stopping a batch means killing the Spark application, so
>>> all bets are off as to what happens there.
>>>
>>>
>>> On Wed, Jan 24, 2018 at 1:08 PM, Alex Bozarth <ajboz...@us.ibm.com>
>>> wrote:
>>>
>>>> You are correct that you are using the term Job incorrectly (at least
>>>> according to how Spark/Livy uses it). Each spark-submit is a a single Spark
>>>> Application and can include many jobs (which are broken down themselves
>>>> into stages and tasks). In Livy using sessions would be like using
>>>> spark-shell rather than spark-submit, you probably want to use batches
>>>> instead (which utilize spark-submit), then you would use that delete
>>>> command as mentioned earlier. As for the result being listed as FAILED and
>>>> not CANCELLED, that is as intended. When a Livy Session is stopped
>>>> (deleted) is sends a command to all the running jobs (in your case each of
>>>> you apps only have one "Job") to set as failed.
>>>>
>>>> @Marcelo you wrote the code that does this, do you remember why you had
>>>> Jobs killed instead of cancelled when a Livy session is stopped? Otherwise
>>>> we may be able to open a JIRA and change this, but I am unsure of any
>>>> potential consequences.
>>>>
>>>>
>>>> *Alex Bozarth*
>>>> Software Engineer
>>>> Spark Technology Center
>>>> --
>>>> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>>>> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>>>>
>>>>
>>>> 505 Howard Street
>>>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>>>> San Francisco, CA 94105
>>>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>>>> United States
>>>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>>>>
>>>>
>>>>
>>>> [image: Inactive hide details for kant kodali ---01/23/2018 11:44:26
>>>> PM---I tried POST to sessions/{session id}/jobs/{job id}/cancel a]kant
>>>> kodali ---01/23/2018 11:44:26 PM---I tried POST to sessions/{sess
Re: How to cancel the running streaming job using livy?
Ok Let me paste some code to try and avoid the confusion. In the below code
I am running two streaming queries. Now here are my two simple questions.
1) Does each Streaming Query below spawn one job or multiple jobs?
2) What should I do if I need to kill everything related to streaming
query1 but not streaming query2?
public Void call(JobContext ctx) throws Exception {
SparkSession sparkSession = ctx.sparkSession();
Dataset df = sparkSession.readStream().format("kafka").load();
df.createOrReplaceTempView("table");
Dataset resultSet1 = sparkSession.sql("select * from table");
resultSet1.writeStream().format("console").start(); //Streaming
query1 started
Dataset resultSet2 = sparkSession.sql("select count(*) from
table"); //Streaming query2 started
sparkSession.streams().awaitAnyTermination();
return null;
}
Thanks!
On Wed, Jan 24, 2018 at 1:47 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> I'm a little confused about what is meant as a job here, after all this
> discussion...
>
> For "interactive sessions", stopping a session means stopping the
> SparkContext. So the final state of any running jobs in that session should
> be the same as if you stopped the SparkContext without explicitly stopping
> the jobs in a normal, non-Livy application.
>
> For batches, stopping a batch means killing the Spark application, so all
> bets are off as to what happens there.
>
>
> On Wed, Jan 24, 2018 at 1:08 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
>> You are correct that you are using the term Job incorrectly (at least
>> according to how Spark/Livy uses it). Each spark-submit is a a single Spark
>> Application and can include many jobs (which are broken down themselves
>> into stages and tasks). In Livy using sessions would be like using
>> spark-shell rather than spark-submit, you probably want to use batches
>> instead (which utilize spark-submit), then you would use that delete
>> command as mentioned earlier. As for the result being listed as FAILED and
>> not CANCELLED, that is as intended. When a Livy Session is stopped
>> (deleted) is sends a command to all the running jobs (in your case each of
>> you apps only have one "Job") to set as failed.
>>
>> @Marcelo you wrote the code that does this, do you remember why you had
>> Jobs killed instead of cancelled when a Livy session is stopped? Otherwise
>> we may be able to open a JIRA and change this, but I am unsure of any
>> potential consequences.
>>
>>
>> *Alex Bozarth*
>> Software Engineer
>> Spark Technology Center
>> --
>> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>>
>>
>> 505 Howard Street
>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>> San Francisco, CA 94105
>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>> United States
>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>>
>>
>>
>> [image: Inactive hide details for kant kodali ---01/23/2018 11:44:26
>> PM---I tried POST to sessions/{session id}/jobs/{job id}/cancel a]kant
>> kodali ---01/23/2018 11:44:26 PM---I tried POST to sessions/{session
>> id}/jobs/{job id}/cancel and that doesn't seem to cancel either.
>>
>> From: kant kodali <kanth...@gmail.com>
>> To: user@livy.incubator.apache.org
>> Date: 01/23/2018 11:44 PM
>>
>> Subject: Re: How to cancel the running streaming job using livy?
>> --
>>
>>
>>
>> I tried POST to sessions/{session id}/jobs/{job id}/cancel and that
>> doesn't seem to cancel either. I think first of all the word "job" is used
>> in so many context that it might be misleading.
>>
>> Imagine for a second I don't have livy and I just use spark-submit
>> command line to spawn . say I do that following
>>
>> spark-submit hello1.jar // streaming job1 (runs forever)
>> spark-submit hello2.jar //streaming job2 (runs forever)
>>
>> The number of jobs I spawned is two and now I want to be able to cancel
>> one of them..These jobs reads data from kafka and will be split into stages
>> and task now sometimes these tasks are also called jobs according to SPARK
>> UI for some reason. And looks like live may be is cancelling those with the
>> above end point.
>>
>> It would be great help if someone could try from their
Re: How to cancel the running streaming job using livy?
I looked into livy 0.5 and 0.6 and I still don't see any changes there
w.r.t to job cancellation. isn't Job cancellation an important API? If not,
I am not sure how to turn off long running streaming jobs. please help
On Tue, Jan 23, 2018 at 11:43 PM, kant kodali <kanth...@gmail.com> wrote:
> I tried POST to sessions/{session id}/jobs/{job id}/cancel and that
> doesn't seem to cancel either. I think first of all the word "job" is used
> in so many context that it might be misleading.
>
> Imagine for a second I don't have livy and I just use spark-submit command
> line to spawn . say I do that following
>
> spark-submit hello1.jar // streaming job1 (runs forever)
> spark-submit hello2.jar //streaming job2 (runs forever)
>
> The number of jobs I spawned is two and now I want to be able to cancel
> one of them..These jobs reads data from kafka and will be split into stages
> and task now sometimes these tasks are also called jobs according to SPARK
> UI for some reason. And looks like live may be is cancelling those with the
> above end point.
>
> It would be great help if someone could try from their end and see if they
> are able to cancel the jobs?
>
> Thanks!
>
> On Fri, Jan 19, 2018 at 4:03 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
>
>> Ah, that's why I couldn't find cancel in JobHandle, but it was
>> implemented in all it's implementations, which all implement it as would be
>> expected.
>>
>>
>> *Alex Bozarth*
>> Software Engineer
>> Spark Technology Center
>> --
>> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>>
>>
>> 505 Howard Street
>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>> San Francisco, CA 94105
>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>> United States
>> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>>
>>
>>
>> [image: Inactive hide details for Marcelo Vanzin ---01/19/2018 03:55:43
>> PM---A JobHandle (which you get by submitting a Job) is a Futur]Marcelo
>> Vanzin ---01/19/2018 03:55:43 PM---A JobHandle (which you get by submitting
>> a Job) is a Future, and Futures have a "cancel()" method.
>>
>> From: Marcelo Vanzin <van...@cloudera.com>
>> To: user@livy.incubator.apache.org
>> Date: 01/19/2018 03:55 PM
>>
>> Subject: Re: How to cancel the running streaming job using livy?
>> --
>>
>>
>>
>> A JobHandle (which you get by submitting a Job) is a Future, and Futures
>> have a "cancel()" method.
>>
>> I don't remember the details about how "cancel()" is implemented in Livy,
>> though.
>>
>> On Fri, Jan 19, 2018 at 3:52 PM, Alex Bozarth <*ajboz...@us.ibm.com*
>> <ajboz...@us.ibm.com>> wrote:
>>
>>Ok so I looked into this a bit more. I misunderstood you a bit
>>before, the delete call is for ending livy sessions using the rest API,
>> not
>>jobs and not via the Java API. As for the Job state that makes sense, if
>>you end the session the session kills all currently running jobs. What you
>>want to to send cancel requests to the jobs the session is running. From
>> my
>>research I found that there is a way to do this via the REST API, but it
>>isn't documented for some reason. Doing a POST to /{session id}/jobs/{job
>>id}/cancel will cancel a job. As for the Java API, the feature isn't part
>>of the Java interface, but most implementations of it add it, such as the
>>Scala API which ScalaJobHandle class on sumbit which has a cancel
>> function.
>>I'm not sure how you're submitting you jobs, but there should be a cancel
>>function available to you somewhere depending on the client you're using.
>>From this discussion I've realized our current documentation is even more
>>lacking that I had thought.
>>
>>
>> *Alex Bozarth*
>> Software Engineer
>> Spark Technology Center
>> --
>> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
>> *GitHub: **github.com/ajbozarth*
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=dcr5yrMAHblD8Ur9vfpBsXcOzNGHtaEF9jk5yMBv4Kk=gMcUXOnL9YD3_CIOpwNX4jqFVWhx0l6DAsJYTKN9HVU=>
>>
&
Re: How to cancel the running streaming job using livy?
aEF9jk5yMBv4Kk=Iu4BJQb_gsqB3B1AXW2WTuFJsI-peBqIQyczkuK3MMU=> > *United States* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D505-2BHoward-2BStreet-2BSan-2BFrancisco-2C-2BCA-2B94105-2BUnited-2BStates-26entry-3Dgmail-26source-3Dg=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=dcr5yrMAHblD8Ur9vfpBsXcOzNGHtaEF9jk5yMBv4Kk=Iu4BJQb_gsqB3B1AXW2WTuFJsI-peBqIQyczkuK3MMU=> > > > > >[image: Inactive hide details for kant kodali ---01/18/2018 06:09:59 >PM---Also just tried the below and got the state. It ended up in "]kant >kodali ---01/18/2018 06:09:59 PM---Also just tried the below and got the >state. It ended up in "FAILED" stated when I expected it to be > >From: kant kodali <*kanth...@gmail.com* <kanth...@gmail.com>> >To: *user@livy.incubator.apache.org* <user@livy.incubator.apache.org> >Date: 01/18/2018 06:09 PM >Subject: Re: How to cancel the running streaming job using livy? > >------ > > > >Also just tried the below and got the state. It ended up in "FAILED" >stated when I expected it to be in "CANCELLED" state. Also from the docs it >is not clear if it kills the session or the job? if it kills the session I >can't spawn any other Job. Sorry cancelling job had been a bit confusing >for me. >DELETE /sessions/0 > > > >On Thu, Jan 18, 2018 at 5:55 PM, kant kodali <*kanth...@gmail.com* ><kanth...@gmail.com>> wrote: > oh this raises couple questions. > > 1) Is there a programmatic way to cancel a job? > > 2) is there any programmatic way to get session id? If not, how > do I get a sessionId when I spawn multiple jobs or multiple sessions? > > > On Thu, Jan 18, 2018 at 5:39 PM, Alex Bozarth < > *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>> wrote: > You make a DELETE call as detailed here: > *http://livy.apache.org/docs/latest/rest-api.html#response* > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__livy.apache.org_docs_latest_rest-2Dapi.html-23response=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=nI9x8SjWSOnoLQr05P15W5ofGJayNWwF3InumEtLhVo=eAcZY6sAN_mkDv5Ves9UtZaotVvvUc3BBdkCEV_CqVg=> > *Alex Bozarth* >Software Engineer >Spark Technology Center > > -- > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=nI9x8SjWSOnoLQr05P15W5ofGJayNWwF3InumEtLhVo=EV7HPze6ToE8xgFtDOw9zE2b3sGYWSW1rB-7ZhiJRok=> > > > *505 Howard Street* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D505-2BHoward-2BStreet-2BSan-2BFrancisco-2C-2BCA-2B94105-2BUnited-2BStates-26entry-3Dgmail-26source-3Dg=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=nI9x8SjWSOnoLQr05P15W5ofGJayNWwF3InumEtLhVo=uy43iGDrczqx4GGhTSYqjjIeyjGpxPQ0611WcWeaB_s=> > *San Francisco, CA 94105* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D505-2BHoward-2BStreet-2BSan-2BFrancisco-2C-2BCA-2B94105-2BUnited-2BStates-26entry-3Dgmail-26source-3Dg=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=nI9x8SjWSOnoLQr05P15W5ofGJayNWwF3InumEtLhVo=uy43iGDrczqx4GGhTSYqjjIeyjGpxPQ0611WcWeaB_s=> > *United States* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D505-2BHoward-2BStreet-2BSan-2BFrancisco-2C-2BCA-2B94105-2BUnited-2BStates-26entry-3Dgmail-26source-3Dg=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=nI9x8SjWSOnoLQr05P15W5ofGJayNWwF3InumEtLhVo=uy43iGDrczqx4GGhTSYqjjIeyjGpxPQ0611WcWeaB_s=> > > > > > [image: Inactive hide details for kant kodali ---01/18/2018 > 05:34:07 PM---Hi All, I was able to submit a streaming job to livy > however]kant > kodali ---01/18/2018 05:34:07 PM---Hi All, I was able to submit a > streaming > job to livy however I wasn't able to find > > From: kant kodali <*kanth...@gmail.com* <kanth...@gmail.com>> > To: *user@livy.incubator.apache.org* > <user@livy.incubator.apache.org> > Date: 01/18/2018 05:34 PM > Subject: How to cancel the running streaming job using livy? > -- > > > > Hi All, > > I was able to submit a streaming job to livy however I wasn't > able to find any way to cancel the running the job? Please let me > know. > > Thanks! > > > > > > > > > > -- > Marcelo > > >
Re: How to cancel the running streaming job using livy?
oh this raises couple questions. 1) Is there a programmatic way to cancel a job? 2) is there any programmatic way to get session id? If not, how do I get a sessionId when I spawn multiple jobs or multiple sessions? On Thu, Jan 18, 2018 at 5:39 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote: > You make a DELETE call as detailed here: http://livy.apache.org/docs/ > latest/rest-api.html#response > > > *Alex Bozarth* > Software Engineer > Spark Technology Center > -- > *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com> > *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth> > > > 505 Howard Street > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g> > San Francisco, CA 94105 > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g> > United States > <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g> > > > > [image: Inactive hide details for kant kodali ---01/18/2018 05:34:07 > PM---Hi All, I was able to submit a streaming job to livy however]kant > kodali ---01/18/2018 05:34:07 PM---Hi All, I was able to submit a streaming > job to livy however I wasn't able to find > > From: kant kodali <kanth...@gmail.com> > To: user@livy.incubator.apache.org > Date: 01/18/2018 05:34 PM > Subject: How to cancel the running streaming job using livy? > -- > > > > Hi All, > > I was able to submit a streaming job to livy however I wasn't able to find > any way to cancel the running the job? Please let me know. > > Thanks! > > >
How to set driverMemory, driverCores, executorMemory using livy?
Hi All, I do see POST /sessions API where I can pass driverMemory, driverCores, executorMemory as part of the request body but I am using programmatic API to submit upload the Jar and submit my job so how do I set values for driverMemory, driverCores, executorMemory ? Thanks!
Why does jobContext.sparkSession doesn't return a SparkSession Object?
Hi All,
Why does jobContext.sparkSession() doesn't return a SparkSession Object
instead it returns a parametrized type?
jobContext.sc(); //returns JavaSparkContext so this is good
jobContext.sqlContext();// returns SqlContext so this is good
jobContext.steamingContext(); // returns StreamingContext so this is good
jobContext.sparkSession(); // returns any parameterized type. why?
since it returns a parametrized type I can assign it to anything I like
that wouldn't make any sense.
Integer k = jobContext.sparkSession()
or
Long l = jobContext.sparkSession()
Below is the livy interface
package org.apache.livy;
import java.io.File;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.sql.hive.HiveContext;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
public interface JobContext {
JavaSparkContext sc();
SQLContext sqlctx();
HiveContext hivectx();
JavaStreamingContext streamingctx();
void createStreamingContext(long var1);
void stopStreamingCtx();
File getLocalTmpDir();
E sparkSession() throws Exception;
}
Re: How to submit the job via REST ?
Got it! That works for now!
On Fri, Dec 1, 2017 at 1:07 PM, Stefan Miklosovic <mikloso...@gmail.com>
wrote:
> yes you put it into myApp.jar and you have to make pijar accessible
> for myapp jar to upload that file. if you do not want to add anything
> to java -jar myApp.jar (as a classpath deps with livy client classes),
> you could make a fat jar but that is optional.
>
> On Fri, Dec 1, 2017 at 6:28 PM, kant kodali <kanth...@gmail.com> wrote:
> > hmm...I am confused now. I typically run all my java applications using
> java
> > -jar myApp.jar
> >
> > From what you have said PiJob class is in a seperate jar so lets call
> that
> > as PiJar which means all PiJob is the only class in PiJar.
> >
> > Now, where do I put the following code? say I create another class called
> > MyLivyClient and put the below code and bundle it as MyApp.jar
> >
> >
> > LivyClient client = new LivyClientBuilder()
> > .setURI(new URI(livyUrl))
> > .build();
> >
> > try {
> > System.err.printf("Uploading %s to the Spark context...\n", piJar);
> > client.uploadJar(new File(piJar)).get();
> >
> > System.err.printf("Running PiJob with %d samples...\n", samples);
> > double pi = client.submit(new PiJob(samples)).get();
> >
> > System.out.println("Pi is roughly: " + pi);
> > } finally {
> > client.stop(true);
> > }
> >
> >
> > so I can finally run java -jar myApp.jar.
> >
> >
> >
> > On Fri, Dec 1, 2017 at 8:43 AM, Stefan Miklosovic <mikloso...@gmail.com>
> > wrote:
> >>
> >> I dont understand what "LivyClient jar" is in your context. You are just
> >> using that api of Livy when you have that dependency on the class path.
> You
> >> are not bundling there anything unless you make a fat standalone jar
> with
> >> all you need to use LivyClient by which you would upload that job
> class. But
> >> you dont upload that fat jar itself.
> >>
> >> On 1 Dec 2017 17:38, "kant kodali" <kanth...@gmail.com> wrote:
> >>>
> >>> @Stefan Sorry if I wasn't clear I wasn't talking about uploading. I can
> >>> have LivyClient Jar and PiJob jar and I can use LivyClient Jar to run
> on my
> >>> machine which internally uploads PiJob Jar to LivyServer. correct?
> >>>
> >>> On Fri, Dec 1, 2017 at 8:34 AM, Stefan Miklosovic <
> mikloso...@gmail.com>
> >>> wrote:
> >>>>
> >>>> They are separate, livy client is pary of livy distribution by which
> you
> >>>> talk to livy server. There is no need to upload that into livy
> server. Why
> >>>> would you do that ...
> >>>>
> >>>> I dont know about other approach you show. It it suits your case feel
> >>>> free to use it.
> >>>>
> >>>> On 1 Dec 2017 17:11, "kant kodali" <kanth...@gmail.com> wrote:
> >>>>>
> >>>>> can we just do
> >>>>> sparkConf.setJars(JavaSparkContext.jarOfClass(PiJob.class)); ?
> >>>>>
> >>>>> On Fri, Dec 1, 2017 at 7:51 AM, kant kodali <kanth...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>
> >>>>>> If I understand this correctly. Is LivyClient and PiJob are separate
> >>>>>> classes within the same Jar such that LivyClient will have the main
> method?
> >>>>>> or LivyClient and PiJob are separate classes in separate Jars ? If
> so which
> >>>>>> class will have the main method?
> >>>>>>
> >>>>>> I believe Spark also does this but in a much more easy fashion
> >>>>>>
> >>>>>> public static String[] jarOfClass(Class cls)
> >>>>>>
> >>>>>>
> >>>>>> https://spark.apache.org/docs/2.1.1/api/java/index.html?org/
> apache/spark/api/java/JavaSparkContext.html
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Dec 1, 2017 at 7:45 AM, Stefan Miklosovic
> >>>>>> <mikloso...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> The last paragraph holds if your job class is in src/test/java as
> >>>>>>> mine
> >>>>>>> is, I am using Livy for submitting my jobs programmatically as a
> part
> >>>>>>> of mvn test.
&g
Re: Does Apache Livy support Spark Structured Streaming 2.2.0?
Sorry I am new to this. By Job API I assume you meant programmatic API <https://livy.incubator.apache.org/docs/latest/programmatic-api.html> what is interactive query? On Tue, Nov 28, 2017 at 11:25 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > Livy doesn't support WebSocket. I think for your scenario, you need to use > Livy's JobAPI instead of interactive query, Livy doesn't push results back > to client in real time. So this maybe slightly different from what you want > (IIUC). > > 2017-11-29 14:34 GMT+08:00 kant kodali <kanth...@gmail.com>: > >> Nice! so if I submit a streaming query over REST can I get the results >> back over REST or Websocket? >> >> On Tue, Nov 28, 2017 at 10:29 PM, Saisai Shao <sai.sai.s...@gmail.com> >> wrote: >> >>> Livy doesn't add any restriction on how user uses Spark API, so of >>> course Structured Streaming is supported. >>> >>> 2017-11-29 14:21 GMT+08:00 kant kodali <kanth...@gmail.com>: >>> >>>> Hi All, >>>> >>>> Does Apache Livy support Spark Structured Streaming 2.2.0? If so, any >>>> examples please? preferably in Java. >>>> >>>> Thanks, >>>> kant >>>> >>> >>> >> >
Re: Does Apache Livy support Spark Structured Streaming 2.2.0?
Nice! so if I submit a streaming query over REST can I get the results back over REST or Websocket? On Tue, Nov 28, 2017 at 10:29 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > Livy doesn't add any restriction on how user uses Spark API, so of course > Structured Streaming is supported. > > 2017-11-29 14:21 GMT+08:00 kant kodali <kanth...@gmail.com>: > >> Hi All, >> >> Does Apache Livy support Spark Structured Streaming 2.2.0? If so, any >> examples please? preferably in Java. >> >> Thanks, >> kant >> > >
Does Apache Livy support Spark Structured Streaming 2.2.0?
Hi All, Does Apache Livy support Spark Structured Streaming 2.2.0? If so, any examples please? preferably in Java. Thanks, kant
