You are correct that you are using the term Job incorrectly (at least
according to how Spark/Livy uses it). Each spark-submit is a a single Spark
Application and can include many jobs (which are broken down themselves
into stages and tasks). In Livy using sessions would be like using
spark-shell rather than spark-submit, you probably want to use batches
instead (which utilize spark-submit), then you would use that delete
command as mentioned earlier. As for the result being listed as FAILED and
not CANCELLED, that is as intended. When a Livy Session is stopped
(deleted) is sends a command to all the running jobs (in your case each of
you apps only have one "Job") to set as failed.
@Marcelo you wrote the code that does this, do you remember why you had
Jobs killed instead of cancelled when a Livy session is stopped? Otherwise
we may be able to open a JIRA and change this, but I am unsure of any
potential consequences.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
From: kant kodali <kanth...@gmail.com>
To: user@livy.incubator.apache.org
Date: 01/23/2018 11:44 PM
Subject: Re: How to cancel the running streaming job using livy?
I tried POST to sessions/{session id}/jobs/{job id}/cancel and that
doesn't seem to cancel either. I think first of all the word "job" is used
in so many context that it might be misleading.
Imagine for a second I don't have livy and I just use spark-submit command
line to spawn . say I do that following
spark-submit hello1.jar // streaming job1 (runs forever)
spark-submit hello2.jar //streaming job2 (runs forever)
The number of jobs I spawned is two and now I want to be able to cancel one
of them..These jobs reads data from kafka and will be split into stages and
task now sometimes these tasks are also called jobs according to SPARK UI
for some reason. And looks like live may be is cancelling those with the
above end point.
It would be great help if someone could try from their end and see if they
are able to cancel the jobs?
Thanks!
On Fri, Jan 19, 2018 at 4:03 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
Ah, that's why I couldn't find cancel in JobHandle, but it was
implemented in all it's implementations, which all implement it as would
be expected.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
Inactive hide details for Marcelo Vanzin ---01/19/2018 03:55:43 PM---A
JobHandle (which you get by submitting a Job) is a FuturMarcelo Vanzin
---01/19/2018 03:55:43 PM---A JobHandle (which you get by submitting a
Job) is a Future, and Futures have a "cancel()" method.
From: Marcelo Vanzin <van...@cloudera.com>
To: user@livy.incubator.apache.org
Date: 01/19/2018 03:55 PM
Subject: Re: How to cancel the running streaming job using livy?
A JobHandle (which you get by submitting a Job) is a Future, and Futures
have a "cancel()" method.
I don't remember the details about how "cancel()" is implemented in Livy,
though.
On Fri, Jan 19, 2018 at 3:52 PM, Alex Bozarth <ajboz...@us.ibm.com>
wrote:
Ok so I looked into this a bit more. I misunderstood you a bit
before, the delete call is for ending livy sessions using the rest
API, not jobs and not via the Java API. As for the Job state that
makes sense, if you end the session the session kills all currently
running jobs. What you want to to send cancel requests to the jobs
the session is running. From my research I found that there is a
way to do this via the REST API, but it isn't documented for some
reason. Doing a POST to /{session id}/jobs/{job id}/cancel will
cancel a job. As for the Java API, the feature isn't part of the
Java interface, but most implementations of it add it, such as the
Scala API which ScalaJobHandle class on sumbit which has a cancel
function. I'm not sure how you're submitting you jobs, but there
should be a cancel function available to you somewhere depending on
the client you're using. From this discussion I've realized our
current documentation is even more lacking that I had thought.
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
Inactive hide details for kant kodali ---01/18/2018 06:09:59
PM---Also just tried the below and got the state. It ended up in "
kant kodali ---01/18/2018 06:09:59 PM---Also just tried the below
and got the state. It ended up in "FAILED" stated when I expected
it to be
From: kant kodali <kanth...@gmail.com>
To: user@livy.incubator.apache.org
Date: 01/18/2018 06:09 PM
Subject: Re: How to cancel the running streaming job using livy?
Also just tried the below and got the state. It ended up in
"FAILED" stated when I expected it to be in "CANCELLED" state. Also
from the docs it is not clear if it kills the session or the job?
if it kills the session I can't spawn any other Job. Sorry
cancelling job had been a bit confusing for me.
DELETE /sessions/0
On Thu, Jan 18, 2018 at 5:55 PM, kant kodali <kanth...@gmail.com>
wrote:
oh this raises couple questions.
1) Is there a programmatic way to cancel a job?
2) is there any programmatic way to get session id? If
not, how do I get a sessionId when I spawn multiple
jobs or multiple sessions?
On Thu, Jan 18, 2018 at 5:39 PM, Alex Bozarth <
ajboz...@us.ibm.com> wrote:
You make a DELETE call as detailed here:
http://livy.apache.org/docs/latest/rest-api.html#response
Alex Bozarth
Software Engineer
Spark Technology Center
E-mail: ajboz...@us.ibm.com
GitHub: github.com/ajbozarth
505 Howard
Street
San Francisco, CA
94105
United
States
Inactive hide details for kant kodali ---01/18/2018
05:34:07 PM---Hi All, I was able to submit a streaming
job to livy however kant kodali ---01/18/2018 05:34:07
PM---Hi All, I was able to submit a streaming job to
livy however I wasn't able to find
From: kant kodali <kanth...@gmail.com>
To: user@livy.incubator.apache.org
Date: 01/18/2018 05:34 PM
Subject: How to cancel the running streaming job using
livy?
Hi All,
I was able to submit a streaming job to livy however I
wasn't able to find any way to cancel the running the
job? Please let me know.
Thanks!
--
Marcelo
