See if https://spark.apache.org/docs/latest/monitoring.html helps.
Essentially whether you are running an app as spark-shell, via spark-submit (local, Spark-Cluster, YARN, Kubernetes, mesos), the driver will provide a UI on port 4040. You can monitor via the UI and via a REST API E.g. running a job locally on your laptop, you can run something like this http://127.0.0.1:4040/api/v1/applications To see the “jobs”, you can use something like this (local-1544110095543 is just the id of my spark-shell on my laptop which I got from the command above) http://127.0.0.1:4040/api/v1/applications/local-1544110095543/jobs If you are only look for job completion, you can just monitor if there is a listener on the port. Once job completes/fails, the driver and the listener will exit and hence it means the job is complete. As far as giving you the percentage complete for the application, there is no such thing as unlike mapreduce, a Spark app is not a single step/job. Using the REST API, you can see which job/stage is running and determine what percentage of your job is complete. Even when getting the stage info, you only get the number of “tasks” complete v/s percentage complete. From: kant kodali <kanth...@gmail.com> Date: Thursday, December 6, 2018 at 4:40 AM To: Mark Hamstra <m...@clearstorydata.com> Cc: <pmatp...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: How to track batch jobs in spark ? Thanks for all responses. 1) I am not using YARN. I am using Spark Standalone. 2) yes I want to be able to kill the whole Application. 3) I want to be able to monitor the status of the Application which is running a batch query and expected to run for an hour or so, therefore, I am looking for some mechanism where I can monitor the progress like a percentage or something. Thanks! On Wed, Dec 5, 2018 at 3:12 PM Mark Hamstra <m...@clearstorydata.com<mailto:m...@clearstorydata.com>> wrote: That will kill an entire Spark application, not a batch Job. On Wed, Dec 5, 2018 at 3:07 PM Priya Matpadi <pmatp...@gmail.com<mailto:pmatp...@gmail.com>> wrote: if you are deploying your spark application on YARN cluster, 1. ssh into master node 2. List the currently running application and retreive the application_id yarn application --list 3. Kill the application using application_id of the form application_xxxxx_xxxx from output of list command yarn application --kill <application_id> On Wed, Dec 5, 2018 at 1:42 PM kant kodali <kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote: Hi All, How to track batch jobs in spark? For example, is there some id or token i can get after I spawn a batch job and use it to track the progress or to kill the batch job itself? For Streaming, we have StreamingQuery.id() Thanks!
