ExecutorId in JAVA_OPTS

[surbhi.mungre]

I was trying to profile some Spark jobs and I want to collect Java Flight
Recorder(JFR) files from each executor. I am running my job on a YARN
cluster with several nodes, so I cannot manually collect JRF file for each
run. 

MR provides a way to name JFR files generated by each task with taskId. It
replaces '@task@' with TaskId in Java opts. With this I can get a unique
name for JFR files created by each task and the since TaskId also has JobId,
I can parse it to distinguish files generated by different MR jobs. 

I am wondering, if Spark has something similar. Does Spark provides a way to
determine executorId in Java opts? Has anyone else has tried to do something
similar and found a better way collect all JFR files for a Spark job? 



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/ExecutorId-in-JAVA-OPTS-tp25280.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Preemption with Spark on Yarn

[surbhi.mungre]

Hi All, 

I am new to Spark and I am trying to understand how preemption works with
Spark on Yarn. My goal is to determine amount of re-work a Spark application
has to do if an executor is preempted. 

For my test, I am using a 4 node cluster with Cloudera VM running Spark
1.3.0. I am running PageRank spark example. I tried to run tests with both
Capacity Scheduler and Fair Scheduler and I can tell from Resource Manager
and Application Master logs that containers are getting preempted. However,
I am not able to see any task/executor failures in Spark UI. I checked logs
for Driver(in yarn-client mode), Application Master and preempted Container
but I am not able to answer the question I have. The main questions which I
want to answer are: 

1. What happens to the tasks which were killed due to preemption? Why I do
not see any failure for these tasks on history server UI?  
2. What happens to the tasks which are already completed by the executor
which was preempted? Are there any cases when these tasks will be
recomputed? 
3. What happens to the tasks which are pending to be picked up by executors
which was preempted? I am guessing these are scheduled on other executors
but I cannot tell that from logs. 

It will be great if I get some help to answer these questions. 

Thanks,
Surbhi








--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Preemption-with-Spark-on-Yarn-tp25146.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org