Mike Chan created SPARK-27264:
---------------------------------
Summary: spark sql released all executor but the job is not done
Key: SPARK-27264
URL: https://issues.apache.org/jira/browse/SPARK-27264
Project: Spark
Issue Type: Question
Components: SQL
Affects Versions: 2.4.0
Environment: Azure HDinsight spark 2.4 on Azure storage SQL: Read and
Join some data and finally write result to a Hive metastore
Reporter: Mike Chan
I have a spark sql that used to execute < 10 mins now running at 3 hours after
a cluster migration and need to deep dive on what it's actually doing. I'm new
to spark and please don't mind if I'm asking something unrelated.
Increased spark.executor.memory but no luck. Env: Azure HDinsight spark 2.4 on
Azure storage SQL: Read and Join some data and finally write result to a Hive
metastore
The sparl.sql ends with below code:
.write.mode("overwrite").saveAsTable("default.mikemiketable")
Application Behavior: Within the first 15 mins, it loads and complete most
tasks (199/200); left only 1 executor process alive and continually to shuffle
read / write data. Because now it only leave 1 executor, we need to wait 3
hours until this application finish.
[!https://i.stack.imgur.com/6hqvh.png!|https://i.stack.imgur.com/6hqvh.png]
Left only 1 executor alive
[!https://i.stack.imgur.com/55162.png!|https://i.stack.imgur.com/55162.png]
Not sure what's the executor doing:
[!https://i.stack.imgur.com/TwhuX.png!|https://i.stack.imgur.com/TwhuX.png]
>From time to time, we can tell the shuffle read increased:
>[!https://i.stack.imgur.com/WhF9A.png!|https://i.stack.imgur.com/WhF9A.png]
Therefore I increased the spark.executor.memory to 20g, but nothing changed.
From Ambari and YARN I can tell the cluster has many resources left.
[!https://i.stack.imgur.com/pngQA.png!|https://i.stack.imgur.com/pngQA.png]
Release of almost all executor
[!https://i.stack.imgur.com/pA134.png!|https://i.stack.imgur.com/pA134.png]
Any guidance is greatly appreciated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
