Davies Liu created SPARK-3478:
---------------------------------
Summary: Profile Python tasks stage by stage in worker
Key: SPARK-3478
URL: https://issues.apache.org/jira/browse/SPARK-3478
Project: Spark
Issue Type: New Feature
Components: PySpark
Reporter: Davies Liu
The Python code in driver is easy to profile by users, but the code run in
worker is distributed in clusters, is not easy to profile by users.
So we need a way to do the profiling in worker and aggregate all the result
together for users.
This also can be used to analys the bottleneck in PySpark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
