[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630080#comment-17630080
]
Alfred Davidson commented on SPARK-40281:
-----------------------------------------
+1 in general this would be good regardless of access/usage pattern. Also
having the option to have this exposed in the UI would be a bonus and will
consolidate metrics (I can imagine many doing their own thing). I look forward
to seeing what’s next
> Memory Profiler on Executors
> ----------------------------
>
> Key: SPARK-40281
> URL: https://issues.apache.org/jira/browse/SPARK-40281
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 3.4.0
> Reporter: Xinrong Meng
> Priority: Major
>
> Profiling is critical to performance engineering. Memory consumption is a key
> indicator of how efficient a PySpark program is. There is an existing effort
> on memory profiling of Python programs, Memory Profiler
> ([https://pypi.org/project/memory-profiler/).|https://pypi.org/project/memory-profiler/]
> PySpark applications run as independent sets of processes on a cluster,
> coordinated by the SparkContext object in the driver program. On the driver
> side, PySpark is a regular Python process, thus, we can profile it as a
> normal Python program using Memory Profiler.
> However, on the executors side, we are missing such memory profiler. Since
> executors are distributed on different nodes in the cluster, we need to
> aggregate profiles. Furthermore, Python worker processes are spawned per
> executor for the Python/Pandas UDF execution, which makes the memory
> profiling more intricate.
> The ticket proposes to implement a Memory Profiler on Executors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
