[
https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272956#comment-17272956
]
Apache Spark commented on SPARK-34265:
--------------------------------------
User 'LucaCanali' has created a pull request for this issue:
https://github.com/apache/spark/pull/31367
> Add SQLMetrics instrumentation to Python UDF
> --------------------------------------------
>
> Key: SPARK-34265
> URL: https://issues.apache.org/jira/browse/SPARK-34265
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.1.1
> Reporter: Luca Canali
> Priority: Minor
> Attachments: PythonSQLMetrics_Jira_Picture.png
>
>
> This proposes to add SQLMetrics instrumentation for Python UDF. This is aimed
> at improving monitoring and performance troubleshooting of Python code called
> by Spark, via UDF, Pandas UDF or with MapPartittions.
> The introduced metrics are exposed to the end users via the WebUI interface,
> in the SQL tab for execution steps related to Python UDF execution, namely
> BatchEvalPython, ArrowEvalPython, AggregateInPandas, FlaMapGroupsInPandas,
> FlatMapsCoGroupsInPandas, WindowsInPandas.
> See also the attached screenshot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
