[ https://issues.apache.org/jira/browse/SPARK-31903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Takuya Ueshin updated SPARK-31903: ---------------------------------- Description: When calling {{toPandas}}, usually Query UI shows each plan node's metric and corresponding Stage ID and Task ID: {code:java} >>> df = spark.createDataFrame([(1, 10, 'abc'), (2, 20, 'def')], schema=['x', >>> 'y', 'z']) >>> df.toPandas() x y z 0 1 10 abc 1 2 20 def {code} !Screen Shot 2020-06-03 at 4.47.07 PM.png! but if Arrow execution is enabled, it shows only plan nodes: {code:java} >>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True) >>> df.toPandas() x y z 0 1 10 abc 1 2 20 def{code} !Screen Shot 2020-06-03 at 4.47.27 PM.png! was: When calling {{toPandas}}, usually Query UI shows each plan node's metric and corresponding Stage ID and Task ID: {code:java} >>> df = spark.createDataFrame([(1, 10, 'abc'), (2, 20, 'def')], schema=['x', >>> 'y', 'z']) >>> df.toPandas() x y z 0 1 10 abc 1 2 20 def {code} !Screen Shot 2020-06-03 at 4.47.07 PM.png! but if Arrow execution is enabled, it shows only plan nodes: {code:java} >>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True) >>> df.toPandas() x y z 0 1 10 abc 1 2 20 def{code} !Screen Shot 2020-06-03 at 4.47.27 PM.png! > toPandas with Arrow enabled doesn't show metrics in Query UI. > ------------------------------------------------------------- > > Key: SPARK-31903 > URL: https://issues.apache.org/jira/browse/SPARK-31903 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.4.5, 3.0.0 > Reporter: Takuya Ueshin > Priority: Major > Attachments: Screen Shot 2020-06-03 at 4.47.07 PM.png, Screen Shot > 2020-06-03 at 4.47.27 PM.png > > > When calling {{toPandas}}, usually Query UI shows each plan node's metric and > corresponding Stage ID and Task ID: > {code:java} > >>> df = spark.createDataFrame([(1, 10, 'abc'), (2, 20, 'def')], schema=['x', > >>> 'y', 'z']) > >>> df.toPandas() > x y z > 0 1 10 abc > 1 2 20 def > {code} > !Screen Shot 2020-06-03 at 4.47.07 PM.png! > but if Arrow execution is enabled, it shows only plan nodes: > {code:java} > >>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True) > >>> df.toPandas() > x y z > 0 1 10 abc > 1 2 20 def{code} > > !Screen Shot 2020-06-03 at 4.47.27 PM.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org