[
https://issues.apache.org/jira/browse/SPARK-55534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18083008#comment-18083008
]
rishavsinha commented on SPARK-55534:
-------------------------------------
I tried reproducing this issue on current master and observed that
{{queryExecution.executedPlan}} for INSERT queries is still available and
contains the write subtree wrapped in {{{}CommandResultExec{}}}.
Repro:
{{spark.sql("DROP TABLE IF EXISTS tt")
spark.sql("CREATE TABLE tt(id INT) USING parquet")
val insertDf = spark.sql("INSERT INTO tt VALUES (1)")
println(insertDf.queryExecution.executedPlan)}}
Observed physical plan:
{{CommandResult <empty>
+- Execute InsertIntoHadoopFsRelationCommand ...
+- WriteFiles
+- LocalTableScan ...}}
So the executed write plan does appear to be preserved inside
{{{}CommandResultExec{}}}, although the command itself is eagerly executed
through {{eagerlyExecuteCommands()}} in {{{}QueryExecution{}}}.
This may indicate the original issue has already been addressed partially, or
the remaining inconsistency is more specifically about the returned DataFrame
semantics rather than loss of the executed physical plan itself.
> executedPlan for Insert/Update query
> ------------------------------------
>
> Key: SPARK-55534
> URL: https://issues.apache.org/jira/browse/SPARK-55534
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.1.1
> Reporter: Shekhar Prasad Rajak
> Priority: Major
> Labels: extension, parser, query-engine, sql
>
> We should get consistent result for any kind of SQL query for API
> df.queryExecution.executedPlan
>
> // For SELECT queries
> val df = spark.sql("SELECT * FROM t")
> val plan = df.queryExecution.executedPlan // Works - returns DataFrame with
> plan
> // For INSERT/WRITE commands
> val df = spark.sql("INSERT INTO t VALUES (1)")
> // df is an EMPTY DataFrame with just row count metrics
> // The actual write plan is ALREADY EXECUTED and gone, so we can not find the
> executed plan
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
