Anurag Mantripragada created SPARK-59943:
--------------------------------------------
Summary: Carry the CREATE/REPLACE TABLE write distribution and
ordering on TableSpec
Key: SPARK-59943
URL: https://issues.apache.org/jira/browse/SPARK-59943
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Anurag Mantripragada
SPARK-34586 (https://github.com/apache/spark/pull/58153) adds
writeDistributionMode and writeOrdering as parameters of CreateTable,
CreateTableAsSelect, ReplaceTable and ReplaceTableAsSelect, and of the
CREATE/REPLACE TABLE execs. That changes the plan and exec signatures, so
extensions that match or construct them positionally stop compiling (for
example Apache Paimon's PaimonCreateTableAsSelectStrategy).
Carry the pair on TableSpec instead, as the SPARK-43529 follow-up (7e94f2a5433)
did for its parameter, so the four plan signatures and the exec signatures go
back to what they were before SPARK-34586.
Once a custom strategy compiles again it skips
WriteDistributionAndOrdering.validateCatalogForWriteDistributionAndOrdering,
which runs in DataSourceV2Strategy, so move that check into the execs or into
CheckAnalysis. TableSpec.withNewLocation also rebuilds TableSpec positionally
and has to keep the pair. Should land before 4.4.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
