Gengliang Wang created SPARK-59841:
--------------------------------------
Summary: Align table and changelog state-option handling and
metadata reuse
Key: SPARK-59841
URL: https://issues.apache.org/jira/browse/SPARK-59841
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
TableCatalog currently has different option-handling contracts for ordinary
table reads and changelog reads:
* {{tableStateOptionKeys()}} declares the connector options that select table
state. Spark passes only those options to {{loadTable(Identifier, TableContext,
stateOptions)}} and uses them for table metadata reuse. Complete per-read
options remain available for scan planning.
* {{loadChangelog(Identifier, ChangelogContext, options)}} receives all query
options, and changelog resolution bypasses the corresponding state-aware
table-loading/cache path. There is no separate changelog state-option
declaration.
State options do reach CDC connectors, but Spark does not apply the same option
projection and metadata-reuse rules to the two paths. This complicates
deduplicating loads and refreshes, including the ongoing work on concurrent
table refresh.
For example, a current-table read and a changes-from-version-1 read with
matching table-state options should be able to share the underlying captured
table state. Their scan options and changelog range still describe distinct
reads. Different state options must remain isolated, and different changelog
contexts must not be treated as equivalent reads.
The goal is to define consistent state-option, caching, and refresh semantics
for ordinary and changelog reads, while retaining complete scan options and
preserving the requested CDC context through refresh.
The API approach is undecided. This ticket does not prescribe changing the
Changelog hierarchy, adding a baseTable accessor, or any particular wrapper or
refresh-request abstraction. The implementation should be evaluated for
compatibility with existing connector APIs for the Spark 4.3.0 release.
References:
* [TableCatalog option
contracts|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java#L104]
* [Current changelog
resolution|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala#L382]
* [Exploratory prototype for discussion; API not
decided|https://github.com/gengliangwang/spark/pull/20]
Target release: Spark 4.3.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
