yyanyy opened a new pull request, #57799:
URL: https://github.com/apache/spark/pull/57799
### What changes were proposed in this pull request?
SPARK-58389 changed DSv2 relation caching to include all read options. This
is required
to preserve each reference's complete option bag, but it also means
references that differ
only in scan-specific options can independently call `loadTable` and receive
different
concrete table versions within one query.
This PR separates relation reuse from table-state pinning:
* Add the evolving `SupportsTableStateOptions` catalog capability. Catalogs
use
`tableStateOptionKeys()` to identify raw options that may affect the table
state selected
by `loadTable`.
* Add a query-scoped `AnalysisContext.tableCache`, keyed by catalog
instance, identifier,
parsed time-travel specification, and projected table-state options.
* Keep `AnalysisContext.relationCache` keyed by all options so each
reference preserves its
complete option bag.
* Continue requiring all options to match when consulting
`sharedRelationCache`, which is
backed by `CacheManager` entries created by operations such as `CACHE
TABLE`.
* Before a table pin exists, allow a full-option shared relation cache match
by `Table.id`
after loading the current table identity. After a pin exists, require the
cached relation
to contain the exact pinned `Table` instance.
* Apply the same table-state projection and first-resolution-wins behavior
to cacheable
`V2TableReference` resolution and execution refresh.
* Keep write targets outside both query-scoped read caches. Parsed time
travel remains a
separate component of the table-state key.
Catalogs that do not implement `SupportsTableStateOptions` are handled
conservatively:
all raw options are treated as table-state-affecting.
Generic `spark.read.format(...).load()` resolution remains outside this
change.
### Why are the changes needed?
A query can reference the same table more than once with different
scan-specific options,
for example different split sizes. Those references must retain their own
options, but they
should use one concrete `Table` instance when the options select the same
table state.
Without a separate table-state cache, independently resolving those
references can observe
different external table versions within one analyzed query. This PR
preserves full-option
relation semantics while restoring first-resolution-wins consistency within
each table-state
domain.
Different state domains, including different parsed time-travel
specifications, continue to
resolve and pin independently.
### Does this PR introduce _any_ user-facing change?
Yes, for catalog implementors only.
This adds the evolving `SupportsTableStateOptions` catalog capability. There
is no new SQL syntax
or configuration, and catalogs that do not implement it retain the
conservative behavior where all
raw options are considered table-state-affecting.
The table-consistency fix otherwise addresses behavior introduced on the
unreleased master branch
by SPARK-58389.
### How was this patch tested?
Added coverage for state-option projection and fallback, query-scoped table
pinning, shared relation
cache matching, refresh, time travel, streaming, nested views, and
write-target bypass.
* `build/sbt 'catalyst/testOnly *CatalogV2UtilSuite'` — 10/10 passed.
* `build/sbt 'catalyst/testOnly *TableLookupCacheSuite'` — 3/3 passed.
* `build/sbt 'sql/testOnly *PlanResolutionSuite'` — 95/95 passed.
* `build/sbt 'sql/testOnly *DataSourceV2OptionSuite'` — 36/36 passed.
* `build/sbt 'sql/testOnly *DataSourceV2DataFrameSuite'` — 203/203 passed.
* Focused transaction and changelog tests passed.
* Catalyst/SQL compilation, scalastyle, and `git diff --check` passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
