Yan Yan created SPARK-59163:
-------------------------------
Summary: Reuse projected table state options across table loading
and cache lookup
Key: SPARK-59163
URL: https://issues.apache.org/jira/browse/SPARK-59163
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Yan Yan
h3. Background
SPARK-58392 and PR #57585 introduced table-state option projection for relation
loading. During relation resolution, Spark already materializes the projected
state-option map for the per-query table cache key.
h3. Problem
Some subsequent paths derive the same projection again before catalog loading
or shared relation cache lookup. This repeats normalization and materialization
and makes correctness depend on independently computed maps remaining
equivalent.
h3. Proposed change
Reuse the already-projected state-option map for the table cache key, catalog
load, shared relation cache lookup, and execution refresh. Also apply a minor
internal refactoring so method names make it explicit when callers pass a
complete option map versus already-projected state options.
There is no user-facing behavior change.
Related issue: SPARK-58392
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
