[
https://issues.apache.org/jira/browse/SPARK-59163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-59163:
----------------------------
Description:
h3. Background
SPARK-58392 and PR #57585 introduced table-state option projection for relation
loading. During relation resolution, Spark already materializes the projected
state-option map for the per-query table cache key.
h3. Problem
Some subsequent paths derive the same projection again before catalog loading
or shared relation cache lookup. This repeats normalization and materialization
and makes correctness depend on independently computed maps remaining
equivalent.
h3. Proposed change
Reuse the already-projected state-option map for the table cache key, catalog
load, shared relation cache lookup, and execution refresh. Also apply a minor
internal refactoring so method names make it explicit when callers pass a
complete option map versus already-projected state options.
There is no user-facing behavior change.
Related issue: SPARK-58392
PR: https://github.com/apache/spark/pull/58462
was:
h3. Background
SPARK-58392 and PR #57585 introduced table-state option projection for relation
loading. During relation resolution, Spark already materializes the projected
state-option map for the per-query table cache key.
h3. Problem
Some subsequent paths derive the same projection again before catalog loading
or shared relation cache lookup. This repeats normalization and materialization
and makes correctness depend on independently computed maps remaining
equivalent.
h3. Proposed change
Reuse the already-projected state-option map for the table cache key, catalog
load, shared relation cache lookup, and execution refresh. Also apply a minor
internal refactoring so method names make it explicit when callers pass a
complete option map versus already-projected state options.
There is no user-facing behavior change.
Related issue: SPARK-58392
> Reuse projected table state options across table loading and cache lookup
> -------------------------------------------------------------------------
>
> Key: SPARK-59163
> URL: https://issues.apache.org/jira/browse/SPARK-59163
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> h3. Background
> SPARK-58392 and PR #57585 introduced table-state option projection for
> relation loading. During relation resolution, Spark already materializes the
> projected state-option map for the per-query table cache key.
> h3. Problem
> Some subsequent paths derive the same projection again before catalog loading
> or shared relation cache lookup. This repeats normalization and
> materialization and makes correctness depend on independently computed maps
> remaining equivalent.
> h3. Proposed change
> Reuse the already-projected state-option map for the table cache key, catalog
> load, shared relation cache lookup, and execution refresh. Also apply a minor
> internal refactoring so method names make it explicit when callers pass a
> complete option map versus already-projected state options.
> There is no user-facing behavior change.
> Related issue: SPARK-58392
> PR: https://github.com/apache/spark/pull/58462
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
