[
https://issues.apache.org/jira/browse/SPARK-59163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-59163.
---------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 58462
[https://github.com/apache/spark/pull/58462]
> Reuse projected table state options across table loading and cache lookup
> -------------------------------------------------------------------------
>
> Key: SPARK-59163
> URL: https://issues.apache.org/jira/browse/SPARK-59163
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Yan Yan
> Assignee: Yan Yan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h3. Background
> SPARK-58392 and PR #57585 introduced table-state option projection for
> relation loading. During relation resolution, Spark already materializes the
> projected state-option map for the per-query table cache key.
> h3. Problem
> Some subsequent paths derive the same projection again before catalog loading
> or shared relation cache lookup. This repeats normalization and
> materialization and makes correctness depend on independently computed maps
> remaining equivalent.
> Safely reusing one projected map also requires it to remain stable after it
> is passed to connector code. {{CaseInsensitiveStringMap}} rejects direct
> mutation, but its normalized collection views were mutable. This makes an
> instance unsafe to retain as cache or deduplication identity after exposing
> it to a connector.
> h3. Proposed change
> Reuse the already-projected state-option map for the table cache key, catalog
> load, shared relation cache lookup, and execution refresh. Also apply a minor
> internal refactoring so method names make it explicit when callers pass a
> complete option map versus already-projected state options.
> Make {{CaseInsensitiveStringMap}} consistently read-only by backing its
> normalized entries with an unmodifiable map. The same projected instance can
> then be passed to catalog code and retained for cache or
> refresh-deduplication identity. Avoiding a reconstructed defensive copy also
> preserves original-case entries and the effective value already selected when
> input contains keys that differ only by case.
> h3. User-facing impact
> {{CaseInsensitiveStringMap}} is an {{@Experimental}} connector-facing API
> that was already intended to be read-only. Attempts to mutate it through
> {{keySet()}}, {{values()}}, or {{entrySet()}} now throw
> {{UnsupportedOperationException}}, consistent with its existing direct
> mutators. Read behavior is unchanged.
> Related issue: SPARK-58392
> PR: https://github.com/apache/spark/pull/58462
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
