[
https://issues.apache.org/jira/browse/SPARK-59163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-59163:
----------------------------
Description:
h3. Background
SPARK-58392 and PR #57585 introduced table-state option projection for relation
loading. During relation resolution, Spark already materializes the projected
state-option map for the per-query table cache key.
h3. Problem
Some subsequent paths derive the same projection again before catalog loading
or shared relation cache lookup. This repeats normalization and materialization
and makes correctness depend on independently computed maps remaining
equivalent.
Safely reusing one projected map also requires it to remain stable after it is
passed to connector code. {{CaseInsensitiveStringMap}} rejects direct mutation,
but its normalized collection views were mutable. This makes an instance unsafe
to retain as cache or deduplication identity after exposing it to a connector.
h3. Proposed change
Reuse the already-projected state-option map for the table cache key, catalog
load, shared relation cache lookup, and execution refresh. Also apply a minor
internal refactoring so method names make it explicit when callers pass a
complete option map versus already-projected state options.
Make {{CaseInsensitiveStringMap}} consistently read-only by backing its
normalized entries with an unmodifiable map. The same projected instance can
then be passed to catalog code and retained for cache or refresh-deduplication
identity. Avoiding a reconstructed defensive copy also preserves original-case
entries and the effective value already selected when input contains keys that
differ only by case.
h3. User-facing impact
{{CaseInsensitiveStringMap}} is an {{@Experimental}} connector-facing API that
was already intended to be read-only. Attempts to mutate it through
{{keySet()}}, {{values()}}, or {{entrySet()}} now throw
{{UnsupportedOperationException}}, consistent with its existing direct
mutators. Read behavior is unchanged.
Related issue: SPARK-58392
PR: https://github.com/apache/spark/pull/58462
was:
h3. Background
SPARK-58392 and PR #57585 introduced table-state option projection for relation
loading. During relation resolution, Spark already materializes the projected
state-option map for the per-query table cache key.
h3. Problem
Some subsequent paths derive the same projection again before catalog loading
or shared relation cache lookup. This repeats normalization and materialization
and makes correctness depend on independently computed maps remaining
equivalent.
h3. Proposed change
Reuse the already-projected state-option map for the table cache key, catalog
load, shared relation cache lookup, and execution refresh. Also apply a minor
internal refactoring so method names make it explicit when callers pass a
complete option map versus already-projected state options.
There is no user-facing behavior change.
Related issue: SPARK-58392
PR: https://github.com/apache/spark/pull/58462
> Reuse projected table state options across table loading and cache lookup
> -------------------------------------------------------------------------
>
> Key: SPARK-59163
> URL: https://issues.apache.org/jira/browse/SPARK-59163
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> h3. Background
> SPARK-58392 and PR #57585 introduced table-state option projection for
> relation loading. During relation resolution, Spark already materializes the
> projected state-option map for the per-query table cache key.
> h3. Problem
> Some subsequent paths derive the same projection again before catalog loading
> or shared relation cache lookup. This repeats normalization and
> materialization and makes correctness depend on independently computed maps
> remaining equivalent.
> Safely reusing one projected map also requires it to remain stable after it
> is passed to connector code. {{CaseInsensitiveStringMap}} rejects direct
> mutation, but its normalized collection views were mutable. This makes an
> instance unsafe to retain as cache or deduplication identity after exposing
> it to a connector.
> h3. Proposed change
> Reuse the already-projected state-option map for the table cache key, catalog
> load, shared relation cache lookup, and execution refresh. Also apply a minor
> internal refactoring so method names make it explicit when callers pass a
> complete option map versus already-projected state options.
> Make {{CaseInsensitiveStringMap}} consistently read-only by backing its
> normalized entries with an unmodifiable map. The same projected instance can
> then be passed to catalog code and retained for cache or
> refresh-deduplication identity. Avoiding a reconstructed defensive copy also
> preserves original-case entries and the effective value already selected when
> input contains keys that differ only by case.
> h3. User-facing impact
> {{CaseInsensitiveStringMap}} is an {{@Experimental}} connector-facing API
> that was already intended to be read-only. Attempts to mutate it through
> {{keySet()}}, {{values()}}, or {{entrySet()}} now throw
> {{UnsupportedOperationException}}, consistent with its existing direct
> mutators. Read behavior is unchanged.
> Related issue: SPARK-58392
> PR: https://github.com/apache/spark/pull/58462
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
