[
https://issues.apache.org/jira/browse/SPARK-59751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Rielau resolved SPARK-59751.
----------------------------------
Resolution: Won't Do
Obsolete
> Preserve CHAR/VARCHAR scan policy across cache mutation, refresh, and rename
> ----------------------------------------------------------------------------
>
> Key: SPARK-59751
> URL: https://issues.apache.org/jira/browse/SPARK-59751
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Serge Rielau
> Priority: Major
>
> First-class CHAR/VARCHAR under SPARK-58794 binds an analyzed scan mode
> ({{{}PreserveNative{}}} / {{{}SparkStandard{}}}) onto relation identity so
> incompatible scans cannot reuse each other's cached results. SPARK-58814 / PR
> 58317 covers format round-trips and ORC truncation, plus the cache-lifecycle
> repairs that were required to keep that identity from serving stale rows on
> the paths already in that PR.
> This follow-up is the remaining cache-policy contract. Today {{None}} still
> means "unbound / not first-class". A cache created with first-class flags off
> can be rebound when a later session with first-class modes recaches or
> renames the table. V1 and Hive rename still snapshot one cache entry, so
> coexisting PreserveNative and SparkStandard variants can lose a mode or
> storage level. Hive refresh coexistence and a simultaneous three-policy
> mutation matrix are also not complete.
> Scope:
> * Represent the pre-first-class policy as an explicit bound state (or
> equivalent), distinct from unbound.
> * Preserve that policy, plus PreserveNative and SparkStandard, through
> recache, refreshTable, and rename for V1, V2, and Hive.
> * Keep ordinary cache substitution mode-sensitive; ignore scan mode only for
> mutation/rename discovery.
> * Restore only direct table caches (including analyzer CHAR/VARCHAR padding
> Projects); do not promote dependent-query or time-travel caches.
> * Reproduce with coexisting variants in one SharedState, including a
> mutation/rename from a session using a different policy.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
