[
https://issues.apache.org/jira/browse/SPARK-58389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Okolnychyi resolved SPARK-58389.
--------------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57582
[https://github.com/apache/spark/pull/57582]
> Pass all options while loading tables
> -------------------------------------
>
> Key: SPARK-58389
> URL: https://issues.apache.org/jira/browse/SPARK-58389
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yan Yan
> Assignee: Yan Yan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> SPARK-56961 changed TableCatalog.loadChangelog to receive all user-specified
> options (via a ChangelogContext + CaseInsensitiveStringMap), so connectors can
> see the options a query passes. The regular table read path was not updated:
> TableCatalog.loadTable and its overloads do not forward the user's options to
> the catalog, so a connector cannot access them when a table is loaded.
> This is a follow-up that does the same for table reads. It adds
> ```
> TableCatalog.loadTable(Identifier ident, TableContext context,
> CaseInsensitiveStringMap options)
> ```
> where TableContext carries the parsed, Spark-recognized load parameters (time
> travel and write privileges) and the CaseInsensitiveStringMap carries all raw
> user options. The new method has a default implementation that delegates to
> the
> existing loadTable overloads, so existing connectors keep working unchanged.
> TableContext models time travel with a new public connector type, TimeTravel
> (a sealed interface with Version/Timestamp), mirroring how SPARK-56961
> introduced ChangelogContext/ChangelogRange rather than exposing the internal
> catalyst TimeTravelSpec.
> This is needed to make the read API usable in connectors like Iceberg and
> Delta. The functionality is unreleased.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
