[
https://issues.apache.org/jira/browse/SPARK-58389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-58389:
----------------------------
Description:
SPARK-56961 changed TableCatalog.loadChangelog to receive user-specified
options so connectors can access query options. The regular table-read path did
not provide an equivalent options-aware API.
This follow-up adds:
TableCatalog.loadTable(
Identifier ident,
TableContext context,
CaseInsensitiveStringMap stateOptions)
TableContext carries Spark-parsed load parameters such as time travel and write
privileges. Catalogs declare connector-specific options that select table
state, such as a branch, tag, snapshot, or version, through
TableCatalog.tableStateOptionKeys(). Spark passes only those declared options
in stateOptions.
The complete user option map remains on the resolved relation for subsequent
scan and write planning. Spark reuses one table instance only when references
have matching table-state options.
The new loadTable method defaults to the existing overloads, and
tableStateOptionKeys() defaults to an empty set, preserving compatibility for
existing connectors. An implementation that overrides the new method must honor
the time-travel and write-privilege context itself.
TableContext models time travel with the public TimeTravel connector type
rather than exposing the internal Catalyst TimeTravelSpec.
The functionality was unreleased when introduced.
was:
SPARK-56961 changed TableCatalog.loadChangelog to receive all user-specified
options (via a ChangelogContext + CaseInsensitiveStringMap), so connectors can
see the options a query passes. The regular table read path was not updated:
TableCatalog.loadTable and its overloads do not forward the user's options to
the catalog, so a connector cannot access them when a table is loaded.
This is a follow-up that does the same for table reads. It adds
```
TableCatalog.loadTable(Identifier ident, TableContext context,
CaseInsensitiveStringMap options)
```
where TableContext carries the parsed, Spark-recognized load parameters (time
travel and write privileges) and the CaseInsensitiveStringMap carries all raw
user options. The new method has a default implementation that delegates to the
existing loadTable overloads, so existing connectors keep working unchanged.
TableContext models time travel with a new public connector type, TimeTravel
(a sealed interface with Version/Timestamp), mirroring how SPARK-56961
introduced ChangelogContext/ChangelogRange rather than exposing the internal
catalyst TimeTravelSpec.
This is needed to make the read API usable in connectors like Iceberg and
Delta. The functionality is unreleased.
> Pass table state options while loading tables
> ---------------------------------------------
>
> Key: SPARK-58389
> URL: https://issues.apache.org/jira/browse/SPARK-58389
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yan Yan
> Assignee: Yan Yan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> SPARK-56961 changed TableCatalog.loadChangelog to receive user-specified
> options so connectors can access query options. The regular table-read path
> did not provide an equivalent options-aware API.
> This follow-up adds:
> TableCatalog.loadTable(
> Identifier ident,
> TableContext context,
> CaseInsensitiveStringMap stateOptions)
> TableContext carries Spark-parsed load parameters such as time travel and
> write privileges. Catalogs declare connector-specific options that select
> table state, such as a branch, tag, snapshot, or version, through
> TableCatalog.tableStateOptionKeys(). Spark passes only those declared options
> in stateOptions.
> The complete user option map remains on the resolved relation for subsequent
> scan and write planning. Spark reuses one table instance only when references
> have matching table-state options.
> The new loadTable method defaults to the existing overloads, and
> tableStateOptionKeys() defaults to an empty set, preserving compatibility for
> existing connectors. An implementation that overrides the new method must
> honor the time-travel and write-privilege context itself.
> TableContext models time travel with the public TimeTravel connector type
> rather than exposing the internal Catalyst TimeTravelSpec.
> The functionality was unreleased when introduced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
