pepijnve opened a new pull request, #27210:
URL: https://github.com/apache/datafusion/pull/27210
## Which issue does this PR close?
- Closes #27209.
- Relates to #7292 which is an earlier time travel discussion
- Relates to #24758 which suggests to reject table modifiers the default
planner ignores, including version clauses
## Rationale for this change
See #27209 description for full details. Adding first class support for time
travel queries in the core library simplifies the implementation work for
extensions that want to support this functionality.
## What changes are included in this PR?
- Table resolution becomes version aware. This impacts the table
pre-planning table resolution in `SessionState::statement_to_plan` and the
subsequent lookup of tables during SQL to logical plan translation.
- `ContextProvider` gains `evaluate_table_version` and `get_table_source_at`
to support SQL to logical translation.
- In order to not break existing `RelationPlanner` based version handlers
like Paimon, the unversioned table is also prefetched when relation planners
have been registered.
- `SchemaProvider::table_at(name, &TableVersion)` is added which returns a
specific version of
a table. The default returns a "not implemented" error, so catalogs without
time travel support fail instead of silently returning the current version.
- `TableProvider::version()` / `TableSource::version()` report the resolved
version that will actually be read. Providers returned by `table_at` must
report one, which may differ from the requested version: a timestamp or a tag
typically resolves to a snapshot id. `ListingTable`, `MemTable`, `ViewTable`
and `EmptyTable` gain `with_version` to make it easy for existing
implementations to adopt versioning support.
- `TableScan` equality, ordering and hashing include the version reported by
its source, which fixes the subquery described in the issue . This also covers
scans built by `RelationPlanner` extensions, as long as their providers
implement `TableProvider::version()`
### Known limitations
- Substrait: versioned scans are rejected, as Substrait cannot represent
them.
- Prepared statements: parameters are not supported in time travel clauses,
and
clauses that depend on the time of execution, such as `now()`, are rejected
in `PREPARE`, as they would be evaluated when the statement is prepared.
## What is the testing strategy for this PR?
- `time_travel.slt`, with a test catalog containing a table with three
versions,
covers all syntaxes, version and timestamp resolution, combining versions
in
one query (joins, `EXCEPT`, subqueries), views, `EXPLAIN`, prepared
statements
and error cases. It includes the subquery queries above as regression
tests:
they return wrong results when the version is removed from `TableScan`
equality.
- `relation_planner.rs`: extensions can still handle time travel themselves,
and the current version is only fetched when relation planners are
registered.
- `sql_integration.rs` / `plan_to_sql.rs`: planning with a mock provider, the
source version check, and unparsing for each dialect style.
- `resolve.rs`: collecting versioned table references, including tables in
subqueries inside time travel clauses.
- `roundtrip_logical_plan.rs`: protobuf round trips of versioned scans.
## Are there any user-facing changes?
Yes:
- New SQL feature: time travel queries, for the dialects whose sqlparser
implementation parses these clauses (Snowflake, Databricks, BigQuery, MsSQL).
The generic dialect needs a sqlparser change, which will follow upstream.
- Behaviour change: time travel clauses that were silently ignored are now
planned, or rejected if the catalog does not support time travel.
- Documentation:
- the SQL reference (`select.md`) and the catalog guide
(`catalogs.md`) gain time travel sections.
- API changes documented in the 57.0.0 upgrade guide
## AI Usage Disclosure
This PR was authored in collaboration with Claude Code. I've reviewed and
understand the changes proposed here and will handle any review comments myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
