yihua opened a new pull request, #846:
URL: https://github.com/apache/hudi-rs/pull/846
## Description
An engine that plans a Hudi read on its driver already holds the table
properties, the query instant and the table schema, and ships each task its
file slices. Until now a task could build a `FileGroupReader` only through
`new_with_options`, which fetches `hoodie.properties` from storage, or through
a `Table`, which lists the timeline and reads the latest commit metadata; and
only the `Table` path could hand the reader the table schema, since
`set_data_schema` is crate-private. A reader built from paths alone pushes a
projection straight into the base file, so a column added after that file was
written fails the read instead of being null-filled.
This adds two public pieces to `FileGroupReader`:
- `new_with_resolved_options(base_uri, hudi_options, storage_options)`
builds a reader from table properties and read options the caller already
holds, with no storage access. The options are validated the way
`new_with_options` validates what it loads. A pinned instant goes in as
`hoodie.read.end.timestamp`; `hoodie.read.as.of.timestamp` is rejected, since
only a `Table` can resolve it against the timeline and nothing on this path
would honour it. Below table version 8 the log-block completion gate stays off
on this path, the same as for the cxx bridge; the docs say so.
- `with_data_schema(schema)` sets the table schema on such a reader, so an
evolved table reads against its current schema rather than each base file's,
the same as a reader from `Table`.
Nothing changes for existing callers. The driver-side use is the Apache Hudi
Spark integration with DataFusion Comet (apache/hudi#20170), where each task
currently opens a `Table` to get the schema; with this it ships the properties
and schema in the plan and the task touches storage only for the data files.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_new_with_resolved_options_reads_nothing_from_storage` builds a reader
for a base URI that does not exist,
`test_new_with_resolved_options_validates_the_given_properties` checks an
unsupported table version is rejected,
`test_new_with_resolved_options_rejects_as_of_timestamp` checks the as-of
option is refused with a pointer to the end timestamp, and
`test_with_data_schema_null_fills_a_column_added_after_the_base_file` reads the
added-column fixture from paths with and without the schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
