yihua commented on PR #698: URL: https://github.com/apache/hudi-rs/pull/698#issuecomment-6379819463
Thanks for the thorough review @linliu-code. All the concerns are addressed at `f86e4638`: both blocking items, the three "worth fixing before merge" items, and the non-blocking ones, with two small follow-ups noted at the end. Details below. **Blocking 1 (Lance).** Went with a fourth option: Lance stays on by default, with no feature gate. Every Lance release is still on arrow 58, so `crates/core` now also depends on Lance's arrow (renamed `arrow-array` / `arrow-schema` 58) and crosses batches through the Arrow C data interface in `file_group/base_file/lance_arrow.rs`. `--all-features` builds, the Lance tests run unconditionally, and the Python wheel and cxx bridge keep Lance. With no feature gate, items 3 and 4 no longer apply, and the `protoc` comment you flagged is accurate again. Bridge overhead and the reasoning are in the PR description. **Blocking 2 (pyo3).** Migrated: `downcast` to `cast`, `PyCapsule::new_with_value`, and `from_py_object` on the cloned pyclasses. **5 (`locations`).** Took your suggestion, with a plan test that rejects two locations. **Non-blocking.** `crates/test` no longer sets `default-features = false`. The `resolver.rs` fallthrough went away with the feature gate. Tracking issue for `ParquetObjectReader` is #843, named from the code comment. Tests were added for the `locations` handling, for routing sizeless base files away from the parquet source, and for the Lance bridge round trips. Two items I have not changed: `HudiDataSource::options()` still returns the options exactly as the caller passed them (`Debug` keeps omitting it; I can add a doc note that credentials must reach workers out of band), and `HudiScanExec` still implements `partition_statistics` rather than `statistics_from_inputs`, which I would leave for a follow-up since behavior is unchanged. CI runs on this head now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
