This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hudi-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 2dab9e99 docs: fix stale API claims in README, reader spec, and agent
guides (#716)
2dab9e99 is described below
commit 2dab9e99f1baa004192239b15153926d3bc6b31e
Author: Y Ethan Guo <[email protected]>
AuthorDate: Thu Sep 3 19:20:49 2026 -0700
docs: fix stale API claims in README, reader spec, and agent guides (#716)
---
AGENTS.md | 7 +++++--
README.md | 8 ++++----
docs/reader-spec.md | 2 +-
python/AGENTS.md | 4 ++--
python/hudi/_internal.pyi | 5 +++--
5 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 9b26c5ce..9cd1d0a3 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -9,13 +9,16 @@ Arrow-native, `object_store` for all I/O, timeline-based MVCC.
```
crates/
- core/ hudi-core — config, expr, file_group, merge, metadata, schema,
storage, table, timeline, hfile
+ core/ hudi-core — avro_to_arrow, config, expr, file_group, hfile,
keygen, merge, metadata,
+ record, schema, statistics, storage, table, timeline, util
datafusion/ hudi-datafusion — DataFusion TableProvider (feature:
datafusion)
hudi/ public umbrella crate; re-exports core + (optional) datafusion
+ jvm-ffi/ hudi-jvm-ffi — plain C ABI for JVM callers via the Arrow C
Data Interface
test/ shared test fixtures
python/ PyO3 bindings (module hudi._internal); tests in python/tests
cpp/ cxx bindings; bridge in cpp/src/lib.rs
benchmark/tpch/ TPC-H benchmark harness
+benchmark/filegroup/ file-group read benchmark (fg-bench / fg-gen)
```
## Commands
@@ -126,7 +129,7 @@ single translation point exists to prevent.
Storage backends route by URI scheme (`file://`, `s3://`, `az://`, `gs://`)
through
[`object_store`](https://docs.rs/object_store) — don't hand-roll per-scheme
paths. Table options
-are typed: `HudiTableConfig`, `HudiReadConfig` (also Python enums). Prefer
enum members over raw
+are typed: `HudiTableConfig`, `HudiReadConfig`, `HudiPlanConfig` (also Python
enums). Prefer enum members over raw
string keys; bulk variants (`with_hudi_options` / `with_options`) currently
expect string keys.
## Code review
diff --git a/README.md b/README.md
index e5a3ca6c..4be8911f 100644
--- a/README.md
+++ b/README.md
@@ -109,7 +109,7 @@ async fn main() -> Result<()> {
let columns = vec!["rider", "city", "ts", "fare"];
for col_name in columns {
let idx = batch.schema().index_of(col_name).unwrap();
- println!("{}: {}", col_name, batch.column(idx));
+ println!("{col_name}: {:?}", batch.column(idx));
}
Ok(())
}
@@ -343,8 +343,8 @@ All read APIs accept a `ReadOptions` (Rust) /
`HudiReadOptions` (Python) value.
| Stage | API
| Description
|
|-----------------|------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
| Query planning | `get_file_slices(options)`
| Get the file slices the read targets, dispatched on
`options.query_type`. To bucket for parallel reads, call
`hudi::util::collection::split_into_chunks` on the result. |
-| | `compute_table_stats(options)`
| Estimated `(num_rows, byte_size)` for scan planning. Snapshot (default)
uses the metadata table; incremental aggregates from changed file slices.
Returns `None` when stats cannot be computed. |
-| Query execution | `create_file_group_reader_with_options(read_options,
extra_storage_overrides)` | Create a file group reader with the table's
configs. Both args are optional. Timestamps are resolved automatically (e.g.
`AsOfTimestamp` → `EndTimestamp`), so callers can pass the same options used
for `get_file_slices`. |
+| | `compute_table_stats(options)`
| Estimated `(num_rows, byte_size)` for scan planning, derived from the
metadata table. Snapshot only. Returns `None` for incremental queries, and
whenever the estimate cannot be computed (no metadata table, non-Parquet base
files, footer sampling failure). |
+| Query execution | `create_file_group_reader_with_options(read_options,
extra_storage_overrides)` | Create a file group reader with the table's
configs. In Python both args are optional; in Rust `read_options` is an
`Option` and `extra_storage_overrides` takes a (possibly empty) iterator.
Timestamps are resolved automatically (e.g. `AsOfTimestamp` → `EndTimestamp`),
so callers can pass the same options used for `get_file_slices`. |
| | `read(options)` / `read_stream(options)`
| Record-read APIs. Dispatch on `options.query_type`. `read_stream`
errors on `Incremental` for now. Per-slice streaming lives on
`FileGroupReader`. |
### Read configs
@@ -451,7 +451,7 @@ ctx.sql("SELECT max(fare), city from trips group by city
order by 1 desc").show(
Hudi is also integrated with
-- [Daft](https://www.getdaft.io/projects/docs/en/stable/integrations/hudi/)
+- [Daft](https://docs.daft.ai/en/stable/connectors/hudi/)
-
[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi)
### Work with cloud storage
diff --git a/docs/reader-spec.md b/docs/reader-spec.md
index 54405f12..1eae8581 100644
--- a/docs/reader-spec.md
+++ b/docs/reader-spec.md
@@ -195,7 +195,7 @@ Available pairs: `with_hudi_option` / `with_hudi_options`,
`with_storage_option`
| `FileSlice::base_file_relative_path()` / `log_file_relative_path(&LogFile)`
| Paths relative to the table base URI.
|
| `FileSlice::total_size_bytes()`
| Sum of base + log file on-disk sizes.
Missing metadata contributes 0. |
-`FileSlice` uses the base file's `num_records` as the record count for the
slice. Log file records are not counted separately because they merge into the
base file record batch during reads — the base file count represents the
materialized output. This applies to both snapshot and incremental stats.
`total_size_bytes()` includes log file on-disk sizes since they contribute to
I/O cost. File slices without a base file (log-only groups) are not yet
supported.
+`FileSlice` uses the base file's `num_records` as the record count for the
slice. Log file records are not counted separately because they merge into the
base file record batch during reads — the base file count represents the
materialized output. This applies to both snapshot and incremental stats.
`total_size_bytes()` includes log file on-disk sizes since they contribute to
I/O cost. A file slice without a base file (log-only group) is readable under
file group reader version 2 (the de [...]
## 5. Python API
diff --git a/python/AGENTS.md b/python/AGENTS.md
index a68cabff..e50298b6 100644
--- a/python/AGENTS.md
+++ b/python/AGENTS.md
@@ -53,11 +53,11 @@ Tests in `python/tests/`.
```python
import pytest
-from hudi import HudiTableBuilder
+from hudi import HudiReadOptions, HudiTableBuilder
def test_read_snapshot_with_filters():
table = HudiTableBuilder.from_base_uri("/tmp/test").build()
- batches = table.read_snapshot(filters=[("city", "=", "test")])
+ batches = table.read(HudiReadOptions(filters=[("city", "=", "test")]))
assert len(batches) > 0
def test_invalid_path_raises():
diff --git a/python/hudi/_internal.pyi b/python/hudi/_internal.pyi
index 2184a305..2ec130f8 100644
--- a/python/hudi/_internal.pyi
+++ b/python/hudi/_internal.pyi
@@ -146,8 +146,9 @@ class HudiFileGroupReader:
Reads a file slice as a stream of record batches.
For COW tables or read-optimized mode, this yields batches as they are
read
- without loading all data into memory. For MOR tables with log files,
this
- currently falls back to a single merged batch.
+ without loading all data into memory. For MOR slices with log files,
file
+ group reader version 2 (the default) yields merged chunks
incrementally;
+ version 1 falls back to a single merged batch.
"""
...
