yihua opened a new issue, #858:
URL: https://github.com/apache/hudi-rs/issues/858
### Feature Description
#666 added a native write path to `hudi-core`: `Table::create`, `append`,
`upsert`, `update`, `delete`, `delete_keys`, `overwrite`, and
`dynamic_partition_overwrite`, with the metadata table and record-level index
maintained on every commit. The Python package still exposes only the read
APIs, so `pip install hudi` cannot create or mutate a table.
Expose the write path through the PyO3 binding:
```python
import pyarrow as pa
from hudi import HudiTable
table = HudiTable.create(
"/tmp/trips",
record_key="id",
partition_by="city",
ordering_field="ts",
table_type="cow", # or "mor"
)
table.append(pa.table({"id": ["t1", "t2"], "city": ["sf", "sf"], "ts": [1,
1], "fare": [30.0, 45.0]}))
result = table.upsert(pa.table({"id": ["t2", "t3"], "city": ["sf", "nyc"],
"ts": [2, 2], "fare": [50.0, 15.0]}))
print(result.instant, result.num_updates, result.num_inserts)
table.update("city = 'nyc'", {"fare": 99.0})
table.delete("id IN ('t1')")
table.delete_keys(["t2"])
table.overwrite(...)
table.dynamic_partition_overwrite(...)
```
Proposed surface:
- `HudiTable.create(base_uri, *, record_key, partition_by=None,
ordering_field=None, table_type="cow", table_name=None, record_index=None,
metadata=None, options=None, storage_options=None, ...)`, a static method
wrapping `TableCreateBuilder`. Only arguments the caller passes are forwarded,
so the Rust defaults (table version 9, metadata table and record index on) stay
the defaults.
- Write verbs on `HudiTable`: `append(data)`, `upsert(data,
update_columns=None)`, `update(filter, set)`, `delete(filter)`,
`delete_keys(keys)`, `overwrite(data)`, `dynamic_partition_overwrite(data)`.
- `HudiWriteResult` with `instant`, `num_writes`, `num_updates`,
`num_inserts`, `num_deletes`.
- `data` accepts `pyarrow.Table`, `pyarrow.RecordBatch`,
`pyarrow.RecordBatchReader`, a list of record batches, and any object
implementing `__arrow_c_stream__` (Polars, pandas 2.2+), converted once in Rust
through `arrow::pyarrow`. `Utf8View` and `LargeUtf8` columns (what Polars and
pandas export) are cast to `Utf8` on the way in, since the writer requires
`Utf8` keys.
- `update(filter, set)` takes a `dict` of column to scalar, or a one-row
batch.
- `delete_keys` takes record keys as `str` or `(record_key, partition_path)`
tuples.
- `hudi.write_hudi(base_uri, data, mode="append" | "upsert" | "overwrite",
record_key=..., ...)` as a convenience that creates the table on first write.
- Every verb releases the GIL and maps `CoreError` to a Python exception; no
panic reaches Python.
Acceptance:
- `.pyi` stubs and docstrings; `make check` green (ruff, mypy --strict,
clippy)
- `python/tests/test_table_write.py` covering each verb, each input type,
COW and MOR, and the error paths
- README "Writing Tables" shows the Python call next to each Rust call
### Additional context
The C++ binding is tracked separately. The DataFusion provider stays
read-only for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
