QuakeWang commented on code in PR #538: URL: https://github.com/apache/paimon-rust/pull/538#discussion_r3608020714
########## docs/src/python-binding.md: ########## @@ -0,0 +1,455 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Python Integration + +The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html). + +## Prerequisites + +- Python 3.10 or later +- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64) Review Comment: The supported-platform list does not match the published artifacts: the release workflow builds a Windows wheel, and a win_amd64 wheel is available on PyPI. ########## docs/src/python-binding.md: ########## @@ -0,0 +1,455 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Python Integration + +The Python integration is a binding built on top of Apache Paimon Rust, allowing you to access Paimon tables from Python programs. It uses [PyArrow](https://arrow.apache.org/docs/python/) for zero-copy data transfer via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html). + +## Prerequisites + +- Python 3.10 or later +- Supported platforms: Linux (amd64, arm64), macOS (amd64, arm64) + +## Installation + +```bash +pip install pypaimon-rust pyarrow +``` + +The pre-built native library is embedded in the package and automatically loaded at runtime — no manual build step is needed. [PyArrow](https://arrow.apache.org/docs/python/) is a required peer dependency and must be installed separately. + +## Creating a Catalog + +Use `PaimonCatalog` with a dictionary of options to create a catalog. The catalog type is determined by the `metastore` option (default: `filesystem`). + +```python +from pypaimon_rust.datafusion import PaimonCatalog + +# Local filesystem +catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"}) + +# List databases and tables +print(catalog.list_databases()) +print(catalog.list_tables("default")) + +# Get a table handle +table = catalog.get_table("default.my_table") +``` + +### Alibaba Cloud OSS + +```python +catalog = PaimonCatalog({ + "warehouse": "oss://bucket/warehouse", + "fs.oss.accessKeyId": "your-access-key-id", + "fs.oss.accessKeySecret": "your-access-key-secret", + "fs.oss.endpoint": "oss-cn-hangzhou.aliyuncs.com", +}) +``` + +### REST Catalog + +```python +catalog = PaimonCatalog({ + "metastore": "rest", + "uri": "http://localhost:8080";, + "warehouse": "my_warehouse", +}) +``` + +## SQL Context + +`SQLContext` supports registering multiple Paimon catalogs and executing SQL queries with DataFusion. + +```python +from pypaimon_rust.datafusion import SQLContext + +ctx = SQLContext() +ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"}) + +# DDL and DML +ctx.sql("CREATE SCHEMA paimon.my_db") +ctx.sql("CREATE TABLE paimon.my_db.t (id INT, name STRING)") +ctx.sql("INSERT INTO paimon.my_db.t VALUES (1, 'alice'), (2, 'bob')") + +# Query returns a list of PyArrow RecordBatches +batches = ctx.sql("SELECT * FROM paimon.my_db.t") +for batch in batches: + print(batch) +``` + +## Reading a Table + +Paimon Python uses a **scan-then-read** pattern: first scan the table to produce splits, then read data from those splits as PyArrow RecordBatches. + +```python +import pyarrow as pa +from pypaimon_rust.datafusion import PaimonCatalog + +catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"}) +table = catalog.get_table("default.my_table") + +# Create a read builder +rb = table.new_read_builder() + +# Step 1: Scan — produces a Plan containing Splits +scan = rb.new_scan() +plan = scan.plan() +splits = plan.splits() + +# Step 2: Read — consumes splits and returns PyArrow RecordBatches +read = rb.new_read() +batches = read.read(splits) + +for batch in batches: + print(batch) +``` + +Alternatively, read via SQL using `SQLContext`: + +```python +from pypaimon_rust.datafusion import SQLContext + +ctx = SQLContext() +ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"}) + +batches = ctx.sql("SELECT id, name FROM paimon.default.my_table") +for batch in batches: + print(batch) +``` + +## Writing to a Table + +Paimon Python uses a **write-then-commit** pattern: write PyArrow RecordBatches to a writer, prepare commit messages, then commit. + +```python +import pyarrow as pa +from pypaimon_rust.datafusion import PaimonCatalog + +catalog = PaimonCatalog({"warehouse": "/path/to/warehouse"}) +table = catalog.get_table("default.my_table") + +# Build a batch matching the table schema +batch = pa.record_batch( + [pa.array([1, 2, 3], pa.int32()), pa.array(["a", "b", "c"], pa.string())], + names=["id", "name"], +) + +# Create a write builder (shared commit_user for writer and committer) +wb = table.new_write_builder() + +# Write batches +write = wb.new_write() +write.write_arrow(batch) + +# Prepare commit messages +messages = write.prepare_commit() + +# Commit +wb.new_commit().commit(messages) +``` + +Alternatively, write via SQL using `SQLContext`: + +```python +from pypaimon_rust.datafusion import SQLContext + +ctx = SQLContext() +ctx.register_catalog("paimon", {"warehouse": "/path/to/warehouse"}) + +ctx.sql("INSERT INTO paimon.default.my_table VALUES (1, 'alice'), (2, 'bob')") +``` + +!!! warning "Schema Validation" + The input batch schema is strictly validated against the table schema: field count, order, names, and types must match exactly. A `ValueError` is raised on mismatch. + +!!! note "Write Builder Consistency" + The writer and committer must come from the same `WriteBuilder` — they share a `commit_user` for duplicate-commit detection. Passing messages from one builder's writer to another builder's committer will raise a `ValueError`. + +## Column Projection + +Use `with_projection` to select specific columns. Only the requested columns are read, reducing I/O. + +```python +rb = table.new_read_builder() +rb.with_projection(["id", "name"]) + +# Continue with scan-then-read as above... +``` + +## Limit + +Use `with_limit` to set a hint for the number of rows returned. A limit of `0` returns zero rows. + +```python +rb = table.new_read_builder() +rb.with_limit(100) +``` + +!!! warning + `with_limit` is a scan-planning hint, not an exact row cap. When all rows fall within a single split, the entire split is returned regardless of the limit value. Callers should apply application-level limiting if an exact upper bound is required. + +## Case Sensitivity + +Use `with_case_sensitive` to control whether column-name matching in projections and predicates is case-sensitive. Defaults to `True` (exact match). Set to `False` for case-insensitive matching (ASCII case-folding). + +```python +rb = table.new_read_builder() +rb.with_case_sensitive(False) +``` + +!!! note + `with_case_sensitive` must be called **before** `with_filter` to affect predicate construction. The predicate is built using the case-sensitivity setting at the time `with_filter` is invoked; changing it afterward has no effect on an already-constructed predicate. + +## Filter Push-Down + +Filter push-down prunes data at two levels: + +1. **Scan planning** — skips partitions, buckets, and data files based on file-level statistics (min/max). +2. **Read-side** — applies row-level filtering via Parquet native row filters for leaf predicates. + +!!! warning + Filter push-down is a **best-effort** optimization. The returned results may still contain rows that do not satisfy the filter condition. Callers should always apply residual filtering on the returned records to ensure correctness. Review Comment: The filter semantics described here do not match the current implementation. Scan pruning can be conservative, but the read path applies exact residual filtering and does not return rows that fail the predicate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
