QuakeWang commented on code in PR #33: URL: https://github.com/apache/paimon-website/pull/33#discussion_r3527101337
########## community/docs/releases/paimon-rust-release-0.2.0.md: ########## @@ -0,0 +1,213 @@ +--- +title: "Paimon Rust Release 0.2.0" +type: release +version: Rust 0.2.0 +--- + +# Paimon Rust 0.2.0 Available + +May 2026 - Yuxia Luo ([email protected]) + +The Apache Paimon PMC is pleased to announce the release of Paimon Rust 0.2.0. This release includes +59 commits: 54 from 12 human contributors and 5 dependency update commits from dependabot. We would like +to express our sincere gratitude to all the developers who have participated in the contribution! + +## What's New in 0.2.0? + +Building on the foundation laid by 0.1.0, this release significantly advances feature parity with the +Java implementation. The highlights include primary key table write support, the partial-update merge +engine, Vortex columnar format, Blob type, Lumina vector index, comprehensive system tables, and an +expanded DML/DDL SQL surface through DataFusion. + +### Supported Table Types + +| Table Type | Scan | Read | Write | +|------------------------------------------|-----------|-----------|----------------------------------------| +| Append-Only Table | Supported | Supported | Supported (INSERT, COW DML) | +| Primary Key Table | Supported | Supported | **Supported (New in 0.2.0)** | +| Data Evolution Table (Append-Only) | Supported | Supported | Supported (MERGE & UPDATE) | + +- **Append-Only Table**: Now supports copy-on-write DML operations (DELETE, UPDATE, MERGE INTO) in + addition to INSERT INTO/OVERWRITE. +- **Primary Key Table**: Now supports read/write with sort-merge deduplication, fixed and dynamic bucket + assignment, and postpone bucket mode. Supports Deduplicate, PartialUpdate, and FirstRow merge engines. + Note: partial-update with `deletion-vectors.enabled=true` is not yet supported. +- **Data Evolution Table**: Continues to support MERGE & UPDATE with enhanced row-ID validation. + +## New Features + +### Primary Key Table Write + +The most significant addition in 0.2.0 is primary key table write support: + +- **Sort-Merge Deduplication** — merge-on-read primary key semantics with full sort-merge writer + implementation. +- **Fixed Bucket Assignment** — deterministic row-to-bucket mapping based on primary key hash. +- **Dynamic Bucket Assignment** — automatic bucket scaling based on data volume, allowing + tables to grow without manual resharding. +- **Postpone Bucket (bucket=-2)** — deferred bucket assignment for cross-partition primary key tables. + +### Partial-Update Merge Engine + +Introduced the partial-update merge engine, supporting field-by-field merge semantics for both +fixed-bucket and dynamic-bucket primary key tables. This enables incremental column updates +without rewriting entire rows — a critical feature for wide-table ETL pipelines. + +### Vortex Columnar Format + +Added [Vortex](https://github.com/spiraldb/vortex) as an optional columnar file format alongside +Parquet, ORC, and Avro. Vortex provides compressed columnar storage with predicate pushdown and +selection support. Enable it via the `vortex` feature flag. + +### Blob Type + +Introduced the Blob type for storing large binary objects (images, documents, model weights, etc.) +with DDL semantics. Supports blob descriptor write for append-only tables and blob file reading +in data evolution flows. + +### Lumina Vector Index + +Added vector index read infrastructure via the Lumina module, bringing vector similarity search +capabilities to Paimon Rust. This lays the groundwork for AI/ML workloads on Paimon tables. + +### Copy-on-Write DML for Append-Only Tables + +Append-only tables now support full DML operations (DELETE, UPDATE, MERGE INTO) via a copy-on-write +strategy, enabling mutable semantics on immutable storage. + +### COUNT(*) Pushdown + +Enabled exact `COUNT(*)` pushdown via partition statistics, avoiding full data scans for count-only +queries when partition-level row counts are available. + +### Avro Reader Optimization + +Replaced `serde_avro_fast` with `apache-avro` and introduced a custom Avro OCF reader for manifest +parsing with filtered decoding, improving manifest read performance and correctness. + +## DataFusion Integration Enhancements + +### System Tables + +Added comprehensive system tables for metadata inspection via standard SQL: + +| System Table | Description | +|-----------------------------|----------------------------------------------| +| `$schemas` | Table schema version history | +| `$snapshots` | Snapshot metadata and history | +| `$tags` | Tag definitions and snapshot links | +| `$manifests` | Manifest entries with partition stats | +| `$partitions` | Partition metadata and statistics | +| `$files` | Data file listing and metadata | +| `$table_indexes` | Index definitions | +| `$branches` | Branch metadata | +| `$options` | Table configuration options | +| `$referenced_files_size` | Space used by files in the latest snapshot | +| `$physical_files_size` | Actual disk usage of table data | + +Example: + +```sql +SELECT * FROM paimon.my_db.`my_table$snapshots` ORDER BY snapshot_id DESC LIMIT 5; +``` + +### DML & SQL + +Significant SQL surface expansion in this release: + +- **Hive-style INSERT OVERWRITE PARTITION** — overwrite specific partitions while preserving others. +- **TRUNCATE TABLE** — remove all data from a table. +- **DROP PARTITION** — drop specific partitions. +- **CALL Procedures** — `CALL tag(...)` for tagging and `CALL rollback(...)` for rollback operations. +- **Session-Scoped Dynamic Options** — `SET` / `RESET` for runtime configuration overrides. + +### SQLContext with Multi-Catalog Support + +Renamed `PaimonSqlHandler` to `SQLContext` with multi-catalog support, enabling users to register +and query across multiple Paimon catalogs in a single DataFusion session. + +### Branch Manager + +Added branch manager support, enabling branch-based development workflows for Paimon tables. + +### Temporary Tables + +Added catalog-level temporary table support for intermediate computation results that do not +need to be persisted. + +### Table-Valued Functions + +Introduced table-valued functions auto-registered when a catalog is registered with SQLContext: + +- `vector_search` — vector similarity search via Lumina index. +- `full_text_search` — full-text search via Tantivy index (feature-gated with `fulltext`). + +### Other Improvements + +- **Reject typed columns in PARTITIONED BY** — improved DDL validation. +- **Error logging for catalog methods** — better observability for silent failures. +- **RESTCatalog list_partitions** — support listing and paginating partitions via REST API. + +## Getting Started + +Add dependencies to your `Cargo.toml`: + +```toml +[dependencies] +paimon = "0.2.0" +paimon-datafusion = "0.2.0" +tokio = { version = "1", features = ["full"] } +``` + +### Write to a Primary Key Table with DataFusion SQL + +```rust +use std::sync::Arc; +use paimon::{CatalogOptions, FileSystemCatalog, Options}; +use paimon_datafusion::SQLContext; + +#[tokio::main] +async fn main() { + // 1. Create a catalog and register it with SQLContext + let mut options = Options::new(); + options.set(CatalogOptions::WAREHOUSE, "/path/to/warehouse"); + let catalog = FileSystemCatalog::new(options).unwrap(); + + let mut ctx = SQLContext::new(); + ctx.register_catalog("paimon", Arc::new(catalog)).await.unwrap(); + + // 2. Create a database and a primary key table + ctx.sql("CREATE SCHEMA paimon.my_db").await.unwrap(); + ctx.sql("CREATE TABLE paimon.my_db.users ( + id BIGINT, + name STRING, + age INT, + PRIMARY KEY (id) NOT ENFORCED + )").await.unwrap(); + + // 3. Insert data + ctx.sql("INSERT INTO paimon.my_db.users VALUES (1, 'Alice', 30), (2, 'Bob', 25)") + .await.unwrap() + .collect().await.unwrap(); + + // 4. Query the table + ctx.sql("SELECT * FROM paimon.my_db.users") + .await.unwrap() + .show().await.unwrap(); + + // 5. Inspect metadata via system tables + ctx.sql("SELECT * FROM paimon.my_db.`users$snapshots`") + .await.unwrap() + .show().await.unwrap(); +} +``` + +For more information, visit the [documentation](https://paimon.apache.org/docs/rust/) and the +[GitHub repository](https://github.com/apache/paimon-rust). + +## Contributors + +Thanks to all the contributors who made this release possible: + +Arpit Jain, Jiajia Li, Jingsong Lee, Jiwen liu, QuakeWang, XiaoHongbo, Zach, Review Comment: Zach are nickname for the QuakeWang account, so just keep QuakeWang. Thanks ########## community/docs/releases/paimon-rust-release-0.2.0.md: ########## @@ -0,0 +1,213 @@ +--- +title: "Paimon Rust Release 0.2.0" +type: release +version: Rust 0.2.0 +--- + +# Paimon Rust 0.2.0 Available + +May 2026 - Yuxia Luo ([email protected]) + +The Apache Paimon PMC is pleased to announce the release of Paimon Rust 0.2.0. This release includes +59 commits: 54 from 12 human contributors and 5 dependency update commits from dependabot. We would like +to express our sincere gratitude to all the developers who have participated in the contribution! + +## What's New in 0.2.0? + +Building on the foundation laid by 0.1.0, this release significantly advances feature parity with the +Java implementation. The highlights include primary key table write support, the partial-update merge +engine, Vortex columnar format, Blob type, Lumina vector index, comprehensive system tables, and an +expanded DML/DDL SQL surface through DataFusion. + +### Supported Table Types + +| Table Type | Scan | Read | Write | +|------------------------------------------|-----------|-----------|----------------------------------------| +| Append-Only Table | Supported | Supported | Supported (INSERT, COW DML) | +| Primary Key Table | Supported | Supported | **Supported (New in 0.2.0)** | +| Data Evolution Table (Append-Only) | Supported | Supported | Supported (MERGE & UPDATE) | + +- **Append-Only Table**: Now supports copy-on-write DML operations (DELETE, UPDATE, MERGE INTO) in + addition to INSERT INTO/OVERWRITE. +- **Primary Key Table**: Now supports read/write with sort-merge deduplication, fixed and dynamic bucket + assignment, and postpone bucket mode. Supports Deduplicate, PartialUpdate, and FirstRow merge engines. + Note: partial-update with `deletion-vectors.enabled=true` is not yet supported. +- **Data Evolution Table**: Continues to support MERGE & UPDATE with enhanced row-ID validation. + +## New Features + +### Primary Key Table Write + +The most significant addition in 0.2.0 is primary key table write support: + +- **Sort-Merge Deduplication** — merge-on-read primary key semantics with full sort-merge writer + implementation. +- **Fixed Bucket Assignment** — deterministic row-to-bucket mapping based on primary key hash. +- **Dynamic Bucket Assignment** — automatic bucket scaling based on data volume, allowing + tables to grow without manual resharding. +- **Postpone Bucket (bucket=-2)** — deferred bucket assignment for cross-partition primary key tables. + +### Partial-Update Merge Engine + +Introduced the partial-update merge engine, supporting field-by-field merge semantics for both +fixed-bucket and dynamic-bucket primary key tables. This enables incremental column updates +without rewriting entire rows — a critical feature for wide-table ETL pipelines. + +### Vortex Columnar Format + +Added [Vortex](https://github.com/spiraldb/vortex) as an optional columnar file format alongside +Parquet, ORC, and Avro. Vortex provides compressed columnar storage with predicate pushdown and +selection support. Enable it via the `vortex` feature flag. + +### Blob Type + +Introduced the Blob type for storing large binary objects (images, documents, model weights, etc.) +with DDL semantics. Supports blob descriptor write for append-only tables and blob file reading +in data evolution flows. + +### Lumina Vector Index + +Added vector index read infrastructure via the Lumina module, bringing vector similarity search +capabilities to Paimon Rust. This lays the groundwork for AI/ML workloads on Paimon tables. + +### Copy-on-Write DML for Append-Only Tables + +Append-only tables now support full DML operations (DELETE, UPDATE, MERGE INTO) via a copy-on-write +strategy, enabling mutable semantics on immutable storage. + +### COUNT(*) Pushdown + +Enabled exact `COUNT(*)` pushdown via partition statistics, avoiding full data scans for count-only +queries when partition-level row counts are available. + +### Avro Reader Optimization + +Replaced `serde_avro_fast` with `apache-avro` and introduced a custom Avro OCF reader for manifest +parsing with filtered decoding, improving manifest read performance and correctness. + +## DataFusion Integration Enhancements + +### System Tables + +Added comprehensive system tables for metadata inspection via standard SQL: + +| System Table | Description | +|-----------------------------|----------------------------------------------| +| `$schemas` | Table schema version history | +| `$snapshots` | Snapshot metadata and history | +| `$tags` | Tag definitions and snapshot links | +| `$manifests` | Manifest entries with partition stats | +| `$partitions` | Partition metadata and statistics | +| `$files` | Data file listing and metadata | +| `$table_indexes` | Index definitions | +| `$branches` | Branch metadata | +| `$options` | Table configuration options | +| `$referenced_files_size` | Space used by files in the latest snapshot | +| `$physical_files_size` | Actual disk usage of table data | + +Example: + +```sql +SELECT * FROM paimon.my_db.`my_table$snapshots` ORDER BY snapshot_id DESC LIMIT 5; +``` + +### DML & SQL + +Significant SQL surface expansion in this release: + +- **Hive-style INSERT OVERWRITE PARTITION** — overwrite specific partitions while preserving others. +- **TRUNCATE TABLE** — remove all data from a table. +- **DROP PARTITION** — drop specific partitions. +- **CALL Procedures** — `CALL tag(...)` for tagging and `CALL rollback(...)` for rollback operations. Review Comment: These procedure names do not match the 0.2.0 API. The supported forms are `CALL sys.create_tag(...)`, `CALL sys.delete_tag(...)`, `CALL sys.rollback_to(...)`, `CALL sys.rollback_to_timestamp(...)`, etc. ########## community/docs/releases/paimon-rust-release-0.2.0.md: ########## @@ -0,0 +1,213 @@ +--- +title: "Paimon Rust Release 0.2.0" +type: release +version: Rust 0.2.0 +--- + +# Paimon Rust 0.2.0 Available + +May 2026 - Yuxia Luo ([email protected]) + +The Apache Paimon PMC is pleased to announce the release of Paimon Rust 0.2.0. This release includes +59 commits: 54 from 12 human contributors and 5 dependency update commits from dependabot. We would like +to express our sincere gratitude to all the developers who have participated in the contribution! + +## What's New in 0.2.0? + +Building on the foundation laid by 0.1.0, this release significantly advances feature parity with the +Java implementation. The highlights include primary key table write support, the partial-update merge +engine, Vortex columnar format, Blob type, Lumina vector index, comprehensive system tables, and an +expanded DML/DDL SQL surface through DataFusion. + +### Supported Table Types + +| Table Type | Scan | Read | Write | +|------------------------------------------|-----------|-----------|----------------------------------------| +| Append-Only Table | Supported | Supported | Supported (INSERT, COW DML) | +| Primary Key Table | Supported | Supported | **Supported (New in 0.2.0)** | +| Data Evolution Table (Append-Only) | Supported | Supported | Supported (MERGE & UPDATE) | + +- **Append-Only Table**: Now supports copy-on-write DML operations (DELETE, UPDATE, MERGE INTO) in + addition to INSERT INTO/OVERWRITE. +- **Primary Key Table**: Now supports read/write with sort-merge deduplication, fixed and dynamic bucket + assignment, and postpone bucket mode. Supports Deduplicate, PartialUpdate, and FirstRow merge engines. + Note: partial-update with `deletion-vectors.enabled=true` is not yet supported. +- **Data Evolution Table**: Continues to support MERGE & UPDATE with enhanced row-ID validation. + +## New Features + +### Primary Key Table Write + +The most significant addition in 0.2.0 is primary key table write support: + +- **Sort-Merge Deduplication** — merge-on-read primary key semantics with full sort-merge writer + implementation. +- **Fixed Bucket Assignment** — deterministic row-to-bucket mapping based on primary key hash. +- **Dynamic Bucket Assignment** — automatic bucket scaling based on data volume, allowing + tables to grow without manual resharding. +- **Postpone Bucket (bucket=-2)** — deferred bucket assignment for cross-partition primary key tables. + +### Partial-Update Merge Engine + +Introduced the partial-update merge engine, supporting field-by-field merge semantics for both +fixed-bucket and dynamic-bucket primary key tables. This enables incremental column updates +without rewriting entire rows — a critical feature for wide-table ETL pipelines. + +### Vortex Columnar Format + +Added [Vortex](https://github.com/spiraldb/vortex) as an optional columnar file format alongside +Parquet, ORC, and Avro. Vortex provides compressed columnar storage with predicate pushdown and +selection support. Enable it via the `vortex` feature flag. + +### Blob Type + +Introduced the Blob type for storing large binary objects (images, documents, model weights, etc.) +with DDL semantics. Supports blob descriptor write for append-only tables and blob file reading +in data evolution flows. + +### Lumina Vector Index + +Added vector index read infrastructure via the Lumina module, bringing vector similarity search +capabilities to Paimon Rust. This lays the groundwork for AI/ML workloads on Paimon tables. + +### Copy-on-Write DML for Append-Only Tables + +Append-only tables now support full DML operations (DELETE, UPDATE, MERGE INTO) via a copy-on-write +strategy, enabling mutable semantics on immutable storage. + +### COUNT(*) Pushdown + +Enabled exact `COUNT(*)` pushdown via partition statistics, avoiding full data scans for count-only +queries when partition-level row counts are available. + +### Avro Reader Optimization + +Replaced `serde_avro_fast` with `apache-avro` and introduced a custom Avro OCF reader for manifest +parsing with filtered decoding, improving manifest read performance and correctness. + +## DataFusion Integration Enhancements + +### System Tables + +Added comprehensive system tables for metadata inspection via standard SQL: + +| System Table | Description | +|-----------------------------|----------------------------------------------| +| `$schemas` | Table schema version history | +| `$snapshots` | Snapshot metadata and history | +| `$tags` | Tag definitions and snapshot links | +| `$manifests` | Manifest entries with partition stats | +| `$partitions` | Partition metadata and statistics | +| `$files` | Data file listing and metadata | +| `$table_indexes` | Index definitions | +| `$branches` | Branch metadata | +| `$options` | Table configuration options | +| `$referenced_files_size` | Space used by files in the latest snapshot | Review Comment: This description is inaccurate. `$referenced_files_size` summarizes referenced files across retained snapshot scopes, including main branch, tags, and other branches, not only the latest snapshot. ########## community/docs/releases/paimon-rust-release-0.2.0.md: ########## @@ -0,0 +1,213 @@ +--- +title: "Paimon Rust Release 0.2.0" +type: release +version: Rust 0.2.0 Review Comment: Please add an explicit `weight`. The website sorts releases by `weight` and uses the first item as latest; without it, Rust 0.2.0 may not appear before Rust 0.1.0 in `/releases`. ########## community/docs/releases/paimon-rust-release-0.2.0.md: ########## @@ -0,0 +1,213 @@ +--- +title: "Paimon Rust Release 0.2.0" +type: release +version: Rust 0.2.0 +--- + +# Paimon Rust 0.2.0 Available + +May 2026 - Yuxia Luo ([email protected]) + +The Apache Paimon PMC is pleased to announce the release of Paimon Rust 0.2.0. This release includes +59 commits: 54 from 12 human contributors and 5 dependency update commits from dependabot. We would like +to express our sincere gratitude to all the developers who have participated in the contribution! + +## What's New in 0.2.0? + +Building on the foundation laid by 0.1.0, this release significantly advances feature parity with the +Java implementation. The highlights include primary key table write support, the partial-update merge +engine, Vortex columnar format, Blob type, Lumina vector index, comprehensive system tables, and an +expanded DML/DDL SQL surface through DataFusion. + +### Supported Table Types + +| Table Type | Scan | Read | Write | +|------------------------------------------|-----------|-----------|----------------------------------------| +| Append-Only Table | Supported | Supported | Supported (INSERT, COW DML) | +| Primary Key Table | Supported | Supported | **Supported (New in 0.2.0)** | +| Data Evolution Table (Append-Only) | Supported | Supported | Supported (MERGE & UPDATE) | + +- **Append-Only Table**: Now supports copy-on-write DML operations (DELETE, UPDATE, MERGE INTO) in + addition to INSERT INTO/OVERWRITE. +- **Primary Key Table**: Now supports read/write with sort-merge deduplication, fixed and dynamic bucket + assignment, and postpone bucket mode. Supports Deduplicate, PartialUpdate, and FirstRow merge engines. + Note: partial-update with `deletion-vectors.enabled=true` is not yet supported. +- **Data Evolution Table**: Continues to support MERGE & UPDATE with enhanced row-ID validation. + +## New Features + +### Primary Key Table Write + +The most significant addition in 0.2.0 is primary key table write support: + +- **Sort-Merge Deduplication** — merge-on-read primary key semantics with full sort-merge writer + implementation. +- **Fixed Bucket Assignment** — deterministic row-to-bucket mapping based on primary key hash. +- **Dynamic Bucket Assignment** — automatic bucket scaling based on data volume, allowing + tables to grow without manual resharding. +- **Postpone Bucket (bucket=-2)** — deferred bucket assignment for cross-partition primary key tables. + +### Partial-Update Merge Engine + +Introduced the partial-update merge engine, supporting field-by-field merge semantics for both +fixed-bucket and dynamic-bucket primary key tables. This enables incremental column updates +without rewriting entire rows — a critical feature for wide-table ETL pipelines. + +### Vortex Columnar Format + +Added [Vortex](https://github.com/spiraldb/vortex) as an optional columnar file format alongside +Parquet, ORC, and Avro. Vortex provides compressed columnar storage with predicate pushdown and +selection support. Enable it via the `vortex` feature flag. + +### Blob Type + +Introduced the Blob type for storing large binary objects (images, documents, model weights, etc.) +with DDL semantics. Supports blob descriptor write for append-only tables and blob file reading +in data evolution flows. + +### Lumina Vector Index + +Added vector index read infrastructure via the Lumina module, bringing vector similarity search +capabilities to Paimon Rust. This lays the groundwork for AI/ML workloads on Paimon tables. + +### Copy-on-Write DML for Append-Only Tables + +Append-only tables now support full DML operations (DELETE, UPDATE, MERGE INTO) via a copy-on-write +strategy, enabling mutable semantics on immutable storage. + +### COUNT(*) Pushdown + +Enabled exact `COUNT(*)` pushdown via partition statistics, avoiding full data scans for count-only +queries when partition-level row counts are available. + +### Avro Reader Optimization + +Replaced `serde_avro_fast` with `apache-avro` and introduced a custom Avro OCF reader for manifest +parsing with filtered decoding, improving manifest read performance and correctness. + +## DataFusion Integration Enhancements + +### System Tables + +Added comprehensive system tables for metadata inspection via standard SQL: + +| System Table | Description | +|-----------------------------|----------------------------------------------| +| `$schemas` | Table schema version history | +| `$snapshots` | Snapshot metadata and history | +| `$tags` | Tag definitions and snapshot links | +| `$manifests` | Manifest entries with partition stats | +| `$partitions` | Partition metadata and statistics | +| `$files` | Data file listing and metadata | +| `$table_indexes` | Index definitions | +| `$branches` | Branch metadata | +| `$options` | Table configuration options | +| `$referenced_files_size` | Space used by files in the latest snapshot | +| `$physical_files_size` | Actual disk usage of table data | + +Example: + +```sql +SELECT * FROM paimon.my_db.`my_table$snapshots` ORDER BY snapshot_id DESC LIMIT 5; +``` + +### DML & SQL + +Significant SQL surface expansion in this release: + +- **Hive-style INSERT OVERWRITE PARTITION** — overwrite specific partitions while preserving others. +- **TRUNCATE TABLE** — remove all data from a table. +- **DROP PARTITION** — drop specific partitions. +- **CALL Procedures** — `CALL tag(...)` for tagging and `CALL rollback(...)` for rollback operations. +- **Session-Scoped Dynamic Options** — `SET` / `RESET` for runtime configuration overrides. + +### SQLContext with Multi-Catalog Support + +Renamed `PaimonSqlHandler` to `SQLContext` with multi-catalog support, enabling users to register +and query across multiple Paimon catalogs in a single DataFusion session. + +### Branch Manager + +Added branch manager support, enabling branch-based development workflows for Paimon tables. + +### Temporary Tables + +Added catalog-level temporary table support for intermediate computation results that do not +need to be persisted. + +### Table-Valued Functions + +Introduced table-valued functions auto-registered when a catalog is registered with SQLContext: Review Comment: “Introduced” is a bit misleading here: `full_text_search` already existed in 0.1.0. 0.2.0 adds `vector_search` and SQLContext auto-registration for built-in TVFs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
