JingsongLi commented on PR #591:
URL: https://github.com/apache/paimon-rust/pull/591#issuecomment-5069223475
1. **High: DROP PARTITION is inconsistent with Java semantics**
- Rust only supports single, complete partitions: `sql_context.rs:1064`
rejects multiple specs, and `sql_context.rs:1734` uses `require_complete=true`.
- Java supports dropping multiple partitions at once and also supports
arbitrary partial specifications such as `dt=...` and `hh=...`, which are
expanded to include all matching leaf partitions.
- Impact: Commands executable in Java, such as `DROP PARTITION
(dt=‘20260715’)` and batch DROP operations, fail outright in Rust.
Recommendation: Follow Java’s approach—pre-check complete specs using
`list-by-names`; expand partial specs after a single catalog traversal.
2. **High: Catalog-managed scans lack partition pruning via the REST
endpoint**
- Rust’s `format_table_scan.rs:216` unconditionally uses
`list_partitions` to retrieve all partitions in the table before executing the
predicate locally.
- Java `FormatTableScan.java:230` extracts the leading equality prefix
and the full predicate; `CatalogFormatTablePartitionManager.java:68` pushes the
pattern/predicate and page size (1000) down to REST.
- Even when querying a single partition in a table with many partitions,
all partition metadata is downloaded and parsed, which may cause significant
latency and memory issues.
3. **Note: Boolean partition values are incompatible with Java**
- Repair preserves and registers the original values in the catalog, but
Rust `format_partition.rs:216` uses `str::parse::<bool>()`, which only accepts
lowercase `true/false`.
- Java ignores case and accepts `t/y/yes/1` and `f/n/no/0`.
- Therefore, if a catalog entry `active=TRUE` is registered via Rust
MSCK, subsequent `scan` or `SHOW` operations will report “invalid catalog
partition metadata”; Java can read it normally. It is recommended to adopt the
Java-compatible rules and perform additional cross-platform testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
