suxiaogang223 opened a new issue, #1382: URL: https://github.com/apache/paimon-rust/issues/1382
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. Searched issues and PRs for incremental C, incremental FFI, audit_log, and audit C. #506 and #1042 concern the DataFusion table function, not an ordinary C FFI entry point. No directly matching C API issue was found as of November 12, 2026. ### Motivation Apache Doris plans Paimon scans in its Java frontend and assigns logical splits to backend workers. The backend already uses the C binding for native Paimon reads. To support `table$audit_log` over this path, each worker needs to execute its assigned splits with audit semantics and stream Arrow batches, without replanning the whole table. The Rust core supports current-state audit reads through `AuditLogRead`, and snapshot-range planning with Delta, Changelog, Auto, and Diff modes. It exposes `to_incremental_arrow` and `to_audit_log_arrow`. Current-state audit reads and incremental audit reads are distinct APIs; exposing only the latter would leave ordinary `table$audit_log` queries uncovered. These outputs have different contracts. Ordinary Diff returns after images and omits deleted keys. Audit-log Diff can return `+I`, `-U`, `+U`, and `-D` through its rowkind column. Consumers maintaining downstream state need that distinction. At main `f91fdb8ab034887547a5ee2f26c1979e30250558`, the C ReadBuilder does not expose equivalent incremental-plan and audit-log read APIs. Its tests explicitly reject `incremental-between` passed as a scan option. Allowing that string alone would not expose the missing plan and output contracts. ### Solution Provide additive C entry points for current-state audit reading and for planning a bounded snapshot interval and reading its ordinary or audit-log output, reusing the existing core implementation. Define ownership of read/plan handles and the produced Arrow streams. Distributed consumers also need to execute only externally assigned work units, without each worker replanning the complete table or resolving latest independently. An audit reader over imported DataSplits would cover current-state execution, provided those splits satisfy the audit planner's grouping requirements. DataFusion already demonstrates this execution pattern with `AuditLogRead::new(read)?.to_arrow(splits)`. Incremental import additionally needs a resolved mode and validated work-unit representation. Java 2.1.0 `IncrementalSplit` contains before/after files and deletion files and is not a DataSplit; the existing DataSplit byte import cannot be assumed to accept it. Define a compatible importer or an equivalent public representation mapping to `IncrementalPlan::try_new`, preserving each DiffPair as a complete work unit. Current-state binding and incremental transport can be separate contributions under the same API discussion. The public contract should describe `(start_exclusive, end_inclusive]`, equal endpoints, required retained snapshots, mode-specific output, rowkind/schema export, and error behavior for unsupported tables or missing history. Preserve the existing core restrictions rather than silently broadening support. Coverage should include inserts, updates, deletes, repeated keys, deletion vectors, first-row split grouping where supported, relevant schema evolution, malformed intervals, missing endpoints, empty results, system-field-only projection, independent execution of assigned splits, and resource release after errors or cancellation. Ordinary reads followed by a synthetic `+I` column are not a substitute for AuditLogRead's retained-retract semantics. The Diff contract needs explicit cross-version documentation: Java 2.1.0 `IncrementalDiffSplitRead` emits changed after images and retained deletes, whereas the Rust audit Diff API emits before/after update pairs. This is an interoperability question to resolve with shared fixtures, not a claim that the Rust implementation is necessarily incorrect. ### Anything else? References: [core incremental-reading documentation](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/docs/src/incremental-reading.md) and [C binding](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/bindings/c/src/table.rs#L2641). Further references: [DataFusion audit execution](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/integrations/datafusion/src/physical_plan/audit_log.rs), [Java IncrementalSplit](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/source/IncrementalSplit.java), and [Java Diff reader](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/source/splitread/IncrementalDiffSplitRead.java). This issue does not add binlog pairing/array output, MV scheduling, progress publication, or a new changelog producer. Those are separate reader, consumer, or writer capabilities. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
