suxiaogang223 opened a new issue, #1382:
URL: https://github.com/apache/paimon-rust/issues/1382

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-rust/issues) and found nothing 
similar.
   
   Searched issues and PRs for incremental C, incremental FFI, audit_log, and 
audit C. #506 and #1042 concern the DataFusion table function, not an ordinary 
C FFI entry point. No directly matching C API issue was found as of November 
12, 2026.
   
   ### Motivation
   
   Apache Doris plans Paimon scans in its Java frontend and assigns logical 
splits to backend workers. The backend already uses the C binding for native 
Paimon reads. To support `table$audit_log` over this path, each worker needs to 
execute its assigned splits with audit semantics and stream Arrow batches, 
without replanning the whole table.
   
   The Rust core supports current-state audit reads through `AuditLogRead`, and 
snapshot-range planning with Delta, Changelog, Auto, and Diff modes. It exposes 
`to_incremental_arrow` and `to_audit_log_arrow`. Current-state audit reads and 
incremental audit reads are distinct APIs; exposing only the latter would leave 
ordinary `table$audit_log` queries uncovered.
   
   These outputs have different contracts. Ordinary Diff returns after images 
and omits deleted keys. Audit-log Diff can return `+I`, `-U`, `+U`, and `-D` 
through its rowkind column. Consumers maintaining downstream state need that 
distinction.
   
   At main `f91fdb8ab034887547a5ee2f26c1979e30250558`, the C ReadBuilder does 
not expose equivalent incremental-plan and audit-log read APIs. Its tests 
explicitly reject `incremental-between` passed as a scan option. Allowing that 
string alone would not expose the missing plan and output contracts.
   
   ### Solution
   
   Provide additive C entry points for current-state audit reading and for 
planning a bounded snapshot interval and reading its ordinary or audit-log 
output, reusing the existing core implementation. Define ownership of read/plan 
handles and the produced Arrow streams.
   
   Distributed consumers also need to execute only externally assigned work 
units, without each worker replanning the complete table or resolving latest 
independently. An audit reader over imported DataSplits would cover 
current-state execution, provided those splits satisfy the audit planner's 
grouping requirements. DataFusion already demonstrates this execution pattern 
with `AuditLogRead::new(read)?.to_arrow(splits)`.
   
   Incremental import additionally needs a resolved mode and validated 
work-unit representation. Java 2.1.0 `IncrementalSplit` contains before/after 
files and deletion files and is not a DataSplit; the existing DataSplit byte 
import cannot be assumed to accept it. Define a compatible importer or an 
equivalent public representation mapping to `IncrementalPlan::try_new`, 
preserving each DiffPair as a complete work unit. Current-state binding and 
incremental transport can be separate contributions under the same API 
discussion.
   
   The public contract should describe `(start_exclusive, end_inclusive]`, 
equal endpoints, required retained snapshots, mode-specific output, 
rowkind/schema export, and error behavior for unsupported tables or missing 
history. Preserve the existing core restrictions rather than silently 
broadening support.
   
   Coverage should include inserts, updates, deletes, repeated keys, deletion 
vectors, first-row split grouping where supported, relevant schema evolution, 
malformed intervals, missing endpoints, empty results, system-field-only 
projection, independent execution of assigned splits, and resource release 
after errors or cancellation. Ordinary reads followed by a synthetic `+I` 
column are not a substitute for AuditLogRead's retained-retract semantics.
   
   The Diff contract needs explicit cross-version documentation: Java 2.1.0 
`IncrementalDiffSplitRead` emits changed after images and retained deletes, 
whereas the Rust audit Diff API emits before/after update pairs. This is an 
interoperability question to resolve with shared fixtures, not a claim that the 
Rust implementation is necessarily incorrect.
   
   ### Anything else?
   
   References: [core incremental-reading 
documentation](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/docs/src/incremental-reading.md)
 and [C 
binding](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/bindings/c/src/table.rs#L2641).
   
   Further references: [DataFusion audit 
execution](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/integrations/datafusion/src/physical_plan/audit_log.rs),
 [Java 
IncrementalSplit](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/source/IncrementalSplit.java),
 and [Java Diff 
reader](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/source/splitread/IncrementalDiffSplitRead.java).
   
   This issue does not add binlog pairing/array output, MV scheduling, progress 
publication, or a new changelog producer. Those are separate reader, consumer, 
or writer capabilities.
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to