suxiaogang223 opened a new issue, #1383:
URL: https://github.com/apache/paimon-rust/issues/1383

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-rust/issues) and found nothing 
similar.
   
   Related issue [#1382](https://github.com/apache/paimon-rust/issues/1382) 
already tracks audit-log and incremental reads through C FFI. This issue covers 
the remaining system-table coverage and shared native API contracts; it does 
not request a second implementation of #1382.
   
   ### Motivation
   
   Source comparison: paimon-rust main 
`f91fdb8ab034887547a5ee2f26c1979e30250558` against Java Paimon 2.1.0 
`3a5ffe7870697e2c2605e63413fb5ac5800aae7e`, checked on November 12, 2026. No 
directly matching issue or C API PR was found in the pre-submission search. 
Core SystemRead already exists through #1137/#1138; those implementations are 
reused here. Related #679 concerns referenced_files_size, which is outside this 
Java table-level scope.
   
   Apache Doris exposes Java Paimon's table-level system tables and would like 
to execute their reads through paimon-rust. Its Java frontend can continue to 
bind relations and plan work, while native backend workers read their assigned 
inputs and return Arrow batches. This requires public APIs usable independently 
of DataFusion SQL.
   
   Rust already implements many metadata tables through `SystemRead`, and 
Python exposes a thin binding. However, the C binding does not expose 
SystemRead. Some Java system-table relations are also missing from the public 
core API, and an existing reader does not always expose the complete Java 
output schema. As a result, native consumers cannot currently cover the same 
table-level system-table surface.
   
   The compatibility baseline here is the 19 table-level tables registered by 
Java Paimon 2.1.0. It excludes catalog-wide global tables. The source 
comparison found the following gaps:
   
   | Tables / capability | Existing support | Remaining gap |
   | --- | --- | --- |
   | `branches`, `buckets`, `consumers`, `files`, `file_key_ranges`, 
`manifests`, `options`, `partitions`, `schemas`, `snapshots`, `table_indexes`, 
`tags` | Registered in core SystemRead | No C SystemRead creation, schema, 
configuration, execution, or release APIs |
   | `aggregation_fields` | Implemented in the DataFusion integration | No 
shared public core reader or C entry point |
   | `file_indexes`, `statistics` | Not registered in core SystemRead or the 
DataFusion system-table registry | Missing corresponding system-table readers 
and C exposure; file-index internals and ordinary scan metrics do not supply 
these relations |
   | `binlog` | Audit support provides part of the underlying functionality | 
Missing binlog update pairing, array output, and public native entry points |
   | `snapshots.operation` | Java exposes field ID 14; Rust preserves 
writer_version as ID 15 | Rust Snapshot and the snapshots output schema lack 
operation |
   | System-table and metadata predicates | C predicate constructors resolve 
fields from the base table schema | Cannot construct predicates against a 
system relation's own fields; row-tracking metadata projection is available, 
but those metadata predicates are not exposed by the same C constructors |
   | Large metadata results | SystemRead returns one materialized RecordBatch | 
No public streaming/work-unit execution interface; wrapping an already 
materialized batch in a stream does not avoid its allocation peak |
   
   `audit_log` is covered by #1382. The two remaining table names, `ro` and 
`row_tracking`, should not be treated as entirely missing reader APIs: 
externally planned read-optimized DataSplits can reuse ordinary reading, and C 
projection can already select `_ROW_ID` and `_SEQUENCE_NUMBER` on appropriate 
tables. Their existing capabilities and limitations should be documented 
alongside the support matrix.
   
   ### Solution
   
   The missing public contracts are:
   
   1. **Metadata system-table reads through C.** Expose the existing core 
readers, their logical output schema, projection, filtering, limits, Arrow 
results, and ownership/release rules. Predicate construction must use the 
target relation's complete schema, including filter-only columns, rather than 
the base table's field positions. Basic queries may leave unsupported filters 
in the consuming engine.
   2. **Shared readers for uncovered relations.** Make aggregation_fields 
available outside DataFusion, and provide the Java-compatible file_indexes and 
statistics relations. File indexes must distinguish embedded and external index 
metadata; statistics must expose persisted table statistics rather than 
substitute scan counters.
   3. **Binlog output semantics.** Preserve Java-compatible scalar system 
fields and array-valued user fields. Java 2.1.0 wraps non-streaming DataSplit 
values in one-element arrays; streaming updates are packed into before/after 
arrays. The paired output kind follows the second record, while an enabled 
sequence number comes from the first. Pair within the producer's logical 
stream, preserving repeated updates rather than grouping an entire interval by 
primary key. Pairing must work across Arrow batch boundaries with valid buffer 
ownership; incomplete-event behavior and supported split/mode combinations need 
a Java-compatible contract. This is distinct from returning audit rows with 
scalar user fields.
   4. **Output schema compatibility.** Preserve field identities, names, order, 
types, and nullability, including nested index metadata. In particular, expose 
snapshots.operation from the corresponding snapshot metadata; commit_kind or a 
constant NULL is not an equivalent substitute. Compatibility should be stated 
against a specific Java baseline rather than assumed from matching table names.
   5. **Read-state compatibility.** Allow native system reads to consume a 
resolved table environment with a selected branch and snapshot, including a 
captured empty view, while preserving the schema bound by the consuming engine. 
Document supported point-in-time and interval semantics per table. Ordinary 
time-travel options or timestamp-window support do not establish equivalence 
for every incremental system-table query, particularly partitions.
   6. **Incremental production of large metadata results.** Provide a contract 
for producing batches without first collecting the complete output, with 
resource accounting and cancellation. Distributed execution needs 
non-overlapping assigned work units or an equivalent partitioning contract, so 
workers do not each enumerate the full table. Bucket/partition aggregates must 
remain correct when inputs are divided. Small metadata relations may continue 
to use a single execution unit.
   
   These requirements reuse existing core capabilities where present. They do 
not require every system table to share the same physical reader or to become a 
DataSplit, nor do they require a Rust implementation of engine scheduling.
   
   ### Anything else?
   
   Useful cross-language coverage includes full and reordered projections, 
filter-only columns, empty and zero-column outputs, nullable and nested values, 
snapshot/branch selection, and supported interval modes. Additional cases 
include non-null snapshots.operation, file-index storage variants, absent 
persisted statistics, and binlog update pairs crossing batch boundaries. For 
large metadata reads, verify that batches are produced before full result 
materialization and that cancellation releases retained resources. These are 
validation requirements; this issue does not claim runtime reproductions or 
measured memory failures.
   
   Related boundaries:
   
   - Audit-log and incremental C reader work remains in #1382; this issue can 
reference that coverage.
   - General memory integration is already discussed in 
[#911](https://github.com/apache/paimon-rust/issues/911); the gap here is 
applying an appropriate contract to metadata reads.
   - Java-planned `ro` reads and basic row-tracking projections can reuse 
existing C APIs. A standalone Rust read-optimized planner is a separate 
requirement if consumers also migrate planning.
   - Catalog-wide all_tables, all_partitions, all_table_options, and 
catalog_options are outside this table-level scope, as are changelog 
production, MV scheduling, and write operations.
   
   Source references:
   
   - [Java table-level 
registry](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/system/SystemTableLoader.java#L72),
 [Rust 
SystemRead](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/paimon/src/table/system_table/mod.rs),
 and [C 
APIs](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/bindings/c/src/table.rs).
   - [DataFusion 
aggregation_fields](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/integrations/datafusion/src/system_tables/aggregation_fields.rs),
 [Java 
FileIndexesTable](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/system/FileIndexesTable.java),
 and [Java 
StatisticTable](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/system/StatisticTable.java).
   - [Java 
BinlogTable](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/system/BinlogTable.java),
 [Java snapshots 
schema](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/table/system/SnapshotsTable.java#L111),
 and [Rust snapshots 
schema](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/paimon/src/table/system_table/snapshots.rs#L28).
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to