JingsongLi opened a new pull request, #10451: URL: https://github.com/apache/paimon/pull/10451
## Summary Align existing PK row and input-changelog writes with Java, and exercise Native batch changelog packing and sharding end to end. - Preserve `InternalRow.get_row_kind()` when converting a row to Arrow for either the Python writer or the thin Native wrapper. A configured `rowkind.field` retains precedence, and ignored retractions are filtered before Python dynamic-bucket assignment. - Normalize accepted Arrow batches to the declared top-level nullability and metadata so row and Arrow APIs can share a buffered writer. Reuse already matching batches and preserve their zero-copy behavior. - Record every accepted input in separate rolling changelog files before the merge engine folds equal keys, matching Java `MergeTreeWriter`. Previously Python wrote the folded data into the input changelog, silently losing earlier same-key inputs. - Share file-statistics calculation between data and input-log files. Advance a raw-log chunk only after its file and metadata are complete, retain published logs across prepare retries, and hand off ownership at successful prepare. - Cover combined batch packing and sharding with fixed/dynamic buckets, deletion vectors, all four row kinds, timestamp boundaries, global LIMIT, residual filters, nested projection, partition pruning, schema evolution, compact/overwrite selection and merge engines. - Native planning/reading tests explicitly fail on Python fallback. Mixed row/Arrow writes cover both call orders, producer none/input and cross-partition migration deletes. This contribution uses existing APIs; batch planning and write normalization remain in Rust core. The Native Python binding/wrapper only converts inputs and calls core. ## Dependency **Depends on https://github.com/apache/paimon-rust/pull/1079.** Native CI continues to build paimon-rust from **main**. Until that PR merges, the new sharded changelog and explicit cross-partition row-kind regressions require its local build. This PR is therefore opened as a draft. The ordinary Python writer repairs are independently covered without Native. ## Java reference Compared with `IncrementalDeltaStartingScanner` / `SnapshotReaderImpl` for complete-window batch packing, worker selection, timestamp boundaries and snapshot filtering; `RowKindGenerator` / `TableWriteImpl` for row-kind precedence and filtering; and `MergeTreeWriter` for separate raw input changelog and merged data writers. ## Verification Using the current local Rust extension containing the dependency: - Native test selection with planning/reading/writing/updating/committing enabled: **2537 passed, 13 skipped**. - Ordinary Python writer, changelog, file-cleanup and Arrow-schema regressions with all Native switches disabled: **116 passed, 1 skipped**. - Legacy `tests/py36` regression cases on the local Python 3.13 runtime: **79 passed** (this is not a Python 3.6 runtime verification). - Python binding read/write tests in the Rust repository: **168 passed**. - Flake8 for changed files, license-header validation and `git diff --check`: passed. Fault injection covers failure on the second rolled changelog chunk and failure on a data file after logs publish. Retried prepare preserves every input exactly once, correct sequence metadata, and prepared files after writer close. Independent verification exercised all four row kinds and both failure paths, plus the mixed Native row/Arrow regressions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
