wangzhigang1999 opened a new pull request, #10435:
URL: https://github.com/apache/paimon/pull/10435
### Purpose
Fixes #10433
When the Python primary-key writer persists DELETE or UPDATE_BEFORE rows,
the common writer records `delete_row_count=0`. A single-file split can then be
marked raw-convertible, causing a snapshot read to return retract rows. For
example, writing and committing one DELETE row currently returns that row when
the table is reopened; with this fix, the read returns no rows.
Count DELETE and UPDATE_BEFORE rows in the final Arrow table for each data
and input changelog file. This records the physical retract count after buffer
merging and file rolling. Reuse the same counting logic for postpone writes
while preserving their sequence bounds. Return zero for non-primary-key data or
batches without `_VALUE_KIND`.
This change applies to newly written files. Existing files with incorrect
zero counts are outside its scope.
### Tests
Add 12 end-to-end cases across Parquet and ORC in
`delete_row_count_test.py`. They cover all four row kinds, buffer merging and
file rolling, a point lookup for a deleted key, and deletion across commits.
Each case uses real writes and commits, reopens the table, and compares
prepared metadata and persisted manifests with the physical file contents,
including changelog files.
All 12 cases pass with the fix. Against the unmodified baseline, eight fail:
six expose incorrect query results and two expose incorrect persisted metadata.
The following Python regression selection passed: **323 tests**, including
the new cases, postpone writes and the existing mosaic writer options
regression.
```sh
PYTHONPATH=paimon-python python -m pytest -q \
paimon-python/pypaimon/tests/write \
paimon-python/pypaimon/tests/mosaic_writer_options_test.py \
paimon-python/pypaimon/tests/reader_split_generator_test.py \
paimon-python/pypaimon/tests/reader_primary_key_test.py \
paimon-python/pypaimon/tests/reader_append_only_test.py \
paimon-python/pypaimon/tests/manifest/manifest_manager_test.py \
paimon-python/pypaimon/tests/manifest/manifest_schema_test.py
```
Flake8 with the repository configuration and `git diff --check` passed. Ruff
and Pyright reported no new diagnostics relative to the baseline; the checked
existing files still have pre-existing diagnostics. Validation used Python
3.12.6 and PyArrow 19.0.1. The Rust native runtime was not tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
