sundapeng opened a new pull request, #8509: URL: https://github.com/apache/paimon/pull/8509
### Purpose PyPaimon used the projected read type both as the user-visible output schema and as the row layout for predicate evaluation. When a filter referenced a column omitted by `with_projection()`, the predicate could be skipped or evaluated against the wrong row layout, silently returning unfiltered rows. This PR widens the internal scan schema with predicate-only columns, applies filters on that internal schema, then projects results back to the requested output columns. It also keeps `_row_kind` intact when output projection is used on the parallel Arrow path. ### Brief change log - `TableRead`: track predicate-only scan fields separately from output fields and project Arrow batches back to the requested schema. - `SplitRead` and reader wrappers: rewrite predicate indices for projected row layouts, preserve batch-reader field metadata, and project data-evolution reader output before iterator exposure. - `field_indices.py`: share BLOB/VECTOR field-index helpers across projection wrappers. - Tests: add regressions for primary-key, append-only, and data-evolution reads with filters on non-projected columns, plus parallel `_row_kind` preservation. ### Tests - `python -m pytest paimon-python/pypaimon/tests/projection_predicate_index_test.py -q` - `python -m pytest paimon-python/pypaimon/tests/data_evolution_test.py -q -k 'not test_with_blob'` - `python -m pytest paimon-python/pypaimon/tests/reader_parallel_test.py -q` - `python -m pytest paimon-python/pypaimon/tests/test_outer_projection_record_reader.py -q` - `flake8 --config paimon-python/dev/cfg.ini <touched python files>` - `git diff --check` ### API and Format - [x] This PR does not change any public API - [x] This PR does not change any storage format ### Documentation - [x] No documentation change needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
