wangzhigang1999 opened a new pull request, #9858:
URL: https://github.com/apache/paimon/pull/9858
### Purpose
Fixes #9857.
Java table reads can lose matching rows when a filter references a column
outside the read projection. With four rows `(id, status) = (0, A), (1, B), (2,
null), (3, A)`,
`newReadBuilder().withFilter(statusEqualsA).withReadType(idOnly)` returns no
rows on the baseline, with or without a bitmap file index.
Filter predicates in `ParquetReaderFactory` before converting them to
Parquet filters. Split top-level AND conditions and retain only conjuncts whose
referenced fields are projected. Keep each OR conjunct intact and discard it if
any referenced field is unprojected. Field matching follows the reader's
case-sensitivity setting.
This follows the projection-filtering direction suggested in #5385. It
preserves the read projection, physical type resolution, and Parquet page-index
implementation. Non-projected conditions are omitted even when the physical
file lacks those columns, so the reader may return extra candidates under the
existing best-effort filtering contract. No public API or storage format
changes.
### Tests
Added two parameterized methods in existing test classes (four cases total).
The public Java table-read regression covers bitmap file indexing enabled and
disabled; both cases fail on the baseline. The format-level method checks that
an AND retains projected-column pruning and an OR preserves matches from an
unprojected column; the OR case fails on the baseline.
On the fixed `43e9ad7` baseline, all 52 Parquet read/write tests and the two
public API cases pass per Surefire phase, with no failures, errors, or skips.
Checkstyle, Spotless, Enforcer, and root RAT pass. All builds and tests ran on
JDK 17 with Java 8 source compatibility and an isolated Maven repository,
without `fast-build`.
```sh
mvn -B -pl paimon-core -am \
-Dtest=ParquetReadWriteTest,AppendOnlySimpleTableTest#testParquetFilterOnUnprojectedColumn
\
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
-DwildcardSuites=none verify
mvn -B -N apache-rat:check
```
After merging upstream master at `3a858842a498c61f1c385291873c81002bb90856`,
the [fork CI
run](https://github.com/wangzhigang1999/paimon/actions/runs/34926140386) passed
the Java matrix, including Core, Flink, Spark, E2E, and Iceberg, as well as
artifact licensing. The Python 3.6.15 job was cancelled and the aggregate CI
check failed, so the full run is not green. The earlier dependency download
failure and Iceberg test error did not recur in this run.
The reproduction and validation cover Java table and file-format reads. They
do not establish a Spark/Flink SQL failure or exact row-level filtering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
