wangzhigang1999 opened a new issue, #9857:
URL: https://github.com/apache/paimon/issues/9857
### Paimon version
Reproduced on master at `43e9ad701ebee6d9af8316f35533f4d0836a1fc0`
(`2.2-SNAPSHOT`), with Parquet `1.16.0`.
### Compute Engine
Java API on JDK 17, through `FileStoreTable.newReadBuilder()`.
### Minimal reproduce step
Create an append-only table with `bucket = -1`, `file.format = parquet`, and
`write-only = true`. Use schema `id INT, status STRING` and commit these four
rows through the Java batch write API:
| id | status |
| --- | --- |
| 0 | A |
| 1 | B |
| 2 | NULL |
| 3 | A |
Read with `status = 'A'` as the filter and only `id` in the read type:
```java
ReadBuilder readBuilder =
table.newReadBuilder()
.withFilter(
new PredicateBuilder(rowType)
.equal(1, BinaryString.fromString("A")))
.withReadType(rowType.project(new int[] {0}));
List<Integer> ids = new ArrayList<>();
try (RecordReader<InternalRow> reader =
readBuilder.newRead().createReader(readBuilder.newScan().plan().splits())) {
reader.forEachRemaining(row -> ids.add(row.getInt(0)));
}
```
Actual result: `ids` is empty. The same reproduction fails with
`file-index.bitmap.columns = status` enabled.
### What doesn't meet your expectations?
The result must contain IDs `0` and `3`. The Java read API allows
best-effort filtering and extra candidate rows, but matching rows must not be
lost.
### Anything else?
Related closed PR: #5385, which described the same empty-result behavior. In
that PR, @JingsongLi suggested considering only projected fields when
converting Parquet filters and discarding other filters. The author's
implementation added filter fields to the read projection; both commits
preceded the review comments, and the PR was closed without a subsequent
implementation of that suggestion.
I reproduced the problem through the public Java table-read API on the
commit above. In that version, `ParquetReaderFactory` converts predicates
against the physical file schema without first removing conditions on
unprojected fields. During page pruning, Parquet treats a field absent from the
requested columns as a missing column with null values, even when that field
exists in the file.
The [candidate
fix](https://github.com/wangzhigang1999/paimon/commit/f83f08c867b83b0a89e2d11331d536051f152185)
follows the suggested direction: retain AND conjuncts covered by the read
projection before Parquet conversion, and discard an entire OR conjunct if it
references an unprojected field. It keeps the requested columns and the Parquet
page-index implementation unchanged.
Spark/Flink SQL reproduction has not been established. PyPaimon's PyArrow
reader does not call this Java reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
