TheR1sing3un commented on issue #14181:
URL: https://github.com/apache/hudi/issues/14181#issuecomment-3466234847
> theoretically, all the filters can be pushed into the columnar readers
right, not just pk fields.
We distinguish between two situations：
1. only need to read base file:
- COW table
- MOR table with `read_optimized` query type
2. need to perform merging during read:
- MOR table with `snapshot` query type
Case1 can safely push all filters down to a file because the latest result
of each row of record is in that file.
However, Case2 does not work because the final result of each row may come
from different files. We cannot push down non-primary key filtering conditions
within each file. Please consider the following scenario:
```
- pk: id
- ordering value / precombine key: ts
|----------base file----------| |-----------log file---------|
| id: 1 | col_a: 2 | ts: 2 | | id: 1 | col_a: 1 | ts: 1 |
final result: (id:1, col_a:2, ts:2)
If we query like: select * from tbl where col_a = 1
In fact, there is no row with `col_a = 1` in the latest result, so the
expected result is empty.
But if we push down the filter: `col_a = 1` to file format,
then we scan base file get empty,
and scan log file get `(id:1,col_a:1,ts:1)`,
at this point, when reading, there is only one record, so the merged record
is the old data read from the log,
and next plan `FilterExec` can't filter out the `(id:1,col_a:1,ts:1)`.
So the final result after we push down no-pk filter to file format is
`(id:1,col_a:1,ts:1)`, which is an incorrect result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
