cshuo opened a new issue, #20653:
URL: https://github.com/apache/hudi/issues/20653

   Flink snapshot queries on MOR tables can return deleted records when 
primary-key predicates are pushed down to native delete logs.
   
   **Example scenario**
   
   1. Create a version 10 MOR table using native logs, with primary key `id` 
and ordering field `ts`.
   2. Write `id = 'k', ts = 100` to a base file.
   3. Delete `k` with a newer ordering value, producing an uncompacted 
`.deletes.parquet` file.
   4. Run:
   
      ```sql
      SELECT * FROM t WHERE id = 'k';
      ```
   
   **Expected behavior:** No rows are returned.
   
   **Actual behavior:** The old base-file row for `k` can be returned.
   
   **Root cause**
   
   `FlinkRowDataReaderContext` pushes business primary-key predicates into 
native delete logs. These files contain `_hoodie_record_key` and ordering 
fields, but generally lack the business-key columns. Parquet can therefore 
discard the tombstones before merging.
   
   **Validation**
   
   Reproduced using actual Parquet files and the Flink file-group reader on 
Flink 1.20.1 / Java 11. The unfiltered read correctly applies the deletion.
   
   **Suggested fix**
   
   Skip query-predicate pushdown for native delete logs while preserving safe 
primary-key pushdown for base files and data logs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to