Indian-boult opened a new issue, #20741:
URL: https://github.com/apache/hudi/issues/20741

   ### Bug Description
   
   **What happened:**
   
   Data skipping re-applies a filter's term to the min and max stats of each 
file (column stats) or partition (partition stats). That is only right when the 
term is order-preserving in the column. `ConstantOperandRule` 
(hudi-spark-common, `org.apache.spark.sql.hudi.dataskipping.rules`) claims it 
for two kinds of terms where it does not hold, so files and partitions that 
hold matching rows are pruned and the query returns too few rows.
   
   - `BitwiseOr` (`a | c` and `c | a`) is not monotonic, even with a constant: 
`1 | 2 = 3`, but the larger `2 | 2 = 2`. A file that holds 1 and 2 gets the 
bounds `[3, 2]`, so `(a | 2) = 3` prunes it, though its row 1 matches. The rule 
already notes this (`BitwiseOr is not monotonic`, #20449), and the oracle in 
#20439 has it as the `bitwise_or` family.
   - `DateAdd`, `DateSub`, `DateDiff` (end date) and `ShiftRight` accept any 
second operand. When that operand references the column, `TermRange` re-binds 
it to the same bound, so the term is evaluated at the min and at the max, but 
it is not order-preserving:
     - `date_add(d, dayofmonth(d))` maps 2024-01-31 to 2024-03-02, and the 
later 2024-02-01 to 2024-02-02.
     - `datediff(d, trunc(d, 'MM'))` maps 2024-01-31 to 30, and 2024-02-01 to 0.
     - `shiftright(a, a)` on a bigint maps 64 to 64 and 65 to 32 (a bigint 
shifts by the low 6 bits of the amount).
   
   `Add`, `Multiply` and `ShiftLeft` already need a constant operand (#20457, 
#20532).
   
   **What you expected:**
   
   Every file and partition that holds a matching row stays a candidate. Terms 
with a constant operand, such as `date_add(d, 1)` or `shiftright(a, 2)`, can 
still prune.
   
   **Steps to reproduce:**
   
   ```sql
   create table t (id int, a bigint, d date, ts long, dt string) using hudi
     tblproperties (primaryKey = 'id', orderingFields = 'ts', type = 'cow')
     partitioned by (dt);
   set hoodie.metadata.index.column.stats.enable = true;
   set hoodie.metadata.index.column.stats.column.list = a,d;
   insert into t values (1, 1, date'2024-01-31', 1, 'par1'), (2, 2, 
date'2024-02-01', 1, 'par1');
   insert into t values (3, 64, date'2024-05-10', 1, 'par2'), (4, 65, 
date'2024-05-11', 1, 'par2');
   set hoodie.enable.data.skipping = true;
   select id from t where (a | 2) = 3;                                     -- 
expected 1, returns nothing
   select id from t where shiftright(a, a) = 64;                           -- 
expected 3, returns nothing
   select id from t where date_add(d, dayofmonth(d)) = date'2024-03-02';  -- 
expected 1, returns nothing
   select id from t where datediff(d, trunc(d, 'MM')) = 30;               -- 
expected 1, returns nothing
   ```
   
   With `hoodie.enable.data.skipping = false` each query returns its row. The 
result is the same with partition stats on or off.
   
   ### Environment
   
   - Hudi version: master (1.3.0-SNAPSHOT)
   - Query engine: Spark 3.5
   - Relevant configs: `hoodie.enable.data.skipping=true`, column stats on the 
filtered column
   
   ### Logs and Stack Trace
   
   No error. The files are pruned and the rows are missing.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to