Indian-boult opened a new issue, #20741:
URL: https://github.com/apache/hudi/issues/20741
### Bug Description
**What happened:**
Data skipping re-applies a filter's term to the min and max stats of each
file (column stats) or partition (partition stats). That is only right when the
term is order-preserving in the column. `ConstantOperandRule`
(hudi-spark-common, `org.apache.spark.sql.hudi.dataskipping.rules`) claims it
for two kinds of terms where it does not hold, so files and partitions that
hold matching rows are pruned and the query returns too few rows.
- `BitwiseOr` (`a | c` and `c | a`) is not monotonic, even with a constant:
`1 | 2 = 3`, but the larger `2 | 2 = 2`. A file that holds 1 and 2 gets the
bounds `[3, 2]`, so `(a | 2) = 3` prunes it, though its row 1 matches. The rule
already notes this (`BitwiseOr is not monotonic`, #20449), and the oracle in
#20439 has it as the `bitwise_or` family.
- `DateAdd`, `DateSub`, `DateDiff` (end date) and `ShiftRight` accept any
second operand. When that operand references the column, `TermRange` re-binds
it to the same bound, so the term is evaluated at the min and at the max, but
it is not order-preserving:
- `date_add(d, dayofmonth(d))` maps 2024-01-31 to 2024-03-02, and the
later 2024-02-01 to 2024-02-02.
- `datediff(d, trunc(d, 'MM'))` maps 2024-01-31 to 30, and 2024-02-01 to 0.
- `shiftright(a, a)` on a bigint maps 64 to 64 and 65 to 32 (a bigint
shifts by the low 6 bits of the amount).
`Add`, `Multiply` and `ShiftLeft` already need a constant operand (#20457,
#20532).
**What you expected:**
Every file and partition that holds a matching row stays a candidate. Terms
with a constant operand, such as `date_add(d, 1)` or `shiftright(a, 2)`, can
still prune.
**Steps to reproduce:**
```sql
create table t (id int, a bigint, d date, ts long, dt string) using hudi
tblproperties (primaryKey = 'id', orderingFields = 'ts', type = 'cow')
partitioned by (dt);
set hoodie.metadata.index.column.stats.enable = true;
set hoodie.metadata.index.column.stats.column.list = a,d;
insert into t values (1, 1, date'2024-01-31', 1, 'par1'), (2, 2,
date'2024-02-01', 1, 'par1');
insert into t values (3, 64, date'2024-05-10', 1, 'par2'), (4, 65,
date'2024-05-11', 1, 'par2');
set hoodie.enable.data.skipping = true;
select id from t where (a | 2) = 3; --
expected 1, returns nothing
select id from t where shiftright(a, a) = 64; --
expected 3, returns nothing
select id from t where date_add(d, dayofmonth(d)) = date'2024-03-02'; --
expected 1, returns nothing
select id from t where datediff(d, trunc(d, 'MM')) = 30; --
expected 1, returns nothing
```
With `hoodie.enable.data.skipping = false` each query returns its row. The
result is the same with partition stats on or off.
### Environment
- Hudi version: master (1.3.0-SNAPSHOT)
- Query engine: Spark 3.5
- Relevant configs: `hoodie.enable.data.skipping=true`, column stats on the
filtered column
### Logs and Stack Trace
No error. The files are pruned and the rows are missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]