Flyangz opened a new issue, #2369:
URL: https://github.com/apache/auron/issues/2369
**Describe the bug**
ORC predicate pushdown produces wrong results for a WHERE clause of the form
A OR B when one disjunct cannot be converted to an ORC Predicate.
In orc_exec.rs, collect_or_predicates silently drops any OR disjunct that
fails to convert (convert_expr_to_orc returns None), then builds
Predicate::or(...) from only the surviving disjuncts. This narrows the
pushed predicate into a subset of the real filter. Since the pushed
predicate is used to skip row groups whose statistics prove no match, the
narrowed predicate skips row groups that actually contain matching
rows → rows are lost, results are wrong.
A common trigger is comparing a string column with a numeric literal: b =
900000 is coerced to cast(b as double) = 900000.0 (a CastExpr, not
convertible), while a sibling disjunct like id = 5 (Column = Literal) is
convertible. The mixed OR id = 5 OR b = 900000 then pushes down only id =
5, and every row group whose id statistics exclude 5 is skipped — dropping the
row where b = '900000'.
**To Reproduce**
```scala
test("test OR pushdown with an unconvertible disjunct for orc table") {
withTable("orc_or") {
sql("create table orc_or(id int, b string) using orc")
// enough rows so `id` statistics differ across row groups and pruning
kicks in
sql("insert into orc_or select cast(id as int), cast(id as string)
from range(0, 1000000)")
// `b = 900000` (string col vs int literal) -> cast(b as double)=2.0
-> not convertible
// `id = 5` -> convertible
// OR drops the b-branch -> pushes only `id = 5` -> row groups without
id=5 are skipped,
// losing the row where b='900000'
checkSparkAnswerAndOperator("select * from orc_or where id = 5 or b =
900000")
}
}
```
**Expected behavior**
Auron returns the same rows as vanilla Spark (both the id = 5 row and the b
= '900000' row). An OR predicate must be pushed down all-or-nothing: if any
disjunct cannot be converted, the whole OR must not be pushed,
while the convertible AND conjuncts around it still push down safely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
