Vivek1106-04 commented on PR #58912: URL: https://github.com/apache/spark/pull/58912#issuecomment-5773198638
Thank you for the review @uros-b. I went with the second option: unwrap now returns a form a data source can express, so `DataSourceV2Strategy` no longer matches the rule's encoding. `UnwrapCastInBinaryComparison` gains a second entry point, `unwrapCastInFilter`, and a `filterSemantics` flag threaded through `unwrapCast`. Under it `falseIfNotNull(e)` returns `FalseLiteral` and `trueIfNotNull(e)` returns `IsNotNull(e)`, so unwrapping a filter yields only `EqualTo`/`GreaterThan`/`FalseLiteral`/`IsNotNull` and the like. The flag defaults to false, so the rule's own path is unchanged. Both encodings now stay in the file that defines them. `unwrapCastInFilter` walks the `And`/`Or` tree rather than the root only, and leaves the subtrees below a `Not` to the plain unwrapping. The substitution is the filter's equivalence, not the expression's: a filter only has to tell the rows it keeps from the ones it drops, and within a `Not`-free `And`/`Or` tree a null result drops the row exactly like a false one (`null AND true` and `false AND true` both drop; `null OR false` and `false OR false` both drop). `Not` inverts that, hence the carve-out. A disjunction over the partition column reaches this too, e.g. ```sql WHERE part = (SELECT max(val) FROM dim) OR part > 100 ``` which unwrapped to `Or(And(IsNull(part), null), part > 100)` and was pushed with the `BOOLEAN_EXPRESSION(null)` child you describe. Walking the tree covers it, and I added your tests as written: - `EqualTo(Cast(cint, LongType), Literal(Int.MaxValue + 1L)) AND other` -> `AND(FALSE, other)` - `LessThan(Cast(cint, LongType), Literal(Int.MaxValue + 1L)) AND other` -> `AND(IS_NOT_NULL(cint), other)` - `NOT (EqualTo(Cast(cint, LongType), Literal(Int.MaxValue + 1L)))` is translated, is not `NOT(FALSE)`, and matches what the rule's own unwrapping translates to `UnwrapCastInBinaryComparisonSuite` and `DataSourceV2StrategySuite` pass, along with the end-to-end pruning test. PR description updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
