Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21556
@rdblue, so basically you mean it looks both equality comparison and
nullsafe equality comparison are identically pushed down and looks it should be
distinguished; otherwise, there could be a potential problem? If so, yup. I
agree with it.
I think we won't have actually a chance to push down equality comparison or
nullsafe equality comparison with actual `null` value by the optimizer.
However, sure, I think we shouldn't relay on it. I think actually we should
disallow one of both nullsafe equality comparison or equality comparison with
`null` in `ParquetFilters`.
Thing is, I remember I checked the inside of Parquet's equality comparison
API itself is actually nullsafe a long ago like few years ago - this of course
should be double checked.
Since this PR doesn't change the existing behaviour on this and looks
needing some more investigation (e.g., checking if it is still (or it has been)
true what I remembered and checked about Parquet's equality comparison),
probably, it might be okay to leave it as is.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
