Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/21556
  
    @rdblue, so basically  you mean it looks both equality comparison and 
nullsafe equality comparison are identically pushed down and looks it should be 
distinguished; otherwise, there could be a potential problem? If so, yup. I 
agree with it.
    
    I think we won't have actually a chance to push down equality comparison or 
nullsafe equality comparison with actual `null` value by the optimizer. 
However, sure, I think we shouldn't relay on it. I think actually we should 
disallow one of both nullsafe equality comparison or equality comparison with 
`null` in `ParquetFilters`.
    
    Thing is, I remember I checked the inside of Parquet's equality comparison 
API itself is actually nullsafe a long ago like few years ago - this of course 
should be double checked.
    
    Since this PR doesn't change the existing behaviour on this and looks 
needing some more investigation (e.g., checking if it is still (or it has been) 
true what I remembered and checked about Parquet's equality comparison), 
probably, it might be okay to leave it as is.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to