maropu commented on issue #24765: [SPARK-27915][SQL][WIP] Update logical Filter's output nullability based on IsNotNull conditions URL: https://github.com/apache/spark/pull/24765#issuecomment-511653938 Yea, thanks for revisiting this, @JoshRosen! I remember we have the two suggestions from @gatorsmile and @cloud-fan in the previous discussion; [1) nullability is just a hint for the optimizer and it might be good to add a new trait for this hint](https://github.com/apache/spark/pull/18576#issuecomment-326050659). And, [2) the optimization for `Filter.output` is not common in use cases and it is more important to fix the same issue in `Join.output`](https://github.com/apache/spark/pull/18576#discussion_r177674483). So, I'm currently not sure that this is a right approach (I agree to fix this issue though).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
