Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21070
Yup, https://github.com/apache/spark/pull/21070#issuecomment-386793202,
just for clarification, the given attribute and literal is castable but they
are not being as so, right?
I believe this is a known issue and there were several tries:
One approach was directly casting always and it was reverted (roughly 2
years ago?). Another approach was constant folding at optimizer level but it
was rejected as it's too messy. Another approach was directly casting and
comparing both values but it was also rejected since it sounded unsafe.
It was a long old story so probably worth double checking the history but I
feel sure that I remember this story correctly.
The key point IIUC was that `translateFilter` should be super conservative
and sounds we better need to check every possibility.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
