Dobiasd commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-584630973 For me, neither "Because the software has worked this way", nor "other similar software", are valid arguments. For me, it's just plainly wrong to be able to filter a dataframe on a column that does not exist in this dataframe. I think this behavior is an issue because it means one can not simply look at the schema of a dataframe to determine if an operation with it is valid. Instead one has to consider the whole history of how the dataframe was created/derived. This leads to the effect that refactorings, e.g., changing the way of creation of a dataframe, will break one's code, even though the refactoring should be totally OK because it results in the exact same dataframe schema.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
