holdenk commented on PR #45802: URL: https://github.com/apache/spark/pull/45802#issuecomment-2038750162
It is. In general I think since we still apply the filter post projection if a user has created a projection with a named field and then filtered on that field the user is probably doing that intentionally since they don't want to double eval the filter. That plus some basic cost heuristics (simple math is cheap udfs can be expensive and so can regexes) should be a net win. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
