LorenzoMartini commented on PR #32921: URL: https://github.com/apache/spark/pull/32921#issuecomment-1418837654
Hey @aokolnychyi thank you for the answer. I see that those sources have special optimizations. However we do have instances of data transformations being incredibly slower using spark's v2 datasources and the only difference in the query plans compared to those same transformation ran using v1 datasources is the absence of the dynamic pruning expressions. Do you have any suggestions on how to improve those use cases if not trying to implement `SupportsRuntimeFiltering` for the spark sources? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
