maryannxue commented on pull request #29726: URL: https://github.com/apache/spark/pull/29726#issuecomment-792915413
If I understand correctly (please make it clearer in your PR description), you are trying to enable DPP even if the filter is not broadcast-able but is small enough, I do not think it is a proper solution to hack through this "broadcastOnly" flag. Instead, you should improve the cost model in `pruningHasBenefit` in this PR and change the config `broadcastOnly` to `false` in your own application. And if you can further prove that with the improved cost model, you get performance improvement across the board, you can suggest flipping the default conf value in this PR or another follow-up one. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
