revans2 opened a new pull request #29067: URL: https://github.com/apache/spark/pull/29067
I would love to see this go into the 3.1 release. It also applies cleanly to the 3.0 branch if someone wants to pull it in there too. This may look big, but it is mostly refactoring to get all of the code associated with transforming a batch around cacheing into a single location. I did clean up a few things along the way to separate out the actual transformation from the metrics. I ran some simple tests manually to see if I could find any performance difference between the old and new code, and if there is any it is within error. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
