LantaoJin commented on pull request #29869: URL: https://github.com/apache/spark/pull/29869#issuecomment-701491881
I can add something in the user facing changes part. But let me explain why I written No from another viewpoint. This patch actually separates all accumulators to two types: merged before entering event loop and merged after entering event loop. Without this patch all accumulators are the second type. With this patch, whatever name user set, if accumulators with a certain name go to the first type, they will benefited by this patch, avoid triggering fullGC. If their with another certain name go to the second type, these accumulators behivous are just like without patch, nothing will go worse. Of course, change a certain name to match the first type will be good. but this patch is just give an opportunity for developers to avoid driver full GC. They can keep their old accumulator name if the driver are good enough. As long as this patch not make things worse, I call that no user facing changed. Of course, it is not a bug, it’s should be a improvement or feature. if this approach acknowledged, I will document it somewhere. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
