Jasraj-Dhanoa commented on PR #59177: URL: https://github.com/apache/spark/pull/59177#issuecomment-5976442769
Closing this after a closer look. Guarding these paths would only be worth it if the failure were recoverable, meaning the same executor keeps running tasks afterward, so a leaked buffer would actually be lost. In practice it isn't: with UnsafeRowSerializer, the realistic failure on these paths is a direct-memory allocation failure (OutOfDirectMemoryError), which is fatal. The executor exits through SparkUncaughtExceptionHandler, and the memory is freed when it restarts. A leak could only persist after a non-fatal exception, which would need a separate bug first. As purely defensive error handling, it isn't worth adding. Thanks to everyone who took a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
