pan3793 commented on PR #3555: URL: https://github.com/apache/celeborn/pull/3555#issuecomment-3659310987
lz4 is famous for its ultra-fast speed, the upgrade is not free, my test shows it has perf impact - https://github.com/apache/spark/pull/53453 I understand that security takes precedence over performance, so I'm fine with this change. for the suggestion of 'moving to fastestInstance.safeDecompressor', I think we can NOT do that blindly - Celeborn Spark/Flink clients use the lz4-java libs provided by the engine libs, since we support a wide range of Spark/Flink versions, it's possible that the engine still ships old lz4-java jar, we may need to dynamiclly check and bind the `fastDecompressor` or `safeDecompressor` based on runtime version of lz4-java -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
