JinHelin404 commented on PR #2335: URL: https://github.com/apache/incubator-celeborn/pull/2335#issuecomment-1965946595
> > celeborn should be compatible with spark conf spark.shuffle.compress > > should we? > > the options are not same actually, Celeborn only supports LZ4 and ZSTD Hi pan, thanks for reviewing! The compression format of celeborn may be inconsistent with spark. But does celeborn client need to be compatible with common configurations like spark.shuffle.compress? For example, my sparkapp is configured with spark.shuffle.compress=false but celeborn still compresses the shuffle data, which is beyond my expectation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
