FMX commented on PR #2335: URL: https://github.com/apache/incubator-celeborn/pull/2335#issuecomment-1965959637
> > > celeborn should be compatible with spark conf spark.shuffle.compress > > > > > > should we? > > the options are not same actually, Celeborn only supports LZ4 and ZSTD > > @pan3793 Hi pan, thanks for reviewing! > > The compression format of celeborn may be inconsistent with spark. But does celeborn client need to be compatible with common configurations like spark.shuffle.compress? > > For example, my sparkapp is configured with spark.shuffle.compress=false but celeborn still compresses the shuffle data, which is beyond my expectation. Why will you need uncompressed shuffle data in Celeborn? Celeborn will compress shuffle data by default because there is a performance benefit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
