JinHelin404 commented on PR #2335: URL: https://github.com/apache/incubator-celeborn/pull/2335#issuecomment-1965950957
> > > celeborn should be compatible with spark conf spark.shuffle.compress > > > > > > should we? > > the options are not same actually, Celeborn only supports LZ4 and ZSTD > > @pan3793 Hi pan, thanks for reviewing! > > The compression format of celeborn may be inconsistent with spark. But does celeborn client need to be compatible with common configurations like spark.shuffle.compress? > > For example, my sparkapp is configured with spark.shuffle.compress=false but celeborn still compresses the shuffle data, which is beyond my expectation. @pan3793 Or may we need some logs to show that spark.shuffle.xxx is not effective when using celeborn? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
