pan3793 commented on PR #2335: URL: https://github.com/apache/incubator-celeborn/pull/2335#issuecomment-1965961276
> But does celeborn client need to be compatible with common configurations like spark.shuffle.compress? > > For example, my sparkapp is configured with spark.shuffle.compress=false but celeborn still compresses the shuffle data, which is beyond my expectation. I don't think we should, we expect that users search the [Celeborn configuration page](https://celeborn.apache.org/docs/latest/configuration/) with the keywords `compression` if they want to tune the compression algorithm and levels. additionally, conditional mapping spark internal configuration may involve a lot of inconsistency. for example, should we map `spark.shuffle.service.port` to celeborn's something? due the different implementation details, Celeborn won't benefit from zstd buffer pool feature, if we map `spark.shuffle.compress` to `celeborn.shuffle.compression.codec`, how could the user know that `spark.io.compression.zstd.bufferPool.enabled` does not take effect? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
