yahoNanJing commented on PR #576: URL: https://github.com/apache/arrow-ballista/pull/576#issuecomment-1366319709
Hi @Dandandan, could we make this feature optional? And I think the benefits of compression will be reflected at two aspects. The network throughput will be exactly much higher than before. While for the overall time costing, the transforming speed will be faster. However, it may also cost more CPUs for compression and decompression. So whether the overall total time cost can be reduced may depend on the size of the shuffle data. How about making the decision depending on some size of the shuffling data? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
