sunchao commented on issue #102: URL: https://github.com/apache/arrow-datafusion-comet/issues/102#issuecomment-1962424008
I think for native shuffle, the majority of shuffle write happens on the native side. The native `ShuffleWriterExec` takes the input columnar batch from the previous operators and write them out to different shuffle files based on the output partitions. The feature is limited however: it only supports hash-based shuffle but not sort-based shuffle, e.g., sort the shuffle rows based on the partitions they belong to and write data from different partitions in a single sorted file instead of separate files per partition (this happens by default when # of partitions is > 200). It only supports hash partitioning but not other partitioning types like range or round-robin. On the other hand, for the columnar shuffle, the JVM side does the heavy duty work but only the writing of sorted shuffle files happen on the native side. In addition, several optimizations were made on the JVM side, including async shuffle mode (the sorting of shuffle data and writing sorted data to disk are de-coupled). Columnar shuffle supports both hash-based shuffle and sort-based shuffle, and also support all (I think?) partitioning types (hash/range/round-robin). In addition, columnar shuffle also support a non-native operator as input, unlike native shuffle whose previous operator must be a native Comet operator. The downside of columnar shuffle, I think, is that it has to do extra row-to-columnar conversion regardless of whether the input is columnar batch (from native Comet operator), or rows (e.g., from Spark operator). Therefore, performance may not be as good as native shuffle. However, I think it is relatively more robust and has better coverage. We are trying to move columnar shuffle as the default mode and prioritize this more for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
