This is an automated email from the ASF dual-hosted git repository.
liuneng pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
from d566aacea [GLUTEN-5341] fix fail 3.5 ut of VeloxParquetWriteSuite
(#5463)
add 4fca32594 [CH]feat: Support external sort shuffle, reduce shuffle
memory usage when the number of partitions is high (#5279)
No new revisions were added by this update.
Summary of changes:
.../vectorized/CHShuffleSplitterJniWrapper.java | 15 +-
.../spark/shuffle/CHColumnarShuffleWriter.scala | 8 +-
...tenClickHouseColumnarSortShuffleAQESuite.scala} | 3 +-
.../local-engine/Shuffle/CachedShuffleWriter.cpp | 22 ++-
cpp-ch/local-engine/Shuffle/CachedShuffleWriter.h | 4 +-
cpp-ch/local-engine/Shuffle/PartitionWriter.cpp | 159 ++++++++++++++++++---
cpp-ch/local-engine/Shuffle/PartitionWriter.h | 46 +++++-
cpp-ch/local-engine/Shuffle/SelectorBuilder.cpp | 56 +++++---
cpp-ch/local-engine/Shuffle/SelectorBuilder.h | 13 +-
cpp-ch/local-engine/Shuffle/ShuffleSplitter.h | 4 +
.../Shuffle/SortedPartitionDataMerger.cpp | 139 ++++++++++++++++++
.../Shuffle/SortedPartitionDataMerger.h | 91 ++++++++++++
cpp-ch/local-engine/local_engine_jni.cpp | 11 +-
.../scala/org/apache/gluten/GlutenConfig.scala | 29 ++++
14 files changed, 531 insertions(+), 69 deletions(-)
copy
backends-clickhouse/src/test/scala/org/apache/gluten/execution/{GlutenClickHouseColumnarShuffleAQESuite.scala
=> GlutenClickHouseColumnarSortShuffleAQESuite.scala} (97%)
create mode 100644 cpp-ch/local-engine/Shuffle/SortedPartitionDataMerger.cpp
create mode 100644 cpp-ch/local-engine/Shuffle/SortedPartitionDataMerger.h
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
