This is an automated email from the ASF dual-hosted git repository.
yuanzhou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 1f6154b11b [VL] Increase the spill start partition bit to 48 to avoid
overlapping with the hash bits (#9703)
1f6154b11b is described below
commit 1f6154b11bec2a28232fe4ad15633fdf5ce7a3e8
Author: Hongze Zhang <[email protected]>
AuthorDate: Tue May 20 22:04:34 2025 +0100
[VL] Increase the spill start partition bit to 48 to avoid overlapping with
the hash bits (#9703)
Velox now set this number to 48 by default. The patch does the same update
on Gluten's code.
It's known that the check VELOX_CHECK_LT(sizeBits_ - 1,
spillInputStartPartitionBit); in Velox code could be failed if this option is
set with a small number.
---
cpp/velox/compute/WholeStageResultIterator.cc | 2 +-
docs/get-started/Velox.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/cpp/velox/compute/WholeStageResultIterator.cc
b/cpp/velox/compute/WholeStageResultIterator.cc
index 79d7669737..57f2f8af1d 100644
--- a/cpp/velox/compute/WholeStageResultIterator.cc
+++ b/cpp/velox/compute/WholeStageResultIterator.cc
@@ -526,7 +526,7 @@ std::unordered_map<std::string, std::string>
WholeStageResultIterator::getQueryC
configs[velox::core::QueryConfig::kSpillReadBufferSize] =
std::to_string(veloxCfg_->get<int32_t>(kSpillReadBufferSize, 1L * 1024
* 1024));
configs[velox::core::QueryConfig::kSpillStartPartitionBit] =
- std::to_string(veloxCfg_->get<uint8_t>(kSpillStartPartitionBit, 29));
+ std::to_string(veloxCfg_->get<uint8_t>(kSpillStartPartitionBit, 48));
configs[velox::core::QueryConfig::kSpillNumPartitionBits] =
std::to_string(veloxCfg_->get<uint8_t>(kSpillPartitionBits, 3));
configs[velox::core::QueryConfig::kSpillableReservationGrowthPct] =
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index 533ed3cf62..c40a1ebb77 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -424,7 +424,7 @@ Using the following configuration options to customize
spilling:
| spark.gluten.sql.columnar.backend.velox.orderBySpillEnabled |
true | Whether spill is enabled on sorts
|
| spark.gluten.sql.columnar.backend.velox.maxSpillLevel | 4
| The max allowed spilling level with zero being the initial
spilling level
|
| spark.gluten.sql.columnar.backend.velox.maxSpillFileSize |
1GB | The max allowed spill file size. If it is zero, then there is
no limit
|
-| spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit |
29 | The start partition bit which is used with 'spillPartitionBits'
together to calculate the spilling partition number
|
+| spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit |
48 | The start partition bit which is used with 'spillPartitionBits'
together to calculate the spilling partition number
|
| spark.gluten.sql.columnar.backend.velox.spillPartitionBits | 2
| The number of bits used to calculate the spilling partition
number. The number of spilling partitions will be power of two
|
| spark.gluten.sql.columnar.backend.velox.spillableReservationGrowthPct |
25 | The spillable memory reservation growth percentage of the
previous memory reservation size
|
| spark.gluten.sql.columnar.backend.velox.spillThreadNum | 0
| (Experimental) The thread num of a dedicated thread pool to do
spill
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
