This is an automated email from the ASF dual-hosted git repository.
yuanzhou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new aeed56a225 [VL] CI: Q97 OOM test passed, stop ignoring its return code
in CI (#7294)
aeed56a225 is described below
commit aeed56a225b3d6394781a85551b2942e14469963
Author: Hongze Zhang <[email protected]>
AuthorDate: Fri Sep 20 16:23:15 2024 +0800
[VL] CI: Q97 OOM test passed, stop ignoring its return code in CI (#7294)
* [VL] CI: Q97 OOM test passed, make it be able to fail CI
The case started to pass due to the merge of:
Fix [GLUTEN-7243][VL] Fix Q97 cross-task spilling hangs #7244
Rebase [GLUTEN-6887][VL] Daily Update Velox Version (2024_09_18) #7259
(investigating)
---
.github/workflows/velox_backend.yml | 20 ++++----------------
1 file changed, 4 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/velox_backend.yml
b/.github/workflows/velox_backend.yml
index 505cd6f548..45b247c9ce 100644
--- a/.github/workflows/velox_backend.yml
+++ b/.github/workflows/velox_backend.yml
@@ -389,8 +389,7 @@ jobs:
-d=FLUSH_MODE:DISABLED,spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false,spark.gluten.sql.columnar.backend.velox.maxPartialAggregationMemoryRatio=1.0,spark.gluten.sql.columnar.backend.velox.maxExtendedPartialAggregationMemoryRatio=1.0,spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinPct=100,spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinRows=0
\
-d=FLUSH_MODE:ABANDONED,spark.gluten.sql.columnar.backend.velox.maxPartialAggregationMemoryRatio=1.0,spark.gluten.sql.columnar.backend.velox.maxExtendedPartialAggregationMemoryRatio=1.0,spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinPct=0,spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinRows=0
\
-d=FLUSH_MODE:FLUSHED,spark.gluten.sql.columnar.backend.velox.maxPartialAggregationMemoryRatio=0.05,spark.gluten.sql.columnar.backend.velox.maxExtendedPartialAggregationMemoryRatio=0.1,spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinPct=100,spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinRows=0
- - name: (To be fixed) TPC-DS SF30.0 Parquet local spark3.2 Q97 low
memory, IO threads off
- continue-on-error: true # OOM
+ - name: TPC-DS SF30.0 Parquet local spark3.2 Q97 low memory
run: |
cd tools/gluten-it \
&& GLUTEN_IT_JVM_ARGS=-Xmx3G sbin/gluten-it.sh parameterized \
@@ -400,20 +399,9 @@ jobs:
-d=ISOLATION:OFF,spark.gluten.memory.isolation=false \
-d=ISOLATION:ON,spark.gluten.memory.isolation=true,spark.memory.storageFraction=0.1
\
-d=OFFHEAP_SIZE:2g,spark.memory.offHeap.size=2g \
- -d=OFFHEAP_SIZE:1g,spark.memory.offHeap.size=1g
- - name: (To be fixed) TPC-DS SF30.0 Parquet local spark3.2 Q97 low
memory, IO threads on
- continue-on-error: true # Timeout
- timeout-minutes: 15 #
https://github.com/apache/incubator-gluten/issues/7161
- run: |
- cd tools/gluten-it \
- && GLUTEN_IT_JVM_ARGS=-Xmx3G sbin/gluten-it.sh parameterized \
- --local --preset=velox --benchmark-type=ds --error-on-memleak
--queries=q97 -s=30.0 --threads=12 --shuffle-partitions=72 --iterations=1 \
- --data-gen=skip -m=OffHeapExecutionMemory \
- --extra-conf=spark.gluten.sql.columnar.backend.velox.IOThreads=12 \
- -d=ISOLATION:OFF,spark.gluten.memory.isolation=false \
-
-d=ISOLATION:ON,spark.gluten.memory.isolation=true,spark.memory.storageFraction=0.1
\
- -d=OFFHEAP_SIZE:2g,spark.memory.offHeap.size=2g \
- -d=OFFHEAP_SIZE:1g,spark.memory.offHeap.size=1g
+ -d=OFFHEAP_SIZE:1g,spark.memory.offHeap.size=1g \
+
-d=IO_THREADS:12,spark.gluten.sql.columnar.backend.velox.IOThreads=12 \
+ -d=IO_THREADS:0,spark.gluten.sql.columnar.backend.velox.IOThreads=0
run-tpc-test-ubuntu-randomkill:
needs: build-native-lib-centos-7
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
