wForget opened a new issue, #11018:
URL: https://github.com/apache/incubator-gluten/issues/11018
### Backend
VL (Velox)
### Bug description
error:
```
W20251102 22:06:55.722939 2682850 HashProbe.cpp:1837] Can't reclaim from
hash probe operator, state_[WAIT_FOR_BUILD], nonReclaimableSection_[0],
inputSpiller_[nullptr], table_[nullptr], table_ numDistinct[nullptr],
op.3.0.0.HashProbe, usage: 1.94MB, node pool reservation: 8.00MB
25/11/02 22:06:55 INFO NativePlanEvaluator: NativePlanEvaluator-5: Spilled
1335885824 / 52953088 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 /
80425779 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 /
80425779 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk
7340032 / 80425779 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 /
73085747 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 /
73085747 bytes of data.
25/11/02 22:06:59 INFO NativeMemoryManager: NativeMemoryManager: Shrunk 0 /
73085747 bytes of data.
W20251102 22:07:03.134801 2682850 HashProbe.cpp:1837] Can't reclaim from
hash probe operator, state_[WAIT_FOR_BUILD], nonReclaimableSection_[0],
inputSpiller_[nullptr], table_[nullptr], table_ numDistinct[nullptr],
op.3.0.0.HashProbe, usage: 1.89MB, node pool reservation: 11.00MB
25/11/02 22:07:03 INFO NativePlanEvaluator: NativePlanEvaluator-5: Spilled
1349517312 / 73085747 bytes of data.
E20251102 22:07:03.571292 2682850 Exceptions.h:66] Line:
/work/ep/build-velox/build/velox_ep/velox/exec/Spill.cpp:469,
Function:SpillPartitionId, Expression: Spill level 4 exceeds max spill level
3, Source: RUNTIME, ErrorCode: INVALID_STATE
E20251102 22:07:03.584228 2682850 Exceptions.h:66] Line:
/work/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp:574,
Function:operator(), Expression: Operator::getOutput failed for [operator:
ValueStream, plan node ID: 0]: Error during calling Java code from native code:
org.apache.gluten.exception.GlutenException:
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Spill level 4 exceeds max spill level 3
Retriable: False
Context: Operator: HashBuild[3] 2
Additional Context: Operator: ValueStream[0] 0
Function: SpillPartitionId
File: /work/ep/build-velox/build/velox_ep/velox/exec/Spill.cpp
Line: 469
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1
_ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2
_ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4exec16SpillPartitionIdC1ES2_j
# 4
_ZN8facebook5velox4exec11SpillerBase11finishSpillERSt3mapINS1_16SpillPartitionIdESt10unique_ptrINS1_14SpillPartitionESt14default_deleteIS6_EESt4lessIS4_ESaISt4pairIKS4_S9_EEE
# 5 _ZN8facebook5velox4exec9HashBuild15finishHashBuildEv
# 6 _ZN8facebook5velox4exec9HashBuild19noMoreInputInternalEv
# 7 _ZN8facebook5velox4exec9HashBuild20postHashBuildProcessEv
# 8
_ZN8facebook5velox4exec9HashBuild9isBlockedEPN5folly10SemiFutureINS3_4UnitEEE
# 9
_ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE0_clEv
# 10
_ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 11
_ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 12 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 13 _ZN6gluten24WholeStageResultIterator4nextEv
# 14 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 15 0x00007fab220da0a8
at
org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at
org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at
org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at
org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at
org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:95)
at
scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at
org.apache.gluten.vectorized.ColumnarBatchInIterator.hasNext(ColumnarBatchInIterator.java:36)
at
org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native
Method)
at
org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at
org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at
org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at
org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at
org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at
org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:95)
at
org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:122)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at
scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at
org.apache.gluten.vectorized.ColumnarBatchInIterator.hasNext(ColumnarBatchInIterator.java:36)
at
org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native
Method)
at
org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
at
org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at
org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:154)
at
org.apache.gluten.iterator.IteratorsV1$ReadTimeAccumulator.hasNext(IteratorsV1.scala:122)
at
org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:38)
at
org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:66)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at
org.apache.spark.shuffle.writer.VeloxUniffleColumnarShuffleWriter.writeImpl(VeloxUniffleColumnarShuffleWriter.java:149)
at
org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:328)
at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
