ayushi-agarwal commented on PR #11452:
URL:
https://github.com/apache/incubator-gluten/pull/11452#issuecomment-3953101632
@rui-mo @FelixYBW We see this issue again: there were few log lines added so
that we know it goes in the new destructor added.
The code went there and came out of it, reserved bytes are 0 but numPools
are still 4 because of which it fails.
I20260224 14:23:26.637977 108415 VeloxMemoryManager.cc:402] There are still
outstanding Velox memory allocations. Waiting for 14550 ms to let possible
async tasks done...
E20260224 14:23:26.645238 108415 VeloxMemoryManager.cc:408] Failed to
release Velox memory manager after 43350ms as there are still outstanding
memory resources.
E20260224 14:23:26.645354 108415 Exceptions.h:66] Line:
/root/incubator-gluten/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp:153,
Function:~MemoryManager, Expression: pools_.size() != 0 (1 vs 0). There are
unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools
4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 44.00MB
task.Gluten_Stage_7_TID_1204_VTID_27 usage 0B reserved 0B peak 43.00MB
node.0 usage 0B reserved 0B peak 10.00MB
op.0.0.0.TableScan usage 0B reserved 0B peak 9.99MB
refcount 3
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 1048576 allocated
pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0
numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0
reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity
0B freeReservedCapacity 0B], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of
'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools
allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools
4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 44.00MB
task.Gluten_Stage_7_TID_1204_VTID_27 usage 0B reserved 0B peak 43.00MB
node.0 usage 0B reserved 0B peak 10.00MB
op.0.0.0.TableScan usage 0B reserved 0B peak 9.99MB
refcount 3
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 1048576 allocated
pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0
numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0
reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity
0B freeReservedCapacity 0B]
Retriable: False
Function: ~MemoryManager
File:
/root/incubator-gluten/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp
Line: 153
Stack trace:
```
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1
_ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2
_ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 0x0000000001bf53d9
# 4 _ZN6gluten18VeloxMemoryManagerD1Ev
# 5 _ZN6gluten18VeloxMemoryManagerD0Ev
# 6 _ZN6gluten13MemoryManager7releaseEPS0_
# 7 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 8 0x00007fd1a7a0f8f0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
