wangyum opened a new issue, #8362:
URL: https://github.com/apache/incubator-gluten/issues/8362

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   Log:
   ```
   E20241226 23:24:39.978191 324668 VeloxMemoryManager.cc:401] Failed to 
release Velox memory manager after 43350ms as there are still outstanding 
memory resources. 
   E20241226 23:24:39.978293 324668 MemoryPool.cpp:442] [MEM] Memory leak (Used 
memory): Memory Pool[default_leaf LEAF root[root] parent[root] MALLOC 
track-usage thread-safe]<unlimited max capacity capacity 1.51GB used 1.50GB 
available 7.99MB reservation [used 1.50GB, reserved 1.51GB, min 0B] counters 
[allocs 2, frees 0, reserves 0, releases 0, collisions 0])>
   E20241226 23:24:39.979336 324668 Exceptions.h:66] Line: 
/work/cpp/velox/memory/VeloxMemoryManager.cc:102, Function:removePool, 
Expression: pool->reservedBytes() == 0 (1619001344 vs. 0), Source: RUNTIME, 
ErrorCode: INVALID_STATE
   terminate called after throwing an instance of 
'facebook::velox::VeloxRuntimeError'
     what():  Exception: VeloxRuntimeError
   Error Source: RUNTIME
   Error Code: INVALID_STATE
   Reason: (1619001344 vs. 0)
   Retriable: False
   Expression: pool->reservedBytes() == 0
   Function: removePool
   File: /work/cpp/velox/memory/VeloxMemoryManager.cc
   Line: 102
   Stack trace:
   # 0  _ZN8facebook5velox7process10StackTraceC1Ei
   # 1  
_ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
   # 2  
_ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
   # 3  
_ZN6gluten20ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
   # 4  _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
   # 5  _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
   # 6  _ZN6gluten18VeloxMemoryManagerD1Ev
   # 7  _ZN6gluten18VeloxMemoryManagerD0Ev
   # 8  _ZN6gluten13MemoryManager7releaseEPS0_
   # 9  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
   # 10 0x00007ff6b0bc7be4
   
   
   
   .
   Driver stacktrace:
        at 
org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:45)
        at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:341)
        at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:198)
        at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at 
org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:83)
        at 
org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:67)
        at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:55)
        at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:198)
        at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:193)
        at 
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at 
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:207)
        at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
   Caused by: org.apache.spark.SparkException: Job aborted due to stage 
failure: Task 152 in stage 9.0 failed 4 times, most recent failure: Lost task 
152.3 in stage 9.0 (TID 24199, 10.10.1.127, executor 3): ExecutorLostFailure 
(executor 3 exited caused by one of the running tasks) Reason: Container from a 
bad node: container_e2320_1734105600630_7003_01_000015 on host: 10.10.1.127. 
Exit status: 134. Diagnostics: :61)
        at 
org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.wrapForeignAllocation(BaseAllocator.java:241)
        at 
org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.wrapForeignAllocation(RootAllocator.java:29)
        at 
org.apache.arrow.c.ReferenceCountedArrowArray.unsafeAssociateAllocation(ReferenceCountedArrowArray.java:67)
        at 
org.apache.arrow.c.BufferImportTypeVisitor.importBuffer(BufferImportTypeVisitor.java:99)
        at 
org.apache.arrow.c.BufferImportTypeVisitor.importData(BufferImportTypeVisitor.java:120)
        at 
org.apache.arrow.c.BufferImportTypeVisitor.visit(BufferImportTypeVisitor.java:208)
        at 
org.apache.arrow.c.BufferImportTypeVisitor.visit(BufferImportTypeVisitor.java:61)
        at 
org.apache.gluten.shaded.org.apache.arrow.vector.types.pojo.ArrowType$Utf8.accept(ArrowType.java:739)
        at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:126)
        ... 138 more
   
   ```
   
   ### Spark version
   
   Spark-3.5.x
   
   ### Spark configurations
   
   key|value
   -- | --
   spark.gluten.memory.conservative.task.offHeap.size.in.bytes | 1640438897
   spark.executor.memory|45g
   spark.executor.cores|18
   spark.gluten.memory.dynamic.offHeap.sizing.enabled | false
   spark.gluten.memory.offHeap.size.in.bytes | 59055800320
   spark.gluten.memory.task.offHeap.size.in.bytes | 3280877795
   spark.gluten.memoryOverhead.size.in.bytes | 15032385536
   
   
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to