zml1206 opened a new issue, #8752: URL: https://github.com/apache/incubator-gluten/issues/8752
### Backend VL (Velox) ### Bug description 1.3.0 <img width="1028" alt="Image" src="https://github.com/user-attachments/assets/78bacbea-7cb6-4ab8-84e6-6f854f95e12d"; /> 2024-10-09 <img width="1018" alt="Image" src="https://github.com/user-attachments/assets/dd209366-4ff2-4eff-af22-3b8157fba151"; /> I used the main version of 2024-10-09 and 1.3.0 to run the same series of tasks to test the stability and performance of version 1.3.0. During the test, I found that the executor of 1.3.0 frequently crashed. executor log ``` pure virtual method called terminate called without an active exception ``` core dump ``` (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=139908111537728) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=139908111537728) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=139908111537728, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007f3f99233476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007f3f992197f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x00007f3f98f80b9e in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007f3f98f8c20c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007f3f98f8c277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x00007f3f98f8cfa5 in __cxa_pure_virtual () from /lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x00007f3ee6c5634b in gluten::ListenableArbitrator::growCapacityInternal(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #10 0x00007f3ee6c56993 in gluten::ListenableArbitrator::growCapacity(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #11 0x00007f3ee6e29797 in facebook::velox::memory::MemoryPoolImpl::growCapacity(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #12 0x00007f3ee6e299f3 in facebook::velox::memory::MemoryPoolImpl::incrementReservationThreadSafe(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #13 0x00007f3ee6e299b9 in facebook::velox::memory::MemoryPoolImpl::incrementReservationThreadSafe(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so --Type <RET> for more, q to quit, c to continue without paging--c #14 0x00007f3ee6e299b9 in facebook::velox::memory::MemoryPoolImpl::incrementReservationThreadSafe(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #15 0x00007f3ee6e299b9 in facebook::velox::memory::MemoryPoolImpl::incrementReservationThreadSafe(facebook::velox::memory::MemoryPool*, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #16 0x00007f3ee6e2b66c in facebook::velox::memory::MemoryPoolImpl::reserveThreadSafe(unsigned long, bool) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #17 0x00007f3ee6e1c19d in facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function<void (unsigned long, bool)>, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #18 0x00007f3ee6e302a9 in facebook::velox::memory::MemoryPoolImpl::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, unsigned long) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #19 0x00007f3eea303d5b in facebook::velox::dwio::common::DirectCoalescedLoad::loadData(bool) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #20 0x00007f3ee6ddb026 in facebook::velox::cache::CoalescedLoad::loadOrFuture(folly::SemiFuture<bool>*, bool) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #21 0x00007f3eea302269 in void folly::detail::function::call_<facebook::velox::dwio::common::DirectBufferedInput::makeLoads(std::vector<facebook::velox::dwio::common::LoadRequest*, std::allocator<facebook::velox::dwio::common::LoadRequest*> >, bool)::{lambda()#8}, true, false, void>(, folly::detail::function::Data&) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #22 0x00007f3eea6b71f4 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7f3ee549f010) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/Function.h:370 #23 folly::catch_exception<folly::Function<void ()>&, void (&)(char const*) noexcept, char const*&, void>(folly::Function<void ()>&, void (&)(char const*) noexcept, char const*&) (c=<optimized out>, t=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/lang/Exception.h:290 #24 folly::Executor::invokeCatchingExns<folly::Function<void ()> >(char const*, folly::Function<void ()>) (f=..., p=0x7f3eeacbfa43 "ThreadPoolExecutor: func") at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/Executor.h:234 #25 folly::ThreadPoolExecutor::runTask (this=0x7f3f004e3380, thread=std::shared_ptr<folly::ThreadPoolExecutor::Thread> (use count 8, weak count 0) = {...}, task=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/executors/ThreadPoolExecutor.cpp:142 #26 0x00007f3eea6ad07b in operator() (__closure=0x7f3f025400a0) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/executors/IOThreadPoolExecutor.cpp:149 #27 folly::detail::function::call_<folly::IOThreadPoolExecutor::add(folly::Func, std::chrono::milliseconds, folly::Func)::<lambda()>, false, false, void>(folly::detail::function::Data &) (p=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/Function.h:341 #28 0x00007f3eea6f5e66 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7f3ee549f160) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/Function.h:368 #29 folly::EventBase::FuncRunner::operator()(folly::Function<void ()>&&) (func=..., this=0x7f3f05e055c0) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventBase.cpp:200 #30 folly::detail::invokeConsumerWithTask<folly::Function<void ()>, folly::EventBase::FuncRunner&, void>(folly::EventBase::FuncRunner&, folly::Function<void ()>&&, std::shared_ptr<folly::RequestContext>&&) (rctx=..., task=..., consumer=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/AtomicNotificationQueue-inl.h:254 #31 folly::AtomicNotificationQueue<folly::Function<void ()> >::drive<folly::EventBase::FuncRunner&>(folly::EventBase::FuncRunner&) (this=0x7f3f05e05500, consumer=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/AtomicNotificationQueue-inl.h:337 #32 0x00007f3eea6f764d in non-virtual thunk to folly::EventBaseAtomicNotificationQueue<folly::Function<void ()>, folly::EventBase::FuncRunner>::handlerReady(unsigned short) () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #33 0x00007f3eea6fe1e8 in folly::EventHandler::libeventCallback (fd=<optimized out>, events=<optimized out>, arg=0x7f3f05e05428) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventHandler.cpp:159 #34 0x00007f3eea851fb6 in event_process_active_single_queue () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #35 0x00007f3eea8529ef in event_base_loop () from /tmp/spark-1decd196-73b9-44d2-ac0a-6d376303fac7/executor-f24aeb44-8369-47b5-b6d4-971202f55a9e/gluten-9f3ff136-1ab4-4207-8126-f937a4438b27/jni/9d971c63-f9a3-4dd2-a99b-70f50853eccd/gluten-14997099487791574374/linux/amd64/libvelox.so #36 0x00007f3eea6f077e in (anonymous namespace)::EventBaseBackend::eb_event_base_loop (flags=1, this=<optimized out>) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventBase.cpp:99 #37 folly::EventBase::loopMain (this=0x7f3f05e69280, flags=<optimized out>, options=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventBase.cpp:603 #38 0x00007f3eea6f0e01 in folly::EventBase::loopBody (this=0x7f3f05e69280, flags=0, options=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventBase.cpp:513 #39 0x00007f3eea6f0e86 in folly::EventBase::loop (this=0x7f3f05e69280) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventBase.cpp:474 #40 0x00007f3eea6f2f99 in folly::EventBase::loopForever (this=0x7f3f05e69280) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/io/async/EventBase.cpp:781 #41 0x00007f3eea6ae08c in folly::IOThreadPoolExecutor::threadRun (this=0x7f3f004e3380, thread=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/executors/IOThreadPoolExecutor.cpp:240 #42 0x00007f3eea6bdf19 in std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__f=<optimized out>, __t=<optimized out>, __f=<optimized out>, __t=<optimized out>) at /opt/rh/devtoolset-11/root/usr/include/c++/11/bits/invoke.h:74 #43 std::__invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>) at /opt/rh/devtoolset-11/root/usr/include/c++/11/bits/invoke.h:96 #44 std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (__args=..., this=<optimized out>) at /opt/rh/devtoolset-11/root/usr/include/c++/11/functional:420 #45 std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::operator()<, void>() (this=<optimized out>) at /opt/rh/devtoolset-11/root/usr/include/c++/11/functional:503 #46 folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::function::Data&) (p=...) at /opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.07.01.00-4ae9f6791f.clean/folly/Function.h:341 #47 0x00007f3f98fba253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #48 0x00007f3f99285ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #49 0x00007f3f99316a04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 (gdb) ``` ### Spark version Spark-3.4.x ### Spark configurations _No response_ ### System information _No response_ ### Relevant logs ```bash ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
