zhztheplayer commented on issue #7243:
URL:
https://github.com/apache/incubator-gluten/issues/7243#issuecomment-2406916997
Reopening. Observed in a local TPC benchmark where Velox tasks are usually
broken into several in one Spark task
Error:
```
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f044de29e3e in folly::detail::(anonymous
namespace)::nativeFutexWaitImpl (waitMask=4294967295, absSteadyTime=0x0,
absSystemTime=0x0, expected=4294967293, addr=0x7f05d4d037f8)
at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex.cpp:126
#2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f05d4d037f8,
expected=expected@entry=4294967293, absSystemTime=absSystemTime@entry=0x0,
absSteadyTime=absSteadyTime@entry=0x0,
waitMask=waitMask@entry=4294967295) at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex.cpp:254
#3 0x00007f044de71416 in folly::detail::futexWait<std::atomic<unsigned int>
> (waitMask=4294967295, expected=4294967293, futex=0x7f05d4d037f8)
at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/Futex-inl.h:94
#4 folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>,
std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
(expected=4294967293, waitMask=4294967295, stackToRetain=1024,
timeoutVariationFrac=0.5, idleTimeout=..., fut=std::atomic<unsigned int>
= { 4294967293 })
at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/detail/MemoryIdler.h:128
#5 folly::fibers::Baton::waitThread (this=0x7f05d4d037f8) at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton.cpp:70
#6 0x00007f044de716e0 in
folly::fibers::Baton::wait<folly::fibers::Baton::wait()::<lambda()> >
(mainContextFunc=..., this=0x7f05d4d037f8)
at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton-inl.h:54
#7 folly::fibers::Baton::wait (this=this@entry=0x7f05d4d037f8) at
/opt/gluten/dev/vcpkg/.vcpkg/buildtrees/folly/src/4.05.20.00-935b4335b7.clean/folly/fibers/Baton.cpp:46
#8 0x00007f044a513abe in
folly::futures::detail::waitImpl<folly::SemiFuture<folly::Unit>, folly::Unit>
(f=...)
at
/opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2230
#9 0x00007f044a513f90 in folly::SemiFuture<folly::Unit>::wait() &
(this=0x7f05d4d03928) at
/opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2337
#10 0x00007f044b39aa48 in folly::SemiFuture<folly::Unit>::wait() &&
(this=0x7f05d4d03928) at
/opt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/folly/futures/Future-inl.h:2343
#11 facebook::velox::exec::Task::MemoryReclaimer::reclaimTask
(this=0x7f04807707b0, task=std::shared_ptr<facebook::velox::exec::Task> (use
count 6, weak count 3) = {...}, targetBytes=18035507,
maxWaitMs=0, stats=...) at
/opt/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:2996
#12 0x00007f044b39b0d1 in
facebook::velox::exec::Task::MemoryReclaimer::reclaim (this=0x7f04807707b0,
pool=0x7f02a670f450, targetBytes=18035507, maxWaitMs=0, stats=...)
at /opt/gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:2970
#13 0x00007f044a70a4f9 in facebook::velox::memory::MemoryReclaimer::reclaim
(this=<optimized out>, pool=<optimized out>, targetBytes=<optimized out>,
maxWaitMs=0, stats=...)
at
/opt/gluten/ep/build-velox/build/velox_ep/velox/common/memory/MemoryArbitrator.cpp:266
#14 0x00007f044a539dc5 in gluten::ListenableArbitrator::shrinkCapacity
(this=0x7f04810b4010, targetBytes=18035507, allowSpill=<optimized out>,
allowAbort=<optimized out>)
at /opt/gluten/cpp/velox/memory/VeloxMemoryManager.cc:122
#15 0x00007f044a51c285 in gluten::WholeStageResultIterator::spillFixedSize
(this=0x7f04803627c0, size=<optimized out>) at
/opt/gluten/cpp/velox/compute/WholeStageResultIterator.cc:250
#16 0x00007f05ce621c35 in gluten::ResultIterator::spillFixedSize
(size=18035507, this=0x7f048044b380) at
/opt/gluten/cpp/core/compute/ResultIterator.h:71
#17 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeSpill
(env=0x7f046860aa60, wrapper=<optimized out>, iterHandle=75136157876230,
size=18035507)
at /opt/gluten/cpp/core/jni/JniWrapper.cc:516
#18 0x00007f06dd018427 in ?? ()
#19 0x0000000772211e98 in ?? ()
#20 0x4908b36b977f1f00 in ?? ()
#21 0x0000000600000001 in ?? ()
#22 0x00007f05cce302e8 in ?? ()
#23 0x00007f05d4d03fe8 in ?? ()
#24 0x00007f05d4d03f78 in ?? ()
#25 0x0000000000000000 in ?? ()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
