stenlarsson opened a new issue, #35819:
URL: https://github.com/apache/arrow/issues/35819

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   We have some (complicated) Ruby code that joins a lot of tables, and it 
randomly crashes. Sometimes you get a stack trace, but sometimes the Ruby crash 
handler in turn crashes. This indicates that garbage has been written to the 
stack. I have also seen cases where it doesn't crash, but there is some random 
garbage in the result.
   
   I installed the `libarrow-acero1200-dbgsym` and `libarrow1200-dbgsym` 
packages in our Docker image, and ran Ruby through GDB. This way I'm 
(sometimes) able to capture a stack trace. Here is one example:
   
   ```
   #0  
arrow::compute::ExecBatchBuilder::Visit<arrow::compute::ExecBatchBuilder::AppendSelected(const
 std::shared_ptr<arrow::ArrayData>&, arrow::compute::ResizableArrayData*, int, 
const uint16_t*, arrow::MemoryPool*)::<lambda(int, const uint8_t*, uint32_t)> > 
(process_value_fn=..., row_ids=0xfffeed18d3b8, num_rows=1024, column=<optimized 
out>) at ./cpp/src/arrow/compute/light_array.cc:467
   #1  
arrow::compute::ExecBatchBuilder::AppendSelected(std::shared_ptr<arrow::ArrayData>
 const&, arrow::compute::ResizableArrayData*, int, unsigned short const*, 
arrow::MemoryPool*)
       (source=std::shared_ptr<arrow::ArrayData> (use count 2, weak count 0) = 
{...}, target=0xffff503d2ae0, num_rows_to_append=1024, row_ids=0xfffeed18d3b8, 
pool=<optimized out>) at ./cpp/src/arrow/compute/light_array.cc:578
   #2  0x0000ffff89ca4834 in 
arrow::compute::ExecBatchBuilder::AppendSelected(arrow::MemoryPool*, 
arrow::compute::ExecBatch const&, int, unsigned short const*, int, int const*) 
(this=this@entry=0xaaab2cb144d0, pool=
       0xffff8b339360 <arrow::global_state+88>, batch=..., 
num_rows_to_append=num_rows_to_append@entry=1024, row_ids=0xfffeed18d3b8, 
num_cols=<optimized out>, col_ids=0xaaab2cb15550) at 
/usr/include/c++/10/bits/stl_vector.h:1043
   #3  0x0000ffff89381858 in 
arrow::acero::JoinResultMaterialize::AppendProbeOnly(arrow::compute::ExecBatch 
const&, int, unsigned short const*, int*)
       (this=0xaaab2cb14480, key_and_payload=<optimized out>, 
num_rows_to_append=1024, row_ids=<optimized out>, 
num_rows_appended=0xffff80d44124) at ./cpp/src/arrow/acero/swiss_join.cc:1587
   #4  0x0000ffff89382434 in operator() (num_rows_appended=0xffff80d44124, 
offset=0, num_rows_to_append_left=1024, this=<optimized out>) at 
./cpp/src/arrow/acero/swiss_join_internal.h:586
   #5  
arrow::acero::JoinResultMaterialize::AppendAndOutput<arrow::acero::JoinResultMaterialize::AppendProbeOnly<arrow::acero::JoinProbeProcessor::OnNextBatch(int64_t,
 const arrow::compute::ExecBatch&, arrow::util::TempVectorStack*, 
std::vector<arrow::compute::KeyColumnArray>*)::<lambda(arrow::compute::ExecBatch)>
 >::<lambda(int, int, int*)>, 
arrow::acero::JoinProbeProcessor::OnNextBatch(int64_t, const 
arrow::compute::ExecBatch&, arrow::util::TempVectorStack*, 
std::vector<arrow::compute::KeyColumnArray>*)::<lambda(arrow::compute::ExecBatch)>
 > (output_batch_fn=<synthetic pointer>..., append_rows_fn=<optimized out>, 
num_rows_to_append=1024, this=0xaaab2cb14480) at 
./cpp/src/arrow/acero/swiss_join_internal.h:567
   #6  
arrow::acero::JoinResultMaterialize::AppendProbeOnly<arrow::acero::JoinProbeProcessor::OnNextBatch(int64_t,
 const arrow::compute::ExecBatch&, arrow::util::TempVectorStack*, 
std::vector<arrow::compute::KeyColumnArray>*)::<lambda(arrow::compute::ExecBatch)>
 > (output_batch_fn=..., row_ids=<optimized out>, num_rows_to_append=1024, 
key_and_payload=..., this=0xaaab2cb14480) at 
./cpp/src/arrow/acero/swiss_join_internal.h:590
   #7  arrow::acero::JoinProbeProcessor::OnNextBatch(long, 
arrow::compute::ExecBatch const&, arrow::util::TempVectorStack*, 
std::vector<arrow::compute::KeyColumnArray, 
std::allocator<arrow::compute::KeyColumnArray> >*)
       (this=0xaaab2cae23f0, thread_id=4, keypayload_batch=..., 
temp_stack=0xaaab2cb13f68, temp_column_arrays=0xaaab2cb14538) at 
./cpp/src/arrow/acero/swiss_join.cc:2019
   #8  0x0000ffff893886f0 in arrow::acero::SwissJoin::ProbeSingleBatch(unsigned 
long, arrow::compute::ExecBatch) (this=0xaaab2cae1d10, thread_index=4, 
batch=...) at ./cpp/src/arrow/acero/swiss_join.cc:2145
   #9  0x0000ffff8934be7c in 
arrow::acero::HashJoinNode::OnProbeSideBatch(unsigned long, 
arrow::compute::ExecBatch) (this=this@entry=0xaaab2ca11800, 
thread_index=thread_index@entry=4, batch=...) at 
./cpp/src/arrow/acero/hash_join_node.cc:818
   #10 0x0000ffff8934c284 in 
arrow::acero::HashJoinNode::InputReceived(arrow::acero::ExecNode*, 
arrow::compute::ExecBatch) (this=0xaaab2ca11800, input=0xaaab2ca10ae0, 
batch=...) at ./cpp/src/arrow/acero/hash_join_node.cc:891
   #11 0x0000ffff893716f4 in operator()() const (__closure=0xffff3c51bd20) at 
./cpp/src/arrow/acero/source_node.cc:119
   #12 0x0000ffff893718d0 in std::__invoke_impl<arrow::Status, 
arrow::acero::(anonymous namespace)::SourceNode::SliceAndDeliverMorsel(const 
arrow::compute::ExecBatch&)::<lambda()>&> (__f=<optimized out>) at 
/usr/include/c++/10/bits/invoke.h:59
   #13 std::__invoke_r<arrow::Status, arrow::acero::(anonymous 
namespace)::SourceNode::SliceAndDeliverMorsel(const 
arrow::compute::ExecBatch&)::<lambda()>&> (__fn=<optimized out>) at 
/usr/include/c++/10/bits/invoke.h:115
   #14 std::_Function_handler<arrow::Status(), arrow::acero::(anonymous 
namespace)::SourceNode::SliceAndDeliverMorsel(const 
arrow::compute::ExecBatch&)::<lambda()> >::_M_invoke(const std::_Any_data &) 
(__functor=<optimized out>)
       at /usr/include/c++/10/bits/std_function.h:292
   #15 0x0000ffff89365fd8 in std::function<arrow::Status ()>::operator()() 
const (this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
   #16 arrow::detail::ContinueFuture::operator()<std::function<arrow::Status 
()>&, , arrow::Status, arrow::Future<arrow::internal::Empty> 
>(arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>&) const
       (this=<optimized out>, f=<optimized out>, next=...) at 
./cpp/src/arrow/util/future.h:150
   #17 std::__invoke_impl<void, arrow::detail::ContinueFuture&, 
arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status 
()>&>(std::__invoke_other, arrow::detail::ContinueFuture&, 
arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status ()>&) 
(__f=<optimized out>) at /usr/include/c++/10/bits/invoke.h:60
   #18 std::__invoke<arrow::detail::ContinueFuture&, 
arrow::Future<arrow::internal::Empty>&, std::function<arrow::Status 
()>&>(arrow::detail::ContinueFuture&, arrow::Future<arrow::internal::Empty>&, 
std::function<arrow::Status ()>&)
       (__fn=<optimized out>) at /usr/include/c++/10/bits/invoke.h:95
   #19 std::_Bind<arrow::detail::ContinueFuture 
(arrow::Future<arrow::internal::Empty>, std::function<arrow::Status 
()>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) 
(__args=<optimized out>, this=<optimized out>)
       at /usr/include/c++/10/functional:416
   #20 std::_Bind<arrow::detail::ContinueFuture 
(arrow::Future<arrow::internal::Empty>, std::function<arrow::Status 
()>)>::operator()<, void>() (this=<optimized out>) at 
/usr/include/c++/10/functional:499
   #21 arrow::internal::FnOnce<void 
()>::FnImpl<std::_Bind<arrow::detail::ContinueFuture 
(arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>)> 
>::invoke() (this=<optimized out>) at ./cpp/src/arrow/util/functional.h:152
   #22 0x0000ffff89b3cf38 in arrow::internal::FnOnce<void ()>::operator()() && 
(this=0xffff80d44968) at /usr/include/c++/10/bits/unique_ptr.h:172
   #23 arrow::internal::WorkerLoop (it=Python Exception <class 'gdb.error'> 
value has been optimized out:
   {_M_id = {_M_thread = 281472843141584}}, state=) at 
./cpp/src/arrow/util/thread_pool.cc:269
   #24 operator() (__closure=<optimized out>) at 
./cpp/src/arrow/util/thread_pool.cc:430
   #25 std::__invoke_impl<void, 
arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()> > 
(__f=<optimized out>) at /usr/include/c++/10/bits/invoke.h:60
   #26 
std::__invoke<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()>
 > (__fn=<optimized out>) at /usr/include/c++/10/bits/invoke.h:95
   #27 
std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()>
 > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/10/thread:264
   #28 
std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()>
 > >::operator() (this=<optimized out>) at /usr/include/c++/10/thread:271
   #29 
std::thread::_State_impl<std::thread::_Invoker<std::tuple<arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::<lambda()>
 > > >::_M_run(void) (this=<optimized out>) at /usr/include/c++/10/thread:215
   #30 0x0000ffff89164cac in  () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
   #31 0x0000ffff9018f648 in start_thread (arg=0xffff80d44ad0) at 
pthread_create.c:477
   #32 0x0000ffff8ff3dfdc in thread_start () at 
../sysdeps/unix/sysv/linux/aarch64/clone.S:78
   ```
   
   Because this is so random I have unfortunately not been able to create a 
small reproducible test that I can share with you. I tried setting 
`OMP_NUM_THREADS=1`, but I couldn't get it to crash. Please let me know if 
there is anything more I can do to troubleshoot this.
   
   This problem does not happen in Arrow 10, but I have been able to reproduce 
it in both Arrow 11 and 12, on both amd64 and arm64, and on both macOS and 
Linux.
   
   ### Component(s)
   
   Ruby


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to