[
https://issues.apache.org/jira/browse/KUDU-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655933#comment-15655933
]
Matthew Jacobs commented on KUDU-1745:
--------------------------------------
I found a core on kudu-stress-4.vpc.cloudera.com, though unfortunately I can't
get debugging symbols. Unless something is broken with the core, I don't see
Impala calling the client, so maybe a Kudu client thread (it does async work on
its own threads, right)?
{code}
(gdb) bt
#0 0x00000030e2832625 in raise () from /lib64/libc.so.6
#1 0x00000030e2833e05 in abort () from /lib64/libc.so.6
#2 0x00007f1cdfdd5a55 in os::abort(bool) () from
/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3 0x00007f1cdff55f87 in VMError::report_and_die() () from
/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4 0x00007f1cdfdda96f in JVM_handle_linux_signal () from
/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5 <signal handler called>
#6 0x00007f1cded39018 in std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::basic_string(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libstdc++.so.6
#7 0x00007f1cdeff5b90 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#8 0x00007f1cdeff6662 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#9 0x00007f1cdefd3872 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#10 0x00007f1cdefd39d6 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#11 0x00007f1cdeff7775 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#12 0x00007f1cdeffc4d9 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#13 0x00007f1cdf050be0 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#14 0x00007f1cdf050eb2 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#15 0x00007f1cdf05373d in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#16 0x00007f1cdf045405 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#17 0x00007f1cdf0581b6 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#18 0x00007f1cdf1e5c1b in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#19 0x00007f1cdf1e9324 in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#20 0x00007f1cdf0410fb in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#21 0x00007f1cdf10fcea in ?? () from
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.235/lib/impala/lib/libkudu_client.so.0
#22 0x00000030e2c079d1 in start_thread () from /lib64/libpthread.so.0
#23 0x00000030e28e88fd in clone () from /lib64/libc.so.6
{code}
I also have a minidump that I decoded (attached stacks.out), but it more or
less shows the same thing.
Given that the output from the hs_err jvm dump shows this is in KuduError and
KuduUpsert's destructor, I wonder if the operations could be mismanaged (i.e.
not deleted properly) when living in the KuduError. It's possible that Impala
isn't doing something correctly with the KuduUpserts on another thread, but I
think our interaction with them is minimal.
> Kudu causes Impala to crash under stress
> ----------------------------------------
>
> Key: KUDU-1745
> URL: https://issues.apache.org/jira/browse/KUDU-1745
> Project: Kudu
> Issue Type: Bug
> Reporter: Taras Bobrovytsky
> Priority: Critical
> Attachments: hs_err_pid7761.log, stacks.out
>
>
> There were over 200 queries running, about half of which were selects and the
> rest were upsert and delete queries.
> There was a crash after a few minutes with the following stack trace:
> {code}
> Stack: [0x00007f1629c93000,0x00007f162a694000], sp=0x00007f162a6922b0, free
> space=10236k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [libstdc++.so.6+0xc5018] std::basic_string<char, std::char_traits<char>,
> std::allocator<char> >::basic_string(std::string const&)+0x8
> C [libkudu_client.so.0+0x6f662]
> _ZNSt6vectorIPN4kudu6client9KuduErrorESaIS3_EE19_M_emplace_back_auxIIS3_EEEvDpOT_+0x1dd2
> C [libkudu_client.so.0+0x4c872] _init+0xd022
> C [libkudu_client.so.0+0x4c9d6] _init+0xd186
> C [libkudu_client.so.0+0x70775]
> _ZNSt6vectorIPN4kudu6client9KuduErrorESaIS3_EE19_M_emplace_back_auxIIS3_EEEvDpOT_+0x2ee5
> C [libkudu_client.so.0+0x754d9]
> _ZNSt6vectorIPN4kudu6client9KuduErrorESaIS3_EE19_M_emplace_back_auxIIS3_EEEvDpOT_+0x7c49
> C [libkudu_client.so.0+0xc9be0]
> kudu::client::KuduUpsert::~KuduUpsert()+0x2ae30
> C [libkudu_client.so.0+0xc9eb2]
> kudu::client::KuduUpsert::~KuduUpsert()+0x2b102
> C [libkudu_client.so.0+0xcc73d]
> _ZNSt6vectorIN4kudu5SliceESaIS1_EE19_M_emplace_back_auxIIS1_EEEvDpOT_+0x123d
> C [libkudu_client.so.0+0xbe405]
> kudu::client::KuduUpsert::~KuduUpsert()+0x1f655
> C [libkudu_client.so.0+0xcdc0c]
> _ZNSt6vectorIN4kudu5SliceESaIS1_EE19_M_emplace_back_auxIIS1_EEEvDpOT_+0x270c
> C [libkudu_client.so.0+0x25ec1b]
> _ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE22_M_emplace_hint_uniqueIJRKSt21piecewise_construct_tSt5tupleIJRS1_EESD_IJEEEEESt17_Rb_tree_iteratorIS2_ESt23_Rb_tree_const_iteratorIS2_EDpOT_+0x311b
> C [libkudu_client.so.0+0x262324]
> _ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE22_M_emplace_hint_uniqueIJRKSt21piecewise_construct_tSt5tupleIJRS1_EESD_IJEEEEESt17_Rb_tree_iteratorIS2_ESt23_Rb_tree_const_iteratorIS2_EDpOT_+0x6824
> T_+0x558a
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
