[
https://issues.apache.org/jira/browse/KUDU-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664426#comment-15664426
]
Todd Lipcon commented on KUDU-1745:
-----------------------------------
I'm not following the OOM theory -- looks like there was 5GB in buffer cache at
the time of the crash, so it woudl have purged some of that before refusing a
malloc. I also think an exception would propagate back and cause
std::terminate() to be called, but I don't see that in the stack anywhere.
As for the stack itself, it doesn't make any sense still:
{code}
0 libstdc++.so.6!std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::basic_string(std::string const&) + 0x8
1 libkudu_client.so.0!void std::vector<kudu::client::KuduError*,
std::allocator<kudu::client::KuduError*>
>::_M_emplace_back_aux<kudu::client::KuduError*>(kudu::client::KuduError*&&) +
0x1300
2 libkudu_client.so.0!void std::vector<kudu::client::KuduError*,
std::allocator<kudu::client::KuduError*>
>::_M_emplace_back_aux<kudu::client::KuduError*>(kudu::client::KuduError*&&) +
0x1dd2
3 libkudu_client.so.0!_init + 0xd022
4 libkudu_client.so.0!_init + 0xd186
5 libkudu_client.so.0!void std::vector<kudu::client::KuduError*,
std::allocator<kudu::client::KuduError*>
>::_M_emplace_back_aux<kudu::client::KuduError*>(kudu::client::KuduError*&&) +
0x2ee5
6 libkudu_client.so.0!void std::vector<kudu::client::KuduError*,
std::allocator<kudu::client::KuduError*>
>::_M_emplace_back_aux<kudu::client::KuduError*>(kudu::client::KuduError*&&) +
0x7c49
7 libkudu_client.so.0!kudu::client::KuduUpsert::~KuduUpsert() + 0x2ae30
8 libkudu_client.so.0!kudu::client::KuduUpsert::~KuduUpsert() + 0x2b102
9 libkudu_client.so.0!void std::vector<kudu::Slice,
std::allocator<kudu::Slice> >::_M_emplace_back_aux<kudu::Slice>(kudu::Slice&&)
+ 0x123d
10 libkudu_client.so.0!kudu::client::KuduUpsert::~KuduUpsert() + 0x1f655
11 libkudu_client.so.0!void std::vector<kudu::Slice,
std::allocator<kudu::Slice> >::_M_emplace_back_aux<kudu::Slice>(kudu::Slice&&)
+ 0x5cb6
12 libkudu_client.so.0!std::_Rb_tree_iterator<std::pair<std::string const,
std::string> > std::_Rb_tree<std::string, std::pair<std::string const,
std::string>, std::_Select1st<std::pair<std::string const, std::string> >,
std::less<std::string>, std::allocator<std::pair<std::string const,
std::string> >
13 libkudu_client.so.0!std::_Rb_tree_iterator<std::pair<std::string const,
std::string> > std::_Rb_tree<std::string, std::pair<std::string const,
std::string>, std::_Select1st<std::pair<std::string const, std::string> >,
std::less<std::string>, std::allocator<std::pair<std::string const,
std::string> >
14 libkudu_client.so.0!kudu::client::KuduUpsert::~KuduUpsert() + 0x1b34b
15 libkudu_client.so.0!void std::vector<char*, std::allocator<char*>
>::_M_emplace_back_aux<char*>(char*&&) + 0x558a
{code}
note that this shows std::vector<char*>::emplace_back calling a KuduUpsert
destructor, which is definitely nonsense. The rest of the stack makes just as
little sense (eg the KuduUpsert destructor claims to be calling
vector<KuduError*>::emplace_back, which itself calls the shared library's
constructor _init?
You think there is any possibility that the shared library got replaced in the
middle of the run or something? If you can reproduce, can you reproduce using a
kudu client that doesn't have debug symbols stripped? (I'm not sure if the
toolchain builds strip debug symbols, but we do enable them by default, even in
release builds)
> Kudu causes Impala to crash under stress
> ----------------------------------------
>
> Key: KUDU-1745
> URL: https://issues.apache.org/jira/browse/KUDU-1745
> Project: Kudu
> Issue Type: Bug
> Reporter: Taras Bobrovytsky
> Priority: Critical
> Attachments: hs_err_pid7761.log, stacks.out
>
>
> There were over 200 queries running, about half of which were selects and the
> rest were upsert and delete queries.
> There was a crash after a few minutes with the following stack trace:
> {code}
> Stack: [0x00007f1629c93000,0x00007f162a694000], sp=0x00007f162a6922b0, free
> space=10236k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [libstdc++.so.6+0xc5018] std::basic_string<char, std::char_traits<char>,
> std::allocator<char> >::basic_string(std::string const&)+0x8
> C [libkudu_client.so.0+0x6f662]
> _ZNSt6vectorIPN4kudu6client9KuduErrorESaIS3_EE19_M_emplace_back_auxIIS3_EEEvDpOT_+0x1dd2
> C [libkudu_client.so.0+0x4c872] _init+0xd022
> C [libkudu_client.so.0+0x4c9d6] _init+0xd186
> C [libkudu_client.so.0+0x70775]
> _ZNSt6vectorIPN4kudu6client9KuduErrorESaIS3_EE19_M_emplace_back_auxIIS3_EEEvDpOT_+0x2ee5
> C [libkudu_client.so.0+0x754d9]
> _ZNSt6vectorIPN4kudu6client9KuduErrorESaIS3_EE19_M_emplace_back_auxIIS3_EEEvDpOT_+0x7c49
> C [libkudu_client.so.0+0xc9be0]
> kudu::client::KuduUpsert::~KuduUpsert()+0x2ae30
> C [libkudu_client.so.0+0xc9eb2]
> kudu::client::KuduUpsert::~KuduUpsert()+0x2b102
> C [libkudu_client.so.0+0xcc73d]
> _ZNSt6vectorIN4kudu5SliceESaIS1_EE19_M_emplace_back_auxIIS1_EEEvDpOT_+0x123d
> C [libkudu_client.so.0+0xbe405]
> kudu::client::KuduUpsert::~KuduUpsert()+0x1f655
> C [libkudu_client.so.0+0xcdc0c]
> _ZNSt6vectorIN4kudu5SliceESaIS1_EE19_M_emplace_back_auxIIS1_EEEvDpOT_+0x270c
> C [libkudu_client.so.0+0x25ec1b]
> _ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE22_M_emplace_hint_uniqueIJRKSt21piecewise_construct_tSt5tupleIJRS1_EESD_IJEEEEESt17_Rb_tree_iteratorIS2_ESt23_Rb_tree_const_iteratorIS2_EDpOT_+0x311b
> C [libkudu_client.so.0+0x262324]
> _ZNSt8_Rb_treeISsSt4pairIKSsSsESt10_Select1stIS2_ESt4lessISsESaIS2_EE22_M_emplace_hint_uniqueIJRKSt21piecewise_construct_tSt5tupleIJRS1_EESD_IJEEEEESt17_Rb_tree_iteratorIS2_ESt23_Rb_tree_const_iteratorIS2_EDpOT_+0x6824
> T_+0x558a
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
