[
https://issues.apache.org/jira/browse/IMPALA-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611760#comment-16611760
]
Antoni Ivanov commented on IMPALA-5463:
---------------------------------------
Thanks
The metric I've been using to monitor JVM memory is
jvm.total.peak-current-usage-bytes and jvm.total.current-usage-bytes
And they've been under 20G most of the time (and we've set -Xmx to 32G)
Still decided to double it to 64GB. And noticed that
jvm.ps-eden-space.peak-max-usage-bytes spiked to 21GB very fast and
jvm.ps-old-gen.peak-max-usage-bytes spiked to 43G. And CPU is at 100%
Other symptoms are
* I cannot open the Impala UI - _node:25000_
* And also cannot connect to with impala-shell to that node
* Warnings in logs like _"Missing tables were not received in 120000ms. Load
request will be retried."_
* The stack trace of highest cpu are similar (GC related) But there was also
one like
{quote}
Thread 2 (Thread 0x7ef2fdadc700 (LWP 9467)):
#0 0x00007f042a1e1a9b in recv () from /lib64/libpthread.so.0
#1 0x0000000001b0b3fd in apache::thrift::transport::TSocket::read(unsigned
char*, unsigned int) ()
#2 0x0000000001b0e663 in unsigned int
apache::thrift::transport::readAll<apache::thrift::transport::TSocket>(apache::thrift::transport::TSocket&,
unsigned char*, unsigned int) ()
#3 0x0000000000b5428d in
apache::thrift::transport::TSaslTransport::read(unsigned char*, unsigned int) ()
#4 0x0000000001b14e87 in
apache::thrift::transport::TBufferedTransport::readSlow(unsigned char*,
unsigned int) ()
#5 0x000000000081454e in unsigned int
apache::thrift::transport::readAll<apache::thrift::transport::TBufferBase>(apache::thrift::transport::TBufferBase&,
unsigned char*, unsigned int) ()
#6 0x00000000009ee2e1 in unsigned int
apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>::readStringBody<std::string>(std::string&,
int) ()
#7 0x00000000009ee5ce in
apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>,
apache::thrift::protocol::TProtocolDefaults>::readString_virt(std::string&) ()
#8 0x0000000000da3e26 in
impala::TTopicItem::read(apache::thrift::protocol::TProtocol*) ()
#9 0x0000000000da4898 in
impala::TTopicDelta::read(apache::thrift::protocol::TProtocol*) ()
#10 0x0000000000da6476 in
impala::TUpdateStateRequest::read(apache::thrift::protocol::TProtocol*) ()
#11 0x0000000000da8c9d in
impala::StatestoreSubscriber_UpdateState_args::read(apache::thrift::protocol::TProtocol*)
()
#12 0x0000000000daa3dc in
impala::StatestoreSubscriberProcessor::process_UpdateState(int,
apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*,
void*) ()
#13 0x0000000000da9774 in
impala::StatestoreSubscriberProcessor::dispatchCall(apache::thrift::protocol::TProtocol*,
apache::thrift::protocol::TProtocol*, std::string const&, int, void*) ()
{quote}
> OOM during clone() causes crash in libjvm.so!java_start()
> ---------------------------------------------------------
>
> Key: IMPALA-5463
> URL: https://issues.apache.org/jira/browse/IMPALA-5463
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.8.0
> Reporter: Lars Volker
> Priority: Critical
> Attachments: stack-trace-threads-high-cpu.txt
>
>
> Running out of memory seems to cause a crash in libjvm.so!java_start() right
> after calling clone(). Here is the stack trace of the crashing thread from a
> minidump.
> {noformat}
> 0 libjvm.so!PSParallelCompact::MarkAndPushClosure::do_oop(oopDesc**) + 0x86
> 1 libjvm.so!OopMapSet::all_do(frame const*, RegisterMap const*,
> OopClosure*, void (*)(oopDesc**, oopDesc**), OopClosure*) + 0x2fb
> 2 libjvm.so!frame::oops_do_internal(OopClosure*, CLDClosure*,
> CodeBlobClosure*, RegisterMap*, bool) + 0xa2
> 3 libjvm.so!JavaThread::oops_do(OopClosure*, CLDClosure*, CodeBlobClosure*)
> + 0x161
> 4 libjvm.so!ThreadRootsMarkingTask::do_it(GCTaskManager*, unsigned int) +
> 0x106
> 5 libjvm.so!GCTaskThread::run() + 0x12f
> 6 libjvm.so!java_start(Thread*) + 0x108
> 7 libpthread-2.12.so!start_thread + 0xd1
> 8 libc-2.12.so!clone + 0x6d
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
