[ https://issues.apache.org/jira/browse/IMPALA-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611760#comment-16611760 ]
Antoni Ivanov commented on IMPALA-5463: --------------------------------------- Thanks The metric I've been using to monitor JVM memory is jvm.total.peak-current-usage-bytes and jvm.total.current-usage-bytes And they've been under 20G most of the time (and we've set -Xmx to 32G) Still decided to double it to 64GB. And noticed that jvm.ps-eden-space.peak-max-usage-bytes spiked to 21GB very fast and jvm.ps-old-gen.peak-max-usage-bytes spiked to 43G. And CPU is at 100% Other symptoms are * I cannot open the Impala UI - _node:25000_ * And also cannot connect to with impala-shell to that node * Warnings in logs like _"Missing tables were not received in 120000ms. Load request will be retried."_ * The stack trace of highest cpu are similar (GC related) But there was also one like {quote} Thread 2 (Thread 0x7ef2fdadc700 (LWP 9467)): #0 0x00007f042a1e1a9b in recv () from /lib64/libpthread.so.0 #1 0x0000000001b0b3fd in apache::thrift::transport::TSocket::read(unsigned char*, unsigned int) () #2 0x0000000001b0e663 in unsigned int apache::thrift::transport::readAll<apache::thrift::transport::TSocket>(apache::thrift::transport::TSocket&, unsigned char*, unsigned int) () #3 0x0000000000b5428d in apache::thrift::transport::TSaslTransport::read(unsigned char*, unsigned int) () #4 0x0000000001b14e87 in apache::thrift::transport::TBufferedTransport::readSlow(unsigned char*, unsigned int) () #5 0x000000000081454e in unsigned int apache::thrift::transport::readAll<apache::thrift::transport::TBufferBase>(apache::thrift::transport::TBufferBase&, unsigned char*, unsigned int) () #6 0x00000000009ee2e1 in unsigned int apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>::readStringBody<std::string>(std::string&, int) () #7 0x00000000009ee5ce in apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>, apache::thrift::protocol::TProtocolDefaults>::readString_virt(std::string&) () #8 0x0000000000da3e26 in impala::TTopicItem::read(apache::thrift::protocol::TProtocol*) () #9 0x0000000000da4898 in impala::TTopicDelta::read(apache::thrift::protocol::TProtocol*) () #10 0x0000000000da6476 in impala::TUpdateStateRequest::read(apache::thrift::protocol::TProtocol*) () #11 0x0000000000da8c9d in impala::StatestoreSubscriber_UpdateState_args::read(apache::thrift::protocol::TProtocol*) () #12 0x0000000000daa3dc in impala::StatestoreSubscriberProcessor::process_UpdateState(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) () #13 0x0000000000da9774 in impala::StatestoreSubscriberProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::string const&, int, void*) () {quote} > OOM during clone() causes crash in libjvm.so!java_start() > --------------------------------------------------------- > > Key: IMPALA-5463 > URL: https://issues.apache.org/jira/browse/IMPALA-5463 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 2.8.0 > Reporter: Lars Volker > Priority: Critical > Attachments: stack-trace-threads-high-cpu.txt > > > Running out of memory seems to cause a crash in libjvm.so!java_start() right > after calling clone(). Here is the stack trace of the crashing thread from a > minidump. > {noformat} > 0 libjvm.so!PSParallelCompact::MarkAndPushClosure::do_oop(oopDesc**) + 0x86 > 1 libjvm.so!OopMapSet::all_do(frame const*, RegisterMap const*, > OopClosure*, void (*)(oopDesc**, oopDesc**), OopClosure*) + 0x2fb > 2 libjvm.so!frame::oops_do_internal(OopClosure*, CLDClosure*, > CodeBlobClosure*, RegisterMap*, bool) + 0xa2 > 3 libjvm.so!JavaThread::oops_do(OopClosure*, CLDClosure*, CodeBlobClosure*) > + 0x161 > 4 libjvm.so!ThreadRootsMarkingTask::do_it(GCTaskManager*, unsigned int) + > 0x106 > 5 libjvm.so!GCTaskThread::run() + 0x12f > 6 libjvm.so!java_start(Thread*) + 0x108 > 7 libpthread-2.12.so!start_thread + 0xd1 > 8 libc-2.12.so!clone + 0x6d > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org