[
https://issues.apache.org/jira/browse/IMPALA-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17233703#comment-17233703
]
Wenzhe Zhou commented on IMPALA-10276:
--------------------------------------
Hit this DCHECK error in verification build with following stack:
Thread 430 (crashed)
0 libc-2.23.so + 0x35438
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0x00007f114b4cb438 rbx = 0x0000000000000004
rsi = 0x000000000000356d rdi = 0x000000000001b515
rbp = 0x00007f1054dd8e30 rsp = 0x00007f1054dd8ab8
r8 = 0x0000000000000000 r9 = 0x0000000000000020
r10 = 0x0000000000000008 r11 = 0x0000000000000202
r12 = 0x00000000076f56e0 r13 = 0x000000000000007e
r14 = 0x00000000076f56e4 r15 = 0x00000000076edd00
rip = 0x00007f114b4cb438
Found by: given as instruction pointer in context
1 impalad!google::LogMessage::Flush() + 0x1eb
rbp = 0x00007f1054dd8f10 rsp = 0x00007f1054dd8e40
rip = 0x0000000005247dbb
Found by: previous frame's frame pointer
2 impalad!google::LogMessageFatal::~LogMessageFatal() + 0x9
rbx = 0x0000000000000001 rbp = 0x00007f1054dd8f40
rsp = 0x00007f1054dd8ee0 r12 = 0x00000000076fcc18
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x000000000524b9b9
Found by: call frame info
3 impalad!impala::RuntimeProfileBase::total_time_counter() const
[*runtime-profile.h : 181* + 0xc]
rbx = 0x0000000000000001 rbp = 0x00007f1054dd8f40
rsp = 0x00007f1054dd8ef0 r12 = 0x00000000076fcc18
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x00000000022e3cee
Found by: call frame info
4 impalad!impala::RuntimeProfileBase::ComputeTimeInProfile()
[runtime-profile.cc : 578 + 0xf]
rbx = 0x0000000000000001 rbp = 0x00007f1054dd9020
rsp = 0x00007f1054dd8f50 r12 = 0x00000000cf28b980
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x0000000002711ca5
Found by: call frame info
5 impalad!impala::RuntimeProfileBase::ComputeTimeInProfile()
[runtime-profile.cc : 571 + 0x29]
rbx = 0x0000000000000001 rbp = 0x00007f1054dd9100
rsp = 0x00007f1054dd9030 r12 = 0x00000000cf28b980
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x0000000002711c41
Found by: call frame info
6 impalad!impala::RuntimeProfileBase::ComputeTimeInProfile()
[runtime-profile.cc : 571 + 0x29]
rbx = 0x0000000000000002 rbp = 0x00007f1054dd91e0
rsp = 0x00007f1054dd9110 r12 = 0x00000000cf28b980
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x0000000002711c41
Found by: call frame info
7
impalad!impala::FragmentInstanceState::GetStatusReport(impala::FragmentInstanceExecStatusPB*,
impala::TRuntimeProfileTree*) [fragment-instance-state.cc : 292 + 0x17]
rbx = 0x000000001261da40 rbp = 0x00007f1054dd9480
rsp = 0x00007f1054dd91f0 r12 = 0x00000000cf28b980
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x00000000023485e5
Found by: call frame info
8 impalad!impala::QueryState::ConstructReport(bool,
impala::ReportExecStatusRequestPB*, impala::TRuntimeProfileForest*)
[query-state.cc : 548 + 0x29]
rbx = 0x000000003db11620 rbp = 0x00007f1054dd9580
rsp = 0x00007f1054dd9490 r12 = 0x0000000000000001
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x00000000022ae6a2
Found by: call frame info
9 impalad!impala::QueryState::ReportExecStatus() [query-state.cc : 591 + 0x21]
rbx = 0x0000000000000001 rbp = 0x00007f1054dd9d40
rsp = 0x00007f1054dd9590 r12 = 0x0000000000000001
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x00000000022aea11
Found by: call frame info
10 impalad!impala::QueryState::UpdateBackendExecState() [query-state.cc : 465 +
0xf]
rbx = 0x0000000000000001 rbp = 0x00007f1054dd9e70
rsp = 0x00007f1054dd9d50 r12 = 0x0000000000000001
r13 = 0x0000000000000001 r14 = 0x0000000000000001
r15 = 0x000000003b1c5538 rip = 0x00000000022ada61
Found by: call frame info
11 impalad!impala::QueryState::MonitorFInstances() [query-state.cc : 865 + 0xf]
rbx = 0x0000000000000000 rbp = 0x00007f1054dd9f60
rsp = 0x00007f1054dd9e80 r12 = 0x000000003a6f0b80
r13 = 0x00007f10ef93f790 r14 = 0x000000000e193360
r15 = 0x000000000dff9090 rip = 0x00000000022b1598
Found by: call frame info
12 impalad!impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*)
[query-exec-mgr.cc : 162 + 0xf]
rbx = 0x000000003db11600 rbp = 0x00007f1054dda7b0
rsp = 0x00007f1054dd9f70 r12 = 0x000000003a6f0b80
r13 = 0x00007f10ef93f790 r14 = 0x000000000e193360
r15 = 0x000000000dff9090 rip = 0x00000000022a0f79
Found by: call frame info
13 impalad!boost::_mfi::mf1<void, impala::QueryExecMgr,
impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*)
const [mem_fn_template.hpp : 165 + 0xc]
rbx = 0x000000003db11600 rbp = 0x00007f1054dda7e0
rsp = 0x00007f1054dda7c0 r12 = 0x000000003a6f0b80
r13 = 0x00007f10ef93f790 r14 = 0x000000000e193360
r15 = 0x000000000dff9090 rip = 0x00000000022a988d
Found by: call frame info
> Release build sees SIGSEGV when updating the total time counter
> ---------------------------------------------------------------
>
> Key: IMPALA-10276
> URL: https://issues.apache.org/jira/browse/IMPALA-10276
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0
> Reporter: Joe McDonnell
> Priority: Critical
> Labels: broken-build, flaky
>
> A recent release build saw an Impalad crash with the following stack:
> {noformat}
> Crash reason: SIGSEGV
> Crash address: 0x11
> #0 raise () from /lib64/libc.so.6
> #1 abort () from /lib64/libc.so.6
> #2 os::abort(bool) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #3 VMError::report_and_die() () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #4 JVM_handle_linux_signal () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #5 signalHandler(int, siginfo*, void*) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #6 <signal handler called>
> #7 UpdateCounter (this=0x7f93adc88030) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/util/runtime-profile-counters.h:933
> #8 impala::ScopedTimer<impala::MonotonicStopWatch>::~ScopedTimer
> (this=0x7f93adc88030, __in_chrg=<optimized out>) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/util/runtime-profile-counters.h:954
> #9 impala::Coordinator::GetNext (this=this@entry=0x13ad09e00,
> results=results@entry=0xa7e18460, max_rows=max_rows@entry=-1,
> eos=eos@entry=0x7f93adc880ef,
> block_on_wait_time_us=block_on_wait_time_us@entry=0) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/runtime/coordinator.cc:864
> #10 impala::ClientRequestState::FetchRowsInternal
> (this=this@entry=0x11568000, max_rows=max_rows@entry=-1,
> fetched_rows=fetched_rows@entry=0xa7e18460,
> block_on_wait_time_us=block_on_wait_time_us@entry=0) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/client-request-state.cc:1090
> #11 impala::ClientRequestState::FetchRows (this=0x11568000,
> max_rows=max_rows@entry=-1, fetched_rows=fetched_rows@entry=0xa7e18460,
> block_on_wait_time_us=0) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/client-request-state.cc:938
> #12 impala::ImpalaServer::FetchInternal (this=this@entry=0xdd35b00,
> query_id=..., start_over=start_over@entry=false,
> fetch_size=fetch_size@entry=-1,
> query_results=query_results@entry=0x7f93adc88498) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/impala-beeswax-server.cc:614
> #13 impala::ImpalaServer::fetch (this=0xdd35b00, query_results=...,
> beeswax_handle=..., start_over=<optimized out>, fetch_size=-1) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/impala-beeswax-server.cc:191
> #14 beeswax::BeeswaxServiceProcessor::process_fetch (this=0x102b0c60,
> seqid=0, iprot=<optimized out>, oprot=0xd45d0980, callContext=<optimized
> out>) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3398
> #15 beeswax::BeeswaxServiceProcessor::dispatchCall (this=0x102b0c60,
> iprot=0xd45d1e00, oprot=0xd45d0980, fname=..., seqid=0,
> callContext=0x12ea9a80) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3200
> #16 apache::thrift::TDispatchProcessor::process (this=0x102b0c60, in=...,
> out=..., connectionContext=0x12ea9a80) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/include/thrift/TDispatchProcessor.h:121
> #17 apache::thrift::server::TAcceptQueueServer::Task::run (this=0x11a92e80)
> at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:84
> #18 operator() (a2=<optimized out>, a1=..., p=<optimized out>,
> this=<optimized out>) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:280{noformat}
> The code is updating the total time counter in the runtime profile via a
> ScopedTimer.
> This has been seen once
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
