[
https://issues.apache.org/jira/browse/IMPALA-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093254#comment-18093254
]
Zoltán Borók-Nagy commented on IMPALA-15128:
--------------------------------------------
The stacktrace is:
{noformat}
#0 access_mem (as=<optimized out>, addr=4294967294, val=0x7f4e69ae1348,
write=<optimized out>, arg=<optimized out>) at x86_64/Ginit.c:322
#1 0x0000000003ed2f1b in dwarf_get (val=0x7f4e69ae1348, loc=...,
c=0x7f4e69ae1770) at ../include/tdep-x86_64/libunwind_i.h:202
#2 _ULx86_64_step (cursor=cursor@entry=0x7f4e69ae1770) at x86_64/Gstep.c:255
#3 0x00000000015e5303 in kudu::StackTrace::Collect (this=0x7f4e7a622a70,
skip_frames=0, skip_frames@entry=1) at
/home/systest/impala/be/src/kudu/util/debug-util.cc:608
#4 0x00000000015e53a6 in kudu::(anonymous namespace)::HandleStackTraceSignal
(info=<optimized out>) at
/home/systest/impala/be/src/kudu/util/debug-util.cc:282
#5 <signal handler called>
#6 0x00007f5206bfebbf in poll () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x00007f51e609b096 in Java_sun_nio_ch_Net_poll () from
/usr/lib/jvm/java-17-openjdk-amd64/lib/libnio.so
#8 0x00007f51f57059f6 in ?? ()
#9 0x00000000b5e4c630 in ?? ()
#10 0x0000000000000000 in ?? (){noformat}
Which probably means that a JVM thread was in Java_sun_nio_ch_Net_poll() while
webserver page /stacks was queried.
The trigger and the collision, from the text logs, all inside the final ~3
seconds before the minidump:
* *15:24:03.534:* test_web_pages.py::TestWebPage active
* *15:24:05.938:* /stacks rendered, 1.576 s, client 192.168.124.1
* *15:24:06.0–06.4:* concurrent test_jdbc_data_source: select count(*) from
alltypes_jdbc_datasource (JVM/JDBC threads)
* *~15:24:07:* Minidump in thread [392558] (a JDBC exec thread)
So it's a cross-test collision in the parallel EE phase:
- test_web_pages.py::TestWebPage scrapes the debug pages, including /stacks →
fires SIGRTMIN+10 → StackTrace::Collect unwinds every Impala thread via
libunwind.
- test_ext_data_sources.py::TestExtDataSources::test_jdbc_data_source keeps
exec threads inside the JVM (JDBC/Postgres, blocked in poll()).
- libunwind walks that thread's JVM/JIT frames (no CFI), dereferences a bad
address → SIGSEGV → daemon dies → the ~1,800 downstream "can't connect to
21050" errors.
> test_jdbc_data_source can crash impalad
> ---------------------------------------
>
> Key: IMPALA-15128
> URL: https://issues.apache.org/jira/browse/IMPALA-15128
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: broken-build
>
> During the dockerized end-to-end (EE) test run, the coordinator impalad
> crashed with a SIGSEGV while executing a query from test_jdbc_data_source.
> *Offending test*
> {noformat}
> query_test/test_ext_data_sources.py::TestExtDataSources::test_jdbc_data_source
> ["table_format:text/none" | "protocol:hs2" |
> exec_option:{'abort_on_error':1;'batch_size':0;'disable_codegen':False;
> ...}]{noformat}
> *Crashing query*
> {noformat}
> Crashing SQL: select * from alltypes_jdbc_datasource where id > 10 and
> int_col < 5 limit 5{noformat}
> The ERROR log shows a handled Postgres exception on the JDBC data source
> planning path just before the crash:
>
> {noformat}
> PSQLException: FATAL: password authentication failed for user "UNKNOWN"
> at ...GenericJdbcDatabaseAccessor.getTotalNumberOfRecords
> datasource: jdbc:postgresql://192.168.124.1:5432/functional
> {noformat}
> [https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/5311/]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
