[
https://issues.apache.org/jira/browse/IMPALA-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030921#comment-17030921
]
Fang-Yu Rao commented on IMPALA-8539:
-------------------------------------
We have seen a similar failure recently in a log file.
{code:java}
Log file created at: 2020/02/05 08:55:44
Running on machine:
impala-ec2-centos74-r5-4xlarge-ondemand-0bc8.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0205 08:55:44.269302 28519 logging.cc:147] stderr will be logged to this file.
20/02/05 08:55:45 INFO util.JvmPauseMonitor: Starting JVM pause monitor
=================================================================
==28519==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400007e9e0
at pc 0x000001f586b2 bp 0x7f14c05f0ac0 sp 0x7f14c05f0ab8
READ of size 8 at 0x60400007e9e0 thread T3
Picked up JAVA_TOOL_OPTIONS:
-agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n
{code}
According to the time stamp above, i.e., 08:55:44 and the console output, we
know the error occurred when we are running the following EE tests.
{code:java}
<span class="timestamp"><b>08:55:43</b>
</span>custom_cluster/test_restart_services.py::TestRestart::test_restart_statestore_query_resilience
PASSED
<span class="timestamp"><b>08:56:14</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_idle
PASSED
<span class="timestamp"><b>08:57:23</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_executor
PASSED
<span class="timestamp"><b>08:57:32</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_executor_with_delay
SKIPPED
<span class="timestamp"><b>08:57:42</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_signal
PASSED
<span class="timestamp"><b>08:57:53</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_sending_multiple_shutdown_signals
PASSED
<span class="timestamp"><b>08:58:01</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_graceful_shutdown_script
PASSED
<span class="timestamp"><b>08:58:16</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_coordinator
<- hs2/hs2_test_suite.py PASSED
<span class="timestamp"><b>08:58:25</b>
</span>custom_cluster/test_result_spooling.py::TestDedicatedCoordinator::test_dedicated_coordinator[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none] PASSED
{code}
> ASAN heap-use-after-free failure during TestGracefulShutdown
> ------------------------------------------------------------
>
> Key: IMPALA-8539
> URL: https://issues.apache.org/jira/browse/IMPALA-8539
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: David Rorke
> Assignee: Bikramjeet Vig
> Priority: Major
>
> I'm seeing an ASAN heap-use-after-free failure with no stack trace in the
> stderr. Here's the full ERROR log:
> {noformat}
> Log file created at: 2019/05/10 06:32:22
> Running on machine:
> impala-ec2-centos74-r4-4xlarge-ondemand-08f1.vpc.cloudera.com
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> E0510 06:32:22.212716 102164 logging.cc:147] stderr will be logged to this
> file.
> 19/05/10 06:32:23 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> =================================================================
> ==102164==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x603000032668 at pc 0x0000017aa3dc bp 0x7ff292197ae0 sp 0x7ff292197290
> READ of size 4 at 0x603000032668 thread T3
> Picked up JAVA_TOOL_OPTIONS:
> -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n
> {noformat}
> With no stack trace it's hard to say exactly what triggered it but
> correlating the timing of the error log entry with the Jenkins console logs
> it looks like we were running one of the graceful shutdown tests:
> {noformat}
> 06:32:19
> custom_cluster/test_restart_services.py::TestGracefulShutdown::test_sending_multiple_shutdown_signals
> PASSED
> 06:32:37
> custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_coordinator
> <- hs2/hs2_test_suite.py PASSED
> 06:32:37
> custom_cluster/test_rpc_exception.py::TestRPCException::test_rpc_send_closed_connection[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] SKIPPED
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
