[
https://issues.apache.org/jira/browse/IMPALA-9344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030930#comment-17030930
]
Fang-Yu Rao commented on IMPALA-9344:
-------------------------------------
We have seen a similar failure recently in a log file.
{code:java}
Log file created at: 2020/02/05 08:55:44
Running on machine:
impala-ec2-centos74-r5-4xlarge-ondemand-0bc8.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0205 08:55:44.269302 28519 logging.cc:147] stderr will be logged to this file.
20/02/05 08:55:45 INFO util.JvmPauseMonitor: Starting JVM pause monitor
=================================================================
==28519==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400007e9e0
at pc 0x000001f586b2 bp 0x7f14c05f0ac0 sp 0x7f14c05f0ab8
READ of size 8 at 0x60400007e9e0 thread T3
Picked up JAVA_TOOL_OPTIONS:
-agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n
{code}
According to the time stamp above, i.e., 08:55:44 and the console output, we
know the error occurred when we are running the following EE tests.
{code:java}
<span class="timestamp"><b>08:55:43</b>
</span>custom_cluster/test_restart_services.py::TestRestart::test_restart_statestore_query_resilience
PASSED
<span class="timestamp"><b>08:56:14</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_idle
PASSED
<span class="timestamp"><b>08:57:23</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_executor
PASSED
<span class="timestamp"><b>08:57:32</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_executor_with_delay
SKIPPED
<span class="timestamp"><b>08:57:42</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_signal
PASSED
<span class="timestamp"><b>08:57:53</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_sending_multiple_shutdown_signals
PASSED
<span class="timestamp"><b>08:58:01</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_graceful_shutdown_script
PASSED
<span class="timestamp"><b>08:58:16</b>
</span>custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_coordinator
<- hs2/hs2_test_suite.py PASSED
<span class="timestamp"><b>08:58:25</b>
</span>custom_cluster/test_result_spooling.py::TestDedicatedCoordinator::test_dedicated_coordinator[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none] PASSED
{code}
> TestGracefulShutdown hits ASAN use after free
> ---------------------------------------------
>
> Key: IMPALA-9344
> URL: https://issues.apache.org/jira/browse/IMPALA-9344
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: flaky
> Attachments:
> impalad.impala-ec2-centos74-r5-4xlarge-ondemand-1810.vpc.cloudera.com.jenkins.log.ERROR.20200130-025809.11221,
>
> impalad.impala-ec2-centos74-r5-4xlarge-ondemand-1810.vpc.cloudera.com.jenkins.log.INFO.20200130-025809.11221,
>
> impalad.impala-ec2-centos74-r5-4xlarge-ondemand-1810.vpc.cloudera.com.jenkins.log.WARNING.20200130-025809.11221
>
>
> The test passed but ASAN flagged the issue.
> {noformat}
> =================================================================
> ==11221==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60400007e7a0 at pc 0x000002067b42 bp 0x7fae8d502ac0 sp 0x7fae8d502ab8
> READ of size 8 at 0x60400007e7a0 thread T3
> Picked up JAVA_TOOL_OPTIONS:
> -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n
> {noformat}
> I think it's test_graceful_shutdown_script based based on these command line
> flags and the fact that the signal was logged, and the timing
> {noformat}
> --shutdown_deadline_s=3600
> --shutdown_grace_period_s=1
> ...
> I0130 02:58:16.849383 11650 init.cc:205] Shutdown signal received. Current
> Shutdown Status: shutdown grace period left: 1s000ms, deadline left: 8760h,
> queries registered on coordinator: 0, queries executing: 0, fragment
> instances: 0
> {noformat}
> {noformat}
> 02:57:56
> custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_signal
> PASSED
> 02:58:09
> custom_cluster/test_restart_services.py::TestGracefulShutdown::test_sending_multiple_shutdown_signals
> PASSED
> 02:58:17
> custom_cluster/test_restart_services.py::TestGracefulShutdown::test_graceful_shutdown_script
> PASSED
> 02:58:33
> custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_coordinator
> <- hs2/hs2_test_suite.py PASSED
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
