[
https://issues.apache.org/jira/browse/IMPALA-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931812#comment-17931812
]
ASF subversion and git services commented on IMPALA-12931:
----------------------------------------------------------
Commit 5d2fc70423724e218547d5f428d49464c117ea73 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5d2fc7042 ]
IMPALA-12931: Deflake test_no_hms_event_incremental_refresh
test_no_hms_event_incremental_refresh_transactional_table has been flaky
for not meeting the last assertion. This patch attempt to deflake the
tests by shortening the StatestoreD heartbeat duration and wait a little
bit for StatestoreD propagation after HMS Event processor catch up. This
is the same technique we use to deflake IMPALA-13701.
Testing:
- Loop tests 10 times and pass.
Change-Id: If64c6803c37522cc948d0e3480350eaa8e54c8c3
Reviewed-on: http://gerrit.cloudera.org:8080/22562
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> TestEventProcessingCustomConfigs.test_no_hms_event_incremental_refresh_transactional_table
> fails in some builds
> ---------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12931
> URL: https://issues.apache.org/jira/browse/IMPALA-12931
> Project: IMPALA
> Issue Type: Bug
> Reporter: Daniel Becker
> Assignee: Riza Suminto
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.6.0
>
>
> TestEventProcessingCustomConfigs.test_no_hms_event_incremental_refresh_transactional_table
> fails in some builds:
> h3. Error Message
> {code:java}
> assert [] == ['1', '2', '3'] Right contains more items, first extra item: '1'
> Full diff: - [] + ['1', '2', '3']{code}
> h3. Stacktrace
> {code:java}
> custom_cluster/test_events_custom_configs.py:1246: in
> test_no_hms_event_incremental_refresh_transactional_table
> assert results.data == ["1", "2", "3"]
> E assert [] == ['1', '2', '3']
> E Right contains more items, first extra item: '1'
> E Full diff:
> E - []
> E + ['1', '2', '3']{code}
> h3. Standard Error
>
> {code:java}
> -- 2024-03-19 21:11:27,574 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--state_store_args=None '
> '--catalogd_args=--hms_event_incremental_refresh_transactional_table=false '
> --impalad_args=--default_query_options=
> 21:11:28 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 21:11:28 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 21:11:28 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 21:11:28 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 21:11:28 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 21:11:28 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 21:11:31 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:31 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:31 MainThread: Getting num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> 21:11:31 MainThread: 'backends'
> 21:11:31 MainThread: Waiting for num_known_live_backends=3. Current value:
> None
> 21:11:32 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:32 MainThread: Getting num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> 21:11:32 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 21:11:33 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:33 MainThread: Getting num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> 21:11:33 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 21:11:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:34 MainThread: Getting num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> 21:11:34 MainThread: num_known_live_backends has reached value: 3
> 21:11:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:34 MainThread: Getting num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25001
> 21:11:34 MainThread: num_known_live_backends has reached value: 3
> 21:11:35 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 21:11:35 MainThread: Getting num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25002
> 21:11:35 MainThread: num_known_live_backends has reached value: 3
> 21:11:35 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> -- 2024-03-19 21:11:35,965 DEBUG MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> -- 2024-03-19 21:11:35,965 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25010
> -- 2024-03-19 21:11:35,968 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> -- 2024-03-19 21:11:35,968 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> -- 2024-03-19 21:11:35,970 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2024-03-19 21:11:35,970 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25001
> -- 2024-03-19 21:11:35,971 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2024-03-19 21:11:35,971 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25002
> -- 2024-03-19 21:11:35,973 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_no_hms_event_incremental_refresh_transactional_table;
> -- connecting to: localhost:21000
> -- 2024-03-19 21:11:35,973 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-03-19 21:11:35,974 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2024-03-19 21:11:36,115 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2024-03-19 21:11:36,131 INFO MainThread: Closing active operation
> -- connecting to localhost:11050 with impyla
> SET
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_no_hms_event_incremental_refresh_transactional_table;
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS
> `test_no_hms_event_incremental_refresh_transactional_table_75c9c83d` CASCADE;
> -- 2024-03-19 21:11:36,186 INFO MainThread: Started query
> dc4f96723e39c4e0:83716ca800000000
> SET
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_no_hms_event_incremental_refresh_transactional_table;
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE
> `test_no_hms_event_incremental_refresh_transactional_table_75c9c83d`;
> -- 2024-03-19 21:11:36,307 INFO MainThread: Started query
> 73464f29afa1e868:27e6b8c900000000
> -- 2024-03-19 21:11:36,609 INFO MainThread: Created database
> "test_no_hms_event_incremental_refresh_transactional_table_75c9c83d" for test
> ID
> "custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_no_hms_event_incremental_refresh_transactional_table"
> -- 2024-03-19 21:11:36,610 INFO MainThread: -- executing in HiveServer2
> create transactional table
> test_no_hms_event_incremental_refresh_transactional_table_75c9c83d.tbl (i int)
> -- 2024-03-19 21:11:40,122 INFO MainThread: Waiting until events
> processor syncs to event id:42423
> -- 2024-03-19 21:11:40,967 DEBUG MainThread: Metric last-synced-event-id
> has reached the desired value: 42423
> -- 2024-03-19 21:11:41,006 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> -- 2024-03-19 21:11:41,017 INFO MainThread: Sleeping 1s before next retry.
> -- 2024-03-19 21:11:42,018 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> -- 2024-03-19 21:11:42,021 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2077
> -- 2024-03-19 21:11:42,021 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25001
> -- 2024-03-19 21:11:42,032 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2077
> -- 2024-03-19 21:11:42,032 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25002
> -- 2024-03-19 21:11:42,043 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2077
> -- executing against localhost:21000
> refresh
> test_no_hms_event_incremental_refresh_transactional_table_75c9c83d.tbl;
> -- 2024-03-19 21:11:42,052 INFO MainThread: Started query
> 1a4c50fdf1662cc0:994be68900000000
> -- 2024-03-19 21:11:42,405 INFO MainThread: -- executing in HiveServer2
> insert into
> test_no_hms_event_incremental_refresh_transactional_table_75c9c83d.tbl values
> (1),(2),(3)
> -- 2024-03-19 21:11:56,045 INFO MainThread: Waiting until events
> processor syncs to event id:42427
> -- 2024-03-19 21:11:56,991 DEBUG MainThread: Metric last-synced-event-id
> has reached the desired value: 42427
> -- 2024-03-19 21:11:57,009 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
> -- 2024-03-19 21:11:57,021 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2079
> -- 2024-03-19 21:11:57,021 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25001
> -- 2024-03-19 21:11:57,031 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2079
> -- 2024-03-19 21:11:57,031 INFO MainThread: Getting metric:
> catalog.curr-version from
> impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25002
> -- 2024-03-19 21:11:57,042 INFO MainThread: Metric 'catalog.curr-version'
> has reached desired value: 2079
> -- executing against localhost:21000
> select i from
> test_no_hms_event_incremental_refresh_transactional_table_75c9c83d.tbl;
> -- 2024-03-19 21:11:57,438 INFO MainThread: Started query
> f64b0ed3452de446:e876609700000000
> {code}
> As the test was introduced by IMPALA-12835, [~csringhofer] could you take a
> look?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
