[
https://issues.apache.org/jira/browse/IMPALA-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285806#comment-17285806
]
ASF subversion and git services commented on IMPALA-10497:
----------------------------------------------------------
Commit 490aff51b9e3289f2225d3918734821cab7f28c2 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=490aff5 ]
IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
test_no_fd_caching_on_cached_data has been flaky for not having all of
the data fully cached in the warm-up phase. There is a limit on
concurrency in writing to the cache such that we may fail to cache data
the first time read it. This patch fixes the test by repeating the
warm-up query 5 times. This patch also add a proper start_args to the
test so that each impalad will write their data cache file in their own
directory.
Testing:
- Loop the test manually 100 times and see no more failures.
Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Reviewed-on: http://gerrit.cloudera.org:8080/17054
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_no_fd_caching_on_cached_data failing
> -----------------------------------------
>
> Key: IMPALA-10497
> URL: https://issues.apache.org/jira/browse/IMPALA-10497
> Project: IMPALA
> Issue Type: Bug
> Reporter: Bikramjeet Vig
> Assignee: Riza Suminto
> Priority: Major
> Labels: broken-build
>
> {noformat}
> Error Message
> assert 1 == 0 + where 1 = <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>>() +
> where <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>> =
> <test_hdfs_fd_caching.TestHdfsFdCaching object at
> 0x7f22dfe5aa10>.cached_handles
> Stacktrace
> custom_cluster/test_hdfs_fd_caching.py:202: in
> test_no_fd_caching_on_cached_data
> assert self.cached_handles() == 0
> E assert 1 == 0
> E + where 1 = <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>>()
> E + where <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>> =
> <test_hdfs_fd_caching.TestHdfsFdCaching object at
> 0x7f22dfe5aa10>.cached_handles
> Standard Error
> -- 2021-02-08 06:40:41,413 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--max_cached_file_handles=16
> --unused_file_handle_timeout_sec=5 --data_cache=/tmp:500MB
> --always_use_data_cache=true ' '--state_store_args=None '
> '--catalogd_args=--load_catalog_in_background=false '
> --impalad_args=--default_query_options=
> 06:40:42 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 06:40:42 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 06:40:42 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 06:40:42 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 06:40:42 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 06:40:42 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 06:40:45 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:45 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:45 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> 06:40:45 MainThread: Debug webpage not yet available: ('Connection aborted.',
> error(111, 'Connection refused'))
> 06:40:47 MainThread: Debug webpage did not become available in expected time.
> 06:40:47 MainThread: Waiting for num_known_live_backends=3. Current value:
> None
> 06:40:48 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:48 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> 06:40:48 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 06:40:49 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:49 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> 06:40:49 MainThread: num_known_live_backends has reached value: 3
> 06:40:49 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:49 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25001
> 06:40:49 MainThread: num_known_live_backends has reached value: 3
> 06:40:50 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:50 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25002
> 06:40:50 MainThread: num_known_live_backends has reached value: 3
> 06:40:50 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> -- 2021-02-08 06:40:51,049 DEBUG MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> -- 2021-02-08 06:40:51,049 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25010
> -- 2021-02-08 06:40:51,050 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,052 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> -- 2021-02-08 06:40:51,052 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> -- 2021-02-08 06:40:51,053 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,054 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2021-02-08 06:40:51,054 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25001
> -- 2021-02-08 06:40:51,055 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,056 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2021-02-08 06:40:51,056 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25002
> -- 2021-02-08 06:40:51,057 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,058 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-02-08 06:40:51,307 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-02-08 06:40:51,332 INFO MainThread: Closing active operation
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- connecting to:
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> drop database if exists cachefd cascade;
> -- 2021-02-08 06:40:51,527 INFO MainThread: Started query
> c94d37c9548851e5:7c7fb33300000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> create database cachefd;
> -- 2021-02-08 06:40:51,577 INFO MainThread: Started query
> f445a2e444e61b11:246e8dfd00000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> create table cachefd.simple(id int, col1 int, col2 int) stored as parquet;
> -- 2021-02-08 06:40:51,706 INFO MainThread: Started query
> 504e5f830c41c2ba:efc4cdad00000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> insert into cachefd.simple values(0,0,0), (1,1,1), (2,2,2), (3,3,3), (4,4,4),
> (5,5,5), (6,6,6), (7,7,7), (8,8,8), (9,9,9), (10,10,10), (11,11,11),
> (12,12,12), (13,13,13), (14,14,14), (15,15,15), (16,16,16), (17,17,17),
> (18,18,18), (19,19,19), (20,20,20), (21,21,21), (22,22,22), (23,23,23),
> (24,24,24), (25,25,25), (26,26,26), (27,27,27), (28,28,28), (29,29,29),
> (30,30,30), (31,31,31), (32,32,32), (33,33,33), (34,34,34), (35,35,35),
> (36,36,36), (37,37,37), (38,38,38), (39,39,39), (40,40,40), (41,41,41),
> (42,42,42), (43,43,43), (44,44,44), (45,45,45), (46,46,46), (47,47,47),
> (48,48,48), (49,49,49), (50,50,50), (51,51,51), (52,52,52), (53,53,53),
> (54,54,54), (55,55,55), (56,56,56), (57,57,57), (58,58,58), (59,59,59),
> (60,60,60), (61,61,61), (62,62,62), (63,63,63), (64,64,64), (65,65,65),
> (66,66,66), (67,67,67), (68,68,68), (69,69,69), (70,70,70), (71,71,71),
> (72,72,72), (73,73,73), (74,74,74), (75,75,75), (76,76,76), (77,77,77),
> (78,78,78), (79,79,79), (80,80,80), (81,81,81), (82,82,82), (83,83,83),
> (84,84,84), (85,85,85), (86,86,86), (87,87,87), (88,88,88), (89,89,89),
> (90,90,90), (91,91,91), (92,92,92), (93,93,93), (94,94,94), (95,95,95),
> (96,96,96), (97,97,97), (98,98,98), (99,99,99);
> -- 2021-02-08 06:40:52,482 INFO MainThread: Started query
> 7041325f1c73e372:2ec9e2ef00000000
> -- 2021-02-08 06:40:52,742 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,745 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,749 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,754 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,758 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,761 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> use functional;
> -- 2021-02-08 06:40:52,772 INFO MainThread: Started query
> 194c55cdb4433725:bff1f94d00000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> select * from cachefd.simple;
> -- 2021-02-08 06:40:52,842 INFO MainThread: Started query
> 3a47160da328b7dd:983c9ca700000000
> -- 2021-02-08 06:40:53,053 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,057 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,061 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,064 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,068 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,071 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,074 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,077 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,081 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,094 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,107 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,120 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> use functional;
> -- 2021-02-08 06:41:03,137 INFO MainThread: Started query
> 2a4e414134944c87:f07a223200000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> select * from cachefd.simple;
> -- 2021-02-08 06:41:03,146 INFO MainThread: Started query
> 7a4a8d4994fe200d:ec8c84ee00000000
> -- 2021-02-08 06:41:03,256 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,261 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,265 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
