[
https://issues.apache.org/jira/browse/IMPALA-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309125#comment-17309125
]
ASF subversion and git services commented on IMPALA-10598:
----------------------------------------------------------
Commit 5b27b7ca7232a17d2a099f8567553004248989f2 in impala's branch
refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5b27b7c ]
IMPALA-10598: Deflake test_cache_reload_validation
This patch deflakes the test test_cache_reload_validation in
test_hdfs_caching.py e2e test. The util method which the test relies on to
get the count of list of cache directives by parsing the output of command
"hdfs cacheadmin -listDirectives -stats" does not consider that the output
may contain trailing new lines or headers. Hence the test fails because the
expected number of cache directives does not match the number of lines
of the output.
The fix parses the line "Found <int> entries" in the output when available
and returns the count from that line. If the line is not found, it fallbacks
to the earlier implementation of using the number of lines.
Testing:
1. The test was failing for me when run individually. After the patch, I looped
the test 10 times without any errors.
Change-Id: I2d491e90af461d5db3575a5840958d17ca90901c
Reviewed-on: http://gerrit.cloudera.org:8080/17210
Reviewed-by: Vihang Karajgaonkar <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_cache_reload_validation is flaky
> -------------------------------------
>
> Key: IMPALA-10598
> URL: https://issues.apache.org/jira/browse/IMPALA-10598
> Project: IMPALA
> Issue Type: Test
> Reporter: Vihang Karajgaonkar
> Assignee: Vihang Karajgaonkar
> Priority: Minor
> Labels: flaky-test
>
> I noticed that when I run
> {noformat}
> bin/impala-py.test tests/query_test/test_hdfs_caching.py -k
> test_cache_reload_validation
> {noformat}
> I see a the following failure on master branch.
> {noformat}
> TestHdfsCachingDdl.test_cache_reload_validation[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> tests/query_test/test_hdfs_caching.py:269: in test_cache_reload_validation
> assert num_entries_pre + 4 == get_num_cache_requests(), \
> E AssertionError: Adding the tables should be reflected by the number of
> cache directives.
> E assert (2 + 4) == 7
> E + where 7 = get_num_cache_requests()
> {noformat}
> This failure is reproducible for me every time but I am not sure why the
> jenkins job don't show this test failure. When I looked into this I found
> that the test depends on the method
> get the number of cache directives on the hdfs.
> {noformat}
> def get_num_cache_requests_util():
> rc, stdout, stderr = exec_process("hdfs cacheadmin -listDirectives
> -stats")
> assert rc == 0, 'Error executing hdfs cacheadmin: %s %s' % (stdout,
> stderr)
> return len(stdout.split('\n'))
> {noformat}
> This output of this command when there are no entries is
> {noformat}
> Found 0 entries
> {noformat}
> when there are entries the output looks like
> {noformat}
> Found 4 entries
> ID POOL REPL EXPIRY PATH
> BYTES_NEEDED BYTES_CACHED FILES_NEEDED FILES_CACHED
> 225 testPool 8 never /test-warehouse/cachedb.db/cached_tbl_reload
> 0 0 0 0
> 226 testPool 8 never
> /test-warehouse/cachedb.db/cached_tbl_reload_part 0
> 0 0 0
> 227 testPool 8 never
> /test-warehouse/cachedb.db/cached_tbl_reload_part/j=1 0
> 0 0 0
> 228 testPool 8 never
> /test-warehouse/cachedb.db/cached_tbl_reload_part/j=2 0
> 0 0 0
> {noformat}
> When there are no entries there is also a additional new line which is
> counted.
> So when there are no entries the method outputs 2 and when there are 4
> entries the method outputs 7 which causes the failure because the test
> expects 2+4.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
