[
https://issues.apache.org/jira/browse/IMPALA-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101561#comment-17101561
]
Steve Loughran commented on IMPALA-9702:
----------------------------------------
bq. The second issue is that we intermittently see some files missing. These
are files copied from the snapshot into s3 at the beginning of the test. This
is likely to be some consistency issue. I think I'm going to split that out
into its own issue.
S3 load balancers cache 404s of objects looked for *Even after the object is
created*; lots of work in hadoop 3.3.0 to try and nail this down, with
create(path, overwrite=true) skipping head probes, and work looking at app code
to remove calls to exists(), getFileStatus(), isFile(), etc.
A 404 will stay in cache while clients issue HEAD/GET requests, you need to
allow 20-30+s after the failing HEAD before the cache entry appears to expire.
When S3Guard is enabled and we find an entry in DDB, we use that knowledge to
spin for a while (90+s with linear backoff) awaiting for it to appear.
Unguarded: we just try to eliminate those 404s.
what is the interval between test setup and execute? could you implement the
probe with backoff yourself?
> Incoherent data read issues on S3
> ---------------------------------
>
> Key: IMPALA-9702
> URL: https://issues.apache.org/jira/browse/IMPALA-9702
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Bikramjeet Vig
> Assignee: Joe McDonnell
> Priority: Critical
>
> A bunch of tests with extra rows returned or wrong metadata:
> {noformat}
> metadata/test_ddl.py:445: in test_alter_table
> multiple_impalad=self._use_multiple_impalad(vector))
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,1,'2012/withslash' == 1,1,'2012/withslash'
> E 2,1,'2012' == 2,1,'2012'
> E 2,1,'2012' == 2,1,'2012'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E Number of rows returned (expected vs actual): 11 != 16
> {noformat}
> {noformat}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
> check_cardinality(result.data, '7.30K')
> metadata/test_explain.py:98: in check_cardinality
> query_result, expected_cardinality=expected_cardinality)
> metadata/test_explain.py:86: in check_row_size_and_cardinality
> assert m.groups()[1] == expected_cardinality
> E assert '6.99K' == '7.30K'
> E - 6.99K
> E + 7.30K
> {noformat}
> {noformat}
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1 ==
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> None != 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> Number of rows returned (expected vs actual): 1 != 2
> {noformat}
> {noformat}
> metadata/test_metadata_query_statements.py:70: in test_show_stats
> self.run_test_case('QueryTest/show-stats', vector, "functional")
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1'
> == '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1'
> E '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10'
> == '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10'
> E '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11'
> == '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11'
> E '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12'
> == '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12'
> E '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2'
> == '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2'
> E '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3'
> == '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3'
> E '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4'
> == '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4'
> E '2009','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5'
> != '2009','5',0,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5'
> E '2009','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=6'
> == '2009','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=6'
> E '2009','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=7'
> == '2009','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=7'
> E '2009','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=8'
> == '2009','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=8'
> E '2009','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9'
> == '2009','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9'
> E '2010','1',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=1'
> == '2010','1',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=1'
> E '2010','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=10'
> == '2010','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=10'
> E '2010','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=11'
> == '2010','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=11'
> E '2010','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=12'
> == '2010','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=12'
> E '2010','2',280,1,'18.39KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=2'
> == '2010','2',280,1,'18.39KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=2'
> E '2010','3',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=3'
> == '2010','3',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=3'
> E '2010','4',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=4'
> == '2010','4',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=4'
> E '2010','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=5'
> == '2010','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=5'
> E '2010','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=6'
> == '2010','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=6'
> E '2010','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=7'
> == '2010','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=7'
> E '2010','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=8'
> == '2010','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=8'
> E '2010','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=9'
> == '2010','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=9'
> E 'Total','',7300,24,'478.45KB','0B','','','','' !=
> 'Total','',6990,24,'478.45KB','0B','','','',''
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
