[
https://issues.apache.org/jira/browse/IMPALA-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100175#comment-17100175
]
Joe McDonnell commented on IMPALA-9702:
---------------------------------------
This is two distinct issues.
One issue is a consistent failure in two tests:
metadata.test_partition_metadata.TestMixedPartitions.test_incompatible_avro_partition_in_non_avro_table
metadata.test_ddl.TestDdlStatements.test_alter_table
This fails with the current versions for Hadoop/Hive/etc, but we don't see it
when using a newer set of Hadoop/Hive/etc dependencies. I am looking into
updating the CDP GBN, and hopefully that will fix it. These tests do not
overwrite files, so far as I know.
The second issue is that we intermittently see some files missing. These are
files copied from the snapshot into s3 at the beginning of the test. This is
likely to be some consistency issue. I think I'm going to split that out into
its own issue.
> Incoherent data read issues on S3
> ---------------------------------
>
> Key: IMPALA-9702
> URL: https://issues.apache.org/jira/browse/IMPALA-9702
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Bikramjeet Vig
> Assignee: Joe McDonnell
> Priority: Critical
>
> A bunch of tests with extra rows returned or wrong metadata:
> {noformat}
> metadata/test_ddl.py:445: in test_alter_table
> multiple_impalad=self._use_multiple_impalad(vector))
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,1,'2012/withslash' == 1,1,'2012/withslash'
> E 2,1,'2012' == 2,1,'2012'
> E 2,1,'2012' == 2,1,'2012'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E Number of rows returned (expected vs actual): 11 != 16
> {noformat}
> {noformat}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
> check_cardinality(result.data, '7.30K')
> metadata/test_explain.py:98: in check_cardinality
> query_result, expected_cardinality=expected_cardinality)
> metadata/test_explain.py:86: in check_row_size_and_cardinality
> assert m.groups()[1] == expected_cardinality
> E assert '6.99K' == '7.30K'
> E - 6.99K
> E + 7.30K
> {noformat}
> {noformat}
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1 ==
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> None != 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> Number of rows returned (expected vs actual): 1 != 2
> {noformat}
> {noformat}
> metadata/test_metadata_query_statements.py:70: in test_show_stats
> self.run_test_case('QueryTest/show-stats', vector, "functional")
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1'
> == '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1'
> E '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10'
> == '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10'
> E '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11'
> == '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11'
> E '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12'
> == '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12'
> E '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2'
> == '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2'
> E '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3'
> == '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3'
> E '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4'
> == '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4'
> E '2009','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5'
> != '2009','5',0,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5'
> E '2009','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=6'
> == '2009','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=6'
> E '2009','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=7'
> == '2009','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=7'
> E '2009','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=8'
> == '2009','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=8'
> E '2009','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9'
> == '2009','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9'
> E '2010','1',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=1'
> == '2010','1',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=1'
> E '2010','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=10'
> == '2010','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=10'
> E '2010','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=11'
> == '2010','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=11'
> E '2010','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=12'
> == '2010','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=12'
> E '2010','2',280,1,'18.39KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=2'
> == '2010','2',280,1,'18.39KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=2'
> E '2010','3',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=3'
> == '2010','3',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=3'
> E '2010','4',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=4'
> == '2010','4',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=4'
> E '2010','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=5'
> == '2010','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=5'
> E '2010','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=6'
> == '2010','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=6'
> E '2010','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=7'
> == '2010','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=7'
> E '2010','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=8'
> == '2010','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=8'
> E '2010','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=9'
> == '2010','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=9'
> E 'Total','',7300,24,'478.45KB','0B','','','','' !=
> 'Total','',6990,24,'478.45KB','0B','','','',''
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
