[
https://issues.apache.org/jira/browse/IMPALA-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110599#comment-17110599
]
Sahil Takiar commented on IMPALA-9702:
--------------------------------------
[~joemcdonnell] I think the issue with {{TestDdlStatements.test_alter_table}}
might be that we still use the AWS CLI to copy data from the local warehouse
snapshot to S3. We should probably be using the HDFS CLI to ensure that S3Guard
properly updates the DynamoDB tables for all these files (any operations done
via the AWS CLI won't be reflected in S3Guard). Something like {{hdfs fs
-copyFromLocal -d -f ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}
s3a://${S3_BUCKET}$\{TEST_WAREHOUSE_DIR}}}
In general, we should probably use the HDFS CLI rather than the AWS CLI, since
the HDFS CLI will go through S3Guard. There is a call to {{aws s3 rm
--recursive}} in {{load-test-warehouse-snapshot.sh}} which looks like it should
be replaced by the appropriate HDFS CLI call as well.
> Incoherent data read issues on S3
> ---------------------------------
>
> Key: IMPALA-9702
> URL: https://issues.apache.org/jira/browse/IMPALA-9702
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Bikramjeet Vig
> Assignee: Joe McDonnell
> Priority: Critical
>
> A bunch of tests with extra rows returned or wrong metadata:
> {noformat}
> metadata/test_ddl.py:445: in test_alter_table
> multiple_impalad=self._use_multiple_impalad(vector))
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,1,'2012/withslash' == 1,1,'2012/withslash'
> E 2,1,'2012' == 2,1,'2012'
> E 2,1,'2012' == 2,1,'2012'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E Number of rows returned (expected vs actual): 11 != 16
> {noformat}
> {noformat}
> metadata/test_explain.py:113: in test_explain_validate_cardinality_estimates
> check_cardinality(result.data, '7.30K')
> metadata/test_explain.py:98: in check_cardinality
> query_result, expected_cardinality=expected_cardinality)
> metadata/test_explain.py:86: in check_row_size_and_cardinality
> assert m.groups()[1] == expected_cardinality
> E assert '6.99K' == '7.30K'
> E - 6.99K
> E + 7.30K
> {noformat}
> {noformat}
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1 ==
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> None != 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> Number of rows returned (expected vs actual): 1 != 2
> {noformat}
> {noformat}
> metadata/test_metadata_query_statements.py:70: in test_show_stats
> self.run_test_case('QueryTest/show-stats', vector, "functional")
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1'
> == '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1'
> E '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10'
> == '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10'
> E '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11'
> == '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11'
> E '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12'
> == '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12'
> E '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2'
> == '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2'
> E '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3'
> == '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3'
> E '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4'
> == '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4'
> E '2009','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5'
> != '2009','5',0,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5'
> E '2009','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=6'
> == '2009','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=6'
> E '2009','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=7'
> == '2009','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=7'
> E '2009','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=8'
> == '2009','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=8'
> E '2009','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9'
> == '2009','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9'
> E '2010','1',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=1'
> == '2010','1',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=1'
> E '2010','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=10'
> == '2010','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=10'
> E '2010','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=11'
> == '2010','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=11'
> E '2010','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=12'
> == '2010','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=12'
> E '2010','2',280,1,'18.39KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=2'
> == '2010','2',280,1,'18.39KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=2'
> E '2010','3',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=3'
> == '2010','3',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=3'
> E '2010','4',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=4'
> == '2010','4',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=4'
> E '2010','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=5'
> == '2010','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=5'
> E '2010','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=6'
> == '2010','6',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=6'
> E '2010','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=7'
> == '2010','7',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=7'
> E '2010','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=8'
> == '2010','8',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=8'
> E '2010','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=9'
> == '2010','9',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2010/month=9'
> E 'Total','',7300,24,'478.45KB','0B','','','','' !=
> 'Total','',6990,24,'478.45KB','0B','','','',''
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
