[
https://issues.apache.org/jira/browse/IMPALA-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119219#comment-17119219
]
Quanlong Huang commented on IMPALA-9702:
----------------------------------------
I can reproduce the same errors in the description if I run the test
*{color:#FF0000}twice{color}*, i.e.
{code:java}
tests/run-tests.py metadata/test_ddl.py -k test_alter_table; tests/run-tests.py
metadata/test_ddl.py -k test_alter_table; {code}
The reason for my case is that some files of external tables are not cleaned up
as we expected in the first run. Then in the second run they still exist so we
get more rows than we expected. For example, these files cause the first
failure (metadata/test_ddl.py::TestDdlStatements::test_alter_table) in the
description:
{code:java}
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/_impala_insert_staging
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2012
-rw-r--r-- 3 quanlong supergroup 4 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2012/2648d0433549efd8-d0942da200000003_1025037443_data.0.txt
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2012%2Fwithslash
-rw-r--r-- 3 quanlong supergroup 2 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2012%2Fwithslash/aa4acb8deba45855-17d4997300000003_380328109_data.0.txt
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2013
-rw-r--r-- 3 quanlong supergroup 6 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2013/0240e150175b0785-f76d27b600000003_23879212_data.0.txt
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=1/s=2013bar
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=2
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=2/s=__HIVE_DEFAULT_PARTITION__
-rw-r--r-- 3 quanlong supergroup 2 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=2/s=__HIVE_DEFAULT_PARTITION__/11465dff16c4b362-bf4de2d300000003_283035038_data.0.txt
-rw-r--r-- 3 quanlong supergroup 3 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=2/s=__HIVE_DEFAULT_PARTITION__/ae40687a8c4707da-4063197900000003_308010542_data.0.txt
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=__HIVE_DEFAULT_PARTITION__
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=__HIVE_DEFAULT_PARTITION__/s=2013
-rw-r--r-- 3 quanlong supergroup 10 2020-05-29 10:27
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=__HIVE_DEFAULT_PARTITION__/s=2013/4041f63aaca57e3d-bcfdfff200000003_1649990637_data.0.txt
drwxr-xr-x - quanlong supergroup 0 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=__HIVE_DEFAULT_PARTITION__/s=__HIVE_DEFAULT_PARTITION__
-rw-r--r-- 3 quanlong supergroup 10 2020-05-29 10:34
hdfs:///test-warehouse/test_alter_table_db234c3f.db/t_part_tmp/j=__HIVE_DEFAULT_PARTITION__/s=__HIVE_DEFAULT_PARTITION__/3e42a645b578927d-0dcfb56f00000003_304789615_data.0.txt{code}
I have to manually remove these files to make the test pass. If these files
exist in our snapshots used in S3 tests, I think it will cause the same
failures.
> TestDdlStatements::test_alter_table() and
> TestMixedPartitions::test_incompatible_avro_partition_in_non_avro_table()
> consistently fail on S3
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-9702
> URL: https://issues.apache.org/jira/browse/IMPALA-9702
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0
> Reporter: Bikramjeet Vig
> Assignee: Joe McDonnell
> Priority: Critical
>
> Two tests consistently fail on S3 after the switch to USE_CDP_HIVE=true by
> default.
> metadata/test_ddl.py::TestDdlStatements::test_alter_table():
> {noformat}
> metadata/test_ddl.py:445: in test_alter_table
> multiple_impalad=self._use_multiple_impalad(vector))
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,1,'2012/withslash' == 1,1,'2012/withslash'
> E 2,1,'2012' == 2,1,'2012'
> E 2,1,'2012' == 2,1,'2012'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 3,1,'2013' == 3,1,'2013'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E 4,NULL,'NULL' == 4,NULL,'NULL'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E None != 5,NULL,'2013'
> E Number of rows returned (expected vs actual): 11 != 16
> {noformat}
> metadata/test_partition_metadata.py::TestMixedPartitions.test_incompatible_avro_partition_in_non_avro_table():
> {noformat}
> metadata/test_partition_metadata.py:151: in
> test_incompatible_avro_partition_in_non_avro_table
> test_file_vars={'$MAIN_TABLE_FORMAT': main_table_format})
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1 ==
> 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> E None != 1,false,2,3,4,5,6,7,'1985-07-15','c2','my va',1
> E Number of rows returned (expected vs actual): 1 != 2
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
