[
https://issues.apache.org/jira/browse/IMPALA-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-15290.
-------------------------------------
Fix Version/s: Impala 5.0.0
Resolution: Fixed
> test_execute_remove_orphan_files fails in file existence check
> --------------------------------------------------------------
>
> Key: IMPALA-15290
> URL: https://issues.apache.org/jira/browse/IMPALA-15290
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> Stacktrace
> {noformat}
> query_test/test_iceberg.py:502: in test_execute_remove_orphan_files
> assert self.filesystem_client.exists(os.path.join(DATA_PATH, file_parq1))
> E AssertionError: assert False
> E + where False = <bound method HadoopFsCommandLineClient.exists of
> <tests.util.hdfs_util.HadoopFsCommandLineClient object at
> 0x7fd3a9a06d10>>('/test-warehouse/test_execute_remove_orphan_files_1e8edd5.db/tbl_with_orphan_files/data/00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet')
> E + where <bound method HadoopFsCommandLineClient.exists of
> <tests.util.hdfs_util.HadoopFsCommandLineClient object at 0x7fd3a9a06d10>> =
> <tests.util.hdfs_util.HadoopFsCommandLineClient object at
> 0x7fd3a9a06d10>.exists
> E + where <tests.util.hdfs_util.HadoopFsCommandLineClient object at
> 0x7fd3a9a06d10> = <test_iceberg.TestIcebergTable object at
> 0x7fd3a98e8950>.filesystem_client
> E + and
> '/test-warehouse/test_execute_remove_orphan_files_1e8edd5.db/tbl_with_orphan_files/data/00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet'
> = <function join at
> 0x7fd3b6e34f40>('/test-warehouse/test_execute_remove_orphan_files_1e8edd5.db/tbl_with_orphan_files/data',
>
> '00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet')
> E + where <function join at 0x7fd3b6e34f40> = <module 'posixpath'
> (frozen)>.join
> E + where <module 'posixpath' (frozen)> = os.path
> DATA_PATH =
> '/test-warehouse/test_execute_remove_orphan_files_1e8edd5.db/tbl_with_orphan_files/data'
> METADATA_PATH =
> '/test-warehouse/test_execute_remove_orphan_files_1e8edd5.db/tbl_with_orphan_files/metadata'
> SRC_DIR =
> '/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/testdata/data/iceberg_test/iceberg_mixed_file_format_test/{0}/{1}'
> TABLE_PATH =
> '/test-warehouse/test_execute_remove_orphan_files_1e8edd5.db/tbl_with_orphan_files'
> db_tbl =
> 'test_execute_remove_orphan_files_1e8edd5.tbl_with_orphan_files'
> file_avro1 = '055baf62-de6d-4583-bf21-f187f9482343-m0.avro'
> file_parq1 =
> '00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet'
> impalad_client = <tests.common.impala_connection.ImpylaHS2Connection
> object at 0x7fd3a9aa06d0>
> insert_q = 'insert into {0} values ({1})'
> result = <tests.common.impala_connection.ImpylaHS2ResultSet
> object at 0x7fd3a9aa3590>
> self = <test_iceberg.TestIcebergTable object at 0x7fd3a98e8950>
> tbl_name = 'tbl_with_orphan_files'
> unique_database = 'test_execute_remove_orphan_files_1e8edd5'{noformat}
> The code is:
> {code:python}
> 494 # Copy first set of junk files.
> 495 file_parq1 =
> "00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-" \
> 496 "fff150b6136a-job_16619542960420_0002-1-00001.parquet"
> 497 file_avro1 = "055baf62-de6d-4583-bf21-f187f9482343-m0.avro"
> 498 self.filesystem_client.copy_from_local(
> 499 SRC_DIR.format('data', file_parq1), DATA_PATH)
> 500 self.filesystem_client.copy_from_local(
> 501 SRC_DIR.format('metadata', file_avro1), METADATA_PATH)
> 502 assert self.filesystem_client.exists(os.path.join(DATA_PATH,
> file_parq1))
> 503 assert self.filesystem_client.exists(os.path.join(METADATA_PATH,
> file_avro1)){code}
> https://github.com/apache/impala/blob/2157bc8278dd9db8dcb7dc77d982919a9d61f4b0/tests/query_test/test_iceberg.py#L502
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
