Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19441
Change subject: IMPALA-11864: Iceberg LOAD DATA should not load S3 hidden files ...................................................................... IMPALA-11864: Iceberg LOAD DATA should not load S3 hidden files Loading data from S3 did not skip hidden files because the FileSystemUtil.listFiles() call was returning a RemoteIterator, which compared to RecursingIterator does not filter the hidden files. This would make a load fail because the hidden file likely have invalid magic string. This commit adds an extra condition to skip hidden files when creating the CREATE subquery. Testing: - Added E2E test - Ran E2E test on S3 build Change-Id: Iffd179383c2bb2529f6f9b5f8bf5cba5f3553652 --- M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-load.test M tests/query_test/test_iceberg.py 3 files changed, 27 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/19441/1 -- To view, visit http://gerrit.cloudera.org:8080/19441 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iffd179383c2bb2529f6f9b5f8bf5cba5f3553652 Gerrit-Change-Number: 19441 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate <[email protected]>
