Tamas Mate has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19441 )
Change subject: IMPALA-11864: Iceberg LOAD DATA should not load S3 hidden files ...................................................................... IMPALA-11864: Iceberg LOAD DATA should not load S3 hidden files Loading data from S3 did not skip hidden files because the FileSystemUtil.listFiles() call was returning a RemoteIterator, which compared to RecursingIterator does not filter the hidden files. This would make a load fail because the hidden file likely have invalid magic string. This commit adds an extra condition to skip hidden files when creating the CREATE subquery. Testing: - Added E2E test - Ran E2E test on S3 build Change-Id: Iffd179383c2bb2529f6f9b5f8bf5cba5f3553652 --- M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-load.test M tests/query_test/test_iceberg.py 3 files changed, 25 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/19441/2 -- To view, visit http://gerrit.cloudera.org:8080/19441 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iffd179383c2bb2529f6f9b5f8bf5cba5f3553652 Gerrit-Change-Number: 19441 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate <[email protected]> Gerrit-Reviewer: Gergely Fürnstáhl <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
