Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14806 )
Change subject: IMPALA-9122 : Ignore FileNotFoundException when loading a table ...................................................................... Patch Set 1: (6 comments) http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG@7 PS1, Line 7: nit: no space http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG@9 PS1, Line 9: It is possible that when the file metadata of a table or partition is being loaded, some Could you please format the commit message to be 72 char width? I add this config in my ~/.vimrc for this: filetype on au FileType gitcommit setlocal tw=72 http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG@14 PS1, Line 14: the nit: redundant http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG@18 PS1, Line 18: error state. Should we retry the reload operation? I'm not sure whether this case is possible: - Hive want to remove 10 files - Load table operation has loaded 9 files - Hive remove the 10 files - Load table operation encounters FileNotFoundException for the last file and skipps it Then we have stale file meta for the 9 removed files, which may cause failures in the scanner side. http://gerrit.cloudera.org:8080/#/c/14806/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/14806/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@687 PS1, Line 687: // instance, hive deletes the temporary files in the .hive-staging directory Is it possible that some removed files are loaded before so we end up with some stale metadata in this case? http://gerrit.cloudera.org:8080/#/c/14806/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@691 PS1, Line 691: continue; Should we make this behavior configurable? This looks like a util that may be used in many places. Not sure if any callers want to deal with the FileNotFoundException. -- To view, visit http://gerrit.cloudera.org:8080/14806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecf6b193b0d57de27d41ad6ef6e1719005d9e908 Gerrit-Change-Number: 14806 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Comment-Date: Sat, 30 Nov 2019 02:16:50 +0000 Gerrit-HasComments: Yes
