Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14806 )
Change subject: IMPALA-9122 : Ignore FileNotFoundException when loading a table ...................................................................... Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG@9 PS1, Line 9: It is possible that when the file metadata of a table or partition is > Could you please format the commit message to be 72 char width? Done http://gerrit.cloudera.org:8080/#/c/14806/1//COMMIT_MSG@18 PS1, Line 18: soon as it sees a INSERT_EVENT from metastore. Hive may be still > Should we retry the reload operation? I'm not sure whether this case is pos I think this is a much harder problem to solve and would need a more integrated front end with Hive. In theory, hive could remove the files after load operation and it will still cause failures on the scanner side without a manual refresh. Just fixing this problem during load is not really going to help solve this issue completely. Also, there is a chance that we will keep reloading the table for a table which is updated by Hive at a high frequency. http://gerrit.cloudera.org:8080/#/c/14806/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/14806/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@687 PS1, Line 687: // instance, hive deletes the temporary files in the .hive-staging directory > Is it possible that some removed files are loaded before so we end up with Unfortunately, its possible that the files once loaded are removed later. Currently, I don't see a good way to solve it without support from a integrated front-end across query engines. http://gerrit.cloudera.org:8080/#/c/14806/1/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@691 PS1, Line 691: continue; > Should we make this behavior configurable? This looks like a util that may The only place currently this class gets used is in FileMetadataLoader.load(). Not sure if its worth making it configurable. I can add a constructor which takes in a boolean to throw v/s continue if you think otherwise. -- To view, visit http://gerrit.cloudera.org:8080/14806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iecf6b193b0d57de27d41ad6ef6e1719005d9e908 Gerrit-Change-Number: 14806 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Comment-Date: Wed, 04 Dec 2019 00:32:09 +0000 Gerrit-HasComments: Yes
