Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17171
Change subject: IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil ...................................................................... IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil HDFS FileSystem provides a listStatusIterator() API for listing remote storage using a RemoteIterator. We use it to list files when loading table file metadata. It's not guaranteed that a RemoteIterator can survive when its hasNext() or next() throws IOExceptions. We should stop the loop in this case. Otherwise, we may go into a dead loop. Without HADOOP-16685, it's also not guaranteed that FileSystem.listStatusIterator() will thrown a FileNotFoundException when the path doesn't exist. This patch refactors the file listing iterators so we don't need to depend on these two assumptions. Tests: - Run test_insert_stress.py. Verified the non-existing subdirs are skipped. Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea --- M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java 1 file changed, 37 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/17171/1 -- To view, visit http://gerrit.cloudera.org:8080/17171 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea Gerrit-Change-Number: 17171 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
