Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17171
to look at the new patch set (#2).
Change subject: IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil
......................................................................
IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil
HDFS FileSystem provides a listStatusIterator() API for listing remote
storage using a RemoteIterator. We use it to list files when loading
table file metadata.
It's not guaranteed that a RemoteIterator can survive when its hasNext()
or next() throws IOExceptions. We should stop the loop in this case.
Otherwise, we may go into a dead loop.
Without HADOOP-16685, it's also not guaranteed that
FileSystem.listStatusIterator() will thrown a FileNotFoundException when
the path doesn't exist.
This patch refactors the file listing iterators so we don't need to
depend on these two assumptions.
Tests:
- Run test_insert_stress.py. Verified the non-existing subdirs are
skipped.
Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea
---
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
1 file changed, 37 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/17171/2
--
To view, visit http://gerrit.cloudera.org:8080/17171
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea
Gerrit-Change-Number: 17171
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
