Hello Vihang Karajgaonkar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17171
to look at the new patch set (#5).
Change subject: IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil
......................................................................
IMPALA-10579: Fix usage of RemoteIterator in FileSystemUtil
HDFS FileSystem provides a listStatusIterator() API for listing remote
storage using a RemoteIterator. We use it to list files when loading
table file metadata.
It's not guaranteed that a RemoteIterator can survive when its hasNext()
or next() throws IOExceptions. We should stop the loop in this case.
Otherwise, we may go into a infinite loop.
Without HADOOP-16685, it's also not guaranteed that
FileSystem.listStatusIterator() will throw a FileNotFoundException when
the path doesn't exist.
This patch refactors the file listing iterators so we don't need to
depend on these two assumptions. The basic idea is:
- On one side, we should not depends on other RemoteIterator's behavior
after exception.
- On the other side, we try to make our own iterators more robust on
transient sub-directories. So table loading won't be failed by them.
Tests:
- Loop test_insert_stress.py 100 times. Verified the non-existing
subdirs are skipped and inserts are stable in a high concurrency.
Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea
---
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
1 file changed, 83 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/17171/5
--
To view, visit http://gerrit.cloudera.org:8080/17171
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I859bd4f976c51a34eb6a03cefd2ddcdf11656cea
Gerrit-Change-Number: 17171
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
