Steve Loughran created HADOOP-15748:
---------------------------------------
Summary: S3 listing inconsistency can raise NPE in globber
Key: HADOOP-15748
URL: https://issues.apache.org/jira/browse/HADOOP-15748
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.8.4, 2.9.1
Reporter: Steve Loughran
Assignee: Steve Loughran
FileSystem Globber does a listStatus(path) and then, if only one element is
returned, {{getFileStatus(path).isDirectory()}} to see if it is a dir. The way
getFileStatus() is wrapped, IOEs are downgraded to null
On S3, if the path has had entries deleted, the listing may include files which
are no longer there, so the getFileStatus(path),isDirectory triggers an NPE
While its wrong to glob against S3 when its being inconsistent, we should at
least fail gracefully here.
Proposed
# log all IOEs raised in Globber.getFileStatus @ debug
# catch FNFEs and downgrade to warn
# continue
The alternative would be fail fast on FNFE, but that's more traumatic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
