[
https://issues.apache.org/jira/browse/HADOOP-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894199#comment-16894199
]
Steve Loughran commented on HADOOP-16458:
-----------------------------------------
Exception is raised if either of two conditions is met
# path doesn't exist
# path exists but a filter passed in doesn't match
In FileInputFormat scans, that filter is instantiated from
{{"mapreduce.input.pathFilter.class"}} (old and new APIs);
If you see this exception, check your filter classname as well as the path
existence.
Access permissions on S3A objects do not generate these exceptions. You will
get an AccessDeniedException when not running in S3Guard in auth mode; in Auth
Mode you get success/failure based on the conditions
> LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
> -----------------------------------------------------------------------
>
> Key: HADOOP-16458
> URL: https://issues.apache.org/jira/browse/HADOOP-16458
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Environment: S3 + S3Guard
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is
> using globStatus to find files.
> I'd say "turn s3guard on" except this appears to be the case, and the dataset
> being read is
> over 1h old.
> Which means it is harder than I'd like to blame S3 for what would sound like
> an inconsistency
> We're hampered by the number of debug level statements in the globber code
> being approximately none; there's no debugging to turn on. All we know is
> that globFiles returns null without any explanation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
