Steve Loughran created HADOOP-16863:
---------------------------------------
Summary: Report on S3A cached 404 recovery better
Key: HADOOP-16863
URL: https://issues.apache.org/jira/browse/HADOOP-16863
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran
A big hadoop -fs copyFromLocal is showing that 404 cacheing is still happening.
{code}
20/02/13 01:02:18 WARN s3a.S3AFileSystem: Failed to find file
s3a://dilbert/dogbert/queries_split_1/catberg.q._COPYING_. Either it is not yet
visible, or it has been deleted.
0/02/13 01:02:18 WARN s3a.S3AFileSystem: Failed to find file
s3a://dilbert/dogbert/queries_split_1/catberg.q._COPYING_. Either it is not yet
visible, or it has been deleted.
{noformat}
We are recovering (good) but it's (a) got the people running this code worried
and (b) shouldn't be happening.
Proposed
* error message to -> to a wiki link to a (new) doc on the topic.
* retried clause to increment counter & if count >1 report on #of attempts and
duration
* S3A FS.deleteOnExit to avoid all checks
* and review the copyFromLocal to make sure no other probes are happening'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
