[
https://issues.apache.org/jira/browse/HADOOP-17293?focusedWorklogId=498530&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-498530
]
ASF GitHub Bot logged work on HADOOP-17293:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 09/Oct/20 14:14
Start Date: 09/Oct/20 14:14
Worklog Time Spent: 10m
Work Description: bgaborg commented on a change in pull request #2361:
URL: https://github.com/apache/hadoop/pull/2361#discussion_r501626796
##########
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -2956,55 +2956,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
// a file has been found in a non-auth path and the caller has not said
// they only care about directories
LOG.debug("Metadata for {} found in the non-auth metastore.", path);
- // If the timestamp of the pm is close to "now", we don't need to
- // bother with a check of S3. that means:
- // one of : status modtime is close to now,
- // or pm.getLastUpdated() == now
-
- // get the time in which a status modtime is considered valid
- // in a non-auth metastore
- long validTime =
- ttlTimeProvider.getNow() - ttlTimeProvider.getMetadataTtl();
- final long msModTime = msStatus.getModificationTime();
-
- if (msModTime < validTime) {
- LOG.debug("Metastore entry of {} is out of date, probing S3", path);
- try {
- S3AFileStatus s3AFileStatus = s3GetFileStatus(path,
- key,
- probes,
- tombstones,
- needEmptyDirectoryFlag);
- // if the new status is more current than that in the metastore,
- // it means S3 has changed and the store needs updating
- final long s3ModTime = s3AFileStatus.getModificationTime();
-
- if (s3ModTime > msModTime) {
- // there's new data in S3
- LOG.debug("S3Guard metadata for {} is outdated;"
- + " s3modtime={}; msModTime={} updating metastore",
- path, s3ModTime, msModTime);
- // add to S3Guard
- S3Guard.putAndReturn(metadataStore, s3AFileStatus,
- ttlTimeProvider);
- } else {
- // the modtime of the data is the same as/older than the s3guard
- // value either an old object has been found, or the existing one
- // was retrieved in both cases -refresh the S3Guard entry so the
- // record's TTL is updated.
- S3Guard.refreshEntry(metadataStore, pm, s3AFileStatus,
- ttlTimeProvider);
- }
- // return the value
- // note that the checks for empty dir status below can be skipped
- // because the call to s3GetFileStatus include the checks there
- return s3AFileStatus;
- } catch (FileNotFoundException fne) {
- // the attempt to refresh the record failed because there was
- // no entry. Either it is a new file not visible, or it
- // has been deleted (and therefore S3Guard is out of sync with S3)
- LOG.warn("Failed to find file {}. Either it is not yet visible, or
"
- + "it has been deleted.", path);
+ final long msModTime = pm.getFileStatus().getModificationTime();
+
+ S3AFileStatus s3AFileStatus;
+ try {
+ s3AFileStatus = s3GetFileStatus(path,
+ key,
+ probes,
+ tombstones,
+ needEmptyDirectoryFlag);
+ } catch (FileNotFoundException fne) {
+ s3AFileStatus = null;
Review comment:
a debug log could be done here about that we received an exception and
that's why it's null
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##########
@@ -326,7 +326,7 @@ protected Path path() throws IOException {
* @return a number >= 0.
*/
private int getFileStatusHeadCount() {
- return authMode ? 0 : 0;
Review comment:
interesting, why it was 0 : 0? I really haven't noticed it last time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 498530)
Time Spent: 3h 40m (was: 3.5h)
> S3A to always probe S3 in S3A getFileStatus on non-auth paths
> --------------------------------------------------------------
>
> Key: HADOOP-17293
> URL: https://issues.apache.org/jira/browse/HADOOP-17293
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> an incidental part of HDP-13230 was a fix to innerGetFileStatus, wherein
> after a HEAD request we would update the DDB record, so resetting it's TTL
> Applications which did remote updates of buckets without going through
> s3guard are now triggering failures in applications in the cluster when they
> go to open the file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
