[
https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840849#comment-16840849
]
Aaron Fabbri commented on HADOOP-16279:
---------------------------------------
{quote}With HADOOP-15999 we changed a bit how authoritative mode works. In
S3AFileSystem#innerGetFileStatus if allowAuthoritative is false then it will
send a HEAD request{quote}
You might be able to resolve HADOOP-14468 then.
Caught up on that issue and asked a followup question
[there|https://issues.apache.org/jira/browse/HADOOP-15999?focusedCommentId=16840844&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16840844].
Using TTL or expiry to resolve the OOB problem makes sense.
{quote} AF> why we need more prune() functions added to the MS interface
GB> That prune is for removing expired entries from the ddbms. It uses
last_updated for expiry rather than mod_time.
{quote}
I would err on the side of simplicity and deleting code, especially from public
interfaces like MetadataStore. We want it to be easy to implement and
understand. The prune(optional_prefix, age) contract is basically **deletes as
many entries as possible that are older than age**. (This is not a MUST delete
because some MS implementations may decide to keep some state around for
internal reasons, e.g. dynamo's ancestor requirement, or not wanting to prune
directories due to complexity).
I think you can satisfy the prune() contract with only one of the time bases
(mod time or last updated). It seems like an internal implementation detail
that doesn't need to be exposed.
Interesting related thought: Can we claim that last_updated (metastore write
time) >= mod_time? In general, assuming no clock issues or null fields? Seems
like it. (Argument: you cannot write a FileStatus with mod_time in the future
unless your clocks are messed up).
With that property, can't we just simplify prune to always use last_updated?
Or, either way, it is an internal implementation detail?
{quote}AF> smarter logic that allows you set a policy for handling S3 versus MS
conflicts
GB> So basically what you mean is to add a conflict resolution algorithm when
an entry is expired?
{quote}
Not so much when entry is expired, but when data from S3 conflicts with data
from MS. For example, MS has tombstone but S3 says file exists.
Thanks again for the patch and discussion!
> S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
> -----------------------------------------------------------------------
>
> Key: HADOOP-16279
> URL: https://issues.apache.org/jira/browse/HADOOP-16279
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Gabor Bota
> Assignee: Gabor Bota
> Priority: Major
>
> In HADOOP-15621 we implemented TTL for Authoritative Directory Listings and
> added {{ExpirableMetadata}}. {{DDBPathMetadata}} extends {{PathMetadata}}
> extends {{ExpirableMetadata}}, so all metadata entries in ddb can expire, but
> the implementation is not done yet.
> To complete this feature the following should be done:
> * Add new tests for metadata entry and tombstone expiry to {{ITestS3GuardTtl}}
> * Implement metadata entry and tombstone expiry
> I would like to start a debate on whether we need to use separate expiry
> times for entries and tombstones. My +1 on not using separate settings - so
> only one config name and value.
> ----
> Notes:
> * In HADOOP-13649 the metadata TTL is implemented in LocalMetadataStore,
> using an existing feature in guava's cache implementation. Expiry is set with
> {{fs.s3a.s3guard.local.ttl}}.
> * LocalMetadataStore's TTL and this TTL is different. That TTL is using the
> guava cache's internal solution for the TTL of these entries. This is an
> S3AFileSystem level solution in S3Guard, a layer above all metadata store.
> * This is not the same, and not using the [DDB's TTL
> feature|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html].
> We need a different behavior than what ddb promises: [cleaning once a day
> with a background
> job|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html]
> is not usable for this feature - although it can be used as a general
> cleanup solution separately and independently from S3Guard.
> * Use the same ttl for entries and authoritative directory listing
> * All entries can be expired. Then the returned metadata from the MS will be
> null.
> * Add two new methods pruneExpiredTtl() and pruneExpiredTtl(String keyPrefix)
> to MetadataStore interface. These methods will delete all expired metadata
> from the ms.
> * Use last_updated field in ms for both file metadata and authoritative
> directory expiry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
