[
https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839356#comment-15839356
]
Aaron Fabbri commented on HADOOP-13936:
---------------------------------------
This is part of a general class of problems. In general, since we do not have
transactions around updating the two sources of truth, things can get out of
sync when failures happen.
Transactions are probably not worth the cost here. The biggest issue IMO is
the fact that we're running in the FS client. You really want a long-lived
daemon with durable storage to implement transactions. Another significant
issue is the runtime cost in terms of more round trips.
The current approach is to detect problems when they occur and give folks tools
to repair them. Simply wiping the MetadataStore is one easy way to fix
problems. Another more automatic, but still simple approach, that I plan on
implementing in the future, is to set configurable TTL on entries in the
MetadataStore. We will contribute a patch soon which adds a "expire" or "trim"
command to the CLI, which will delete any entries older than X hours. We could
also add a config knob to the client which ignores entries past a certain TTL.
That doesn't immediately fix inconsistencies, but does cause them to
automatically heal after some time.
> S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation
> -------------------------------------------------------------------------
>
> Key: HADOOP-13936
> URL: https://issues.apache.org/jira/browse/HADOOP-13936
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Priority: Minor
>
> As a part of {{S3AFileSystem.delete}} operation {{innerDelete}} is invoked,
> which deletes keys from S3 in batches (default is 1000). But DynamoDB is
> updated only at the end of this operation. This can cause issues when
> deleting large number of keys.
> E.g, it is possible to get exception after deleting 1000 keys and in such
> cases dynamoDB would not be updated. This can cause DynamoDB to go out of
> sync.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
