[
https://issues.apache.org/jira/browse/HADOOP-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793895#comment-16793895
]
Ben Roling commented on HADOOP-16085:
-------------------------------------
With HADOOP-15625 now complete, I've been preparing a new patch for this
rebased on that work. The biggest changes since my previous patch are:
* with the configurable change detection source supporting ETag or versionId,
I store both in the DynamoDB table
* use the S3Guard ETag or versionId in the CopyObjectRequest during rename()
to ensure the object copied is the expected one
Propagating the eTag and versionId for each contained file during a directory
rename() took a few changes. My latest patch introduces an S3LocatedFileStatus
class to carry this. innerListFiles() returns
RemoteIterator<S3LocatedFileStatus> instead of vanilla LocatedFileStatus.
One challenge I'm bumping into is that ETag sometimes changes on copy
(depending on encryption algorithm). Also versionId seems to always change
with copy. What I mean is the new object produced by the copy has a different
ETag or versionId than the copied object. This means that a client-side check
to ensure the copied object is the object that was expected is not really
feasible. Also, supposing the change detection mode is server, it's not
possible to do a client-side check to detect the condition where the server
doesn't support/honor the ETag/versionId specified on the CopyObjectRequest.
Suppose, for example, a writer overwrites "foo.txt" with some new text, and
then renames to "bar.txt". After this operation, the contents of "bar.txt" may
be from the original "foo.txt", not the new "foo.txt". It could happen if the
server is not honoring the specified ETag or versionId. This could go through
without a RemoteFileChangedException or any other such error. The ETag or
versionId passed on the CopyObjectRequest would reflect the correct "foo.txt"
version, but if the server fails to honor it, we cannot catch it. We cannot
look at the ETag or versionId on "bar.txt" because we don't know what value to
expect.
I'm going to go ahead and post my current version of the patch, which
demonstrates this problem through failing rename() tests when
fs.s3a.change.detection.source=versionid or source=etag and the test uses an
encryption type where etag is not the MD5 digest of the object data (SSE-C or
SSE-KMS). See
[here|https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
I still have more work to do on this:
* update copy handling to skip client-side check
* update documentation to warn that change.detection.mode=client has this
vulnerability with rename()
* update documentation to warn 3rd-party S3 implementation users of this
potential vulnerability with rename() if the implementation doesn't honor the
qualification
* improve documentation further in general
* improve the tests. Existing tests are passing with the exception of the
copy/rename issues mentioned, but I need to consider adding some additional
tests. Also, I need to revisit the TestObjectETag test that I had added in the
last patch. With versionId and eTag support I need something that tests both.
Also, I'll consider if there is a more effective or simpler way to test that
the S3Guard metadata is being captured on write and honored on read.
I'm going to be unavailable next week. I might make a little more progress
with this today but after that I will end up coming back to it on Monday, 3/25.
> S3Guard: use object version or etags to protect against inconsistent read
> after replace/overwrite
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-16085
> URL: https://issues.apache.org/jira/browse/HADOOP-16085
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Ben Roling
> Priority: Major
> Attachments: HADOOP-16085_002.patch, HADOOP-16085_3.2.0_001.patch
>
>
> Currently S3Guard doesn't track S3 object versions. If a file is written in
> S3A with S3Guard and then subsequently overwritten, there is no protection
> against the next reader seeing the old version of the file instead of the new
> one.
> It seems like the S3Guard metadata could track the S3 object version. When a
> file is created or updated, the object version could be written to the
> S3Guard metadata. When a file is read, the read out of S3 could be performed
> by object version, ensuring the correct version is retrieved.
> I don't have a lot of direct experience with this yet, but this is my
> impression from looking through the code. My organization is looking to
> shift some datasets stored in HDFS over to S3 and is concerned about this
> potential issue as there are some cases in our codebase that would do an
> overwrite.
> I imagine this idea may have been considered before but I couldn't quite
> track down any JIRAs discussing it. If there is one, feel free to close this
> with a reference to it.
> Am I understanding things correctly? Is this idea feasible? Any feedback
> that could be provided would be appreciated. We may consider crafting a
> patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
