[
https://issues.apache.org/jira/browse/HADOOP-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798111#comment-16798111
]
Steve Loughran commented on HADOOP-16085:
-----------------------------------------
I've got a short term patch for HADOOP-16190 which I'd like reviewed and in
first, as it includes instructions on testing setup and is straightforward to
backport. Reviews encouraged: https://github.com/apache/hadoop/pull/606
the tracking of etag/version in s3guard should line up with it.
bq. We need to provide these tests to show that the improvement is backward
compatible so no need to do any update manually.
we wouldn't be able to run mixed clients without it, and that's going to
happen. Which is something to bear in mind: old clients will be adding files
without etags or version markers. If you can do a unified roll out of clients
(e.g. its an EC2 hosted cluster where all VMs are on the same image, all will
be well, but in a more heterogenous cluster, that can't hold. Be good to test
for that with some workflow of
* create a file, versioned
* remove the version column values for that entry (may need some s3guard test
methods here
* work with the file again
That's to simulate an overwrite.
bq. S3 eventual consistency manifests in a way where one read sees the new
version and the next read sees the old version. This seems unlikely but I don't
think there is any guarantee it couldn't happen.
seen that happen with openstack swift. It's why all our contract tests set to
have unique names across test cases....accidental contamination of subsequent
tests with the data from old runs. For s3 select, well, users will just have to
deal with it. "Don't overwrite a file you are querying"
> S3Guard: use object version or etags to protect against inconsistent read
> after replace/overwrite
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-16085
> URL: https://issues.apache.org/jira/browse/HADOOP-16085
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Ben Roling
> Priority: Major
> Attachments: HADOOP-16085-003.patch, HADOOP-16085_002.patch,
> HADOOP-16085_3.2.0_001.patch
>
>
> Currently S3Guard doesn't track S3 object versions. If a file is written in
> S3A with S3Guard and then subsequently overwritten, there is no protection
> against the next reader seeing the old version of the file instead of the new
> one.
> It seems like the S3Guard metadata could track the S3 object version. When a
> file is created or updated, the object version could be written to the
> S3Guard metadata. When a file is read, the read out of S3 could be performed
> by object version, ensuring the correct version is retrieved.
> I don't have a lot of direct experience with this yet, but this is my
> impression from looking through the code. My organization is looking to
> shift some datasets stored in HDFS over to S3 and is concerned about this
> potential issue as there are some cases in our codebase that would do an
> overwrite.
> I imagine this idea may have been considered before but I couldn't quite
> track down any JIRAs discussing it. If there is one, feel free to close this
> with a reference to it.
> Am I understanding things correctly? Is this idea feasible? Any feedback
> that could be provided would be appreciated. We may consider crafting a
> patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
