[
https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772965#comment-16772965
]
Steve Loughran commented on HADOOP-15625:
-----------------------------------------
third party stores are always somewhat trouble: if you look @ the tests we
always provide a way to turn off things (encryption, sessions, tokens, &c) we
assume aren't there.
you tend to get some different kinds of store
* Full AWS compatible by people who put in the effort to help test (Kudos to
WDC here)
* ones which are fairly complete, but for which there are the odd corner case
(if modified since etc). A good one: some don't handle a GET of content length
0-0 on a zero byte file, which we naively ask for on a 0 byte file. We should
fix that our side for performance alone.
* work in progress ones which should be using our client as part of their test
suite (example: Ozone's S3 adapter, which is still rounding out stuff like
multipart uploads)
I think we should actually have a configurable policy here for versioning with
a property like fs.s3a.change.detection
* {{none}}: no checks. Turn on if either it's 3rd party or some problem with it
is surfacing
* {{etag}} server side with if-modified since
* {{client}} client etag
* {{warn}} client etag with a warning over a failure
then if versioning support is added, a new option could be added.
Yes, this complicates testing. Imagine in a new parameterized test which would
be skipped entirely if the base test configuration for the test store was set
to "none"
looking at {{ITestS3AMiscOperations}} we have a test there
(testChecksumLengthPastEOF) which relies on the checksum being non-null, hence
the etag. And nobody has complained *yet*. But we do have those checksums
disabled by default as it broke distcp (HADOOP-15297) so it may not have yet
surfaced in the wild. It's actually that if-modified-since check which I worry
about, because even though it's part of the HTTP spec, I can imagine some S3
implementation not doing it. A client-side variant would allow the checks but
softly
The other reason I like "warn" is it how we could downgrade handling of
inconsistencies with s3guard to logging issues, but not failing.
> S3A input stream to use etags to detect changed source files
> ------------------------------------------------------------
>
> Key: HADOOP-15625
> URL: https://issues.apache.org/jira/browse/HADOOP-15625
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
> Priority: Major
> Attachments: HADOOP-15625-001.patch, HADOOP-15625-002.patch,
> HADOOP-15625-003.patch
>
>
> S3A input stream doesn't handle changing source files any better than the
> other cloud store connectors. Specifically: it doesn't noticed it has
> changed, caches the length from startup, and whenever a seek triggers a new
> GET, you may get one of: old data, new data, and even perhaps go from new
> data to old data due to eventual consistency.
> We can't do anything to stop this, but we could detect changes by
> # caching the etag of the first HEAD/GET (we don't get that HEAD on open with
> S3Guard, BTW)
> # on future GET requests, verify the etag of the response
> # raise an IOE if the remote file changed during the read.
> It's a more dramatic failure, but it stops changes silently corrupting things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
