[
https://issues.apache.org/jira/browse/HADOOP-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765059#comment-16765059
]
Steve Loughran commented on HADOOP-15999:
-----------------------------------------
-1
not quite yet I'm afraid, though we are close. And this reminds me that without
resilience to rename/delete failures auth mode still has issues.
h3. Docs
It's going to need an update to the s3guard.md doc, something about "S3Guard
and out-of-band IO".
h3. S3AFileSystem
I think we should add another metric/counter of S3Guard out-of-band events.
That way changes aren't just silently swallowed. It also allows our unit tests
to call getStorageStatistics to measure what S3AFileSystem has seen.
* L2307 "S3aFileStatus is null" is correct from the view of the code, but
meaningless for users.And it doesn't include the actual path. Better
"Failed to find file {}. Either it is not yet visible, or it has been deleted"
* L2313 nit: Space between if and (
h3. S3Guard.java
L239: put a space between if and ()
L241. Don't call toString() on the status, let SLF4J do it on demand. Also, fix
typo on "wtih"
h3. ITestS3GuardOutOfBandOperations
* check your import ordering
* L46: use a <pre> section round the bits of the javadoc you want to stay
formatted all the way through to javadocs. If you are really ambitious: build
the javadocs to see
* L210: you can use ContractTestUtils.writeTextFile() to write that text file.
* L233: overwriteFile shoud use the path() method to return a path for testing.
These are (a) in a subtree unique to each JVM testing in parallel and (b)
cleaned up in test teardown.
I worry about the switching between modes of the filesystem. Would it be better
to create a new FS instance which has been switched to no store, e..g a
guardedFS and a rawFS? That'd make clearer what is happening.
* L312. swap order of assertEquals() args
* L333. again, use path()
* L379 expectExceptionWhenReading(). Use LambdaTestUtils.intercept, and have
the closure actually return the contents of the file
> [s3a] Better support for out-of-band operations
> -----------------------------------------------
>
> Key: HADOOP-15999
> URL: https://issues.apache.org/jira/browse/HADOOP-15999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Sean Mackrory
> Assignee: Gabor Bota
> Priority: Major
> Attachments: HADOOP-15999.001.patch, HADOOP-15999.002.patch,
> out-of-band-operations.patch
>
>
> S3Guard was initially done on the premise that a new MetadataStore would be
> the source of truth, and that it wouldn't provide guarantees if updates were
> done without using S3Guard.
> I've been seeing increased demand for better support for scenarios where
> operations are done on the data that can't reasonably be done with S3Guard
> involved. For example:
> * A file is deleted using S3Guard, and replaced by some other tool. S3Guard
> can't tell the difference between the new file and delete / list
> inconsistency and continues to treat the file as deleted.
> * An S3Guard-ed file is overwritten by a longer file by some other tool. When
> reading the file, only the length of the original file is read.
> We could possibly have smarter behavior here by querying both S3 and the
> MetadataStore (even in cases where we may currently only query the
> MetadataStore in getFileStatus) and use whichever one has the higher modified
> time.
> This kills the performance boost we currently get in some workloads with the
> short-circuited getFileStatus, but we could keep it with authoritative mode
> which should give a larger performance boost. At least we'd get more
> correctness without authoritative mode and a clear declaration of when we can
> make the assumptions required to short-circuit the process. If we can't
> consider S3Guard the source of truth, we need to defer to S3 more.
> We'd need to be extra sure of any locality / time zone issues if we start
> relying on mod_time more directly, but currently we're tracking the
> modification time as returned by S3 anyway.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
