[
https://issues.apache.org/jira/browse/HADOOP-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907289#comment-17907289
]
ASF GitHub Bot commented on HADOOP-19354:
-----------------------------------------
ahmarsuhail opened a new pull request, #7237:
URL: https://github.com/apache/hadoop/pull/7237
### Description of PR
This PR makes some additional changes to the initial PR from Steve:
https://github.com/apache/hadoop/pull/7214.
Not sure if I got all of this right, but here are a few callouts:
* Thought of moving InputStreamCallbacks into S3AStore, but I don't think
that is the right place for it. InputStreamCallBacks uses
`S3AFileSystemOperations`, which uses `S3AStore`, so you end up in a dependency
mess. Instead, move them to a separate class `InputStreamCallbacksImpl`, which
keeps the code out of S3AFileSystem. Maybe there is a better way to do this,
but I couldn't think of anything.
* Adds a new config, `fs.s3a.input.stream.type`. This can be set to
`classic`, `prefetch`, `analytics`. Believe this is better than having
multiple`prefetch.enabled` and `analytics.enabled` flags.
* Could not figure out what was meant by "S3Store to implement the factory
interface, completing final binding operations (callbacks, stats)" in Steve's
original PR, let's discuss.
* This is a draft PR, was just attempting to complete the original PR.
### How was this patch tested?
Not tested. Whoever picks this up for completion can test!
> S3A: InputStreams to be created by factory under S3AStore
> ---------------------------------------------------------
>
> Key: HADOOP-19354
> URL: https://issues.apache.org/jira/browse/HADOOP-19354
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Migrate S3AInputStream creation into a factory pattern, push down into
> S3AStore.
> Proposed factories
> * default: whatever this release has as default
> * classic: current S3AInputStream
> * prefetch: prefetching
> * analytics: new analytics stream
> * other: reads a classname from another prop, instantiates.
> Also proposed
> * stream to implement some stream capability to declare what they are
> (classic, prefetch, analytics, other).
> h2. Implementation
> All callbacks used by the stream also to call directly onto S3AStore.
> S3AFileSystem must not be invoked at all (if it is needed: PR is still not
> ready).
> Some interface from Instrumentation will be passed to factory; this shall
> include a way to create new per-stream
> The factory shall implement org.apache.hadoop.service.Service; S3AStore shall
> do same and become a subclass of CompositeService. It shall attach the
> factory as a child, so they can follow the same lifecycle. We shall do the
> same for anything else that gets pushed down.
> Everything related to stream creation must go from s3afs; and creation of the
> factory itself. This must be done in S3AStore.initialize().
> As usual, this will complicate mocking. But the streams themselves should not
> require changes, at least significant ones.
> Testing.
> * The huge file tests should be tuned so each of the different ones uses a
> different stream, always.
> * use a -Dstream="factory name" to choose factory, rather than the -Dprefetch
> * if not set, whatever is in auth-keys gets picked up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
