[
https://issues.apache.org/jira/browse/HADOOP-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913505#comment-17913505
]
ASF GitHub Bot commented on HADOOP-19354:
-----------------------------------------
rajdchak commented on code in PR #7214:
URL: https://github.com/apache/hadoop/pull/7214#discussion_r1917526565
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/streams/StreamThreadOptions.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl.streams;
+
+/**
+ * Options for threading on this input stream.
+ */
+public class StreamThreadOptions {
+
+ /** Number of shared threads to included in the bounded pool. */
+ private final int sharedThreads;
+
+ /**
+ * How many threads per stream, ignoring vector IO requirements.
+ */
+ private final int streamThreads;
+
+ /**
+ * Flag to enable creation of a future pool around the bounded thread pool.
+ */
+ private final boolean createFuturePool;
+
+ /**
+ * Is vector IO supported (so its thread requirements
+ * included too)?
+ */
+ private final boolean vectorSupported;
+
+ /**
+ * Create the thread options.
+ * @param sharedThreads Number of shared threads to included in the bounded
pool.
+ * @param streamThreads How many threads per stream, ignoring vector IO
requirements.
+ * @param createFuturePool Flag to enable creation of a future pool around
the bounded thread pool.
+ */
Review Comment:
@param vectorSupported missing
> S3A: InputStreams to be created by factory under S3AStore
> ---------------------------------------------------------
>
> Key: HADOOP-19354
> URL: https://issues.apache.org/jira/browse/HADOOP-19354
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Migrate S3AInputStream creation into a factory pattern, push down into
> S3AStore.
> Proposed factories
> * default: whatever this release has as default
> * classic: current S3AInputStream
> * prefetch: prefetching
> * analytics: new analytics stream
> * other: reads a classname from another prop, instantiates.
> Also proposed
> * stream to implement some stream capability to declare what they are
> (classic, prefetch, analytics, other).
> h2. Implementation
> All callbacks used by the stream also to call directly onto S3AStore.
> S3AFileSystem must not be invoked at all (if it is needed: PR is still not
> ready).
> Some interface from Instrumentation will be passed to factory; this shall
> include a way to create new per-stream
> The factory shall implement org.apache.hadoop.service.Service; S3AStore shall
> do same and become a subclass of CompositeService. It shall attach the
> factory as a child, so they can follow the same lifecycle. We shall do the
> same for anything else that gets pushed down.
> Everything related to stream creation must go from s3afs; and creation of the
> factory itself. This must be done in S3AStore.initialize().
> As usual, this will complicate mocking. But the streams themselves should not
> require changes, at least significant ones.
> Testing.
> * The huge file tests should be tuned so each of the different ones uses a
> different stream, always.
> * use a -Dstream="factory name" to choose factory, rather than the -Dprefetch
> * if not set, whatever is in auth-keys gets picked up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
