mosche commented on code in PR #23540:
URL: https://github.com/apache/beam/pull/23540#discussion_r1142969122
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
> To me, introducing a config which has different semantics on different
runners is even more questionable. But it sounds like we need an opinion of
someone who could say if this is a norm for other IOs.
The norm is configuration of IOs is always done via the read / write spec,
see the [IO standard
docs](https://beam.apache.org/documentation/io/io-standards/#all-sdks):
> An I/O should rarely rely on a PipelineOptions subclass to tune internal
parameters. If neccesary, a connector-related pipeline options class should ...
The pipeline option is a just an exceptional workaround to support a valid
use case on some runner (namely the Flink runner).
Maybe we should postpone support for this use case all together. It could be
a follow up in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
