mosche commented on code in PR #23540:
URL: https://github.com/apache/beam/pull/23540#discussion_r1142110054
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
yes, I agree that this is by far not optimal ... I fear without adding it to
the read spec too many users won't even notice that there's EFO support and
configuration of IOs using pipeline options is not common (actually, even
discouraged if not absolutely necessary for exactly that reason).
So I guess reasonable semantics for the configuration would be:
- If you provide a consumerArn via the read spec, EFO will ALWAYS be turned
on (that should be the default use case, either have it on or off).
- To temporarily turn it on (e.g. to backfill), one can temporarily provide
consumer arns via pipeline options and remove them again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
