psolomin commented on code in PR #23540:
URL: https://github.com/apache/beam/pull/23540#discussion_r1142570882
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
> I fear without adding it to the read spec too many users won't even notice
that there's EFO support
How about adding something like `withEnhancedFanOutEnabled()`?
- If set - the source will try to lookup a consumer ARN from PipelineOptions
- If consumer ARN not found - a warning is thrown and non-EFO consumer
starts
- If consumer ARN found - EFO consumer starts
- If not set - the source will try to lookup a consumer ARN from
PipelineOptions anyway
- If consumer ARN not found - start non-EFO consumer without a warning
- If consumer ARN found - EFO consumer starts
When defined like that, it will still serve the purpose of "documenting"
pipeline code, but enabling / disabling / changing consumer ARN will be
configured solely by PipelineOptions.
At the same time, there will be no ARN string sitting anywhere in the
serialised state.
To me, it seems to be a reasonable middle-ground. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
