psolomin commented on code in PR #23540:
URL: https://github.com/apache/beam/pull/23540#discussion_r1142089632
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
This brings 2 issues:
- Having 2 ways of configuring the same thing will be confusing, I think.
And having those 2 ways behaving differently is even more confusing: when you
drop `withConsumerArn()` from read spec, it actually doesn't go away and stays
in the savepoints forever.
- Imagine a scenario: I start from LATEST with EFO, then EFO becomes not
necessary. How do I disable EFO? Let's say I disable via NOT providing
PipelineOptions. Then, `withConsumerArn()` is not a default anymore. And once I
provide `withConsumerArn()` again, I will expect it to be default, but
effectively it won't be.
I know that it's a bit confusing too - NOT to have `withConsumerArn()`, but
the real issue I see is that framework actually assumes `Read` instances not
being effectively editable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
