psolomin commented on code in PR #23540:
URL: https://github.com/apache/beam/pull/23540#discussion_r1142146037
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
> If you provide a consumerArn via the read spec, EFO will ALWAYS be turned
on (that should be the default use case, either have it on or off).
Still, it's very counter-intuitive that dropping `withConsumerArn()` does
not disable EFO. And if the consumer is de-registered by a mistake, and then a
new one is registered, the user will need to provide Pipeline Options.
The worst part is that Flink's `allowNonRestoredState` can't help here too -
the user will need to get rid of a savepoint entirely to have
`withConsumerArn()` (or its absence) to become effective.
I will meditate a bit more on this, your points are sound, too 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
