psolomin commented on code in PR #23540:
URL: https://github.com/apache/beam/pull/23540#discussion_r1142951411
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
> Dataflow
To me, introducing a config which has different semantics on different
runners is even more questionable. But it sounds like we need an opinion of
someone who could say if this is a norm for other IOs.
> --kinesisIOConsumerArns={"stream1":null}
Glad we have a work-around. I was also thinking of making `KinesisIO.Read`
mutated inside `KinesisSource` like this:
```
spec = spec.withConsumerArn(< smth which somes from PipelineOptions >);
```
This way, it could still be possible to "clear up" the persisted state, but
I am afraid of potential races with such - like, what if that happens on
different nodes at different times, and some partition in a savepoint becomes
EFO and the other non-EFO, etc..
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:
##########
@@ -147,6 +150,42 @@
* Read#withCustomRateLimitPolicy(RateLimitPolicyFactory)}. This requires
implementing {@link
* RateLimitPolicy} with a corresponding {@link RateLimitPolicyFactory}.
*
+ * <h4>Enhanced Fan-Out</h4>
+ *
+ * Kinesis IO supports Consumers with Dedicated Throughput (Enhanced Fan-Out,
EFO). This type of
+ * consumer doesn't have to contend with other consumers that are receiving
data from the stream.
+ *
+ * <p>More details can be found here: <a
+ *
href="https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html";>Consumers
with
+ * Dedicated Throughput</a>
+ *
+ * <p>EFO can be enabled for one or more {@link KinesisIO.Read} instances via
pipeline options:
Review Comment:
> Dataflow
To me, introducing a config which has different semantics on different
runners is even more questionable. But it sounds like we need an opinion of
someone who could say if this is a norm for other IOs.
> --kinesisIOConsumerArns={"stream1":null}
Glad we have a work-around. I was also thinking of making `KinesisIO.Read`
mutated inside `KinesisSource` like this:
```
spec = spec.withConsumerArn(< smth which comes from PipelineOptions >);
```
This way, it could still be possible to "clear up" the persisted state, but
I am afraid of potential races with such - like, what if that happens on
different nodes at different times, and some partition in a savepoint becomes
EFO and the other non-EFO, etc..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
