[
https://issues.apache.org/jira/browse/BEAM-3819?focusedWorklogId=85265&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-85265
]
ASF GitHub Bot logged work on BEAM-3819:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Mar/18 13:31
Start Date: 28/Mar/18 13:31
Worklog Time Spent: 10m
Work Description: iemejia commented on issue #4851: [BEAM-3819] Add
withLimit() option to KinesisIO
URL: https://github.com/apache/beam/pull/4851#issuecomment-376887515
Hi, I have forgotten to take a look on this one My excuses Alexey. I think
that this 'knob' is quite useful (and that it can even be required) because
Kinesis can be overwhelmed if the shards are not able to keep with the readers
so rate-limiting make sense. Remember the limits of kinesis are let's put it
mildly 'restrictive'
https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
However I am not sure the suggested approach will work considering that we
cannot know in advance how many workers are actively reading from each shard to
have an accurate level of precision (the total). So I would suggest probably to
let it like it is and let this calculation to the user because we cannot do it
precisely from each individual source/reader.
I agree with @pawel-kaczmarczyk that the name is not that right, maybe
`withMaxRecordsPerRequest`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 85265)
Time Spent: 1h 50m (was: 1h 40m)
> Add withLimit() option to KinesisIO
> -----------------------------------
>
> Key: BEAM-3819
> URL: https://issues.apache.org/jira/browse/BEAM-3819
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
> Reporter: Jean-Baptiste Onofré
> Assignee: Alexey Romanenko
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In some cases, the user might need to set the {{limit}} on the
> {{SimplifiedKinesisClient}}, especially for performance reason, depending of
> the number of records.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
