[
https://issues.apache.org/jira/browse/BEAM-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Moritz Mack updated BEAM-13443:
-------------------------------
Status: Open (was: Triage Needed)
> Poor handling of aggregated records in KinesisIO.read
> -----------------------------------------------------
>
> Key: BEAM-13443
> URL: https://issues.apache.org/jira/browse/BEAM-13443
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
> Reporter: Moritz Mack
> Assignee: Moritz Mack
> Priority: P2
> Labels: aws, aws-sdk-v1, aws-sdk-v2, performance
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The way the Kinesis source is implemented it doesn't play well with
> aggregated records.
> Even using configuration options it's fairly hard to configure it in a way
> that becomes sufficiently performant.
> One of the key issues is around bundle size & record queue size vs the number
> of aggregated records per message. These might, in certain situations, exceed
> the internal queue size by far unnecessarily blocking threads and requiring
> thread pools to be forcefully taken down.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
