Moritz Mack created BEAM-13443:
----------------------------------
Summary: Poor handling of aggregated records in KinesisIO.read
Key: BEAM-13443
URL: https://issues.apache.org/jira/browse/BEAM-13443
Project: Beam
Issue Type: Bug
Components: io-java-aws
Reporter: Moritz Mack
The way the Kinesis source is implemented it doesn't play well with aggregated
records.
Even using configuration options it's fairly hard to configure it in a way that
becomes sufficiently performant.
One of the key issues is around bundle size & record queue size vs the number
of aggregated records per message. These might, in certain situations, exceed
the internal queue size by far unnecessarily blocking threads and requiring
thread pools to be forcefully taken down.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
