[
https://issues.apache.org/jira/browse/KAFKA-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved KAFKA-10846.
------------------------------------
Fix Version/s: 2.8.0
Resolution: Fixed
> FileStreamSourceTask buffer can grow without bound
> --------------------------------------------------
>
> Key: KAFKA-10846
> URL: https://issues.apache.org/jira/browse/KAFKA-10846
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Tom Bentley
> Assignee: Tom Bentley
> Priority: Major
> Fix For: 2.8.0
>
>
> When reading a large file the buffer used by {{FileStreamSourceTask}} can
> grow without bound. Even in the unit test
> org.apache.kafka.connect.file.FileStreamSourceTaskTest#testBatchSize the
> buffer grows from 1,024 to 524,288 bytes just reading 10,000 copies of a line
> of <100 chars.
> The problem is that the condition for growing the buffer is incorrect. The
> buffer is doubled whenever some bytes were read and the used space in the
> buffer == the buffer length.
> The requirement to increase the buffer size should be related to whether
> {{extractLine()}} actually managed to read any lines. It's only when no
> complete lines were read since the last call to {{read()}} that we need to
> increase the buffer size (to cope with the large line).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
