[
https://issues.apache.org/jira/browse/FLINK-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460891#comment-16460891
]
Tzu-Li (Gordon) Tai edited comment on FLINK-9178 at 5/2/18 11:22 AM:
---------------------------------------------------------------------
[~app-tarush]
Sorry for slow late reply here.
Before we move on to an implementation of this inside the connector, have you
looked into Kafka's own client configurations and whether or not they support
this?
IIRC, there are configurations that allow controlling how much data is
returned on each fetch, e.g. {{max.partition.fetch.bytes}}. Wouldn't this work?
was (Author: tzulitai):
[~app-tarush]
Sorry for slow late reply here.
Before we move on to an implementation of this inside the connector, have you
looked into Kafka's own client configurations and whether or not they support
this?
IIRC, there are configurations that allow controlling how much data is returned
on each fetch, i.e. `max.partition.fetch.bytes`. Wouldn't this work?
> Add rate control for kafka source
> ---------------------------------
>
> Key: FLINK-9178
> URL: https://issues.apache.org/jira/browse/FLINK-9178
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Reporter: buptljy
> Assignee: Tarush Grover
> Priority: Major
>
> When I want to run the flink program from the earliest offset in Kafka, it'll
> be very easy to cause OOM if there are too much data, because of too many
> HeapMemorySegment in NetworkBufferPool.
> Maybe we should have some settings to control the rate of the receiving data?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
