[
https://issues.apache.org/jira/browse/FLINK-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461994#comment-16461994
]
Tarush Grover edited comment on FLINK-9178 at 5/3/18 6:31 AM:
--------------------------------------------------------------
[~tzulitai] Actually I had checked this {{max.partition.fetch.bytes}}, there
might be two ways that we can add rate control one way is by specifying number
of bytes and other is by mentioning number of events eg. `{{x events per
fetch}}`. We have the configuration `{{max.partition.fetch.bytes}}` which give
the maximum amount of data per-partition the server. Now we have to decide
whether we want to implement second way or we can go with first one.
was (Author: app-tarush):
[~tzulitai] Actually I had checked this `max.partition.fetch.bytes`, there
might be two ways that we can add rate control one way is by specifying number
of bytes and other is by mentioning number of events eg. `x events per fetch`.
We have the configuration `max.partition.fetch.bytes` which give the maximum
amount of data per-partition the server. Now we have to decide whether we want
to implement second way or we can go with first one.
> Add rate control for kafka source
> ---------------------------------
>
> Key: FLINK-9178
> URL: https://issues.apache.org/jira/browse/FLINK-9178
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Reporter: buptljy
> Assignee: Tarush Grover
> Priority: Major
>
> When I want to run the flink program from the earliest offset in Kafka, it'll
> be very easy to cause OOM if there are too much data, because of too many
> HeapMemorySegment in NetworkBufferPool.
> Maybe we should have some settings to control the rate of the receiving data?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
