[
https://issues.apache.org/jira/browse/NIFI-8085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Turcsanyi updated NIFI-8085:
----------------------------------
Description:
The currently used {{KafkaConsumer.poll(long timeout)}} method is deprecated in
2.x. The method waits for the metadata response indefinitely until we get it
(so in case of an unavailable/stopped kafka cluster until the connection gets
recovered). It means the {{poll(long)}} acquires and holds a thread exclusively
for this period which can lead to starvation of other processors for timer
driven threads.
Use {{poll(Duration timeout)}} instead which is not simply an overloaded
version of the method but has different semantics
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior).
If can return from metadata fetch after the timeout expires and does not occupy
a thread continuously and makes the processor able to yield.
was:
The currently used {{KafkaConsumer.poll(long timeout)}} method is deprecated in
2.x. The method waits for the metadata response indefinitely until we get it
(so in case of an unavailable/stopped kafka cluster until the connection gets
recovered). It means the {{poll(long)}} acquires and holds a thread exclusively
for this period which can lead to starvation of other processors for timer
driven threads.
Use poll(Duration timeout) instead which is not simply an overloaded version of
the method but has different semantics
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior).
If can return from metadata fetch after the timeout expires and does not occupy
a thread continuously and makes the processor able to yield.
> Use poll(Duration) in ConsumeKafka_2_x processors
> --------------------------------------------------
>
> Key: NIFI-8085
> URL: https://issues.apache.org/jira/browse/NIFI-8085
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Peter Turcsanyi
> Assignee: Peter Turcsanyi
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The currently used {{KafkaConsumer.poll(long timeout)}} method is deprecated
> in 2.x. The method waits for the metadata response indefinitely until we get
> it (so in case of an unavailable/stopped kafka cluster until the connection
> gets recovered). It means the {{poll(long)}} acquires and holds a thread
> exclusively for this period which can lead to starvation of other processors
> for timer driven threads.
> Use {{poll(Duration timeout)}} instead which is not simply an overloaded
> version of the method but has different semantics
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior).
> If can return from metadata fetch after the timeout expires and does not
> occupy a thread continuously and makes the processor able to yield.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
