[
https://issues.apache.org/jira/browse/NIFI-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-14301.
-------------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
> ConsumeKinesisStream decouple HttpClient maxConcurrency from "Max Concurrent
> Tasks"
> -----------------------------------------------------------------------------------
>
> Key: NIFI-14301
> URL: https://issues.apache.org/jira/browse/NIFI-14301
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Dariusz Seweryn
> Assignee: Dariusz Seweryn
> Priority: Major
> Fix For: 2.3.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> ConsumeKinesisStream currently uses "Max Concurrent Tasks" value as setting
> for the underlying NettyNioAsyncHttpClient maxConcurrency value.
> On the surface this seems OK from the framework perspective but it seems
> wasteful and non-optimal due to the nature of HTTP work which is generally IO
> bound whereas the framework assumes the tasks are Compute bound.
> ConsumeKinesisStream onTrigger method does two things:
> * at first creates a new thread for a AWS Scheduler class that does all the
> heavy lifting for Kinesis interaction
> * when Scheduler is already created it yields
> Having more Concurrent Tasks yielding does not seem to be a productive usage
> of framework threads.
> Proposal: set NettyNioAsyncHttpClient maxConcurrency to minimum of
> `Runtime.getRuntime().availableProcessors()` and processor's "Max Concurrent
> Tasks"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
