Dariusz Seweryn created NIFI-14301:
--------------------------------------
Summary: ConsumeKinesisStream decouple HttpClient maxConcurrency
from "Max Concurrent Tasks"
Key: NIFI-14301
URL: https://issues.apache.org/jira/browse/NIFI-14301
Project: Apache NiFi
Issue Type: Improvement
Reporter: Dariusz Seweryn
ConsumeKinesisStream currently uses "Max Concurrent Tasks" value as setting for
the underlying NettyNioAsyncHttpClient maxConcurrency value.
On the surface this seems OK from the framework perspective but it seems
wasteful and non-optimal due to the nature of HTTP work which is generally IO
bound whereas the framework assumes the tasks are Compute bound.
ConsumeKinesisStream onTrigger method does two things:
* at first creates a new thread for a AWS Scheduler class that does all the
heavy lifting for Kinesis interaction
* when Scheduler is already created it yields
Having more Concurrent Tasks yielding does not seem to be a productive usage of
framework threads.
Proposal: set NettyNioAsyncHttpClient maxConcurrency to
`Runtime.getRuntime().availableProcessors()`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
