[
https://issues.apache.org/jira/browse/NIFI-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835957#comment-16835957
]
Steven Youtsey commented on NIFI-6278:
--------------------------------------
Once I get my dev environment set up, I'll go through the contributors guide to
figure out how to submit a PR and then do so.
> ListenHTTP - Improve throttling and set idle timeout
> ----------------------------------------------------
>
> Key: NIFI-6278
> URL: https://issues.apache.org/jira/browse/NIFI-6278
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Steven Youtsey
> Priority: Major
>
> The processor will hang on a blocking read indefinitely and thus stop
> ingesting data. This is typically caused by a heavily loaded listening node
> with many incoming Post requests. When
> a Post request times out on the sending node, the listening node has no
> knowledge of the timeout since the connections are reused on the sending
> side, thus never closed. The result is the ListenHTTP will block on the read.
> This has been seen on production systems when using the Max Data Rate
> property, but I cannot verify that it has occurred without using that
> property.
> The LeakyBucketStreamThrottler needs a redesign. Rather than incorporating
> the reads from the socket into the Executor's Runnable (Drain), do the reads
> on the incoming connection's thread prior to making the determination to
> throttle. This will accomplish 2 things:
> # It will eliminate the need to thread context switch for every buffer being
> read;
> # It will reduce the amount of time needed to make the determination to
> throttle, and thus give a much more accurate rate. Incorporating the socket
> read into the timed thread creates a high degree of inaccuracy due to the
> variations in the loading of the listening server, the loading of the client,
> and the congestion/bandwidth of the networks.
> In essence, the Runnable should only be computing the total bytes read for
> that 1 sec. interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
