[
https://issues.apache.org/jira/browse/NIFI-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Youtsey reassigned NIFI-6278:
------------------------------------
Assignee: Steven Youtsey
> ListenHTTP - Improve throttling and set idle timeout
> ----------------------------------------------------
>
> Key: NIFI-6278
> URL: https://issues.apache.org/jira/browse/NIFI-6278
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Steven Youtsey
> Assignee: Steven Youtsey
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The processor will hang on a blocking read indefinitely and thus stop
> ingesting data. This is typically caused by a heavily loaded listening node
> with many incoming Post requests. When a Post request times out on the
> sending node, the listening node has no knowledge of the timeout since the
> connections are reused on the sending side, thus never closed. The result is
> the ListenHTTP will block on the read. This has been seen on production
> systems when using the Max Data Rate property, but I cannot verify that it
> has occurred without using that property.
> The LeakyBucketStreamThrottler needs a redesign. Rather than incorporating
> the reads from the socket into the Executor's Runnable (Drain), do the reads
> on the incoming connection's thread. This will accomplish 2 things:
> # It will eliminate the need to thread context switch for every buffer being
> read;
> # It will reduce the amount of time needed to make the determination to
> throttle, and thus give a much more accurate rate. Incorporating the socket
> read into the timed thread creates a high degree of inaccuracy due to the
> variations in the loading of the listening server, the loading of the client,
> and the congestion/bandwidth of the networks.
> In essence, the Runnable should only be resetting the total bytes read for
> that 1 sec. interval.
> Also, would like to change the "Max Data to Receive per Second" config prop
> to "Max Data Rate (5 min)". All the stat's are on a 5 min interval, so this
> avoids one more calculation by an Ops person.
> # Propose changing the existing prop to a 'dynamic' prop with a new
> description, "Deprecated - use Max Data Rate"
> # Add a getSupportedDynamicProperties method which simply returns the
> existing PropertyDescriptor when the 'propertyDescriptorName' matches so that
> the existing property will be handled by the framework yet able to be removed
> from the processor's configuration.
> # In 'createHttpServer' (the onScheduled method) check for the existing prop
> and if set, log a warning message and use it's value IF the new property is
> not set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
