[
https://issues.apache.org/jira/browse/NIFI-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Youtsey updated NIFI-6278:
---------------------------------
Description:
The processor will hang on a blocking read indefinitely and thus stop ingesting
data. This is typically caused by a heavily loaded listening node with many
incoming Post requests. When a Post request times out on the sending node, the
listening node has no knowledge of the timeout since the connections are reused
on the sending side, thus never closed. The result is the ListenHTTP will block
on the read. This has been seen on production systems when using the Max Data
Rate property, but I cannot verify that it has occurred without using that
property.
The LeakyBucketStreamThrottler needs a redesign. Rather than incorporating the
reads from the socket into the Executor's Runnable (Drain), do the reads on the
incoming connection's thread prior to making the determination to throttle.
This will accomplish 2 things:
# It will eliminate the need to thread context switch for every buffer being
read;
# It will reduce the amount of time needed to make the determination to
throttle, and thus give a much more accurate rate. Incorporating the socket
read into the timed thread creates a high degree of inaccuracy due to the
variations in the loading of the listening server, the loading of the client,
and the congestion/bandwidth of the networks.
In essence, the Runnable should only be computing the total bytes read for that
1 sec. interval.
was:
The processor will hang on a blocking read indefinitely and thus stop ingesting
data. This is typically caused by a heavily loaded listening node with many
incoming Post requests. When
a Post request times out on the sending node, the listening node has no
knowledge of the timeout since the connections are reused on the sending side,
thus never closed. The result is the ListenHTTP will block on the read. This
has been seen on production systems when using the Max Data Rate property, but
I cannot verify that it has occurred without using that property.
The LeakyBucketStreamThrottler needs a redesign. Rather than incorporating the
reads from the socket into the Executor's Runnable (Drain), do the reads on the
incoming connection's thread prior to making the determination to throttle.
This will accomplish 2 things:
# It will eliminate the need to thread context switch for every buffer being
read;
# It will reduce the amount of time needed to make the determination to
throttle, and thus give a much more accurate rate. Incorporating the socket
read into the timed thread creates a high degree of inaccuracy due to the
variations in the loading of the listening server, the loading of the client,
and the congestion/bandwidth of the networks.
In essence, the Runnable should only be computing the total bytes read for that
1 sec. interval.
> ListenHTTP - Improve throttling and set idle timeout
> ----------------------------------------------------
>
> Key: NIFI-6278
> URL: https://issues.apache.org/jira/browse/NIFI-6278
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Steven Youtsey
> Priority: Major
>
> The processor will hang on a blocking read indefinitely and thus stop
> ingesting data. This is typically caused by a heavily loaded listening node
> with many incoming Post requests. When a Post request times out on the
> sending node, the listening node has no knowledge of the timeout since the
> connections are reused on the sending side, thus never closed. The result is
> the ListenHTTP will block on the read. This has been seen on production
> systems when using the Max Data Rate property, but I cannot verify that it
> has occurred without using that property.
> The LeakyBucketStreamThrottler needs a redesign. Rather than incorporating
> the reads from the socket into the Executor's Runnable (Drain), do the reads
> on the incoming connection's thread prior to making the determination to
> throttle. This will accomplish 2 things:
> # It will eliminate the need to thread context switch for every buffer being
> read;
> # It will reduce the amount of time needed to make the determination to
> throttle, and thus give a much more accurate rate. Incorporating the socket
> read into the timed thread creates a high degree of inaccuracy due to the
> variations in the loading of the listening server, the loading of the client,
> and the congestion/bandwidth of the networks.
> In essence, the Runnable should only be computing the total bytes read for
> that 1 sec. interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
