Michael W Moser created NIFI-14386:
--------------------------------------
Summary: Remove ListenHTTP Max Data to Receive per Second
capability
Key: NIFI-14386
URL: https://issues.apache.org/jira/browse/NIFI-14386
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Michael W Moser
ListenHTTP can limit the rate of data that it accepts, which limits the rate
that data is read from a socket. It will eventually read all data from the
socket and commit the file to NiFi repositories. However, perhaps due to data
buffering, the sending side can think it has sent the entire file but get a
SocketTimeoutException because ListenHTTP hasn't finished reading yet.
Typically, after encountering an exception, the sending side can try to send
the file again. This causes duplicate files on the receiving end.
Additionally, large files may never make it through the connection. Because of
this we should remove the Max Data to Receive per Second capability.
There is a viable replacement which should be documented. Follow ListenHTTP
with a ControlRate processor where you set the desired data rate limit.
Configure back pressure on the "success" connection between ListenHTTP and
ControlRate. Back pressure should be limited to an acceptable data buffer size
that is not too large to make the rate limit unnoticeable.
When back pressure is in effect, ListenHTTP will send 503 Service Unavailable
responses as soon as an HTTP connection is made. This indicates to the sender
that the ListenHTTP is overloaded and to try again later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
