[
https://issues.apache.org/jira/browse/NIFI-14386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael W Moser updated NIFI-14386:
-----------------------------------
Description:
ListenHTTP can limit the rate of data that it accepts, which limits the rate
that data is read from a socket. It will eventually read all data from the
socket and commit the file to NiFi repositories. However, perhaps due to data
buffering, the sending side can think it has sent the entire file but get a
SocketTimeoutException because ListenHTTP hasn't finished reading yet.
Typically, after encountering an exception, the sending side can try to send
the file again. This causes duplicate files on the receiving end.
Additionally, large files may never make it through the connection. Because of
this we should remove the Max Data to Receive per Second capability.
There is a viable replacement which should be documented. Follow ListenHTTP
with a ControlRate processor where you set the desired data rate limit.
Configure back pressure on the "success" connection between ListenHTTP and
ControlRate. Back pressure should be limited to an acceptable data buffer size
that is not too large such that the rate limit is unnoticeable.
When back pressure is in effect, ListenHTTP will send 503 Service Unavailable
responses as soon as an HTTP connection is made. This indicates to the sender
that the ListenHTTP is overloaded and to try again later.
was:
ListenHTTP can limit the rate of data that it accepts, which limits the rate
that data is read from a socket. It will eventually read all data from the
socket and commit the file to NiFi repositories. However, perhaps due to data
buffering, the sending side can think it has sent the entire file but get a
SocketTimeoutException because ListenHTTP hasn't finished reading yet.
Typically, after encountering an exception, the sending side can try to send
the file again. This causes duplicate files on the receiving end.
Additionally, large files may never make it through the connection. Because of
this we should remove the Max Data to Receive per Second capability.
There is a viable replacement which should be documented. Follow ListenHTTP
with a ControlRate processor where you set the desired data rate limit.
Configure back pressure on the "success" connection between ListenHTTP and
ControlRate. Back pressure should be limited to an acceptable data buffer size
that is not too large to make the rate limit unnoticeable.
When back pressure is in effect, ListenHTTP will send 503 Service Unavailable
responses as soon as an HTTP connection is made. This indicates to the sender
that the ListenHTTP is overloaded and to try again later.
> Remove ListenHTTP Max Data to Receive per Second capability
> -----------------------------------------------------------
>
> Key: NIFI-14386
> URL: https://issues.apache.org/jira/browse/NIFI-14386
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Michael W Moser
> Assignee: Michael W Moser
> Priority: Minor
>
> ListenHTTP can limit the rate of data that it accepts, which limits the rate
> that data is read from a socket. It will eventually read all data from the
> socket and commit the file to NiFi repositories. However, perhaps due to
> data buffering, the sending side can think it has sent the entire file but
> get a SocketTimeoutException because ListenHTTP hasn't finished reading yet.
> Typically, after encountering an exception, the sending side can try to send
> the file again. This causes duplicate files on the receiving end.
> Additionally, large files may never make it through the connection. Because
> of this we should remove the Max Data to Receive per Second capability.
> There is a viable replacement which should be documented. Follow ListenHTTP
> with a ControlRate processor where you set the desired data rate limit.
> Configure back pressure on the "success" connection between ListenHTTP and
> ControlRate. Back pressure should be limited to an acceptable data buffer
> size that is not too large such that the rate limit is unnoticeable.
> When back pressure is in effect, ListenHTTP will send 503 Service Unavailable
> responses as soon as an HTTP connection is made. This indicates to the
> sender that the ListenHTTP is overloaded and to try again later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
