[
https://issues.apache.org/jira/browse/NIFI-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17943237#comment-17943237
]
David Handermann commented on NIFI-14462:
-----------------------------------------
Thanks for the thorough description and workaround details.
As evidenced by the disabling of connection reuse, the core issue sounds like
NiFi closing the connection after a period of time. NiFi 1 used Jetty 9 and
NiFi 2 uses Jetty 12, so there have been quite a few changes between these
versions.
Reviewing the [ProxyPass
documentation|https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass]
one other parameter worth trying is the {{ttl}} property. The idle timeout in
NiFi Jetty Server is 60 seconds, so the {{ttl}} property should be set to 60
seconds or less. That may allow connection pooling to be used without the other
workarounds.
> Regular getting HTTP 502 Error behind Apache Reverse Proxy
> ----------------------------------------------------------
>
> Key: NIFI-14462
> URL: https://issues.apache.org/jira/browse/NIFI-14462
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 2.3.0
> Environment: Debian 11/12
> Podman 5.3.0/5.4.2
> Apache2: 2.4.62
> Reporter: Fabian Reiber
> Priority: Major
>
> We are using nifi in a secured cluster witch three nodes behind an apache
> reverse proxy. After the update from 1.28.1 to 2.3.0, we regularly get 502
> proxy errors like this in the browser:
> {code:none}
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502
> Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server
> received an invalid response from an upstream server.<br /> The proxy server
> could not handle the request<p>Reason: <strong>Error reading from remote
> server</strong></p></p> <hr> <address>Apache/2.4.62 (Debian) Server at
> nifi.local.example.com Port 443</address> </body></html> {code}
> This did not happen with nifi 1. Nifi fetches various api endpoints while
> running. And each of the endpoint can throw this error. Most of the time the
> requests are ok and nifi runs as expected. But sometimes not. Here is an
> example of the apache error log, after we raise the log level to trace5:
> {code:none}
> [Thu Apr 10 14:58:01.694121 2025] [proxy:trace2] [pid 14770:tid 14793]
> mod_proxy.c(848): [client 192.168.58.1:38278] AH03461: attempting to match
> URI path '/nifi-api/flow/status' against prefix '/' for proxying, referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694125 2025] [proxy:debug] [pid 14770:tid 14784]
> mod_proxy.c(1465): [client 192.168.58.1:38262] AH01143: Running scheme https
> handler (attempt 0), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694139 2025] [proxy:trace1] [pid 14770:tid 14793]
> mod_proxy.c(969): [client 192.168.58.1:38278] AH03464: URI path
> '/nifi-api/flow/status' matches proxy handler
> 'proxy:https://nifi.local.example.com:8443/nifi-api/flow/status', referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694143 2025] [proxy:debug] [pid 14770:tid 14784]
> proxy_util.c(2797): AH00942: https: has acquired connection for
> (nifi.local.example.com:8443)
> [Thu Apr 10 14:58:01.694149 2025] [proxy:debug] [pid 14770:tid 14784]
> proxy_util.c(3242): [client 192.168.58.1:38262] AH00944: connecting
> https://nifi.local.example.com:8443/nifi-api/flow/process-groups/1fb57701-0196-1000-705b-739fe62c8f28?uiOnly=true
> to nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694161 2025] [proxy:debug] [pid 14770:tid 14784]
> proxy_util.c(3450): [client 192.168.58.1:38262] AH00947: connecting
> /nifi-api/flow/process-groups/1fb57701-0196-1000-705b-739fe62c8f28?uiOnly=true
> to 127.0.0.1:8443 (nifi.local.example.com:8443), referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694187 2025] [proxy:trace2] [pid 14770:tid 14784]
> proxy_util.c(3734): https: reusing backend connection
> 127.0.0.1:42282<>127.0.0.1:8443
> [Thu Apr 10 14:58:01.694279 2025] [proxy:trace2] [pid 14770:tid 14793]
> proxy_util.c(2604): [client 192.168.58.1:38278] https: found worker
> https://nifi.local.example.com:8443/ for
> https://nifi.local.example.com:8443/nifi-api/flow/status, referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694300 2025] [proxy:debug] [pid 14770:tid 14793]
> mod_proxy.c(1465): [client 192.168.58.1:38278] AH01143: Running scheme https
> handler (attempt 0), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694328 2025] [proxy:debug] [pid 14770:tid 14793]
> proxy_util.c(2797): AH00942: https: has acquired connection for
> (nifi.local.example.com:8443)
> [Thu Apr 10 14:58:01.694334 2025] [proxy:debug] [pid 14770:tid 14793]
> proxy_util.c(3242): [client 192.168.58.1:38278] AH00944: connecting
> https://nifi.local.example.com:8443/nifi-api/flow/status to
> nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694340 2025] [proxy:debug] [pid 14770:tid 14793]
> proxy_util.c(3450): [client 192.168.58.1:38278] AH00947: connecting
> /nifi-api/flow/status to 127.0.0.1:8443 (nifi.local.example.com:8443),
> referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694745 2025] [proxy:trace2] [pid 14770:tid 14793]
> proxy_util.c(3734): https: reusing backend connection
> 127.0.0.1:42272<>127.0.0.1:8443
> [Thu Apr 10 14:58:01.694887 2025] [proxy_http:error] [pid 14770:tid 14793]
> (70014)End of file found: [client 192.168.58.1:38278] AH01102: error reading
> status line from remote server nifi.local.example.com:8443, referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694905 2025] [proxy:error] [pid 14770:tid 14793] [client
> 192.168.58.1:38278] AH00898: Error reading from remote server returned by
> /nifi-api/flow/status, referer: https://nifi.local.example.com/nifi/{code}
> Actually the last two lines are the most interesting, but don't give much
> information.
> We can reproduce this behavior on two different clusters on different virtual
> machines with different debian and podman versions as well as with a secured
> single node instance from scratch. We are using podman to run nifi in a
> container/pod. We have our own certificate authority to configure the proxy
> with certificates. But I think this is not important in this case. So, the
> location directive of the apache config of the single node instance, which is
> more interesting I guess, looks like this:
> {code:none}
> <Location "/">
> ProxyPass https://nifi.local.example.com:8443/
> ProxyPassReverse https://nifi.local.example.com:8443/
> RequestHeader add X-ProxyScheme "https"
> RequestHeader add X-ProxyHost "nifi.local.example.com"
> RequestHeader add X-ProxyPort "443"
> RequestHeader add X-ProxyContextPath "/"
> </Location>{code}
> This part does not differ a lot to the config of the webserver in our cluster.
> But at least we found two workarounds. If we set "SetEnv
> proxy-initial-not-pooled 1" in the location directive of the apache config,
> we do not get any proxy error again and nifi works as expected. Another
> second possibility is by setting "disablereuse=On" in the "ProxyPass"
> directive.
> The documentation of mod_proxy says about that variable:
> ??This avoids the "proxy: error reading status line from remote server" error
> message caused by the race condition that the backend server closed the
> pooled connection after the connection check by the proxy and before data
> sent by the proxy reached the backend.??
> So, if no pooled connection is reused or not any connection at all is reused
> (with "disablereuse=On"), nifi does not close the connection to early to the
> proxy.
> For more information take a look at the apache documentation for [mod_proxy
> Environmental
> variables|https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html#env] and
> [ProxyPass|https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#ProxyPass]
> If you need more information please let me know :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
