[
https://issues.apache.org/jira/browse/NIFI-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17943277#comment-17943277
]
Fabian Reiber commented on NIFI-14462:
--------------------------------------
Thanks for your response and idea.
First I used a ttl of 60 seconds, but it ends up in the same proxy error as
described. Same with a ttl of 55 and 35 seconds. Than I saw that the [default
ttl in
jetty|https://jetty.org/docs/jetty/12/operations-guide/modules/standard.html]
is 30 seconds. So I tried 30 seconds and 25 seconds. This seems to work
(sometimes it needs some time until the error is thrown, but since a couple of
minutes it works).
Jetty has a thread idle timeout as well. And the default is 60 seconds. Does
nifi use this kind of pool? And does nifi overrides the default idle timeout of
jetty, because you said it has a timeout of 60?
I wonder what the apache web server does, while it has no default ttl!? So, it
has to be more than the idle timeout of the jetty webserver in this case?!
> Regular getting HTTP 502 Error behind Apache Reverse Proxy
> ----------------------------------------------------------
>
> Key: NIFI-14462
> URL: https://issues.apache.org/jira/browse/NIFI-14462
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 2.3.0
> Environment: Debian 11/12
> Podman 5.3.0/5.4.2
> Apache2: 2.4.62
> Reporter: Fabian Reiber
> Priority: Major
>
> We are using nifi in a secured cluster witch three nodes behind an apache
> reverse proxy. After the update from 1.28.1 to 2.3.0, we regularly get 502
> proxy errors like this in the browser:
> {code:none}
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502
> Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server
> received an invalid response from an upstream server.<br /> The proxy server
> could not handle the request<p>Reason: <strong>Error reading from remote
> server</strong></p></p> <hr> <address>Apache/2.4.62 (Debian) Server at
> nifi.local.example.com Port 443</address> </body></html> {code}
> This did not happen with nifi 1. Nifi fetches various api endpoints while
> running. And each of the endpoint can throw this error. Most of the time the
> requests are ok and nifi runs as expected. But sometimes not. Here is an
> example of the apache error log, after we raise the log level to trace5:
> {code:none}
> [Thu Apr 10 14:58:01.694121 2025] [proxy:trace2] [pid 14770:tid 14793]
> mod_proxy.c(848): [client 192.168.58.1:38278] AH03461: attempting to match
> URI path '/nifi-api/flow/status' against prefix '/' for proxying, referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694125 2025] [proxy:debug] [pid 14770:tid 14784]
> mod_proxy.c(1465): [client 192.168.58.1:38262] AH01143: Running scheme https
> handler (attempt 0), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694139 2025] [proxy:trace1] [pid 14770:tid 14793]
> mod_proxy.c(969): [client 192.168.58.1:38278] AH03464: URI path
> '/nifi-api/flow/status' matches proxy handler
> 'proxy:https://nifi.local.example.com:8443/nifi-api/flow/status', referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694143 2025] [proxy:debug] [pid 14770:tid 14784]
> proxy_util.c(2797): AH00942: https: has acquired connection for
> (nifi.local.example.com:8443)
> [Thu Apr 10 14:58:01.694149 2025] [proxy:debug] [pid 14770:tid 14784]
> proxy_util.c(3242): [client 192.168.58.1:38262] AH00944: connecting
> https://nifi.local.example.com:8443/nifi-api/flow/process-groups/1fb57701-0196-1000-705b-739fe62c8f28?uiOnly=true
> to nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694161 2025] [proxy:debug] [pid 14770:tid 14784]
> proxy_util.c(3450): [client 192.168.58.1:38262] AH00947: connecting
> /nifi-api/flow/process-groups/1fb57701-0196-1000-705b-739fe62c8f28?uiOnly=true
> to 127.0.0.1:8443 (nifi.local.example.com:8443), referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694187 2025] [proxy:trace2] [pid 14770:tid 14784]
> proxy_util.c(3734): https: reusing backend connection
> 127.0.0.1:42282<>127.0.0.1:8443
> [Thu Apr 10 14:58:01.694279 2025] [proxy:trace2] [pid 14770:tid 14793]
> proxy_util.c(2604): [client 192.168.58.1:38278] https: found worker
> https://nifi.local.example.com:8443/ for
> https://nifi.local.example.com:8443/nifi-api/flow/status, referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694300 2025] [proxy:debug] [pid 14770:tid 14793]
> mod_proxy.c(1465): [client 192.168.58.1:38278] AH01143: Running scheme https
> handler (attempt 0), referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694328 2025] [proxy:debug] [pid 14770:tid 14793]
> proxy_util.c(2797): AH00942: https: has acquired connection for
> (nifi.local.example.com:8443)
> [Thu Apr 10 14:58:01.694334 2025] [proxy:debug] [pid 14770:tid 14793]
> proxy_util.c(3242): [client 192.168.58.1:38278] AH00944: connecting
> https://nifi.local.example.com:8443/nifi-api/flow/status to
> nifi.local.example.com:8443, referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694340 2025] [proxy:debug] [pid 14770:tid 14793]
> proxy_util.c(3450): [client 192.168.58.1:38278] AH00947: connecting
> /nifi-api/flow/status to 127.0.0.1:8443 (nifi.local.example.com:8443),
> referer: https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694745 2025] [proxy:trace2] [pid 14770:tid 14793]
> proxy_util.c(3734): https: reusing backend connection
> 127.0.0.1:42272<>127.0.0.1:8443
> [Thu Apr 10 14:58:01.694887 2025] [proxy_http:error] [pid 14770:tid 14793]
> (70014)End of file found: [client 192.168.58.1:38278] AH01102: error reading
> status line from remote server nifi.local.example.com:8443, referer:
> https://nifi.local.example.com/nifi/
> [Thu Apr 10 14:58:01.694905 2025] [proxy:error] [pid 14770:tid 14793] [client
> 192.168.58.1:38278] AH00898: Error reading from remote server returned by
> /nifi-api/flow/status, referer: https://nifi.local.example.com/nifi/{code}
> Actually the last two lines are the most interesting, but don't give much
> information.
> We can reproduce this behavior on two different clusters on different virtual
> machines with different debian and podman versions as well as with a secured
> single node instance from scratch. We are using podman to run nifi in a
> container/pod. We have our own certificate authority to configure the proxy
> with certificates. But I think this is not important in this case. So, the
> location directive of the apache config of the single node instance, which is
> more interesting I guess, looks like this:
> {code:none}
> <Location "/">
> ProxyPass https://nifi.local.example.com:8443/
> ProxyPassReverse https://nifi.local.example.com:8443/
> RequestHeader add X-ProxyScheme "https"
> RequestHeader add X-ProxyHost "nifi.local.example.com"
> RequestHeader add X-ProxyPort "443"
> RequestHeader add X-ProxyContextPath "/"
> </Location>{code}
> This part does not differ a lot to the config of the webserver in our cluster.
> But at least we found two workarounds. If we set "SetEnv
> proxy-initial-not-pooled 1" in the location directive of the apache config,
> we do not get any proxy error again and nifi works as expected. Another
> second possibility is by setting "disablereuse=On" in the "ProxyPass"
> directive.
> The documentation of mod_proxy says about that variable:
> ??This avoids the "proxy: error reading status line from remote server" error
> message caused by the race condition that the backend server closed the
> pooled connection after the connection check by the proxy and before data
> sent by the proxy reached the backend.??
> So, if no pooled connection is reused or not any connection at all is reused
> (with "disablereuse=On"), nifi does not close the connection to early to the
> proxy.
> For more information take a look at the apache documentation for [mod_proxy
> Environmental
> variables|https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html#env] and
> [ProxyPass|https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#ProxyPass]
> If you need more information please let me know :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
