Hmm, assuming there is no other reverse proxy/WAF/API gateway "in the mix" other than F5 I guess you might need to ask your team if something else changed other than the target server port (from HTTPS on 8154 to HTTP on 8153).
It'll be clearest as to which header(s) might be the culprit if you're able to capture the request as it arrives at GoCD itself (capturing network traffic, possibly, or could try the stuff here on GoCD side <https://docs.gocd.org/current/advanced_usage/logging.html> - not sure of how up-to-date it is) I'm not really super familiar with use of F5 like this, however I imagine there is some kind of rule/scripting required to do the standard proxypass stuff Apache and NGINX do - possibly when editing it for your target port change (*server_port* in F5 terminology, I think?) something was screwed up on the X-Forwarded-Port rule (using *remote_port* instead of *client_port* or something like that)? In this theory, the "random" port is possibly the TCP source port of your browser/curl/whatever rather than the target port the client is connecting to on the LB (443 for a standard https connection). If there are other network load balancers in the mix, other possibilities can arise where original ports are lost, but since there is no mention of any change in networking setup, I am assuming it's just F5 for now. -Chad On Thu, May 19, 2022 at 11:12 PM Jon Barry <[email protected]> wrote: > Thanks Chad for the response. > > My suspicion is that it is we are missing something on the F5. The port I > referred to earlier is a random port that changes on every request. If I > do the following curl the port gets added after the first request through > the F5. > > $ curl http://<server IP>:8153/ -H 'Host: ci.example.com' -H > 'X-Forwarded-For: 5.6.7.8' -H 'X-Forwarded-Proto: https' --location -v > > * About to connect() to <server IP> port 8153 (#0) > * Trying <server IP>... > * Connected to <server IP> (<server IP>) port 8153 (#0) > > GET / HTTP/1.1 > > User-Agent: curl/7.29.0 > > Accept: */* > > Host: ci.example.com > > X-Forwarded-For: 5.6.7.8 > > X-Forwarded-Proto: https > > > < HTTP/1.1 302 Found > < Date: Thu, 19 May 2022 13:31:06 GMT > < Cache-Control: no-cache, must-revalidate, no-store > < X-XSS-Protection: 1; mode=block > < X-Content-Type-Options: nosniff > < X-Frame-Options: SAMEORIGIN > < X-UA-Compatible: chrome=1 > < Location: https://ci.example.com/go/pipelines > < Content-Length: 0 > < > * Connection #0 to host <server IP> left intact > > * Issue another request to this URL: 'https://ci.example.com/go/pipelines' > * About to connect() to ci.example.com port 443 (#1) > * Trying <F5 IP>... > * Connected to ci.example.com (<F5 IP>) port 443 (#1) > * Initializing NSS with certpath: sql:/etc/pki/nssdb > * CAfile: /etc/pki/tls/certs/ca-bundle.crt > CApath: none > * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 > * Server certificate: > * subject: <***> <the LB's certificate> > * start date: <***> > * expire date: <***> > * common name: ci.example.com > * issuer: <***> > > GET /go/pipelines HTTP/1.1 > > User-Agent: curl/7.29.0 > > Host: ci.example.com > > Accept: */* > > X-Forwarded-For: 5.6.7.8 > > X-Forwarded-Proto: https > > > < HTTP/1.1 302 Found > < Date: Thu, 19 May 2022 13:31:06 GMT > < X-XSS-Protection: 1; mode=block > < X-Content-Type-Options: nosniff > < X-Frame-Options: SAMEORIGIN > < X-UA-Compatible: chrome=1 > < Set-Cookie: JSESSIONID=<***>; Path=/go; Expires=Thu, 02-Jun-2022 > 13:31:06 GMT; Max-Age=1209600; Secure; HttpOnly > < Expires: Thu, 01 Jan 1970 00:00:00 GMT > < Location: https://ci.example.com:35499/go/auth/login > < Content-Length: 0 > < Set-Cookie: LB_GO=<***>; path=/; Httponly; Secure > < Strict-Transport-Security: max-age=31536000; includeSubDomains > < > * Connection #1 to host ci.example.com left intact > > * Issue another request to this URL: ' > https://ci.example.com:35499/go/auth/login' > * Found bundle for host ci.example.com: 0x64a290 > * About to connect() to ci.example.com port 35499 (#2) > * Trying <F5 IP>... > * Connection timed out > * Failed connect to ci.example.com:35499; Connection timed out > * Closing connection 2 > curl: (7) Failed connect to ci.example.com:35499; Connection timed out > > On Thursday, 19 May 2022 at 15:08:15 UTC+1 Chad Wilson wrote: > >> Hi Jon >> >> Are you able to see if there are any differences in the HTTP headers of >> the proxied requests from the F5 before and after the change? >> >> My educated guess (albeit possibly naive-to-GoCD-specifics) is that the >> URLs are being built from the incoming request, which inside GoCD is aware >> of headers such as *Forwarded* (RFC7239) and earlier de-facto standards >> such as *X-Forwarded-Port X-Forwarded-Host X-Forwarded-Proto* etc which >> are generally used to communicate the "real host/port/proto" that the >> client is using. >> >> Done canonically, the left-most value of these headers (when multi-value) >> is taken as the outermost value and used to build redirects etc. >> >> I suspect *X-Forwarded-Port* might be set to 37680 rather than unset, or >> 443. Are you able to capture traffic and take a look? If you find those >> headers there on an incoming request, then maybe you can figure out which >> piece of your env is the misbehaving proxy wrt headers. >> >> -Chad >> >> On Thu, May 19, 2022 at 7:51 PM Jon Barry <[email protected]> wrote: >> >>> Hi everyone, >>> >>> I am looking to see if I can get some help on an issue we are seeing >>> after we have upgraded our GoCD Server from version 19.7 to 20.4. Working >>> on getting to the latest release.... >>> >>> Our GoCD server is sitting behind an F5 VIP. The F5 up to and including >>> version 19.7 has been forwarding the traffic to the GoCD server on port >>> 8154. After upgrading to version 20.4, and the SSL changes per >>> https://github.com/gocd/gocd/issues/7872 we changed the member in the >>> pool behind the VIP to go to port 8153 instead of 8154. >>> >>> Since making that change we are seeing the client port being added into >>> the redirects. So for example after a successfull logon from >>> https://our-gocd-server.com/go/auth/login, we get redirected to a url >>> with a port number in it https://our-gocd-server.com:37680/go/pipelines >>> which fails. >>> >>> If we manually remove the port from the url we get to the correct >>> location https://our-gocd-server.com/go/pipelines >>> >>> We are also seeing missing icons on some pages, as the URL to them also >>> has the port in it. e.g: >>> http://our-gocd-server.com:37680/go/api/plugin_images/cd.go.authentication.ldap/9fdf150dbce01084106d76b18e3215af1f9c2beff6f3638e38083237f433662f >>> >>> The *Site URL* and *Secure Site URL *are both set to >>> https://our-gocd-server.com >>> >>> Anyone done something similar or know what the issue may be? I am not >>> the person who manages the F5 so am at a bit of a loss there, but I expect >>> something may be missing there. >>> >>> Thanks in advance. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/966f0230-cf31-446e-809a-ec47281d4f2bn%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/966f0230-cf31-446e-809a-ec47281d4f2bn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/a59907e8-bd37-45a8-9681-6cdd88f94555n%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/a59907e8-bd37-45a8-9681-6cdd88f94555n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_Qpa94H9tnUL6oP5o3h_QA2MZ8zFUBLvW1zkJ8eTq2nw%40mail.gmail.com.
