ericsyh opened a new issue, #15487: URL: https://github.com/apache/pulsar/issues/15487
For a pulsar cluster with both proxy and broker running on istio with the support, I noticed an interesting problem that all http requests forwarded by pulsar proxies to brokers would fail, but other requests seem to be working fine, including - pulsar binary requests that produce & consume messages - http requests served by proxies directly, like `/status.html`, `/metrics/` - http requests issued from the proxy pod directly using commands like `curl` The error request and response looks just like ``` ➜ ~ curl -v https://proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev/admin/v2/clusters * Trying 198.18.9.176:443... * Connected to proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev (198.18.9.176) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Request CERT (13): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Certificate (11): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=*.sn-grizzly-dev-us-west-2.test.aws.sn2.dev * start date: Feb 9 06:56:33 2022 GMT * expire date: May 10 06:56:32 2022 GMT * subjectAltName: host "proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev" matched cert's "*.sn-grizzly-dev-us-west-2.test.aws.sn2.dev" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. > GET /admin/v2/clusters HTTP/1.1 > Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev > User-Agent: curl/7.79.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 503 Service Unavailable < Date: Thu, 17 Mar 2022 08:02:22 GMT < Content-Type: text/plain < Date: Thu, 17 Mar 2022 08:02:21 GMT < Server: envoy < Content-Length: 95 < * Connection #0 to host proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev left intact upstream connect error or disconnect/reset before headers. reset reason: connection termination% ``` After some investigations, it seems the proxy would copy as much of the original http request as possible and send them to the broker service, as shown in the proxy debug log, including the original `HOST` value which might actually caused the issue: ``` 2022-03-17T07:47:59,957+0000 [pulsar-external-web-5-4] DEBUG org.apache.pulsar.proxy.server.AdminProxyHandler.7c6189d5 - 1945615336 proxying to upstream: GET /admin/v2/clusters HTTP/1.1 Accept: */* User-Agent: curl/7.79.1 Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev HttpRequest[GET /admin/v2/clusters HTTP/1.1]@4759582c Accept: */* User-Agent: curl/7.79.1 Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev Via: 1.1 sn-sn-platform-proxy-0 X-Forwarded-For: 10.80.20.192 X-Forwarded-Proto: https X-Forwarded-Host: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev X-Forwarded-Server: 10.80.10.189 ``` And the issue could be reproduced with `curl` on the proxy pod using the command `curl -v "$brokerWebServiceURL/admin/v2/clusters" -H 'HOST: proxy.sn-grizzly-dev-us-west-2.test.aws.sn2.dev'`, while `curl -v "$brokerWebServiceURL/admin/v2/clusters"` works as expected. On the other hand, specifying the broker hostname when making the http request to the proxy would also work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
