tuhaihe commented on issue #148: URL: https://github.com/apache/cloudberry-pxf/issues/148#issuecomment-5568579262
## Root cause: an `http_proxy` in the environment — not an OS or JDK issue PXF actually starts correctly in both cases. What fails is the readiness probe in `server/pxf-service/src/scripts/pxf`: ```bash until $curl --silent --connect-timeout 1 -I "http://localhost:$PXF_PORT"; | grep 'PXF Server' > /dev/null; do ``` `curl` does **not** automatically bypass proxies for `localhost`. With `http_proxy` set in the shell, this probe is sent to the proxy, which of course cannot reach the loopback interface of the PXF host, and returns `502 Bad Gateway`. `grep 'PXF Server'` therefore never matches, and `pxf start` keeps retrying for the full 300 attempts (`checkWebapp 300 10`) while the service is perfectly healthy. `pxf status` reports `ERROR: PXF is down` for exactly the same reason. ### Evidence (Rocky 9 / Java 17 host; proxy address redacted) The process is alive and listening: ``` $ ps -ef | grep pxf-app | grep -v grep gpadmin 111698 1 2 17:16 pts/0 00:00:15 /usr/lib/jvm/java-17-openjdk/bin/java ... \ -jar /usr/local/cloudberry-pxf/application/pxf-app-2.2.0.jar ... $ ss -lntp | grep 5888 LISTEN 0 100 [::ffff:127.0.0.1]:5888 *:* users:(("java",pid=111698,fd=11)) ``` Proxy variables are set, and `no_proxy` is not set at all: ``` $ env | grep -i proxy https_proxy=http://proxy.internal.example:1080 HTTPS_PROXY=http://proxy.internal.example:1080 HTTP_PROXY=http://proxy.internal.example:1080 http_proxy=http://proxy.internal.example:1080 ``` The probe goes through the proxy and gets a 502: ``` $ curl -v --connect-timeout 1 -I http://localhost:5888 * Uses proxy env variable http_proxy == 'http://proxy.internal.example:1080' > HEAD http://localhost:5888/ HTTP/1.1 < HTTP/1.1 502 Bad Gateway ``` Bypassing the proxy shows the service is up and returns the very header the probe greps for: ``` $ curl -sS --noproxy '*' -D - -o /dev/null http://127.0.0.1:5888 HTTP/1.1 404 Content-Type: application/json Server: PXF Server ``` Note that only the lowercase `http_proxy` matters here: `curl` deliberately ignores `HTTP_PROXY`, and `https_proxy` does not apply to an `http://` URL. One more thing worth recording, since it is misleading while debugging: `$PXF_BASE/logs/pxf-app.out` contains **only** the Spring Boot banner, which looks like an aborted startup. That is expected — the root logger in `pxf-log4j2.xml` only has the `RollingFile` appender, so the real startup log goes to `pxf-service.log`. ### Workaround ```bash export no_proxy="localhost,127.0.0.1,::1,$(hostname),$(hostname -f)" export NO_PROXY="$no_proxy" ``` `curl` only honours the lowercase form, but `pxf-cli` (Go) and the JDBC/HDFS clients read the uppercase one, so set both. This belongs in `~/.bashrc` rather than a single shell, because `pxf cluster ...` runs the script over ssh on each segment host. Also check `~/.curlrc`: the script does not pass `curl -q`, so a `proxy = ...` line there produces identical symptoms even with a clean environment. ### Suggested fix The three `curl` calls in `server/pxf-service/src/scripts/pxf` all target endpoints that are by definition local to the host, so they should never be routed through a proxy: 1. the readiness probe in `waitForSpringBoot` 2. the `/actuator/health` check in `checkWebapp` 3. the `/actuator/shutdown` call in `doStop` — with a proxy set this one silently fails too, so `pxf stop` falls back to `SIGTERM`/`SIGKILL` instead of shutting down gracefully Adding `--noproxy '*'` to all three makes `pxf start`/`status`/`stop` behave correctly regardless of the ambient proxy configuration. I have a patch ready and can open a PR. A separate, smaller observation: the retry message is identical for "not up yet", "failed to start", and "cannot be reached", which is what made this take a while to pin down. Distinguishing "the process died" (pid file / `isRunning`) from "the endpoint is unreachable" would make the output far more actionable. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
