Doris-Breakwater commented on issue #70285: URL: https://github.com/apache/doris/issues/70285#issuecomment-6655103487
Initial triage: the reported timeout is consistent with the Docker entrypoint failing its FE-visible readiness check. The supplied BE activity shows that the process was doing work before the timeout, but does not establish why that check failed. A BE crash or a 4.1.5 engine regression is not demonstrated by these excerpts. Verified in the local `4.1.5-release` source (commit `917b20be83878e98f94e3ebf7efccc80346e3e50`; the scripts shipped in the affected image still need confirmation): - The [BE entrypoint](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/docker/runtime/be/resource/entry_point.sh#L163) runs `mysql -uroot -P9030 -h<selected-FE-IP> -N -e "SHOW BACKENDS"`, then filters for the configured BE IP, configured port, and a `true` token. It discards SQL stderr. After 60 failed attempts with one-second sleeps, it reports the quoted error and exits with status 1, even if the background BE process is running. The timeout is hardcoded in this script. - The selected FE IP is the first FE in `FE_SERVERS`, or `FE_MASTER_IP` in assign mode. The SQL port is hardcoded to 9030; the port supplied in `FE_SERVERS` does not change it. The command supplies no explicit password option. - [Initialization](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/docker/runtime/be/resource/init_be.sh#L169) skips configuration setup and automatic BE registration when `/opt/apache-doris/be/storage/data` already exists. Consequently, reusing storage does not ensure that the current BE address is registered with the current FE. - [FE heartbeat handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java#L303) contacts the registered host and heartbeat port; successful responses drive the BE's `Alive` state. The registration endpoint should use `heartbeat_service_port` (default 9050), distinct from `be_port` (default 9060). Periodic BE logs alone do not establish this state. I also exercised the release entrypoint's check with mocked SQL responses and sleeps disabled: SQL failure, an empty result, a different BE address, and a matching row without a `true` token all exhausted the retries; a matching live row passed. This verifies the check's behavior, not a reproduction of your deployment. Unverified possibilities include SQL connectivity/authentication failure, a stale or mismatched registration, an actual heartbeat failure, or initialization exceeding the wrapper's independent deadline. The current logs cannot distinguish them. Also, the [BE Dockerfile](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/docker/runtime/be/Dockerfile) packages BE and does not start an FE. Please confirm the separate FE deployment when comparing `be-4.1.5` with `4.0.3-all-slim`; the comparison changes the image role as well as the version. Please provide the following to narrow this down: 1. The minimal `docker run` commands or Compose file for FE and BE, including network mode, port mappings, mounts, startup overrides, and relevant environment variables (`FE_SERVERS`/`BE_ADDR` or `FE_MASTER_IP`/`BE_IP`/`BE_PORT`). Include FE and BE versions, image digests, relevant `fe.conf`/`be.conf` settings, and whether FE metadata or BE storage was reused. Redact credentials. 2. During the startup window, run the following **inside the BE container**, using the same FE IP selected by the entrypoint, and preserve stdout, stderr, and exit status: ```bash FE_IP='REPLACE_WITH_SELECTED_FE_IP' command -v mysql mysql -uroot -h"$FE_IP" -P9030 -N -e 'SHOW BACKENDS' echo "mysql exit status=$?" mysql -uroot -h"$FE_IP" -P9030 -e 'SHOW FRONTENDS; SHOW BACKENDS\G' ``` If authentication is required, also compare with an authenticated query, without posting the password. Preserve column names and the relevant `Host`, `HeartbeatPort`, `Alive`, `LastHeartbeat`, `ErrMsg`, and `Version` values; address redaction should retain consistent identities. 3. Complete container startup output, BE logs, and corresponding FE heartbeat errors from startup through timeout, plus Docker's exit code and `OOMKilled` state. Include the affected image's `/usr/local/bin/entry_point.sh` and `/usr/local/bin/init_be.sh` to confirm the source-to-image match. The next step depends on those results: a failed SQL query identifies the probe's connection/authentication path; an absent or mismatched row identifies the registration/address path; a matching row with `Alive=false` calls for the FE heartbeat error and FE-to-BE heartbeat connectivity check. If the exact probe succeeds consistently before the deadline while the wrapper still times out, investigate the image scripts and filtering. Extending the timeout alone is not yet a justified fix. Maintainer focus: Docker startup/readiness, pending these diagnostics. Breakwater-GitHub-Analysis-Slot: slot_6bee17ca48d9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
