Admaing opened a new issue, #18611: URL: https://github.com/apache/dolphinscheduler/issues/18611
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement. ### Description Background / Problem In production behind a statesful firewall / security-group (conntrack based), an HTTP task node calling a long-running backend (no traffic on the wire while the response is being produced, e.g. 150–300 s+) fails with Read timed out. On the same host and against the same backend, curl succeeds. Root-cause The HTTP task builds a per-request Apache HttpClient 4.5 client in HttpTask.createHttpClient() (HttpClients.custom().build()), which creates its sockets with SO_KEEPALIVE disabled. So even when the OS TCP keepalive is properly tuned (see below), the DolphinScheduler socket never emits a keepalive probe while it sits idle waiting for the long-lived response. A conntrack-based security group silently drops an idle TCP session after ~150–300 s. Once dropped, the delayed response is discarded by the middlebox, the client reads nothing and eventually trips its socket read timeout. curl (libcurl) enables TCP keepalive by default, so on a host whose net.ipv4.tcp_keepalive_time has been lowered below the security-group idle timeout, curl's probes refresh the conntrack entry and the request succeeds — DolphinScheduler does not. Code site: dolphinscheduler-task-plugin/dolphinscheduler-task-http/.../HttpTask.java#createHttpClient() (Apache HttpClient 4.5.13 / httpcore 4.4.15, managed via dolphinscheduler-bom). Note: the repo has several other HTTP clients (e.g. OkHttpUtils); this issue is scoped to the HTTP task node, other paths can be a follow-up if reproduced. Verified with a reproducible Java repro Reproduced with a small standalone Apache HttpClient program hitting an endpoint that sleeps 150–300 s before responding behind the same security-group-style environment: default DolphinScheduler-style client (httpclient default build): Read timed out; same client but with SocketConfig.Builder.setSoKeepAlive(true): request succeeds. This matches the root-cause: turning on the OS keepalive probes is what keeps the conntrack entry alive on this environment. Note: enabling SO_KEEPALIVE alone is only effective if the host's net.ipv4.tcp_keepalive_time is lower than the security-group idle timeout (default tcp_keepalive_time = 7200s is far above a 150–300s window, so the probe would never fire in time). On our repro host we first lowered it, e.g.: net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_keepalive_intvl = 10 net.ipv4.tcp_keepalive_probes = 5 With that host-side tuning the keepalive probe fires (~60 s idle) well before the security group drops (~150–300 s). Once SO_KEEPALIVE is on in DolphinScheduler, requests survive; with it off they do not. Suggestion Enable TCP keepalive (SO_KEEPALIVE) on the socket used by the HTTP task, so the socket behaves like curl/libcurl and actually emits keepalive probes when the host has tuned its keepalive times. Also document (near the HTTP-task timeout/connection settings) that relying on this for long requests behind statesful security groups also requires the host net.ipv4.tcp_keepalive_* params to be < the security-group idle timeout. Reproduction Front the backend with a statesful security group that drops idle TCP flows after ~150–300 s, and an endpoint that sleeps > that (e.g. 300 s+) before responding. On the worker host set tcp_keepalive_time low enough (e.g. 60 s — see commands above). Run an HTTP task with a generous socket timeout: fails with Read timed out. curl <url> on the same host succeeds (curl enables TCP keepalive by default); curl --no-keepalive <url> reproduces the failure. Environment OS: Linux (worker); backend reachable through a conntrack / statesful security group / NAT DolphinScheduler: observed on 3.1.9-release (same client logic exists on newer branches; a PR would target dev) Client: Apache HttpClient 4.5.13 / httpcore 4.4.15 Host tuning used during repro: tcp_keepalive_time=60, intvl=10, probes=5 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
