[
https://issues.apache.org/jira/browse/YUNIKORN-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-3470:
-------------------------------------
Labels: pull-request-available (was: )
> Flaky webservice shutdown caused by incomplete HTTP response body cleanup
> -------------------------------------------------------------------------
>
> Key: YUNIKORN-3470
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3470
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: test - unit
> Environment: Linux arm64
> Go 1.27.1
> Race detector enabled
> Tested on commit a38869d52bd089404e05b8e733e16b84de36035e
> Reporter: Yen-Hua Chen
> Assignee: Yen-Hua Chen
> Priority: Minor
> Labels: pull-request-available
>
> {{Test_RedirectDebugHandler}} intermittently fails during webservice shutdown
> under repeated race-enabled test execution.
> h3. Reproduction
> Tested against:
> {code}
> a38869d52bd089404e05b8e733e16b84de36035e
> {code}
> Command:
> {code}
> go test -race ./pkg/webservice \
> -run '^Test_RedirectDebugHandler$' \
> -count=100 -v
> {code}
> Observed 5 failures in 100 iterations.
> Representative failure:
> {code}
> === RUN Test_RedirectDebugHandler/statedump
> === RUN Test_RedirectDebugHandler/stacks
> === NAME Test_RedirectDebugHandler
> webservice_test.go:61: failed to stop webapp
> --- FAIL: Test_RedirectDebugHandler (5.02s)
> --- PASS: Test_RedirectDebugHandler/statedump (0.00s)
> --- PASS: Test_RedirectDebugHandler/stacks (0.01s)
> {code}
> A longer stress run reproduced 37 failures in 500 iterations.
> h3. Root cause
> The successful HTTP responses in {{Test_RedirectDebugHandler}} are closed
> without first consuming their finite response bodies to EOF.
> Under timing-sensitive execution, response draining can overlap the next
> request. This can leave an additional accepted TCP connection in
> {{StateNew}}, which prevents {{http.Server.Shutdown()}} from reaching
> quiescence before YuniKorn's 5-second shutdown deadline.
> A similar response-lifecycle issue exists in two successful GET requests in
> {{Test_RouterHandling}}.
> The surviving connection can also remain associated with the shared
> {{http.DefaultTransport}}, allowing connection state from the retired
> webservice instance to affect the following test.
> h3. Proposed fix
> Fully consume the finite HTTP response bodies before closing them.
> The change is test-only and does not modify production webservice shutdown
> behavior.
> Validation of the proposed fix:
> {code}
> Test_RedirectDebugHandler -race -count=500:
> before: 37 failures / 500
> after: 0 failures / 500
> Test_RedirectDebugHandler + Test_RouterHandling -race -count=100:
> PASS
> pkg/webservice -race -count=20:
> PASS
> make test:
> PASS
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
