Yen-Hua Chen created YUNIKORN-3470:
--------------------------------------
Summary: Flaky webservice shutdown caused by incomplete HTTP
response body cleanup
Key: YUNIKORN-3470
URL: https://issues.apache.org/jira/browse/YUNIKORN-3470
Project: Apache YuniKorn
Issue Type: Bug
Components: test - unit
Environment: Linux arm64
Go 1.27.1
Race detector enabled
Tested on commit a38869d52bd089404e05b8e733e16b84de36035e
Reporter: Yen-Hua Chen
Assignee: Yen-Hua Chen
{{Test_RedirectDebugHandler}} intermittently fails during webservice shutdown
under repeated race-enabled test execution.
h3. Reproduction
Tested against:
{code}
a38869d52bd089404e05b8e733e16b84de36035e
{code}
Command:
{code}
go test -race ./pkg/webservice \
-run '^Test_RedirectDebugHandler$' \
-count=100 -v
{code}
Observed 5 failures in 100 iterations.
Representative failure:
{code}
=== RUN Test_RedirectDebugHandler/statedump
=== RUN Test_RedirectDebugHandler/stacks
=== NAME Test_RedirectDebugHandler
webservice_test.go:61: failed to stop webapp
--- FAIL: Test_RedirectDebugHandler (5.02s)
--- PASS: Test_RedirectDebugHandler/statedump (0.00s)
--- PASS: Test_RedirectDebugHandler/stacks (0.01s)
{code}
A longer stress run reproduced 37 failures in 500 iterations.
h3. Root cause
The successful HTTP responses in {{Test_RedirectDebugHandler}} are closed
without first consuming their finite response bodies to EOF.
Under timing-sensitive execution, response draining can overlap the next
request. This can leave an additional accepted TCP connection in {{StateNew}},
which prevents {{http.Server.Shutdown()}} from reaching quiescence before
YuniKorn's 5-second shutdown deadline.
A similar response-lifecycle issue exists in two successful GET requests in
{{Test_RouterHandling}}.
The surviving connection can also remain associated with the shared
{{http.DefaultTransport}}, allowing connection state from the retired
webservice instance to affect the following test.
h3. Proposed fix
Fully consume the finite HTTP response bodies before closing them.
The change is test-only and does not modify production webservice shutdown
behavior.
Validation of the proposed fix:
{code}
Test_RedirectDebugHandler -race -count=500:
before: 37 failures / 500
after: 0 failures / 500
Test_RedirectDebugHandler + Test_RouterHandling -race -count=100:
PASS
pkg/webservice -race -count=20:
PASS
make test:
PASS
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
