nicole-lihui commented on issue #16344:
URL: https://github.com/apache/dubbo/issues/16344#issuecomment-4873359972
@zrlw @devjue Thanks for working on this
the delayed-reconnect + atomic channel swap approach is the right direction
and clearly addresses the "no provider available" window.
We hit this issue in production and needed a mitigation quickly, so I tested
this patch against that scenario. Note that my testing is currently on Dubbo
v3.2.10 (not this PR's target branch, 3.3), but the core connection-migration
state machine is the same, so I'd expect the findings to apply equally. After
testing, I don't think this patch fully achieves graceful migration yet — the
failure window is narrower than before, but not fully eliminated:
- **The old channel is closed synchronously**, with no drain period. In
AbstractNettyConnectionClient#onConnected(), current.close() runs the instant
the new channel is swapped in. Any request still in flight on the old channel
at that moment gets cut — there's no grace period to let it finish. Under
concurrent load this still produces failures, just fewer than the original bug.
- **Stream-level GOAWAY handling is untouched**. Netty's
Http2MultiplexHandler fans the connection-level GOAWAY frame out as a user
event to every active HTTP/2 stream child channel.
TripleHttp2ClientResponseHandler#userEventTriggered() still calls ctx.close()
on that event, forcibly killing whatever request is currently running on that
stream — independent of whether the connection-level migration in this PR
succeeds. This looks like a separate bug from the one this PR targets, but it
amplifies the same outage under concurrency.
For these two points, I put together a bytecode-injection Java Agent
prototype (old channel gets a configurable drain timeout instead of closing
immediately; the stream-level GOAWAY user event skips ctx.close()), which can
hot-patch an existing 3.2.10 production process without recompiling or
restarting it. In my testing this eliminates the failure window entirely. Code
is here for reference: https://github.com/nicole-lihui/dubbo32-fix-goaway-agent
One caveat: I'm not very experienced with Java or the Dubbo codebase myself
— this patch was built mainly by following this PR's (#16345) approach with AI
assistance, so its rigor and code quality may fall short. It's meant as a
reproduction/reference for the underlying issue, not a drop-in mergeable
implementation — feedback welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
