wu-sheng commented on issue #13958: URL: https://github.com/apache/skywalking/issues/13958#issuecomment-5151410687
We finished the root-cause analysis and the fix. Summary of what is actually happening: **This is not related to the OAP version.** The `Kick Failure (eventfd_write: Bad file descriptor)` errors come entirely from the gRPC client's polling engine at `fork()` time — we reproduced the identical output against a mock collector with no OAP involved at all. (Upgrading from 9.7.0 is still a good idea, but for other reasons.) **It is a grpc-python fork regression, and it is not specific to 1.83.0.** gRPC Python has been migrating to the EventEngine poller since grpcio **1.80.0**; in our tests grpcio 1.82.1 produces the same error spam in the same Gunicorn scenario. What changed in 1.83.0 (grpc/grpc#42828) is that the last opt-out was removed, so the legacy fork recipe (`GRPC_ENABLE_FORK_SUPPORT` + `GRPC_POLL_STRATEGY=poll`) that the Python agent applied in prefork mode — and that gRPC's own `fork_support.md` still documents — no longer works. See upstream grpc/grpc#43055 and grpc/grpc#43062, both open without a fix. **Good news:** in all of our reproductions, tracing kept working despite the errors — workers served traffic and every trace segment was delivered. The spam is stale-file-descriptor noise. However, we also found a worse latent failure mode on some grpcio versions: a forked worker can occasionally deadlock silently inside gRPC's own at-fork handlers (the worker never boots and Gunicorn does not notice), so this needed a structural fix, not a version pin. **The fix is in apache/skywalking-python#409.** The root of every failure mode is a live gRPC channel in the Gunicorn master at `fork()` time, so the agent no longer creates one: with `sw-python run -p gunicorn`, the master now only installs instrumentation, and the full agent (including the gRPC channel) starts in each forked worker — the same model the uWSGI path has always used. This was verified clean on grpcio 1.80.0 / 1.82.1 / 1.83.0 with repeated fork cycles. The fixed agent requires `grpcio >= 1.83` (the generated protobuf stubs already require it at import time). **Workarounds for released agents (<= 1.2.0) until the next release:** - pin `grpcio<1.80` (the last series fully clean of the fork regression), or - switch the reporter to `SW_AGENT_PROTOCOL=http` (or `kafka`), which avoids gRPC entirely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
