yashmayya opened a new pull request, #19876: URL: https://github.com/apache/pinot/pull/19876
## Problem When one mailbox stream of a stage is lost, the receiving stage times out with a message like `Timed out on stage 2 waiting for data from child stage 3`. The message does not say which sender did not finish, so finding the cause means a long search through logs. A stream can be lost because it reached the wrong server (see #19875 and #19873). That server keeps the data in a receiving mailbox that nothing reads. The only trace is an `Evicting dangling receiving mailbox` warning, 300 s later, and there is no metric for it. ## Change - A receive timeout now names the senders whose end of stream did not arrive, with their worker id and address (at most 5). For example: `Timed out on stage 2 waiting for data from child stage 3 (no end of stream from 1 of 46 senders: worker 17 on server-0-1:8101)`. The names are built only on timeout. - New meter `MULTI_STAGE_ORPHANED_RECEIVING_MAILBOXES`, on servers (`ServerMeter`, through `MseMeter`) and on brokers (`BrokerMeter`). It counts receiving mailboxes that the cache evicted without any operator having read them. The eviction warning also says when a mailbox was never read. ## Compatibility No wire or config change. One new meter on servers and brokers. ## Tests - `MailboxReceiveOperatorTest.shouldNameMissingSendersOnTimeout` - `MailboxServiceTest.testOrphanedReceivingMailboxesAreCounted`: server and broker meters; a mailbox that was read is not counted. This is the third of three PRs: #19875 (channel refresh), #19873 (receiver check) and this one. It does not depend on the other two. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
