ankitsultana commented on issue #10555: URL: https://github.com/apache/pinot/issues/10555#issuecomment-1498530597
Took a look and I think this is what's happening: Say we are sorting in MailboxReceiveOperator and there's a single receiving mailbox. Say these blocks were sent in order: `B1, B2, B3, EOS`. Either via callback or during first run of the OpChain, we will consume `B1` and add it to the priority queue. However, even though we consumed something, we return a NoOp block to the top. Since we receive a NoOp block, we will try to yield, in where we'll check whether there's any callback for this OpChain (by looking up `_seenMail`). Say there is, in that case we will schedule the OpChain again and clear the `_seenMail` entry. In the second run we'll consume `B2` and try to yield again. This time there's no callback entry so stay yielded and wait for automatic promotion to happen. A possible fix for this would be to add a new MailboxReceiveOperator, say `SortedMailboxReceiveOperator` which does something similar to `buildBroadcastHashTable` in the join operator, i.e. continuously poll mailboxes until you get null or EOS. There's a separate issue of missed callbacks btw which I have been punting for a while. I'll invest more time on this ticket next week: https://github.com/apache/pinot/issues/10424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
