Vlad, - My understanding is that operators need to take ownership of incoming buffers (using
the vector method transferTo()) - My view is not that receivers are pass-through; instead, I feel that sender & receiver operators should focus on their business logic - It just happens that the unordered-receiver does very little (deserializes the batch through the BatchLoader) - Contrast this with the merge-receiver which needs to consume data from multiple inputs to provide ordered batches - The operator implementation will dictate how many batches are consumed (this should have nothing to do with communication concerns) - Intricacies of buffering, acking, back-pressuring, etc is ideally left to a communication module My intent, is to consistently report on resource usage (I am fine if we exclude pass-through operators as long as we do it consistently). The next enhancement that I am planning to do is to report on the fragment buffered batches. This will enable us to account for such resources when analyzing memory usage. On Fri, Apr 27, 2018 at 9:50 PM, vrozov <g...@git.apache.org> wrote: > Github user vrozov commented on the issue: > > https://github.com/apache/drill/pull/1237 > > IMO, it will be good to understand what other operators do as well. > For example what Project or Filter operators do. Do they take ownership of > incoming batches? And if they do, when is the ownership taken? > > I do not suggest that we change how Sender and Receiver control > **all** aspects of communication, at least not as part of this JIRA/PR. The > difference in my and your approach is whether or not UnorderedReceiver and > other receivers are pass-through operators. My view is that receivers are > not pass-through operators and they are buffering operators as they receive > batches from the network and buffer them before downstream operators are > ready to consume those batches. In your view, receivers are pass-through > operators that get batches from fragment queue or some other queue and pass > them to downstream. As there is no wait and no processing between getting a > batch from fragment queue and passing it to the next operator, I don't see > why a receiver needs to take the ownership. > > > --- >
