Correction for example II as Drill uses a single thread per pipeline (a
batch is fully processed before the next one is; only receive of batches
can happen concurrently):
- Using batch identifiers for more clarity
- t0: (fragment, opr-1, opr-2) = ([b1], [], [])
- t1: (fragment, opr-1, opr-2) =
Another point, I don't see a functional benefit from avoiding a change of
ownership for pass-through operators. Consider the following use-cases:
Example I -
- Single batch of size 8MB is received at time t0 and then is passed
through a set of pass-through operators
- At time t1 owned by operator
Github user vrozov commented on the issue:
https://github.com/apache/drill/pull/1237
IMO, it will be good to understand what other operators do as well. For
example what Project or Filter operators do. Do they take ownership of incoming
batches? And if they do, when is the ownership
Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1237
That was not my intention as my current change aimed at describing the
system the way it is.
@parthchandra, any feedback?
---
Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1237
@vrozov,
**What are we trying to solve / improve**
- Drill is currently not properly reporting memory held in Fragment's
receive queues
- This makes it hard to analyze OOM
Github user vrozov commented on the issue:
https://github.com/apache/drill/pull/1237
IMO, it is better not to report memory usage at all compared to reporting a
wrong number. In case incoming batches are accumulated in a queue, they should
be reported as owned by a receiver. Taking
Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1237
@vrozov, your observation is valid, we need more JIRAs to fix the reporting
problem
**Current Fix**
- At this time, the UnorderedReceiver didn't account for any consumed memory
- This
Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1237
@vrozov, I have implemented the provided suggestions.
---
