Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1237
@vrozov,
**What are we trying to solve / improve**
- Drill is currently not properly reporting memory held in Fragment's
receive queues
- This makes it hard to analyze OOM conditions
This is what I want to see
- Every operator reporting on the resources it is currently using (needed)
- Fragment held resources (other than the ones already reported by the
child operators)
- Drilbit level (metadata caches, web-server, ..)
- I am ok to incrementally reach this goal
**Data Exchange Logistic**
- Ideally, the data exchange fabric should be decoupled from the Drill
Receive / Send operators
- The fabric should be handling all the aspects of pre-fetch / pressuring
and so forth
- It will tune to the speed of producers / consumers when writing / reading
data from it
- This infrastructure should have its own resource management and reporting
capabilities
**Operator based Reporting**
- Receive and Send operators shall not worry about batches they didn't
consume yet
- Doing so is counter productive as the Data Exchange fabric will interpret
a "drain" operation as the operator "needing" more data.
- For example, the merge-receiver should not be managing the receive
queues; it should only advertise the pattern of data consumption and let the
exchange fabric figure out the rest.
The main difference in the two approaches, is that essentially, you are
preaching for Receive and Send operators to control all aspects of
communication whereas I am preaching for decoupling such aspects.
---
