[
https://issues.apache.org/jira/browse/DRILL-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278010#comment-14278010
]
Mehant Baid commented on DRILL-1990:
------------------------------------
Looks good, +1
> Add peak memory allocation in a operator to OperatorStats.
> ----------------------------------------------------------
>
> Key: DRILL-1990
> URL: https://issues.apache.org/jira/browse/DRILL-1990
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Operators
> Reporter: Venki Korukanti
> Assignee: Mehant Baid
> Fix For: 0.8.0
>
> Attachments: DRILL-1990-1.patch
>
>
> Currently we have "localMemoryAllocated" which is always set to zero as we
> try to fill the stats at the end of fragment execution by calling
> allocator.getAllocatedMemory() which at that point has already released the
> allocated memory. Instead if we have a stat for peak memory an allocator has
> seen in the lifetime of the operator execution will be useful (each operator
> has its own allocator).
> Example query on query profile: To find aggregate of peak memory of each
> operator across all minor fragments in a major fragment and list them in
> descending order of peak memory usage
> {code:sql}
> SELECT
> majorFragmentId,
> opProfile['operatorType'] opType,
> sum(opProfile['peakLocalMemoryAllocated'])
> aggPeakMemoryAcrossAllMinorFragments
> FROM
> (SELECT
> majorFragmentId,
> flatten(minorFragProfile['operatorProfile']) opProfile
> FROM
> (SELECT
> majorFragment['majorFragmentId'] majorFragmentId,
> flatten(majorFragment['minorFragmentProfile']) minorFragProfile
> FROM
> (SELECT flatten(fragmentProfile) as majorFragment from
> dfs.`/tmp/a.json`)
> )
> )
> -- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular
> operator
> GROUP BY
> majorFragmentId,
> opProfile['operatorType']
> ORDER BY
> aggPeakMemoryAcrossAllMinorFragments DESC;
> {code}
> {code}
> +-----------------+------------+--------------------------------------+
> | majorFragmentId | opType | aggPeakMemoryAcrossAllMinorFragments |
> +-----------------+------------+--------------------------------------+
> | 1 | 4 | 115065856 |
> | 1 | 3 | 10027008 |
> | 0 | 3 | 1671168 |
> | 3 | 6 | 1536000 |
> | 2 | 6 | 901120 |
> | 1 | 6 | 606208 |
> | 3 | 28 | 393216 |
> | 2 | 28 | 229376 |
> | 3 | 10 | 122880 |
> | 2 | 10 | 81920 |
> | 0 | 11 | 0 |
> | 0 | 10 | 0 |
> | 0 | 13 | 0 |
> | 1 | 10 | 0 |
> | 1 | 11 | 0 |
> +-----------------+------------+--------------------------------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
