[
https://issues.apache.org/jira/browse/DRILL-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276587#comment-14276587
]
Jacques Nadeau commented on DRILL-1990:
---------------------------------------
looks like wrong patch attached.
> Add peak memory allocation in a operator to OperatorStats.
> ----------------------------------------------------------
>
> Key: DRILL-1990
> URL: https://issues.apache.org/jira/browse/DRILL-1990
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Operators
> Reporter: Venki Korukanti
> Assignee: Mehant Baid
> Fix For: 0.8.0
>
> Attachments:
> 0001-DRILL-1991-Code-indendation-and-formatting-cleanup-f.patch
>
>
> Currently we have "localMemoryAllocated" which is always set to zero as the
> we try to fill the stats at the end of execution of fragment by calling
> calling allocator. getAllocatedMemory() which at that point has already
> released the allocated memory. Instead having a peak memory that allocator
> (each operator has its own allocator) has seen in the lifetime of the
> operator execution will be useful.
> Example query on query profile: To find all aggregate peak memory of each
> operator across all minor fragments in a major fragment and list them in
> descending order peak memory usage
> {code:sql}
> SELECT
> majorFragmentId,
> opProfile['operatorType'] opType,
> sum(opProfile['peakLocalMemoryAllocated'])
> aggPeakMemoryAcrossAllMinorFragments
> FROM
> (SELECT
> majorFragmentId,
> flatten(minorFragProfile['operatorProfile']) opProfile
> FROM
> (SELECT
> majorFragment['majorFragmentId'] majorFragmentId,
> flatten(majorFragment['minorFragmentProfile']) minorFragProfile
> FROM
> (SELECT flatten(fragmentProfile) as majorFragment from
> dfs.`/tmp/a.json`)
> )
> )
> -- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular
> operator
> GROUP BY
> majorFragmentId,
> opProfile['operatorType']
> ORDER BY
> aggPeakMemoryAcrossAllMinorFragments DESC;
> {code}
> {code}
> +-----------------+------------+--------------------------------------+
> | majorFragmentId | opType | aggPeakMemoryAcrossAllMinorFragments |
> +-----------------+------------+--------------------------------------+
> | 1 | 4 | 115065856 |
> | 1 | 3 | 10027008 |
> | 0 | 3 | 1671168 |
> | 3 | 6 | 1536000 |
> | 2 | 6 | 901120 |
> | 1 | 6 | 606208 |
> | 3 | 28 | 393216 |
> | 2 | 28 | 229376 |
> | 3 | 10 | 122880 |
> | 2 | 10 | 81920 |
> | 0 | 11 | 0 |
> | 0 | 10 | 0 |
> | 0 | 13 | 0 |
> | 1 | 10 | 0 |
> | 1 | 11 | 0 |
> +-----------------+------------+--------------------------------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
