[
https://issues.apache.org/jira/browse/DRILL-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated DRILL-1990:
-----------------------------------
Description:
Currently we have "localMemoryAllocated" which is always set to zero as we try
to fill the stats at the end of fragment execution by calling
allocator.getAllocatedMemory() which at that point has already released the
allocated memory. Instead if we have a stat for peak memory an allocator has
seen in the lifetime of the operator execution will be useful (each operator
has its own allocator).
Example query on query profile: To find aggregate of peak memory of each
operator across all minor fragments in a major fragment and list them in
descending order of peak memory usage
{code:sql}
SELECT
majorFragmentId,
opProfile['operatorType'] opType,
sum(opProfile['peakLocalMemoryAllocated'])
aggPeakMemoryAcrossAllMinorFragments
FROM
(SELECT
majorFragmentId,
flatten(minorFragProfile['operatorProfile']) opProfile
FROM
(SELECT
majorFragment['majorFragmentId'] majorFragmentId,
flatten(majorFragment['minorFragmentProfile']) minorFragProfile
FROM
(SELECT flatten(fragmentProfile) as majorFragment from
dfs.`/tmp/a.json`)
)
)
-- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular
operator
GROUP BY
majorFragmentId,
opProfile['operatorType']
ORDER BY
aggPeakMemoryAcrossAllMinorFragments DESC;
{code}
{code}
+-----------------+------------+--------------------------------------+
| majorFragmentId | opType | aggPeakMemoryAcrossAllMinorFragments |
+-----------------+------------+--------------------------------------+
| 1 | 4 | 115065856 |
| 1 | 3 | 10027008 |
| 0 | 3 | 1671168 |
| 3 | 6 | 1536000 |
| 2 | 6 | 901120 |
| 1 | 6 | 606208 |
| 3 | 28 | 393216 |
| 2 | 28 | 229376 |
| 3 | 10 | 122880 |
| 2 | 10 | 81920 |
| 0 | 11 | 0 |
| 0 | 10 | 0 |
| 0 | 13 | 0 |
| 1 | 10 | 0 |
| 1 | 11 | 0 |
+-----------------+------------+--------------------------------------+
{code}
was:
Currently we have "localMemoryAllocated" which is always set to zero as the we
try to fill the stats at the end of execution of fragment by calling calling
allocator. getAllocatedMemory() which at that point has already released the
allocated memory. Instead having a peak memory that allocator (each operator
has its own allocator) has seen in the lifetime of the operator execution will
be useful.
Example query on query profile: To find all aggregate peak memory of each
operator across all minor fragments in a major fragment and list them in
descending order peak memory usage
{code:sql}
SELECT
majorFragmentId,
opProfile['operatorType'] opType,
sum(opProfile['peakLocalMemoryAllocated'])
aggPeakMemoryAcrossAllMinorFragments
FROM
(SELECT
majorFragmentId,
flatten(minorFragProfile['operatorProfile']) opProfile
FROM
(SELECT
majorFragment['majorFragmentId'] majorFragmentId,
flatten(majorFragment['minorFragmentProfile']) minorFragProfile
FROM
(SELECT flatten(fragmentProfile) as majorFragment from
dfs.`/tmp/a.json`)
)
)
-- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular
operator
GROUP BY
majorFragmentId,
opProfile['operatorType']
ORDER BY
aggPeakMemoryAcrossAllMinorFragments DESC;
{code}
{code}
+-----------------+------------+--------------------------------------+
| majorFragmentId | opType | aggPeakMemoryAcrossAllMinorFragments |
+-----------------+------------+--------------------------------------+
| 1 | 4 | 115065856 |
| 1 | 3 | 10027008 |
| 0 | 3 | 1671168 |
| 3 | 6 | 1536000 |
| 2 | 6 | 901120 |
| 1 | 6 | 606208 |
| 3 | 28 | 393216 |
| 2 | 28 | 229376 |
| 3 | 10 | 122880 |
| 2 | 10 | 81920 |
| 0 | 11 | 0 |
| 0 | 10 | 0 |
| 0 | 13 | 0 |
| 1 | 10 | 0 |
| 1 | 11 | 0 |
+-----------------+------------+--------------------------------------+
{code}
> Add peak memory allocation in a operator to OperatorStats.
> ----------------------------------------------------------
>
> Key: DRILL-1990
> URL: https://issues.apache.org/jira/browse/DRILL-1990
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Operators
> Reporter: Venki Korukanti
> Assignee: Mehant Baid
> Fix For: 0.8.0
>
> Attachments: DRILL-1990-1.patch
>
>
> Currently we have "localMemoryAllocated" which is always set to zero as we
> try to fill the stats at the end of fragment execution by calling
> allocator.getAllocatedMemory() which at that point has already released the
> allocated memory. Instead if we have a stat for peak memory an allocator has
> seen in the lifetime of the operator execution will be useful (each operator
> has its own allocator).
> Example query on query profile: To find aggregate of peak memory of each
> operator across all minor fragments in a major fragment and list them in
> descending order of peak memory usage
> {code:sql}
> SELECT
> majorFragmentId,
> opProfile['operatorType'] opType,
> sum(opProfile['peakLocalMemoryAllocated'])
> aggPeakMemoryAcrossAllMinorFragments
> FROM
> (SELECT
> majorFragmentId,
> flatten(minorFragProfile['operatorProfile']) opProfile
> FROM
> (SELECT
> majorFragment['majorFragmentId'] majorFragmentId,
> flatten(majorFragment['minorFragmentProfile']) minorFragProfile
> FROM
> (SELECT flatten(fragmentProfile) as majorFragment from
> dfs.`/tmp/a.json`)
> )
> )
> -- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular
> operator
> GROUP BY
> majorFragmentId,
> opProfile['operatorType']
> ORDER BY
> aggPeakMemoryAcrossAllMinorFragments DESC;
> {code}
> {code}
> +-----------------+------------+--------------------------------------+
> | majorFragmentId | opType | aggPeakMemoryAcrossAllMinorFragments |
> +-----------------+------------+--------------------------------------+
> | 1 | 4 | 115065856 |
> | 1 | 3 | 10027008 |
> | 0 | 3 | 1671168 |
> | 3 | 6 | 1536000 |
> | 2 | 6 | 901120 |
> | 1 | 6 | 606208 |
> | 3 | 28 | 393216 |
> | 2 | 28 | 229376 |
> | 3 | 10 | 122880 |
> | 2 | 10 | 81920 |
> | 0 | 11 | 0 |
> | 0 | 10 | 0 |
> | 0 | 13 | 0 |
> | 1 | 10 | 0 |
> | 1 | 11 | 0 |
> +-----------------+------------+--------------------------------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
