[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290193#comment-16290193
]
ASF GitHub Bot commented on TRAFODION-2733:
-------------------------------------------
GitHub user selvaganesang opened a pull request:
https://github.com/apache/incubator-trafodion/pull/1341
[TRAFODION-2733] BMO quota changes
To ensure BMO quota assignment works even when the estimates are way off.
Introduced a CQD BMO_MEMORY_ESTIMATE_OUTLIER_FACTOR. When a BMO memory
estimate exceeds BMO_MEMORY_LIMIT_PER_NODE_IN_MB by this factor,
the BMO memory estimate for the operator is reset to
BMO_MEMORY_LIMIT_PER_NODE_IN_MB * BMO_MEMORY_ESTIMATE_OUTLIER_FACTOR.
Then BMO memory estimate and the quota is assigned based on the
revised estimate.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/selvaganesang/incubator-trafodion
trafodion-2733
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/1341.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1341
----
commit 375eeeab7a178348c5d3ac6e89e40ee8a06cb0d2
Author: selvaganesang <[email protected]>
Date: 2017-12-06T19:50:04Z
[TRAFODION-2733] BMO quota changes
To ensure BMO quota assignment works even when the estimates are way off.
Introduced a CQD BMO_MEMORY_ESTIMATE_OUTLIER_FACTOR. When a BMO memory
estimate exceeds BMO_MEMORY_LIMIT_PER_NODE_IN_MB by this factor,
the BMO memory estimate for the operator is reset to
BMO_MEMORY_LIMIT_PER_NODE_IN_MB * BMO_MEMORY_ESTIMATE_OUTLIER_FACTOR.
Then BMO memory estimate and the quota is assigned based on the
revised estimate.
----
> Provide an improved memory quota assignment for big memory operators (BMO)
> --------------------------------------------------------------------------
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
> Affects Versions: 2.3-incubating
> Reporter: Selvaganesan Govindarajan
> Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
