[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163842#comment-16163842
]
ASF GitHub Bot commented on TRAFODION-2733:
-------------------------------------------
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138491738
--- Diff: core/sql/generator/GenRelJoin.cpp ---
@@ -1784,36 +1784,44 @@ short HashJoin::codeGen(Generator * generator) {
UInt16 numBMOsInFrag = (UInt16)generator->getFragmentDir()->getNumBMOs();
double memQuota = 0;
-
+ double memQuotaRatio;
+ Lng32 numStreams;
+ double bmoMemoryUsagePerNode = getEstimatedRunTimeMemoryUsage(TRUE,
&numStreams).value();
if (mmu != 0) {
memQuota = mmu;
hashj_tdb->setMemoryQuotaMB(mmu);
} else {
// Apply quota system if either one the following two is true:
// 1. the memory limit feature is turned off and more than one BMOs
// 2. the memory limit feature is turned on
- NABoolean mlimitPerCPU = defs.getAsDouble(EXE_MEMORY_LIMIT_PER_CPU) >
0;
-
- if ( mlimitPerCPU || numBMOsInFrag > 1 ||
+ NABoolean mlimitPerNode = defs.getAsDouble(BMO_MEMORY_LIMIT_PER_NODE)
> 0;
+
+ if ( mlimitPerNode || numBMOsInFrag > 1 ||
(numBMOsInFrag == 1 &&
CmpCommon::getDefault(EXE_SINGLE_BMO_QUOTA) == DF_ON)) {
-
memQuota =
computeMemoryQuota(generator->getEspLevel() == 0,
- mlimitPerCPU,
-
generator->getBMOsMemoryLimitPerCPU().value(),
- generator->getTotalNumBMOsPerCPU(),
-
generator->getTotalBMOsMemoryPerCPU().value(),
+ mlimitPerNode,
+
generator->getBMOsMemoryLimitPerNode().value(),
+ generator->getTotalNumBMOs(),
+
generator->getTotalBMOsMemoryPerNode().value(),
numBMOsInFrag,
-
generator->getFragmentDir()->getBMOsMemoryUsage()
+ bmoMemoryUsagePerNode,
+ numStreams,
+ memQuotaRatio
);
-
- Lng32 hjMemoryLowbound =
defs.getAsLong(EXE_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN);
+ }
+ Lng32 hjMemoryLowbound =
defs.getAsLong(BMO_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN);
+ Lng32 memoryUpperbound = defs.getAsLong(BMO_MEMORY_LIMIT_UPPER_BOUND);
- if ( memQuota < hjMemoryLowbound )
- memQuota = hjMemoryLowbound;
-
- hashj_tdb->setMemoryQuotaMB( UInt16(memQuota) );
+ if ( memQuota < hjMemoryLowbound ) {
+ memQuota = hjMemoryLowbound;
+ memQuotaRatio = BMOQuotaRatio::MIN_QUOTA;
}
+ else if (memQuota > memoryUpperbound)
+ memQuota = memoryUpperbound;
+ memQuotaRatio = BMOQuotaRatio::MIN_QUOTA;
+ hashj_tdb->setMemoryQuotaMB( UInt16(memQuota) );
+ hashj_tdb->setBmoQuotaRatio(memQuotaRatio);
--- End diff --
Good Catch. Fixed RelExpr::computeMemoryQuota to return bmoQuotaRatio to
return the ratio correctly. However, this needs to be revisited when the
memory quota is reassigned based on runtime settings. See item 4 in the PR
commit comment.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --------------------------------------------------------------------------
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
> Affects Versions: 2.3-incubating
> Reporter: Selvaganesan Govindarajan
> Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
