[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164721#comment-16164721
]
ASF GitHub Bot commented on TRAFODION-2733:
-------------------------------------------
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138634994
--- Diff: core/sql/generator/GenProbeCache.cpp ---
@@ -470,30 +463,38 @@ CostScalar
ProbeCache::getEstimatedRunTimeMemoryUsage(NABoolean perCPU)
}
const double outputBufferSize = resultSize * numBufferPoolEntries; //
in bytes
- // totalMemory is perCPU at this point of time.
+ // totalMemory is perNode at this point of time.
double totalMemory = cacheSize + outputBufferSize;
-
- if ( perCPU == FALSE ) {
- const PhysicalProperty* const phyProp = getPhysicalProperty();
-
- if (phyProp != NULL)
- {
- PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
- totalMemory *= partFunc->getCountOfPartitions();
- }
+ Lng32 numOfStreams = 1;
+ const PhysicalProperty* const phyProp = getPhysicalProperty();
+ if (phyProp)
+ {
+ PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
+ numOfStreams = partFunc->getCountOfPartitions();
+ if (numOfStreams <= 0)
+ numOfStreams = 1;
+ double memoryLimitPerInstance =
+
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
+ if (totalMemory > memoryLimitPerInstance)
+ totalMemory = memoryLimitPerInstance;
+ totalMemory *= numOfStreams;
}
-
- double memoryLimitPerCpu =
-
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
- if (totalMemory > memoryLimitPerCpu)
- totalMemory = memoryLimitPerCpu;
+ if (numStreams != NULL)
+ *numStreams = numOfStreams;
+ if ( perNode == TRUE )
+ totalMemory /=
MINOF(MAXOF(((NAClusterInfoLinux*)gpClusterInfo)->getTotalNumberOfCPUs(), 1),
numOfStreams);
--- End diff --
While making the change, I realized that I didn't get what is expected. I
believe it is already taking the MINOF(numOfStreams, NumberOfNodes())
> Provide an improved memory quota assignment for big memory operators (BMO)
> --------------------------------------------------------------------------
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
> Affects Versions: 2.3-incubating
> Reporter: Selvaganesan Govindarajan
> Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
