[ https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163926#comment-16163926 ]
ASF GitHub Bot commented on TRAFODION-2733: ------------------------------------------- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138499177 --- Diff: core/sql/generator/GenProbeCache.cpp --- @@ -470,30 +463,38 @@ CostScalar ProbeCache::getEstimatedRunTimeMemoryUsage(NABoolean perCPU) } const double outputBufferSize = resultSize * numBufferPoolEntries; // in bytes - // totalMemory is perCPU at this point of time. + // totalMemory is perNode at this point of time. double totalMemory = cacheSize + outputBufferSize; - - if ( perCPU == FALSE ) { - const PhysicalProperty* const phyProp = getPhysicalProperty(); - - if (phyProp != NULL) - { - PartitioningFunction * partFunc = phyProp -> getPartitioningFunction() ; - totalMemory *= partFunc->getCountOfPartitions(); - } + Lng32 numOfStreams = 1; + const PhysicalProperty* const phyProp = getPhysicalProperty(); + if (phyProp) + { + PartitioningFunction * partFunc = phyProp -> getPartitioningFunction() ; + numOfStreams = partFunc->getCountOfPartitions(); + if (numOfStreams <= 0) + numOfStreams = 1; + double memoryLimitPerInstance = + ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) * 1024 * 1024; + if (totalMemory > memoryLimitPerInstance) + totalMemory = memoryLimitPerInstance; + totalMemory *= numOfStreams; } - - double memoryLimitPerCpu = - ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) * 1024 * 1024; - if (totalMemory > memoryLimitPerCpu) - totalMemory = memoryLimitPerCpu; + if (numStreams != NULL) + *numStreams = numOfStreams; + if ( perNode == TRUE ) --- End diff -- I am curious why is it unsafe? Can you please give some details so that I can modify it accordingly. > Provide an improved memory quota assignment for big memory operators (BMO) > -------------------------------------------------------------------------- > > Key: TRAFODION-2733 > URL: https://issues.apache.org/jira/browse/TRAFODION-2733 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp, sql-exe > Affects Versions: 2.3-incubating > Reporter: Selvaganesan Govindarajan > Assignee: Selvaganesan Govindarajan > Fix For: 2.3-incubating > > > The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort. > Trafodion deploys multiple executor server processes (ESPs) to execute a > query via its data flow architecture. Each ESPs can have an instance of this > BMO operator. Currently, each instance of this operator can potentially have > memory quota of 800 MB assigned to do its BMO operation. However, the memory > allocation is usually limited by the memory pressure when this BMO attempts > to allocate memory within the assigned quota. The assignment doesn't depend > upon the estimation of memory needed by this operation. > Improvement needed in BMO memory assignment are: > 1. Limit the memory quota assignment for these BMO operations per node > 2. Memory quota assigned taking into consideration estimated memory needed > at every operator. > 3. Ensure that the BMO gets the minimum memory needed at least to function > smoothly -- This message was sent by Atlassian JIRA (v6.4.14#64029)