[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163911#comment-16163911
]
ASF GitHub Bot commented on TRAFODION-2733:
-------------------------------------------
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498166
--- Diff: core/sql/executor/ex_exe_stmt_globals.h ---
@@ -307,18 +307,38 @@ class ExExeStmtGlobals : public ex_globals
inline NABoolean grabMemoryQuotaIfAvailable(ULng32 size)
{
+ CliGlobals *cli_globals = GetCliGlobals();
+ if (cli_globals->isEspProcess())
+ return cli_globals->grabMemoryQuotaIfAvailable(size);
if ( unusedBMOsMemoryQuota_ < size ) return FALSE;
unusedBMOsMemoryQuota_ -= size ;
return TRUE;
}
- inline void resetMemoryQuota() { unusedBMOsMemoryQuota_ = 0 ; }
+ inline void resetMemoryQuota()
+ {
+ CliGlobals *cli_globals = GetCliGlobals();
+ if (cli_globals->isEspProcess())
+ return cli_globals->resetMemoryQuota();
+ unusedBMOsMemoryQuota_ = 0 ;
+ }
--- End diff --
In case of master process the unused memory quota is yielded to others
operators within the statement and hence it is stored in statement globals. In
case of ESP, the unused memory quota to yielded to operators in other fragments
within the process and hence it is stored in the default context. The part of
the code deals with the unused memory quota.
I am assuming that you want details about the quota assignment itself.
Based on this assumption:
Each open statement in the master would get its own quota. If it is
non-ESP plan, each BMO operator will be capped to max BMO limit of 1200 MB as
opposed to 10240 MB per node. If there are more than one BMO operator in such a
plan, each BMO operator memory quota will be assigned proportionally as per
item 14 subjected to min and max limits.
it is a ESP plan, the memory quota will be assigned as per item 14
subjected to min and max limit.
Currently, Trafodion won't assign the instances from a same fragment to a
process. It also won't assign the ESP to two different queries either from the
same context or different contexts. Because TDB contains the memory quota
assignment per instance, different instances within the same process should
work irrespective how the instances are assigned.
Yielding the unused memory quota will work fine as long as the ESP is not
shared for different ESPs at the same time. Currently the ESP is released only
when the statement is deallocated. The current plan is not to share ESPs
when it is multi-threaded. It is just that all the fragments and all the
instances within the same fragment from a query will be hosted in the same
process via different threads.
Hence,
1) Yes. Each open statement would get its own quota assignment
2) All ESPs and BMO operator instances memory consumption from a query will
be limited by BMO_MEMORY_LIMIT_PER_NODE. However, memory consumption won't be
exact amount as specified by this CQD because the memory quota assignment is
subjected to min and max limits.
3) ESPs are not shared currently. There is no plan to share in the
multi-threaded ESP environment also.
4) . Each statement will have its own quota irrespective of the context to
which its belongs
> Provide an improved memory quota assignment for big memory operators (BMO)
> --------------------------------------------------------------------------
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
> Affects Versions: 2.3-incubating
> Reporter: Selvaganesan Govindarajan
> Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
