[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164051#comment-16164051
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138511601
--- Diff: core/sql/generator/GenProbeCache.cpp ---
@@ -470,30 +463,38 @@ CostScalar
ProbeCache::getEstimatedRunTimeMemoryUsage(NABoolean perCPU)
}
const double outputBufferSize = resultSize * numBufferPoolEntries; //
in bytes
- // totalMemory is perCPU at this point of time.
+ // totalMemory is perNode at this point of time.
double totalMemory = cacheSize + outputBufferSize;
-
- if ( perCPU == FALSE ) {
- const PhysicalProperty* const phyProp = getPhysicalProperty();
-
- if (phyProp != NULL)
- {
- PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
- totalMemory *= partFunc->getCountOfPartitions();
- }
+ Lng32 numOfStreams = 1;
+ const PhysicalProperty* const phyProp = getPhysicalProperty();
+ if (phyProp)
+ {
+ PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
+ numOfStreams = partFunc->getCountOfPartitions();
+ if (numOfStreams <= 0)
+numOfStreams = 1;
+ double memoryLimitPerInstance =
+
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
+ if (totalMemory > memoryLimitPerInstance)
+totalMemory = memoryLimitPerInstance;
+ totalMemory *= numOfStreams;
}
-
- double memoryLimitPerCpu =
-
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
- if (totalMemory > memoryLimitPerCpu)
- totalMemory = memoryLimitPerCpu;
+ if (numStreams != NULL)
+ *numStreams = numOfStreams;
+ if ( perNode == TRUE )
--- End diff --
Yes, I guess you didn't write this, it's old code. As far as I understand,
TRUE is defined as 1. If perNode happens to be -1, we will not execute the if
statement, but we usually want to treat any value other than 0 as true. That's
why I would either use if (perNode) or if (perNode != FALSE).
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163929#comment-16163929
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138499497
--- Diff: core/sql/generator/GenProbeCache.cpp ---
@@ -470,30 +463,38 @@ CostScalar
ProbeCache::getEstimatedRunTimeMemoryUsage(NABoolean perCPU)
}
const double outputBufferSize = resultSize * numBufferPoolEntries; //
in bytes
- // totalMemory is perCPU at this point of time.
+ // totalMemory is perNode at this point of time.
double totalMemory = cacheSize + outputBufferSize;
-
- if ( perCPU == FALSE ) {
- const PhysicalProperty* const phyProp = getPhysicalProperty();
-
- if (phyProp != NULL)
- {
- PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
- totalMemory *= partFunc->getCountOfPartitions();
- }
+ Lng32 numOfStreams = 1;
+ const PhysicalProperty* const phyProp = getPhysicalProperty();
+ if (phyProp)
+ {
+ PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
+ numOfStreams = partFunc->getCountOfPartitions();
+ if (numOfStreams <= 0)
+numOfStreams = 1;
+ double memoryLimitPerInstance =
+
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
+ if (totalMemory > memoryLimitPerInstance)
+totalMemory = memoryLimitPerInstance;
+ totalMemory *= numOfStreams;
}
-
- double memoryLimitPerCpu =
-
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
- if (totalMemory > memoryLimitPerCpu)
- totalMemory = memoryLimitPerCpu;
+ if (numStreams != NULL)
+ *numStreams = numOfStreams;
+ if ( perNode == TRUE )
+ totalMemory /=
MINOF(MAXOF(((NAClusterInfoLinux*)gpClusterInfo)->getTotalNumberOfCPUs(), 1),
numOfStreams);
--- End diff --
will do
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163926#comment-16163926
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138499177
--- Diff: core/sql/generator/GenProbeCache.cpp ---
@@ -470,30 +463,38 @@ CostScalar
ProbeCache::getEstimatedRunTimeMemoryUsage(NABoolean perCPU)
}
const double outputBufferSize = resultSize * numBufferPoolEntries; //
in bytes
- // totalMemory is perCPU at this point of time.
+ // totalMemory is perNode at this point of time.
double totalMemory = cacheSize + outputBufferSize;
-
- if ( perCPU == FALSE ) {
- const PhysicalProperty* const phyProp = getPhysicalProperty();
-
- if (phyProp != NULL)
- {
- PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
- totalMemory *= partFunc->getCountOfPartitions();
- }
+ Lng32 numOfStreams = 1;
+ const PhysicalProperty* const phyProp = getPhysicalProperty();
+ if (phyProp)
+ {
+ PartitioningFunction * partFunc = phyProp ->
getPartitioningFunction() ;
+ numOfStreams = partFunc->getCountOfPartitions();
+ if (numOfStreams <= 0)
+numOfStreams = 1;
+ double memoryLimitPerInstance =
+
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
+ if (totalMemory > memoryLimitPerInstance)
+totalMemory = memoryLimitPerInstance;
+ totalMemory *= numOfStreams;
}
-
- double memoryLimitPerCpu =
-
ActiveSchemaDB()->getDefaults().getAsLong(EXE_MEMORY_FOR_PROBE_CACHE_IN_MB) *
1024 * 1024;
- if (totalMemory > memoryLimitPerCpu)
- totalMemory = memoryLimitPerCpu;
+ if (numStreams != NULL)
+ *numStreams = numOfStreams;
+ if ( perNode == TRUE )
--- End diff --
I am curious why is it unsafe? Can you please give some details so that I
can modify it accordingly.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163925#comment-16163925
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498956
--- Diff: core/sql/optimizer/RelExpr.cpp ---
@@ -1988,77 +1988,62 @@ NABoolean RelExpr::containsNode(OperatorTypeEnum
nodeType)
}
double RelExpr::computeMemoryQuota(NABoolean inMaster,
- NABoolean perCPU,
- double BMOsMemoryLimit, // in bytes
- UInt16 totalNumBMOs, // per CPU
- double totalBMOsMemoryUsage, // per
CPU, in bytes
+ NABoolean perNode,
+ double BMOsMemoryLimit, // in MB
+ UInt16 totalNumBMOs, // per query
+ double totalBMOsMemoryUsage, // for all
BMOs per node in bytes
UInt16 numBMOsPerFragment, // per
fragment
- double BMOsMemoryUsagePerFragment //
per fragment, in bytes
+ double bmoMemoryUsage, // for the
current BMO/Operator per node in bytes
+ Lng32 numStreams,
+ double
)
{
- if ( perCPU == TRUE ) {
- Lng32 exeMem = Lng32(BMOsMemoryLimit/(1024*1024));
+ if ( perNode == TRUE ) {
+ Lng32 exeMem = Lng32(BMOsMemoryLimit/(1024*1024));
- if ( inMaster && CmpCommon::getDefault(ODBC_PROCESS) == DF_ON ) {
-
-// Limiting the total memory in the master process when in both
-// the per-CPU estimation and the ODBC mode.
-
-NADefaults = ActiveSchemaDB()->getDefaults();
-
-Lng32 inCpuLimitDelta =
-defs.getAsLong(EXE_MEMORY_AVAILABLE_IN_MB)
- -
-defs.getAsLong(EXE_MEMORY_RESERVED_FOR_MXOSRVR_IN_MB);
-
-if ( inCpuLimitDelta < 0 )
- inCpuLimitDelta = 50;
-
-if (exeMem > inCpuLimitDelta)
- exeMem = inCpuLimitDelta;
+ // the quota is allocated in 2 parts
+ // The constant part divided equally across all bmo operators
+ // The variable part allocated in proportion of the given BMO operator
+ // estimated memory usage to the total estimated memory usage of all
BMOs
+
+ // The ratio can be capped by the CQD
+ double equalQuotaShareRatio = 0;
+ equalQuotaShareRatio =
ActiveSchemaDB()->getDefaults().getAsDouble(BMO_MEMORY_EQUAL_QUOTA_SHARE_RATIO);
+/*
+ char *equalQuotaShareRatioStr =
getenv("BMO_MEMORY_EQUAL_QUOTA_SHARE_RATIO");
+ if (equalQuotaShareRatioStr != NULL)
+ equalQuotaShareRatio = atof(equalQuotaShareRatioStr);
+*/
+ double constMemQuota = 0;
+ double variableMemLimit = exeMem;
+ if (equalQuotaShareRatio > 0 && totalNumBMOs > 1) {
+constMemQuota = (exeMem * equalQuotaShareRatio )/ totalNumBMOs;
+variableMemLimit = (1-equalQuotaShareRatio) * exeMem;
}
-
- // the quota is propotional to both the # of BMOs and the estimated
memory
- // usage in the fragment, and evenly distrbuted among BMOs in the
fragment.
- return ((exeMem/2) * (BMOsMemoryUsagePerFragment/totalBMOsMemoryUsage
+
-double(numBMOsPerFragment)/totalNumBMOs)
-) / numBMOsPerFragment;
+ double bmoMemoryRatio = bmoMemoryUsage / totalBMOsMemoryUsage;
+ double capMemoryRatio = 1;
+ if (totalNumBMOs > 1) {
+capMemoryRatio =
ActiveSchemaDB()->getDefaults().getAsDouble(BMO_MEMORY_ESTIMATE_RATIO_CAP);
+if (capMemoryRatio > 0 && capMemoryRatio <=1 && bmoMemoryRatio >
capMemoryRatio)
+ bmoMemoryRatio = capMemoryRatio;
--- End diff --
Yes. That's true. In that case, we need to do multiple pass or adjust the
totalBMOMemoryUsage for the unassigned BMO operator. Will attempt to do this
later
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter:
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163922#comment-16163922
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498790
--- Diff: core/sql/executor/ex_exe_stmt_globals.h ---
@@ -307,18 +307,38 @@ class ExExeStmtGlobals : public ex_globals
inline NABoolean grabMemoryQuotaIfAvailable(ULng32 size)
{
+CliGlobals *cli_globals = GetCliGlobals();
+if (cli_globals->isEspProcess())
+ return cli_globals->grabMemoryQuotaIfAvailable(size);
if ( unusedBMOsMemoryQuota_ < size ) return FALSE;
unusedBMOsMemoryQuota_ -= size ;
return TRUE;
}
- inline void resetMemoryQuota() { unusedBMOsMemoryQuota_ = 0 ; }
+ inline void resetMemoryQuota()
+ {
+CliGlobals *cli_globals = GetCliGlobals();
+if (cli_globals->isEspProcess())
+ return cli_globals->resetMemoryQuota();
+unusedBMOsMemoryQuota_ = 0 ;
+ }
--- End diff --
Thanks very much for the detailed explanation! Now I understand that your
model will work fine. My hope would be that we will share ESPs more, not less,
in the future. A long time ago we designed them to be shared, but a lot of bugs
made us share less and less. Maybe something to keep in mind for the future.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163917#comment-16163917
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498601
--- Diff: core/sql/optimizer/RelExpr.cpp ---
@@ -1988,77 +1988,62 @@ NABoolean RelExpr::containsNode(OperatorTypeEnum
nodeType)
}
double RelExpr::computeMemoryQuota(NABoolean inMaster,
- NABoolean perCPU,
- double BMOsMemoryLimit, // in bytes
- UInt16 totalNumBMOs, // per CPU
- double totalBMOsMemoryUsage, // per
CPU, in bytes
+ NABoolean perNode,
+ double BMOsMemoryLimit, // in MB
+ UInt16 totalNumBMOs, // per query
+ double totalBMOsMemoryUsage, // for all
BMOs per node in bytes
UInt16 numBMOsPerFragment, // per
fragment
- double BMOsMemoryUsagePerFragment //
per fragment, in bytes
+ double bmoMemoryUsage, // for the
current BMO/Operator per node in bytes
+ Lng32 numStreams,
+ double
)
{
- if ( perCPU == TRUE ) {
- Lng32 exeMem = Lng32(BMOsMemoryLimit/(1024*1024));
+ if ( perNode == TRUE ) {
+ Lng32 exeMem = Lng32(BMOsMemoryLimit/(1024*1024));
- if ( inMaster && CmpCommon::getDefault(ODBC_PROCESS) == DF_ON ) {
-
-// Limiting the total memory in the master process when in both
-// the per-CPU estimation and the ODBC mode.
-
-NADefaults = ActiveSchemaDB()->getDefaults();
-
-Lng32 inCpuLimitDelta =
-defs.getAsLong(EXE_MEMORY_AVAILABLE_IN_MB)
- -
-defs.getAsLong(EXE_MEMORY_RESERVED_FOR_MXOSRVR_IN_MB);
-
-if ( inCpuLimitDelta < 0 )
- inCpuLimitDelta = 50;
-
-if (exeMem > inCpuLimitDelta)
- exeMem = inCpuLimitDelta;
+ // the quota is allocated in 2 parts
+ // The constant part divided equally across all bmo operators
+ // The variable part allocated in proportion of the given BMO operator
+ // estimated memory usage to the total estimated memory usage of all
BMOs
+
+ // The ratio can be capped by the CQD
+ double equalQuotaShareRatio = 0;
+ equalQuotaShareRatio =
ActiveSchemaDB()->getDefaults().getAsDouble(BMO_MEMORY_EQUAL_QUOTA_SHARE_RATIO);
+/*
+ char *equalQuotaShareRatioStr =
getenv("BMO_MEMORY_EQUAL_QUOTA_SHARE_RATIO");
+ if (equalQuotaShareRatioStr != NULL)
+ equalQuotaShareRatio = atof(equalQuotaShareRatioStr);
+*/
--- End diff --
done
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163915#comment-16163915
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498231
--- Diff: core/sql/cli/Globals.cpp ---
@@ -1099,6 +1100,52 @@ void CliGlobals::deleteContexts()
}
#endif // _DEBUG
+// The unused BMO memory quota can now be utilized by the other
+// BMO instances from the same or different fragment
+// In case of ESP process, the unused memory quota is maintained
+// at the defaultContext. In case of master process, the ununsed
+// memory quota is maintained in the context of the connection
+
+NABoolean CliGlobals::grabMemoryQuotaIfAvailable(ULng32 size)
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
+ return context->grabMemoryQuotaIfAvailable(size);
+}
+
+void CliGlobals::resetMemoryQuota()
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
+ return context->resetMemoryQuota();
+}
+
+ULng32 CliGlobals::unusedMemoryQuota()
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
+ return context->unusedMemoryQuota();
+}
+
+void CliGlobals::yieldMemoryQuota(ULng32 size)
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
--- End diff --
Thanks, sounds ok, although I don't quite understand why we wouldn't treat
both master and ESP the same. Otherwise, when we switch between serial and
parallel plans we may see an unexpected change in behavior? This leads me to a
more general comment: I wonder whether it would be better to set an overall
limit per connection, not per node. I wonder whether this would be easier to
manage, both for users as well as for the SQL engine.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163911#comment-16163911
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498166
--- Diff: core/sql/executor/ex_exe_stmt_globals.h ---
@@ -307,18 +307,38 @@ class ExExeStmtGlobals : public ex_globals
inline NABoolean grabMemoryQuotaIfAvailable(ULng32 size)
{
+CliGlobals *cli_globals = GetCliGlobals();
+if (cli_globals->isEspProcess())
+ return cli_globals->grabMemoryQuotaIfAvailable(size);
if ( unusedBMOsMemoryQuota_ < size ) return FALSE;
unusedBMOsMemoryQuota_ -= size ;
return TRUE;
}
- inline void resetMemoryQuota() { unusedBMOsMemoryQuota_ = 0 ; }
+ inline void resetMemoryQuota()
+ {
+CliGlobals *cli_globals = GetCliGlobals();
+if (cli_globals->isEspProcess())
+ return cli_globals->resetMemoryQuota();
+unusedBMOsMemoryQuota_ = 0 ;
+ }
--- End diff --
In case of master process the unused memory quota is yielded to others
operators within the statement and hence it is stored in statement globals. In
case of ESP, the unused memory quota to yielded to operators in other fragments
within the process and hence it is stored in the default context. The part of
the code deals with the unused memory quota.
I am assuming that you want details about the quota assignment itself.
Based on this assumption:
Each open statement in the master would get its own quota. If it is
non-ESP plan, each BMO operator will be capped to max BMO limit of 1200 MB as
opposed to 10240 MB per node. If there are more than one BMO operator in such a
plan, each BMO operator memory quota will be assigned proportionally as per
item 14 subjected to min and max limits.
it is a ESP plan, the memory quota will be assigned as per item 14
subjected to min and max limit.
Currently, Trafodion won't assign the instances from a same fragment to a
process. It also won't assign the ESP to two different queries either from the
same context or different contexts. Because TDB contains the memory quota
assignment per instance, different instances within the same process should
work irrespective how the instances are assigned.
Yielding the unused memory quota will work fine as long as the ESP is not
shared for different ESPs at the same time. Currently the ESP is released only
when the statement is deallocated. The current plan is not to share ESPs
when it is multi-threaded. It is just that all the fragments and all the
instances within the same fragment from a query will be hosted in the same
process via different threads.
Hence,
1) Yes. Each open statement would get its own quota assignment
2) All ESPs and BMO operator instances memory consumption from a query will
be limited by BMO_MEMORY_LIMIT_PER_NODE. However, memory consumption won't be
exact amount as specified by this CQD because the memory quota assignment is
subjected to min and max limits.
3) ESPs are not shared currently. There is no plan to share in the
multi-threaded ESP environment also.
4) . Each statement will have its own quota irrespective of the context to
which its belongs
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163913#comment-16163913
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138498212
--- Diff: core/sql/generator/GenPreCode.cpp ---
@@ -2154,34 +2154,10 @@ RelExpr * RelRoot::preCodeGen(Generator * generator,
// Compute the total available memory quota for BMOs
NADefaults= ActiveSchemaDB()->getDefaults();
- // total per CPU
- double m = defs.getAsDouble(EXE_MEMORY_LIMIT_PER_CPU) * (1024*1024);
-
- // total memory usage for all nBMOs
- double m1 = (generator->getTotalNBMOsMemoryPerCPU()).value();
-
- // total memory limit for all BMOs
- double m2 = m-m1;
-
- double ratio =
- defs.getAsDouble(EXE_MEMORY_LIMIT_NONBMOS_PERCENT) / 100;
-
- if ( m2 < 0 ) {
- // EXE_MEMORY_LIMIT_PER_CPU is set too small, set the total
- // memory limit for BMOs to zero. When the memory quota for
- // each BMO is computed (via method
RelExpr::computeMemoryQuota()),
- // the lower-bound for each BMO will kick in and each will receive
- // a quota equal to the lower-bound value.
- m2 = 0;
- } else {
-
- // nBMOs use more memory than the portion, adjust m2 to
- // that of (1-ratio)*m
- if (m1 > m*ratio )
- m2 = m*(1-ratio);
- }
+ // total per node
+ double m = defs.getAsDouble(BMO_MEMORY_LIMIT_PER_NODE) * (1024*1024);
--- End diff --
Will do
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163851#comment-16163851
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138492138
--- Diff: core/sql/cli/Globals.cpp ---
@@ -1099,6 +1100,52 @@ void CliGlobals::deleteContexts()
}
#endif // _DEBUG
+// The unused BMO memory quota can now be utilized by the other
+// BMO instances from the same or different fragment
+// In case of ESP process, the unused memory quota is maintained
+// at the defaultContext. In case of master process, the ununsed
+// memory quota is maintained in the context of the connection
+
+NABoolean CliGlobals::grabMemoryQuotaIfAvailable(ULng32 size)
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
+ return context->grabMemoryQuotaIfAvailable(size);
+}
+
+void CliGlobals::resetMemoryQuota()
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
+ return context->resetMemoryQuota();
+}
+
+ULng32 CliGlobals::unusedMemoryQuota()
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
+ return context->unusedMemoryQuota();
+}
+
+void CliGlobals::yieldMemoryQuota(ULng32 size)
+{
+ ContextCli *context;
+ if (espProcess_)
+ context = defaultContext_;
+ else
+ context = currContext();
--- End diff --
Yes. However currContext() is expected to return the thread specific
context in case of master in case of multi-threaded JDBC T2 application. In
case of ESP, we always want to return the defaultContext so that memory quota
is stored in the default context.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163842#comment-16163842
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138491738
--- Diff: core/sql/generator/GenRelJoin.cpp ---
@@ -1784,36 +1784,44 @@ short HashJoin::codeGen(Generator * generator) {
UInt16 numBMOsInFrag = (UInt16)generator->getFragmentDir()->getNumBMOs();
double memQuota = 0;
-
+ double memQuotaRatio;
+ Lng32 numStreams;
+ double bmoMemoryUsagePerNode = getEstimatedRunTimeMemoryUsage(TRUE,
).value();
if (mmu != 0) {
memQuota = mmu;
hashj_tdb->setMemoryQuotaMB(mmu);
} else {
// Apply quota system if either one the following two is true:
// 1. the memory limit feature is turned off and more than one BMOs
// 2. the memory limit feature is turned on
-NABoolean mlimitPerCPU = defs.getAsDouble(EXE_MEMORY_LIMIT_PER_CPU) >
0;
-
-if ( mlimitPerCPU || numBMOsInFrag > 1 ||
+NABoolean mlimitPerNode = defs.getAsDouble(BMO_MEMORY_LIMIT_PER_NODE)
> 0;
+
+if ( mlimitPerNode || numBMOsInFrag > 1 ||
(numBMOsInFrag == 1 &&
CmpCommon::getDefault(EXE_SINGLE_BMO_QUOTA) == DF_ON)) {
-
memQuota =
computeMemoryQuota(generator->getEspLevel() == 0,
- mlimitPerCPU,
-
generator->getBMOsMemoryLimitPerCPU().value(),
- generator->getTotalNumBMOsPerCPU(),
-
generator->getTotalBMOsMemoryPerCPU().value(),
+ mlimitPerNode,
+
generator->getBMOsMemoryLimitPerNode().value(),
+ generator->getTotalNumBMOs(),
+
generator->getTotalBMOsMemoryPerNode().value(),
numBMOsInFrag,
-
generator->getFragmentDir()->getBMOsMemoryUsage()
+ bmoMemoryUsagePerNode,
+ numStreams,
+ memQuotaRatio
);
-
-Lng32 hjMemoryLowbound =
defs.getAsLong(EXE_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN);
+}
+Lng32 hjMemoryLowbound =
defs.getAsLong(BMO_MEMORY_LIMIT_LOWER_BOUND_HASHJOIN);
+Lng32 memoryUpperbound = defs.getAsLong(BMO_MEMORY_LIMIT_UPPER_BOUND);
-if ( memQuota < hjMemoryLowbound )
- memQuota = hjMemoryLowbound;
-
-hashj_tdb->setMemoryQuotaMB( UInt16(memQuota) );
+if ( memQuota < hjMemoryLowbound ) {
+ memQuota = hjMemoryLowbound;
+ memQuotaRatio = BMOQuotaRatio::MIN_QUOTA;
}
+else if (memQuota > memoryUpperbound)
+ memQuota = memoryUpperbound;
+ memQuotaRatio = BMOQuotaRatio::MIN_QUOTA;
+hashj_tdb->setMemoryQuotaMB( UInt16(memQuota) );
+hashj_tdb->setBmoQuotaRatio(memQuotaRatio);
--- End diff --
Good Catch. Fixed RelExpr::computeMemoryQuota to return bmoQuotaRatio to
return the ratio correctly. However, this needs to be revisited when the
memory quota is reassigned based on runtime settings. See item 4 in the PR
commit comment.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
>
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163817#comment-16163817
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138488567
--- Diff: core/sql/executor/ex_exe_stmt_globals.h ---
@@ -307,18 +307,38 @@ class ExExeStmtGlobals : public ex_globals
inline NABoolean grabMemoryQuotaIfAvailable(ULng32 size)
{
+CliGlobals *cli_globals = GetCliGlobals();
+if (cli_globals->isEspProcess())
+ return cli_globals->grabMemoryQuotaIfAvailable(size);
if ( unusedBMOsMemoryQuota_ < size ) return FALSE;
unusedBMOsMemoryQuota_ -= size ;
--- End diff --
See cli/CliGlobals.cpp line 1103
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[
https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163813#comment-16163813
]
ASF GitHub Bot commented on TRAFODION-2733:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138488146
--- Diff: core/sql/executor/ex_exe_stmt_globals.h ---
@@ -307,18 +307,38 @@ class ExExeStmtGlobals : public ex_globals
inline NABoolean grabMemoryQuotaIfAvailable(ULng32 size)
{
+CliGlobals *cli_globals = GetCliGlobals();
+if (cli_globals->isEspProcess())
+ return cli_globals->grabMemoryQuotaIfAvailable(size);
if ( unusedBMOsMemoryQuota_ < size ) return FALSE;
unusedBMOsMemoryQuota_ -= size ;
--- End diff --
In case of master process the unused memory quota is yielded to others
operators within the statement and hence it is stored in statement globals. In
case of ESP, it is stored in the default context. In case of ESP, the unused
memory quota to yielded to operators in other fragments within the process.
Hence it is stored in the default context. I couldn't find the comment that you
are referring to. The comment needs to be fixed, if you can show it.
> Provide an improved memory quota assignment for big memory operators (BMO)
> --
>
> Key: TRAFODION-2733
> URL: https://issues.apache.org/jira/browse/TRAFODION-2733
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-cmp, sql-exe
>Affects Versions: 2.3-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
> Fix For: 2.3-incubating
>
>
> The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort.
> Trafodion deploys multiple executor server processes (ESPs) to execute a
> query via its data flow architecture. Each ESPs can have an instance of this
> BMO operator. Currently, each instance of this operator can potentially have
> memory quota of 800 MB assigned to do its BMO operation. However, the memory
> allocation is usually limited by the memory pressure when this BMO attempts
> to allocate memory within the assigned quota. The assignment doesn't depend
> upon the estimation of memory needed by this operation.
> Improvement needed in BMO memory assignment are:
> 1. Limit the memory quota assignment for these BMO operations per node
> 2. Memory quota assigned taking into consideration estimated memory needed
> at every operator.
> 3. Ensure that the BMO gets the minimum memory needed at least to function
> smoothly
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2733) Provide an improved memory quota assignment for big memory operators (BMO)
[ https://issues.apache.org/jira/browse/TRAFODION-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163807#comment-16163807 ] ASF GitHub Bot commented on TRAFODION-2733: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1228#discussion_r138487244 --- Diff: core/sql/comexe/ComTdbSort.h --- @@ -201,11 +201,15 @@ class ComTdbSort : public ComTdb Int32 sortPartialKeyLen_; // 56-59 UInt32 minimalSortRecs_; // 60-63 - Float32 sortMemEstInMbPerCpu_;// 64-67 + Float32 sortMemEstInKBPerNode_;// 64-67 Float32 bmoCitizenshipFactor_;// 68-71 Int32 pMemoryContingencyMB_;// 72-75 UInt16 sortGrowthPercent_;// 76-77 - char fillersComTdbSort_[18]; // 78-95 + char filler_1[2]; // 78-79 + Int32 topNThreshold_; // 80-83 + Float32 estMemoryUsage_; // 84-87 + Float32 bmoQuotaRatio_; // 88-92 --- End diff -- done > Provide an improved memory quota assignment for big memory operators (BMO) > -- > > Key: TRAFODION-2733 > URL: https://issues.apache.org/jira/browse/TRAFODION-2733 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-cmp, sql-exe >Affects Versions: 2.3-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > Fix For: 2.3-incubating > > > The big memory operators in Trafodion are HashJoin, HashGroupBy and Sort. > Trafodion deploys multiple executor server processes (ESPs) to execute a > query via its data flow architecture. Each ESPs can have an instance of this > BMO operator. Currently, each instance of this operator can potentially have > memory quota of 800 MB assigned to do its BMO operation. However, the memory > allocation is usually limited by the memory pressure when this BMO attempts > to allocate memory within the assigned quota. The assignment doesn't depend > upon the estimation of memory needed by this operation. > Improvement needed in BMO memory assignment are: > 1. Limit the memory quota assignment for these BMO operations per node > 2. Memory quota assigned taking into consideration estimated memory needed > at every operator. > 3. Ensure that the BMO gets the minimum memory needed at least to function > smoothly -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TRAFODION-2739) RMS semaphore handling need to log the error for easy resolution of the issue
Selvaganesan Govindarajan created TRAFODION-2739: Summary: RMS semaphore handling need to log the error for easy resolution of the issue Key: TRAFODION-2739 URL: https://issues.apache.org/jira/browse/TRAFODION-2739 Project: Apache Trafodion Issue Type: Bug Components: sql-exe Affects Versions: any Reporter: Selvaganesan Govindarajan Assignee: Selvaganesan Govindarajan Fix For: 2.3-incubating Currently, RMS doesn't log the error when semaphore can't be locked or released, though it dumps the core. Because the error code is optimized out it is becoming difficult to get the error code from the core. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TRAFODION-2738) Rowset buffer size during insert/upsert should be limited
Selvaganesan Govindarajan created TRAFODION-2738: Summary: Rowset buffer size during insert/upsert should be limited Key: TRAFODION-2738 URL: https://issues.apache.org/jira/browse/TRAFODION-2738 Project: Apache Trafodion Issue Type: Bug Reporter: Selvaganesan Govindarajan Currently, the rowset size for upsert and insert can be given in a such way that it would be allocate unlimited buffer size to move the data from client to Trafodion SQL engine. This can cause memory corruption and unexpected behavior. Hence, we need a way to limit the number of rows in the rowset. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TRAFODION-2737) Native hbase table access via Trafodion improvements/issues
Selvaganesan Govindarajan created TRAFODION-2737: Summary: Native hbase table access via Trafodion improvements/issues Key: TRAFODION-2737 URL: https://issues.apache.org/jira/browse/TRAFODION-2737 Project: Apache Trafodion Issue Type: Improvement Components: sql-cmp, sql-exe Reporter: Selvaganesan Govindarajan Assignee: Selvaganesan Govindarajan Fix For: 2.3-incubating Native hbase table access via Trafodion needs the following: a) Get a row given a hbase row Id doesn't work at times b) Need a way to specify hbase row id length and hbase row length c) Need a way to support List (Rowset Get in Trafodion terms) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2736) Missing predicates on salt columns when using an index join
[
https://issues.apache.org/jira/browse/TRAFODION-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163525#comment-16163525
]
Hans Zeller commented on TRAFODION-2736:
For regular predicates, Scan::addIndexInfo will generate a list of index-only
scans and of index joins, including the predicates to evaluate on the index and
the base table. Computed predicates don't follow that model. Those get
calculated in Scan::addIndexInfo and then they get stored in the Scan and
FileScan nodes.
For a regular transformation of Scan to FileScan, that's ok. However, when we
transform the scan into a join between an index scan and a base table, there is
a bug. In this case we create two new scans in method
IndexJoinRule1::makeSubstituteFromIndexInfo() in file TransRule.cpp. I don't
see code in that method that transfers the computed predicate from the original
scan to the new scan of the left side of the join. We also won't re-create the
computed predicates on this new scan, since we force an index-only scan for
this node, so Scan::addIndexInfo() will return immediately and do nothing.
> Missing predicates on salt columns when using an index join
> ---
>
> Key: TRAFODION-2736
> URL: https://issues.apache.org/jira/browse/TRAFODION-2736
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 2.0-incubating
> Environment: Any
>Reporter: Hans Zeller
>Assignee: Hans Zeller
>
> When we have salted tables, the compiler is supposed to generate predicates
> on the salt column automatically, based on predicates on the columns used for
> salting.
> Suresh and Benny found a case where we don't do that, related to index joins
> (a join between an index and its base table). Here is an example:
> {code}
> create table tdi(service_id int not null,
> user_number int not null,
> start_time timestamp not null,
> b int,
> primary key(service_id, user_number, start_time))
> division by (date_trunc('day', start_time))
> salt using 4 partitions on (user_number);
> create index tdix on tdi(user_number, start_time) salt like table;
> control query shape join(cut,cut);
> prepare s from
> select * from tdi where user_number = 1234 and start_time between timestamp
> '2017-01-01 00:00:00' and timestamp '2017-01-15 00:00:00';
> explain s;
> {code}
> In the explain, we would expect a predicate on the salt column but there is
> none.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (TRAFODION-2736) Missing predicates on salt columns when using an index join
Hans Zeller created TRAFODION-2736:
--
Summary: Missing predicates on salt columns when using an index
join
Key: TRAFODION-2736
URL: https://issues.apache.org/jira/browse/TRAFODION-2736
Project: Apache Trafodion
Issue Type: Bug
Components: sql-cmp
Affects Versions: 2.0-incubating
Environment: Any
Reporter: Hans Zeller
Assignee: Hans Zeller
When we have salted tables, the compiler is supposed to generate predicates on
the salt column automatically, based on predicates on the columns used for
salting.
Suresh and Benny found a case where we don't do that, related to index joins (a
join between an index and its base table). Here is an example:
{code}
create table tdi(service_id int not null,
user_number int not null,
start_time timestamp not null,
b int,
primary key(service_id, user_number, start_time))
division by (date_trunc('day', start_time))
salt using 4 partitions on (user_number);
create index tdix on tdi(user_number, start_time) salt like table;
control query shape join(cut,cut);
prepare s from
select * from tdi where user_number = 1234 and start_time between timestamp
'2017-01-01 00:00:00' and timestamp '2017-01-15 00:00:00';
explain s;
{code}
In the explain, we would expect a predicate on the salt column but there is
none.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2704) JDBC regression test failed by too many statement on the same session
[ https://issues.apache.org/jira/browse/TRAFODION-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163278#comment-16163278 ] ASF GitHub Bot commented on TRAFODION-2704: --- Github user sandhyasun commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1225#discussion_r138406709 --- Diff: core/conn/odbc/src/odbc/nsksrvrcore/srvrothers.cpp --- @@ -6489,6 +6506,31 @@ odbc_SQLSrvr_ExtractLob_sme_( exception_->u.SQLError.errorList._buffer = QryLobExtractSrvrStmt->sqlError.errorList._buffer; exception_->u.ParamError.ParamDesc = SQLSVC_EXCEPTION_EXECUTE_FAILED; } + +IDL_long_long zeroValue = 0; +snprintf(LobExtractQuery, sizeof(LobExtractQuery), "EXTRACT LOBTOBUFFER(LOB'%s', LOCATION %Ld, SIZE %Ld)", lobHandle, (Int64)lobDataValue, ); + +retcode = QryLobExtractSrvrStmt->ExecDirect(NULL, LobExtractQuery, EXTERNAL_STMT, TYPE_CALL, SQL_ASYNC_ENABLE_OFF, 0); + --- End diff -- About your comment on T2, currently for this release only T4 support is done for LOBs . None of the other drivers support LOB datatype. They need to be addressed one by one. Weiqing, we need separate JIRAs for those too. > JDBC regression test failed by too many statement on the same session > - > > Key: TRAFODION-2704 > URL: https://issues.apache.org/jira/browse/TRAFODION-2704 > Project: Apache Trafodion > Issue Type: Bug >Reporter: Weiqing Xu >Assignee: Weiqing Xu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2704) JDBC regression test failed by too many statement on the same session
[ https://issues.apache.org/jira/browse/TRAFODION-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163277#comment-16163277 ] ASF GitHub Bot commented on TRAFODION-2704: --- Github user sandhyasun commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1225#discussion_r138406464 --- Diff: core/conn/odbc/src/odbc/nsksrvrcore/srvrothers.cpp --- @@ -6489,6 +6506,31 @@ odbc_SQLSrvr_ExtractLob_sme_( exception_->u.SQLError.errorList._buffer = QryLobExtractSrvrStmt->sqlError.errorList._buffer; exception_->u.ParamError.ParamDesc = SQLSVC_EXCEPTION_EXECUTE_FAILED; } + +IDL_long_long zeroValue = 0; +snprintf(LobExtractQuery, sizeof(LobExtractQuery), "EXTRACT LOBTOBUFFER(LOB'%s', LOCATION %Ld, SIZE %Ld)", lobHandle, (Int64)lobDataValue, ); + +retcode = QryLobExtractSrvrStmt->ExecDirect(NULL, LobExtractQuery, EXTERNAL_STMT, TYPE_CALL, SQL_ASYNC_ENABLE_OFF, 0); + --- End diff -- The multiple check extraction is not implemented yet according to Weiqing's previous comment and it should be. Otherwise the application won't be able to extract more than the first 512 MB of data.That is our extract chunk size default in SQL. But he will fix it in with a different JIRA and PR. > JDBC regression test failed by too many statement on the same session > - > > Key: TRAFODION-2704 > URL: https://issues.apache.org/jira/browse/TRAFODION-2704 > Project: Apache Trafodion > Issue Type: Bug >Reporter: Weiqing Xu >Assignee: Weiqing Xu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TRAFODION-2732) UPDATE STATS fails sometimes when a column name is a SQL keyword
[
https://issues.apache.org/jira/browse/TRAFODION-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Wayne Birdsall resolved TRAFODION-2732.
-
Resolution: Fixed
Fix Version/s: 2.3-incubating
> UPDATE STATS fails sometimes when a column name is a SQL keyword
>
>
> Key: TRAFODION-2732
> URL: https://issues.apache.org/jira/browse/TRAFODION-2732
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: any
> Environment: All
>Reporter: David Wayne Birdsall
>Assignee: David Wayne Birdsall
> Fix For: 2.3-incubating
>
>
> On tables having the following characteristics:
> 1. Large table having a column named using a reserved SQL keyword (e.g.
> "YEAR")
> 2. The sample data is too large to fit in memory (or CQD USTAT_INTERNAL_SORT
> is set to 'OFF')
> 3. The table also has a VARCHAR column of longer than 256 characters
> Then UPDATE STATISTICS will fail when it tries to populate a sample table,
> reporting a syntax error on an internally-generated UPSERT statement.
> The following sqlci session demonstrates how to reproduce the problem:
> >>obey repro1j.sql;
> >>?section setup
> >>
> >>drop table UstatsBug;
> *** ERROR[1389] Object TRAFODION.SCH.USTATSBUG does not exist in Trafodion.
> --- SQL operation failed with errors.
> >>
> >>CREATE TABLE UstatsBug
> +> (
> +>ID VARCHAR(32 CHARS) CHARACTER SET UTF8
> +> COLLATE DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE NOT SERIALIZED
> +> , "YEAR" NUMERIC(12, 0) DEFAULT NULL NOT
> SERIALIZED
> +> , JUNK VARCHAR(1000 CHARS) CHARACTER SET UTF8
> +> , PRIMARY KEY (ID ASC)
> +> )
> +> ATTRIBUTES ALIGNED FORMAT
> +> HBASE_OPTIONS
> +> (
> +>DATA_BLOCK_ENCODING = 'FAST_DIFF',
> +>MEMSTORE_FLUSH_SIZE = '1073741824'
> +> )
> +>;
> --- SQL operation complete.
> >>
> >>insert into UstatsBug values
> >>('a',1992,'junk'),('b',1993,'junk'),('c',1994,'junk'),('d',1995,'junk');
> --- 4 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'a', "YEAR", junk from UstatsBug;
> --- 4 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'b', "YEAR", junk from UstatsBug;
> --- 8 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'c', "YEAR", junk from UstatsBug;
> --- 16 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'd', "YEAR", junk from UstatsBug;
> --- 32 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'e', "YEAR", junk from UstatsBug;
> --- 64 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'f', "YEAR", junk from UstatsBug;
> --- 128 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'g', "YEAR", junk from UstatsBug;
> --- 256 row(s) inserted.
> >>
> >>?section doit
> >>
> >>cqd ustat_internal_sort 'OFF';
> --- SQL operation complete.
> >>
> >>UPDATE STATISTICS FOR TABLE UstatsBug create SAMPLE random 50 percent;
> *** ERROR[9200] UPDATE STATISTICS for table TRAFODION.SCH.USTATSBUG
> encountered an error (15001) from statement Process_Query.
> *** ERROR[15001] A syntax error occurred at or before:
> UPSERT USING LOAD INTO
> TRAFODION.SCH.TRAF_SAMPLE_02393802717774510155_150488830
> 9_629934 SELECT ID, YEAR, SUBSTRING(JUNK FOR 64) AS JUNK FROM
> TRAFODION.SCH.UST
> ^ (104 characters from start of SQL statement)
> *** ERROR[8822] The statement was not prepared.
> --- SQL operation failed with errors.
> >>
> >>exit;
> End of MXCI Session
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2732) UPDATE STATS fails sometimes when a column name is a SQL keyword
[
https://issues.apache.org/jira/browse/TRAFODION-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163229#comment-16163229
]
ASF GitHub Bot commented on TRAFODION-2732:
---
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/1227
> UPDATE STATS fails sometimes when a column name is a SQL keyword
>
>
> Key: TRAFODION-2732
> URL: https://issues.apache.org/jira/browse/TRAFODION-2732
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: any
> Environment: All
>Reporter: David Wayne Birdsall
>Assignee: David Wayne Birdsall
>
> On tables having the following characteristics:
> 1. Large table having a column named using a reserved SQL keyword (e.g.
> "YEAR")
> 2. The sample data is too large to fit in memory (or CQD USTAT_INTERNAL_SORT
> is set to 'OFF')
> 3. The table also has a VARCHAR column of longer than 256 characters
> Then UPDATE STATISTICS will fail when it tries to populate a sample table,
> reporting a syntax error on an internally-generated UPSERT statement.
> The following sqlci session demonstrates how to reproduce the problem:
> >>obey repro1j.sql;
> >>?section setup
> >>
> >>drop table UstatsBug;
> *** ERROR[1389] Object TRAFODION.SCH.USTATSBUG does not exist in Trafodion.
> --- SQL operation failed with errors.
> >>
> >>CREATE TABLE UstatsBug
> +> (
> +>ID VARCHAR(32 CHARS) CHARACTER SET UTF8
> +> COLLATE DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE NOT SERIALIZED
> +> , "YEAR" NUMERIC(12, 0) DEFAULT NULL NOT
> SERIALIZED
> +> , JUNK VARCHAR(1000 CHARS) CHARACTER SET UTF8
> +> , PRIMARY KEY (ID ASC)
> +> )
> +> ATTRIBUTES ALIGNED FORMAT
> +> HBASE_OPTIONS
> +> (
> +>DATA_BLOCK_ENCODING = 'FAST_DIFF',
> +>MEMSTORE_FLUSH_SIZE = '1073741824'
> +> )
> +>;
> --- SQL operation complete.
> >>
> >>insert into UstatsBug values
> >>('a',1992,'junk'),('b',1993,'junk'),('c',1994,'junk'),('d',1995,'junk');
> --- 4 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'a', "YEAR", junk from UstatsBug;
> --- 4 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'b', "YEAR", junk from UstatsBug;
> --- 8 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'c', "YEAR", junk from UstatsBug;
> --- 16 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'd', "YEAR", junk from UstatsBug;
> --- 32 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'e', "YEAR", junk from UstatsBug;
> --- 64 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'f', "YEAR", junk from UstatsBug;
> --- 128 row(s) inserted.
> >>
> >>insert into UstatsBug select id || 'g', "YEAR", junk from UstatsBug;
> --- 256 row(s) inserted.
> >>
> >>?section doit
> >>
> >>cqd ustat_internal_sort 'OFF';
> --- SQL operation complete.
> >>
> >>UPDATE STATISTICS FOR TABLE UstatsBug create SAMPLE random 50 percent;
> *** ERROR[9200] UPDATE STATISTICS for table TRAFODION.SCH.USTATSBUG
> encountered an error (15001) from statement Process_Query.
> *** ERROR[15001] A syntax error occurred at or before:
> UPSERT USING LOAD INTO
> TRAFODION.SCH.TRAF_SAMPLE_02393802717774510155_150488830
> 9_629934 SELECT ID, YEAR, SUBSTRING(JUNK FOR 64) AS JUNK FROM
> TRAFODION.SCH.UST
> ^ (104 characters from start of SQL statement)
> *** ERROR[8822] The statement was not prepared.
> --- SQL operation failed with errors.
> >>
> >>exit;
> End of MXCI Session
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TRAFODION-2704) JDBC regression test failed by too many statement on the same session
[ https://issues.apache.org/jira/browse/TRAFODION-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163182#comment-16163182 ] ASF GitHub Bot commented on TRAFODION-2704: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1225#discussion_r138397172 --- Diff: core/conn/odbc/src/odbc/nsksrvrcore/srvrothers.cpp --- @@ -6489,6 +6506,31 @@ odbc_SQLSrvr_ExtractLob_sme_( exception_->u.SQLError.errorList._buffer = QryLobExtractSrvrStmt->sqlError.errorList._buffer; exception_->u.ParamError.ParamDesc = SQLSVC_EXCEPTION_EXECUTE_FAILED; } + +IDL_long_long zeroValue = 0; +snprintf(LobExtractQuery, sizeof(LobExtractQuery), "EXTRACT LOBTOBUFFER(LOB'%s', LOCATION %Ld, SIZE %Ld)", lobHandle, (Int64)lobDataValue, ); + +retcode = QryLobExtractSrvrStmt->ExecDirect(NULL, LobExtractQuery, EXTERNAL_STMT, TYPE_CALL, SQL_ASYNC_ENABLE_OFF, 0); + --- End diff -- I am assuming that the streaming concept is implemented in the JDBC/ODBC driver and the server side just fetches the data in bulk at some size, If the fetch bulk size is greater than the user requested size, it would be serviced from the driver itself or when it is lesser multiple chunks will be fetched to satisfy the user requested size. If so, this change looks ok. However, there is a drawback that this change may not be portable to JDBC T2 driver because there can be only one extract query active at a time. > JDBC regression test failed by too many statement on the same session > - > > Key: TRAFODION-2704 > URL: https://issues.apache.org/jira/browse/TRAFODION-2704 > Project: Apache Trafodion > Issue Type: Bug >Reporter: Weiqing Xu >Assignee: Weiqing Xu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2704) JDBC regression test failed by too many statement on the same session
[ https://issues.apache.org/jira/browse/TRAFODION-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162658#comment-16162658 ] ASF GitHub Bot commented on TRAFODION-2704: --- Github user xwq commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1225#discussion_r138281163 --- Diff: core/conn/odbc/src/odbc/nsksrvrcore/srvrothers.cpp --- @@ -6489,6 +6506,31 @@ odbc_SQLSrvr_ExtractLob_sme_( exception_->u.SQLError.errorList._buffer = QryLobExtractSrvrStmt->sqlError.errorList._buffer; exception_->u.ParamError.ParamDesc = SQLSVC_EXCEPTION_EXECUTE_FAILED; } + +IDL_long_long zeroValue = 0; +snprintf(LobExtractQuery, sizeof(LobExtractQuery), "EXTRACT LOBTOBUFFER(LOB'%s', LOCATION %Ld, SIZE %Ld)", lobHandle, (Int64)lobDataValue, ); + +retcode = QryLobExtractSrvrStmt->ExecDirect(NULL, LobExtractQuery, EXTERNAL_STMT, TYPE_CALL, SQL_ASYNC_ENABLE_OFF, 0); + --- End diff -- For "update lob append", it can be called multi times to append large data to a single LOB. The "extract LOB", maybe I can create another PR for it. > JDBC regression test failed by too many statement on the same session > - > > Key: TRAFODION-2704 > URL: https://issues.apache.org/jira/browse/TRAFODION-2704 > Project: Apache Trafodion > Issue Type: Bug >Reporter: Weiqing Xu >Assignee: Weiqing Xu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
