[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643922#comment-13643922 ] Gopal V commented on HIVE-4068: --- [~navis]: no big deal, man - we all forget stuff :) Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644056#comment-13644056 ] Hudson commented on HIVE-4068: -- Integrated in Hive-trunk-hadoop2 #180 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/180/]) HIVE-4437 Missing file on HIVE-4068 (Navis) (Revision 1476701) Result = FAILURE navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476701 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644059#comment-13644059 ] Hudson commented on HIVE-4068: -- Integrated in Hive-trunk-h0.21 #2083 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2083/]) HIVE-4437 Missing file on HIVE-4068 (Navis) (Revision 1476701) Result = ABORTED navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476701 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643560#comment-13643560 ] Hudson commented on HIVE-4068: -- Integrated in Hive-trunk-h0.21 #2080 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2080/]) HIVE-4068 Size of aggregation buffer which uses non-primitive type is not estimated correctly (Navis) (Revision 1476483) Result = FAILURE navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476483 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBridge.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643704#comment-13643704 ] Hudson commented on HIVE-4068: -- Integrated in Hive-trunk-hadoop2 #177 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/177/]) HIVE-4068 Size of aggregation buffer which uses non-primitive type is not estimated correctly (Navis) (Revision 1476483) Result = FAILURE navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1476483 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBridge.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643768#comment-13643768 ] Gopal V commented on HIVE-4068: --- Missed svn add of ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java before check-in? Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643849#comment-13643849 ] Navis commented on HIVE-4068: - [~gopalv] Thanks. I thinks I should not commit things. Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641603#comment-13641603 ] Namit Jain commented on HIVE-4068: -- +1 [~navis], can you commit it if tests pass Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642504#comment-13642504 ] Navis commented on HIVE-4068: - [~namit] ok, sure. Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640363#comment-13640363 ] Phabricator commented on HIVE-4068: --- njain has commented on the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. can you refresh this ? I missed this. REVISION DETAIL https://reviews.facebook.net/D8859 To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13589329#comment-13589329 ] Phabricator commented on HIVE-4068: --- njain has commented on the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. yes, but can you also make other aggregation buffers extend AbstractAggregationBuffer ? REVISION DETAIL https://reviews.facebook.net/D8859 To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[
https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585722#comment-13585722
]
Phabricator commented on HIVE-4068:
---
njain has commented on the revision HIVE-4068 [jira] Size of aggregation
buffer which uses non-primitive type is not estimated correctly.
I was thinking of a abstract class ---
AbstractAggregationBuffer with the annotation.
The default implementation of estimate() is empty
All known aggregationBuffers extend AbstractAB.
if (agg instanceof GenericUDAFEvaluator.EstimableBuffer) {
return ((GenericUDAFEvaluator.EstimableBuffer)agg).estimate();
}
change instanceof to annotation
Mark AB as deprecated
REVISION DETAIL
https://reviews.facebook.net/D8859
To: JIRA, navis
Cc: njain
Size of aggregation buffer which uses non-primitive type is not estimated
correctly
---
Key: HIVE-4068
URL: https://issues.apache.org/jira/browse/HIVE-4068
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch
Currently, hive assumes an aggregation buffer which holds a map is occupying
just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can
be thrown (especially for 1k buffer).
workaround : set hive.map.aggr.hash.percentmemory=smaller value than
default(0.5)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585414#comment-13585414 ] Phabricator commented on HIVE-4068: --- njain has commented on the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java:581 Instead of checking instanceof, can you add a annotation ? REVISION DETAIL https://reviews.facebook.net/D8859 To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585546#comment-13585546 ] Phabricator commented on HIVE-4068: --- navis has commented on the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java:581 We still need a method for estimation and I thought it's better to have a interface for that. You mean annotation + interface or annotation + reflection on convention? For annotation + interface case, it seemed a little redundant. User should specify the annotation and implement new interface also. For annotation + reflection, it' not safe and need a little more codes. REVISION DETAIL https://reviews.facebook.net/D8859 To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
