[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4068: - Status: Open (was: Patch Available) comments Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4068: Status: Patch Available (was: Open) Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4068: -- Attachment: HIVE-4068.D8859.5.patch navis updated the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. Rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D8859 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D8859?vs=29133id=32877#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBridge.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4068: -- Attachment: HIVE-4068.D8859.4.patch navis updated the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. Addressed comment Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D8859 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D8859?vs=28593id=29133#toc AFFECTED FILES builtins/src/org/apache/hive/builtins/UDAFUnionMap.java pdk/scripts/build-plugin.xml ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBridge.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCount.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java ql/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4068: Status: Patch Available (was: Open) Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4068: - Status: Open (was: Patch Available) comments Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4068: -- Attachment: HIVE-4068.D8859.3.patch navis updated the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. Like this? Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D8859 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D8859?vs=28539id=28593#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java To: JIRA, navis Cc: njain Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4068: Status: Patch Available (was: Open) Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4068: -- Attachment: HIVE-4068.D8859.2.patch navis updated the revision HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. Added sample implementation for ewah_bitmap and histogram_numeric Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D8859 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D8859?vs=28533id=28539#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java To: JIRA, navis Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4068: - Status: Open (was: Patch Available) comments Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-4068: Status: Patch Available (was: Open) Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4068: -- Attachment: HIVE-4068.D8859.1.patch navis requested code review of HIVE-4068 [jira] Size of aggregation buffer which uses non-primitive type is not estimated correctly. Reviewers: JIRA HIVE-4068 Size of aggregation buffer which uses non-primitive type is not estimated correctly Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D8859 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/21519/ To: JIRA, navis Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
