[jira] [Comment Edited] (HIVE-18610) Performance: ListKeyWrapper does not check for hashcode equals, before comparing members
[ https://issues.apache.org/jira/browse/HIVE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383567#comment-16383567 ] Gopal V edited comment on HIVE-18610 at 3/2/18 1:34 PM: The negative failure tests are all failing with {{java.lang.OutOfMemoryError: GC overhead limit exceeded}} {code} ava.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Hashtable.addEntry(Hashtable.java:435) at java.util.Hashtable.put(Hashtable.java:476) at java.util.Properties.setProperty(Properties.java:166) at org.apache.hadoop.conf.Configuration.loadProperty(Configuration.java:2978) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2938) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2734) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2619) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1256) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1228) at org.apache.hadoop.hive.conf.HiveConf.applySystemProperties(HiveConf.java:4753) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:4480) at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:4409) at org.apache.hadoop.hive.ql.QTestUtil.clearTestSideEffects(QTestUtil.java:1020) at org.apache.hadoop.hive.cli.control.CoreNegativeCliDriver.setUp(CoreNegativeCliDriver.java:64) {code} was (Author: gopalv): The negative failure tests are all failing with {{java.lang.OutOfMemoryError: GC overhead limit exceeded}} > Performance: ListKeyWrapper does not check for hashcode equals, before > comparing members > > > Key: HIVE-18610 > URL: https://issues.apache.org/jira/browse/HIVE-18610 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: complextype, performance > Attachments: HIVE-18610.1.patch, HIVE-18610.2.patch > > > ListKeyWrapper::equals() > {code} > @Override > public boolean equals(Object obj) { > if (!(obj instanceof ListKeyWrapper)) { > return false; > } > Object[] copied_in_hashmap = ((ListKeyWrapper) obj).keys; > return equalComparer.areEqual(copied_in_hashmap, keys); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18610) Performance: ListKeyWrapper does not check for hashcode equals, before comparing members
[ https://issues.apache.org/jira/browse/HIVE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381551#comment-16381551 ] Gopal V edited comment on HIVE-18610 at 3/1/18 5:47 AM: This fixes the partial group-by hash function case where the ListKeyWrapper has a deeper Struct (an Object[]) inside it. This is not a correctness issue because if the partial group-by fails to de-dup, the reduce-side GBY will treat identical rows from the same mapper as partial flushes & queries end up producing the right result, but very slowly. was (Author: gopalv): This fixes the partial group-by hash function case where the ListKeyWrapper has a deeper Struct inside it. This is not a correctness issue because if the partial group-by fails to de-dup, the reduce-side GBY will treat identical rows from the same mapper as partial flushes & queries end up producing the right result, but very slowly. > Performance: ListKeyWrapper does not check for hashcode equals, before > comparing members > > > Key: HIVE-18610 > URL: https://issues.apache.org/jira/browse/HIVE-18610 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18610.1.patch, HIVE-18610.2.patch > > > ListKeyWrapper::equals() > {code} > @Override > public boolean equals(Object obj) { > if (!(obj instanceof ListKeyWrapper)) { > return false; > } > Object[] copied_in_hashmap = ((ListKeyWrapper) obj).keys; > return equalComparer.areEqual(copied_in_hashmap, keys); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)