[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Surówka updated HIVE-2500: - Attachment: HIVE-2500.2.patch Thank you for this advice, I updated the diff (and review request) accordingly. Allow Hive to be debugged remotely -- Key: HIVE-2500 URL: https://issues.apache.org/jira/browse/HIVE-2500 Project: Hive Issue Type: New Feature Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Ensure HiveConf includes all properties defined in hive-default.xml
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1423/ --- (Updated 2011-10-14 16:54:00.383612) Review request for hive, Carl Steinbach and John Sichi. Changes --- Rebase the patch on the trunk Summary --- The below configurations are not used in the code base so removed from the hive-default.xml hive.mapjoin.hashtable.initialCapacity hive.mapjoin.hashtable.loadfactor hive.mapjoin.smalltable.filesize hive.optimize.pruner hive.stats.jdbc.atomic hive.concurrency.manager The below configurations are not used in the code base but these configurations are required for the system so need to maintain in the hive-default.xml and no need to add in the HiveConf.java javax.jdo.option.ConnectionDriverName javax.jdo.PersistenceManagerFactoryClass javax.jdo.option.DetachAllOnCommit javax.jdo.option.NonTransactionalRead javax.jdo.option.ConnectionUserName fs.har.impl This addresses bug HIVE-2196. https://issues.apache.org/jira/browse/HIVE-2196 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182263 trunk/conf/hive-default.xml 1182263 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182263 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java 1182263 trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java 1182263 Diff: https://reviews.apache.org/r/1423/diff Testing --- All unit tests passed Thanks, chinna
[jira] [Commented] (HIVE-2196) Ensure HiveConf includes all properties defined in hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127691#comment-13127691 ] Chinna Rao Lalam commented on HIVE-2196: Rebase the patch on the trunk Ensure HiveConf includes all properties defined in hive-default.xml --- Key: HIVE-2196 URL: https://issues.apache.org/jira/browse/HIVE-2196 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0 Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Attachments: HIVE-2196.1.patch, HIVE-2196.2.patch, HIVE-2196.3.patch, HIVE-2196.4.patch, HIVE-2196.5.patch, HIVE-2196.build.log, HIVE-2196.patch There are a bunch of properties that are defined in hive-default.xml but not in HiveConf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2196) Ensure HiveConf includes all properties defined in hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-2196: --- Attachment: HIVE-2196.5.patch Ensure HiveConf includes all properties defined in hive-default.xml --- Key: HIVE-2196 URL: https://issues.apache.org/jira/browse/HIVE-2196 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0 Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Attachments: HIVE-2196.1.patch, HIVE-2196.2.patch, HIVE-2196.3.patch, HIVE-2196.4.patch, HIVE-2196.5.patch, HIVE-2196.build.log, HIVE-2196.patch There are a bunch of properties that are defined in hive-default.xml but not in HiveConf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2196) Ensure HiveConf includes all properties defined in hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127692#comment-13127692 ] jirapos...@reviews.apache.org commented on HIVE-2196: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1423/ --- (Updated 2011-10-14 16:54:00.383612) Review request for hive, Carl Steinbach and John Sichi. Changes --- Rebase the patch on the trunk Summary --- The below configurations are not used in the code base so removed from the hive-default.xml hive.mapjoin.hashtable.initialCapacity hive.mapjoin.hashtable.loadfactor hive.mapjoin.smalltable.filesize hive.optimize.pruner hive.stats.jdbc.atomic hive.concurrency.manager The below configurations are not used in the code base but these configurations are required for the system so need to maintain in the hive-default.xml and no need to add in the HiveConf.java javax.jdo.option.ConnectionDriverName javax.jdo.PersistenceManagerFactoryClass javax.jdo.option.DetachAllOnCommit javax.jdo.option.NonTransactionalRead javax.jdo.option.ConnectionUserName fs.har.impl This addresses bug HIVE-2196. https://issues.apache.org/jira/browse/HIVE-2196 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182263 trunk/conf/hive-default.xml 1182263 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182263 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1182263 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java 1182263 trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java 1182263 Diff: https://reviews.apache.org/r/1423/diff Testing --- All unit tests passed Thanks, chinna Ensure HiveConf includes all properties defined in hive-default.xml --- Key: HIVE-2196 URL: https://issues.apache.org/jira/browse/HIVE-2196 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0 Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Attachments: HIVE-2196.1.patch, HIVE-2196.2.patch, HIVE-2196.3.patch, HIVE-2196.4.patch, HIVE-2196.5.patch, HIVE-2196.build.log, HIVE-2196.patch There are a bunch of properties that are defined in hive-default.xml but not in HiveConf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2196) Ensure HiveConf includes all properties defined in hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-2196: --- Status: Patch Available (was: Open) Ensure HiveConf includes all properties defined in hive-default.xml --- Key: HIVE-2196 URL: https://issues.apache.org/jira/browse/HIVE-2196 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0 Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Attachments: HIVE-2196.1.patch, HIVE-2196.2.patch, HIVE-2196.3.patch, HIVE-2196.4.patch, HIVE-2196.5.patch, HIVE-2196.build.log, HIVE-2196.patch There are a bunch of properties that are defined in hive-default.xml but not in HiveConf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2502) Add whitelist for hosts used in table/partition locations
[ https://issues.apache.org/jira/browse/HIVE-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127695#comment-13127695 ] jirapos...@reviews.apache.org commented on HIVE-2502: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2371/ --- (Updated 2011-10-14 16:56:01.946053) Review request for hive and Paul Yang. Changes --- All lines are within the 100 char limit. Summary --- I added a whitelist configuration variable, whose value should be a comma separated list of hosts. A set of strings is created based on this list and the host name in a location given to a create/alter table/partition command to the HiveMetastore is compared to the values in that set. If the list is nonempty, the table being altered/created or containing the partition is not external/non-native, and the location has an explicit host, the command is failed if the host is not in that set, otherwise the command continues as normal. This addresses bug HIVE-2502. https://issues.apache.org/jira/browse/HIVE-2502 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1182505 trunk/conf/hive-default.xml 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1182505 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1182505 Diff: https://reviews.apache.org/r/2371/diff Testing --- I used Thrift calls to test the following for each of the 4 commands create/alter table/partition: Issuing the command with a location with a host that is not in the whitelist fails. Issuing the command with a location with a host in the whitelist does not fail. Thanks, Kevin Add whitelist for hosts used in table/partition locations - Key: HIVE-2502 URL: https://issues.apache.org/jira/browse/HIVE-2502 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2502.1.patch.txt, HIVE-2502.2.patch.txt Add a whitelist of host names that can be checked before creating/altering a table/partition to verify that the location is acceptable. The whitelist should be empty by default, and should be configurable. The check should default to pass if there is no host in the location, or the whitelist is empty. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2502) Add whitelist for hosts used in table/partition locations
[ https://issues.apache.org/jira/browse/HIVE-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2502: Attachment: HIVE-2502.3.patch.txt Add whitelist for hosts used in table/partition locations - Key: HIVE-2502 URL: https://issues.apache.org/jira/browse/HIVE-2502 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2502.1.patch.txt, HIVE-2502.2.patch.txt, HIVE-2502.3.patch.txt Add a whitelist of host names that can be checked before creating/altering a table/partition to verify that the location is acceptable. The whitelist should be empty by default, and should be configurable. The check should default to pass if there is no host in the location, or the whitelist is empty. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Patch need review and commit
Hi committers, These patches need review and commit. HIVE-2465 HIVE-2196 HIVE-2178 Thanks in advance, Chinna Rao Lalam
[jira] [Commented] (HIVE-2178) Log related Check style Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127714#comment-13127714 ] Ashutosh Chauhan commented on HIVE-2178: @Chinna, Looks all the error messages has lost information. In all the *.q.out files which this patch updates I was expecting those error messages to get better, but nearly all of them have become worse and not printing useful messages any more which they used to print. Log related Check style Comments fixes -- Key: HIVE-2178 URL: https://issues.apache.org/jira/browse/HIVE-2178 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2178.1.patch, HIVE-2178.2.patch, HIVE-2178.3.patch, HIVE-2178.patch Fix Log related Check style Comments -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2501) When new instance of Hive (class) is created, the current database is reset to default (current database shouldn't be changed).
[ https://issues.apache.org/jira/browse/HIVE-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127729#comment-13127729 ] Robert Surówka commented on HIVE-2501: -- True, I am sorry for the overlook. Yet, it would be good if any patch for that bug were committed. When new instance of Hive (class) is created, the current database is reset to default (current database shouldn't be changed). --- Key: HIVE-2501 URL: https://issues.apache.org/jira/browse/HIVE-2501 Project: Hive Issue Type: Bug Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2501.1.patch This bug manifested to me, when first thing I did after starting Hive, was to call use db_name; but then calling show tables; was still showing tables from the default database (and I had to call use again to actually change database). This bug might have manifested only due to a specific Hive deployment I am using (I didn't investigate this issue that deeply). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2178) Log related Check style Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-2178: - Status: Open (was: Patch Available) I agree with Ashutosh. It's good to be chaining the actual exception objects, but apparently Hive is ignoring them at the top level, so for now it's necessary to also append their string form to the message. Log related Check style Comments fixes -- Key: HIVE-2178 URL: https://issues.apache.org/jira/browse/HIVE-2178 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2178.1.patch, HIVE-2178.2.patch, HIVE-2178.3.patch, HIVE-2178.patch Fix Log related Check style Comments -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2223: Add group by support for keys of type ARRAY and MAP.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2381/#review2595
---
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
https://reviews.apache.org/r/2381/#comment5833
Since this is a list, sequence is significant, e.g. [1,2,3] != [3,2,1]. So
perhaps we should make the hash code reflect this?
Java's AbstractList does this via:
while (i.hasNext()) {
E obj = i.next();
hashCode = 31*hashCode + (obj==null ? 0 : obj.hashCode());
}
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
https://reviews.apache.org/r/2381/#comment5834
For java.util.HashMap, they xor the key's hashcode with the value's
hashcode for each entry, and then sum over all entries. I suppose that's to
distinguish
{a-b, c-d} from {a-d, b-c}.
although it fails to distinguish
{a-b} from {b-a}
- John
On 2011-10-13 16:50:53, Jonathan Chang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2381/
---
(Updated 2011-10-13 16:50:53)
Review request for hive.
Summary
---
Adds hash codes for List and Map object inspectors.
This addresses bug HIVE-2223.
https://issues.apache.org/jira/browse/HIVE-2223
Diffs
-
ql/src/test/queries/clientpositive/groupby_complex_types.q PRE-CREATION
ql/src/test/results/clientpositive/groupby_complex_types.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
2d45aba
Diff: https://reviews.apache.org/r/2381/diff
Testing
---
Added unittest.
Thanks,
Jonathan
[jira] [Commented] (HIVE-2223) support grouping on complex types in Hive
[
https://issues.apache.org/jira/browse/HIVE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127747#comment-13127747
]
jirapos...@reviews.apache.org commented on HIVE-2223:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2381/#review2595
---
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
https://reviews.apache.org/r/2381/#comment5833
Since this is a list, sequence is significant, e.g. [1,2,3] != [3,2,1]. So
perhaps we should make the hash code reflect this?
Java's AbstractList does this via:
while (i.hasNext()) {
E obj = i.next();
hashCode = 31*hashCode + (obj==null ? 0 : obj.hashCode());
}
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
https://reviews.apache.org/r/2381/#comment5834
For java.util.HashMap, they xor the key's hashcode with the value's
hashcode for each entry, and then sum over all entries. I suppose that's to
distinguish
{a-b, c-d} from {a-d, b-c}.
although it fails to distinguish
{a-b} from {b-a}
- John
On 2011-10-13 16:50:53, Jonathan Chang wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2381/
bq. ---
bq.
bq. (Updated 2011-10-13 16:50:53)
bq.
bq.
bq. Review request for hive.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Adds hash codes for List and Map object inspectors.
bq.
bq.
bq. This addresses bug HIVE-2223.
bq. https://issues.apache.org/jira/browse/HIVE-2223
bq.
bq.
bq. Diffs
bq. -
bq.
bq.ql/src/test/queries/clientpositive/groupby_complex_types.q PRE-CREATION
bq.ql/src/test/results/clientpositive/groupby_complex_types.q.out
PRE-CREATION
bq.
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
2d45aba
bq.
bq. Diff: https://reviews.apache.org/r/2381/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Added unittest.
bq.
bq.
bq. Thanks,
bq.
bq. Jonathan
bq.
bq.
support grouping on complex types in Hive
-
Key: HIVE-2223
URL: https://issues.apache.org/jira/browse/HIVE-2223
Project: Hive
Issue Type: New Feature
Reporter: Kate Ting
Assignee: Jonathan Chang
Priority: Minor
Attachments: HIVE-2223.patch
Creating a query with a GROUP BY statement when an array type column is part
of the column list is not yet supported:
CREATE TABLE test_group_by ( key INT, group INT, terms ARRAYSTRING);
SELECT key, terms, count(group) FROM test_group_by GROUP BY key, terms;
...
Hash code on complex types not supported yet.
java.lang.RuntimeException: Error while closing operators
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:232)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Hash code on complex types not supported yet.
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:799)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:462)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:211)
... 4 more
Caused by: java.lang.RuntimeException: Hash code on complex types not
supported yet.
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hashCode(ObjectInspectorUtils.java:348)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:187)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:746)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:780)
... 9 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more
[jira] [Updated] (HIVE-2223) support grouping on complex types in Hive
[ https://issues.apache.org/jira/browse/HIVE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-2223: - Status: Open (was: Patch Available) See comments in review board regarding the hash codes. support grouping on complex types in Hive - Key: HIVE-2223 URL: https://issues.apache.org/jira/browse/HIVE-2223 Project: Hive Issue Type: New Feature Reporter: Kate Ting Assignee: Jonathan Chang Priority: Minor Attachments: HIVE-2223.patch Creating a query with a GROUP BY statement when an array type column is part of the column list is not yet supported: CREATE TABLE test_group_by ( key INT, group INT, terms ARRAYSTRING); SELECT key, terms, count(group) FROM test_group_by GROUP BY key, terms; ... Hash code on complex types not supported yet. java.lang.RuntimeException: Error while closing operators at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:232) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:799) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:462) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:211) ... 4 more Caused by: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hashCode(ObjectInspectorUtils.java:348) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:187) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598) at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:746) at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:780) ... 9 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2223: Add group by support for keys of type ARRAY and MAP.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2381/ --- (Updated 2011-10-14 18:38:00.199892) Review request for hive. Changes --- Make hash match algorithms used by Java. Summary --- Adds hash codes for List and Map object inspectors. This addresses bug HIVE-2223. https://issues.apache.org/jira/browse/HIVE-2223 Diffs (updated) - ql/src/test/queries/clientpositive/groupby_complex_types.q PRE-CREATION ql/src/test/results/clientpositive/groupby_complex_types.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2d45aba Diff: https://reviews.apache.org/r/2381/diff Testing --- Added unittest. Thanks, Jonathan
[jira] [Commented] (HIVE-2223) support grouping on complex types in Hive
[ https://issues.apache.org/jira/browse/HIVE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127765#comment-13127765 ] jirapos...@reviews.apache.org commented on HIVE-2223: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2381/ --- (Updated 2011-10-14 18:38:00.199892) Review request for hive. Changes --- Make hash match algorithms used by Java. Summary --- Adds hash codes for List and Map object inspectors. This addresses bug HIVE-2223. https://issues.apache.org/jira/browse/HIVE-2223 Diffs (updated) - ql/src/test/queries/clientpositive/groupby_complex_types.q PRE-CREATION ql/src/test/results/clientpositive/groupby_complex_types.q.out PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2d45aba Diff: https://reviews.apache.org/r/2381/diff Testing --- Added unittest. Thanks, Jonathan support grouping on complex types in Hive - Key: HIVE-2223 URL: https://issues.apache.org/jira/browse/HIVE-2223 Project: Hive Issue Type: New Feature Reporter: Kate Ting Assignee: Jonathan Chang Priority: Minor Attachments: HIVE-2223.patch Creating a query with a GROUP BY statement when an array type column is part of the column list is not yet supported: CREATE TABLE test_group_by ( key INT, group INT, terms ARRAYSTRING); SELECT key, terms, count(group) FROM test_group_by GROUP BY key, terms; ... Hash code on complex types not supported yet. java.lang.RuntimeException: Error while closing operators at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:232) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:799) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:462) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:211) ... 4 more Caused by: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hashCode(ObjectInspectorUtils.java:348) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:187) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598) at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:746) at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:780) ... 9 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2223) support grouping on complex types in Hive
[ https://issues.apache.org/jira/browse/HIVE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127781#comment-13127781 ] John Sichi commented on HIVE-2223: -- +1. Will commit when tests pass. support grouping on complex types in Hive - Key: HIVE-2223 URL: https://issues.apache.org/jira/browse/HIVE-2223 Project: Hive Issue Type: New Feature Reporter: Kate Ting Assignee: Jonathan Chang Priority: Minor Attachments: HIVE-2223.patch, HIVE-2223.patch.2 Creating a query with a GROUP BY statement when an array type column is part of the column list is not yet supported: CREATE TABLE test_group_by ( key INT, group INT, terms ARRAYSTRING); SELECT key, terms, count(group) FROM test_group_by GROUP BY key, terms; ... Hash code on complex types not supported yet. java.lang.RuntimeException: Error while closing operators at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:232) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:799) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:462) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:211) ... 4 more Caused by: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hashCode(ObjectInspectorUtils.java:348) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:187) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598) at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:746) at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:780) ... 9 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2503) HiveServer should provide per session configuration
[ https://issues.apache.org/jira/browse/HIVE-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127789#comment-13127789 ] Ashutosh Chauhan commented on HIVE-2503: @Navis, Can you elaborate bit more about the problem, which session you are referring to here? HiveServer should provide per session configuration --- Key: HIVE-2503 URL: https://issues.apache.org/jira/browse/HIVE-2503 Project: Hive Issue Type: Bug Components: CLI, Server Infrastructure Reporter: Navis Currently ThriftHiveProcessorFactory returns same HiveConf instance to HiveServerHandler, making impossible to use per sesssion configuration. Just wrapping 'conf' - 'new HiveConf(conf)' seemed to solve this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2502) Add whitelist for hosts used in table/partition locations
[ https://issues.apache.org/jira/browse/HIVE-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127844#comment-13127844 ] Paul Yang commented on HIVE-2502: - +1 Will test and commit. Add whitelist for hosts used in table/partition locations - Key: HIVE-2502 URL: https://issues.apache.org/jira/browse/HIVE-2502 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2502.1.patch.txt, HIVE-2502.2.patch.txt, HIVE-2502.3.patch.txt Add a whitelist of host names that can be checked before creating/altering a table/partition to verify that the location is acceptable. The whitelist should be empty by default, and should be configurable. The check should default to pass if there is no host in the location, or the whitelist is empty. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Support with rollup option for group by
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1637/ --- (Updated 2011-10-14 21:14:02.700645) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Changes --- It's been a while, so I ran svn up. Summary --- If a user runs a query that includes group by ... with rollup, the behavior is like that of MySQL, see the task. I had to implement 4 different ways of providing this behavior to fit in with the 4 different ways of implementing group by depending on whether map aggregation is allowed, and whether the data is known to be skewed. If map aggregation is allowed, it is a simple matter of adding new keys with an increasing number of NULLs to the hash map to collect the data for the new rows as part of the map side hash aggregation. If map aggregation is not allowed and the data is not skewed, I perform a reduce job which performs a hash aggregation very similar to the way it is performed on the map side. I then perform a mergepartial reduce job to perform a final aggregation on the hash aggregation. If map aggregation is not allowed and the data is skewed and there are no distinct aggregations for the group by, I aggregate data for the new rows with NULLs as part of the non-hash aggregation. This was as simple as adding the new functionality to the map side hash aggregation. If map aggregation is not allowed and the data is skewed and there are distinct aggregations fro the group by, I perform a reduce job which performs a hash aggregation, and then use the same implementation that is used when there is no rollup option set. I have done my best not to detract from any optimizations that were made for each of the four different implementations of group by, but, r for the ones where I add a new reduce job, I am not sure how successful I was. Currently, the optimizations for multiple group bys is not supported for queries with the rollup option set, but I am continuing to look into this. This addresses bug HIVE-2397. https://issues.apache.org/jira/browse/HIVE-2397 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 1183502 trunk/ql/src/test/queries/clientpositive/groupby10_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby11_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_nomap_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby9_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby_map_ppr_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby_map_ppr_withrollup.q PRE-CREATION
[jira] [Commented] (HIVE-2397) Support with rollup option for group by
[ https://issues.apache.org/jira/browse/HIVE-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127866#comment-13127866 ] jirapos...@reviews.apache.org commented on HIVE-2397: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1637/ --- (Updated 2011-10-14 21:14:02.700645) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Changes --- It's been a while, so I ran svn up. Summary --- If a user runs a query that includes group by ... with rollup, the behavior is like that of MySQL, see the task. I had to implement 4 different ways of providing this behavior to fit in with the 4 different ways of implementing group by depending on whether map aggregation is allowed, and whether the data is known to be skewed. If map aggregation is allowed, it is a simple matter of adding new keys with an increasing number of NULLs to the hash map to collect the data for the new rows as part of the map side hash aggregation. If map aggregation is not allowed and the data is not skewed, I perform a reduce job which performs a hash aggregation very similar to the way it is performed on the map side. I then perform a mergepartial reduce job to perform a final aggregation on the hash aggregation. If map aggregation is not allowed and the data is skewed and there are no distinct aggregations for the group by, I aggregate data for the new rows with NULLs as part of the non-hash aggregation. This was as simple as adding the new functionality to the map side hash aggregation. If map aggregation is not allowed and the data is skewed and there are distinct aggregations fro the group by, I perform a reduce job which performs a hash aggregation, and then use the same implementation that is used when there is no rollup option set. I have done my best not to detract from any optimizations that were made for each of the four different implementations of group by, but, r for the ones where I add a new reduce job, I am not sure how successful I was. Currently, the optimizations for multiple group bys is not supported for queries with the rollup option set, but I am continuing to look into this. This addresses bug HIVE-2397. https://issues.apache.org/jira/browse/HIVE-2397 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1183502 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 1183502 trunk/ql/src/test/queries/clientpositive/groupby10_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby11_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_nomap_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_withrollup.q PRE-CREATION
[jira] [Updated] (HIVE-2397) Support with rollup option for group by
[ https://issues.apache.org/jira/browse/HIVE-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2397: Attachment: HIVE-2397.5.patch.txt Support with rollup option for group by --- Key: HIVE-2397 URL: https://issues.apache.org/jira/browse/HIVE-2397 Project: Hive Issue Type: New Feature Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2397.2.patch.txt, HIVE-2397.3.patch.txt, HIVE-2397.4.patch.txt, HIVE-2397.5.patch.txt We should support the ROLLUP operator similar to the way MySQL is implemented. Exerted from MySQL documents: mysql SELECT year, country, product, SUM(profit) - FROM sales - GROUP BY year, country, product WITH ROLLUP; +--+-++-+ | year | country | product| SUM(profit) | +--+-++-+ | 2000 | Finland | Computer |1500 | | 2000 | Finland | Phone | 100 | | 2000 | Finland | NULL |1600 | | 2000 | India | Calculator | 150 | | 2000 | India | Computer |1200 | | 2000 | India | NULL |1350 | | 2000 | USA | Calculator | 75 | | 2000 | USA | Computer |1500 | | 2000 | USA | NULL |1575 | | 2000 | NULL| NULL |4525 | | 2001 | Finland | Phone | 10 | | 2001 | Finland | NULL | 10 | | 2001 | USA | Calculator | 50 | | 2001 | USA | Computer |2700 | | 2001 | USA | TV | 250 | | 2001 | USA | NULL |3000 | | 2001 | NULL| NULL |3010 | | NULL | NULL| NULL |7535 | +--+-++-+ http://dev.mysql.com/doc/refman/5.0/en/group-by-modifiers.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127896#comment-13127896 ] Robert Surówka commented on HIVE-2500: -- Apparently this diff is still wrong (I will have to do small fix to MapReduceTask.java, when debug off, it will add just one if check for a key being in a map per execute() call of that task, and only if new JVM is about to be started). As I am working on it I plan to add following features: -user could specify port on which Hive JVM will wait (8000 will be default). -user could specify whether child JVMs should be also started in debug mode (default: not). If started in debug mode, child JVMs will wait on a different ports than parent (and I will try to make it in such a way, so probability of ports collision would be very small (to make it impossible one would need to use some between process communication, since children can start their own children, and I don't want to do that yet (especially that HIVE may be deployed on various os-es). User will know on which port new JVMs are waiting, since upon start they print that information. New JVMs will be started on ports higher than port of the initial JVM - so if someone needs, he can set initial port to be higher than any other already used TCP port). ETA: by mid of next week (hopefully) Allow Hive to be debugged remotely -- Key: HIVE-2500 URL: https://issues.apache.org/jira/browse/HIVE-2500 Project: Hive Issue Type: New Feature Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2500) Allow Hive to be debugged remotely
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127899#comment-13127899 ] Robert Surówka commented on HIVE-2500: -- Sorry, I meant MapRedTask. Allow Hive to be debugged remotely -- Key: HIVE-2500 URL: https://issues.apache.org/jira/browse/HIVE-2500 Project: Hive Issue Type: New Feature Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2500) Allow Hive to be debugged remotely
[ https://issues.apache.org/jira/browse/HIVE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Surówka updated HIVE-2500: - Status: Open (was: Patch Available) Allow Hive to be debugged remotely -- Key: HIVE-2500 URL: https://issues.apache.org/jira/browse/HIVE-2500 Project: Hive Issue Type: New Feature Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2500.1.patch, HIVE-2500.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127905#comment-13127905 ] Ashutosh Chauhan commented on HIVE-2504: Can't you achieve this already by setting dfs.umask to 002 in hdfs-site.xml Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2497) partition pruning prune some right partition under specific conditions
[ https://issues.apache.org/jira/browse/HIVE-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127949#comment-13127949 ] Ashutosh Chauhan commented on HIVE-2497: Binlijin, It will be good to also include a test case in the patch for regression purposes. partition pruning prune some right partition under specific conditions --- Key: HIVE-2497 URL: https://issues.apache.org/jira/browse/HIVE-2497 Project: Hive Issue Type: Bug Affects Versions: 0.7.1 Reporter: binlijin Labels: patch Fix For: 0.9.0 Attachments: hive-2497.patch create table src3(key string, value string) partitioned by (pt string) row format delimited fields terminated by ','; ALTER TABLE src3 ADD IF NOT EXISTS PARTITION (pt='2011091100') ; ALTER TABLE src3 ADD IF NOT EXISTS PARTITION (pt='2011091200') ; ALTER TABLE src3 ADD IF NOT EXISTS PARTITION (pt='2011091300') ; explain extended select user_id from ( select cast(key as int) as user_id ,case when (value like 'aaa%' or value like 'vvv%') then 1 else 0 end as tag_student from src3 ) sub where sub.tag_student 0; STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Alias - Map Operator Tree: sub:src3 TableScan alias: src3 Filter Operator isSamplingPred: false predicate: expr: (CASE WHEN (((value like 'aaa%') or (value like 'vvv%'))) THEN (1) ELSE (0) END 0) type: boolean Select Operator expressions: expr: UDFToInteger(key) type: int expr: CASE WHEN (((value like 'aaa%') or (value like 'vvv%'))) THEN (1) ELSE (0) END type: int outputColumnNames: _col0, _col1 Filter Operator isSamplingPred: false predicate: expr: (_col1 0) type: boolean Select Operator expressions: expr: _col0 type: int outputColumnNames: _col0 File Output Operator compressed: false GlobalTableId: 0 directory: hdfs://localhost:54310/tmp/hive-tianzhao/hive_2011-10-11_19-26-12_894_9085644225727185586/-ext-10001 NumFilesPerFileSink: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: columns _col0 columns.types int serialization.format 1 TotalFiles: 1 MultiFileSpray: false Needs Tagging: false Stage: Stage-0 Fetch Operator limit: -1 if we set hive.optimize.ppd=false; STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 is a root stage STAGE PLANS: Stage: Stage-1 Map Reduce Alias - Map Operator Tree: sub:src3 TableScan alias: src3 Select Operator expressions: expr: UDFToInteger(key) type: int expr: CASE WHEN (((value like 'aaa%') or (value like 'vvv%'))) THEN (1) ELSE (0) END type: int outputColumnNames: _col0, _col1 Filter Operator isSamplingPred: false predicate: expr: (_col1 0) type: boolean Select Operator expressions: expr: _col0 type: int outputColumnNames: _col0 File Output Operator compressed: false GlobalTableId: 0 directory: hdfs://localhost:54310/tmp/hive-tianzhao/hive_2011-10-11_19-27-22_527_1729287213481398480/-ext-10001 NumFilesPerFileSink: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: columns _col0 columns.types int
[jira] [Commented] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127992#comment-13127992 ] Carl Steinbach commented on HIVE-2504: -- @Ashutosh: Yes, but that then shifts the configuration burden to the administrator. The point of this ticket is to reduce the configuration burden on admins/users by providing a sensible set of default configuration parameters. I'm also selfishly interested in heading-off the inevitable stream of emails to hive-user from folks wondering why Hive won't let them insert data into the table they just created. Do you disagree with this approach? Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2505) Update junit jar in testlibs
Update junit jar in testlibs Key: HIVE-2505 URL: https://issues.apache.org/jira/browse/HIVE-2505 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: Update junit jar in testlibs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2391/ --- Review request for hive and Yongqiang He. Summary --- I updated the junit jar to the latest available in the maven repository. This addresses bug HIVE-2505. https://issues.apache.org/jira/browse/HIVE-2505 Diffs - trunk/ivy/libraries.properties 1183425 trunk/testlibs/junit-3.8.1.LICENSE.txt 1183425 trunk/testlibs/junit-3.8.1.jar UNKNOWN trunk/testlibs/junit-4.10.LICENSE.txt PRE-CREATION trunk/testlibs/junit-4.10.jar UNKNOWN Diff: https://reviews.apache.org/r/2391/diff Testing --- I ran ant test and the tests continued to pass. Thanks, Kevin
[jira] [Commented] (HIVE-2505) Update junit jar in testlibs
[ https://issues.apache.org/jira/browse/HIVE-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127999#comment-13127999 ] jirapos...@reviews.apache.org commented on HIVE-2505: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2391/ --- Review request for hive and Yongqiang He. Summary --- I updated the junit jar to the latest available in the maven repository. This addresses bug HIVE-2505. https://issues.apache.org/jira/browse/HIVE-2505 Diffs - trunk/ivy/libraries.properties 1183425 trunk/testlibs/junit-3.8.1.LICENSE.txt 1183425 trunk/testlibs/junit-3.8.1.jar UNKNOWN trunk/testlibs/junit-4.10.LICENSE.txt PRE-CREATION trunk/testlibs/junit-4.10.jar UNKNOWN Diff: https://reviews.apache.org/r/2391/diff Testing --- I ran ant test and the tests continued to pass. Thanks, Kevin Update junit jar in testlibs Key: HIVE-2505 URL: https://issues.apache.org/jira/browse/HIVE-2505 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Update junit jar in testlibs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2391/ --- (Updated 2011-10-15 00:20:46.329179) Review request for hive and Yongqiang He. Summary --- I updated the junit jar to the latest available in the maven repository. This addresses bug HIVE-2505. https://issues.apache.org/jira/browse/HIVE-2505 Diffs - trunk/ivy/libraries.properties 1183425 trunk/pdk/build.xml 1183425 trunk/testlibs/junit-3.8.1.LICENSE.txt 1183425 trunk/testlibs/junit-3.8.1.jar UNKNOWN trunk/testlibs/junit-4.10.LICENSE.txt PRE-CREATION trunk/testlibs/junit-4.10.jar UNKNOWN Diff: https://reviews.apache.org/r/2391/diff Testing --- I ran ant test and the tests continued to pass. Thanks, Kevin
[jira] [Updated] (HIVE-2505) Update junit jar in testlibs
[ https://issues.apache.org/jira/browse/HIVE-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2505: Attachment: HIVE-2505.1.patch.txt Update junit jar in testlibs Key: HIVE-2505 URL: https://issues.apache.org/jira/browse/HIVE-2505 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2505.1.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2505) Update junit jar in testlibs
[ https://issues.apache.org/jira/browse/HIVE-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128004#comment-13128004 ] jirapos...@reviews.apache.org commented on HIVE-2505: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2391/ --- (Updated 2011-10-15 00:20:46.329179) Review request for hive and Yongqiang He. Summary --- I updated the junit jar to the latest available in the maven repository. This addresses bug HIVE-2505. https://issues.apache.org/jira/browse/HIVE-2505 Diffs - trunk/ivy/libraries.properties 1183425 trunk/pdk/build.xml 1183425 trunk/testlibs/junit-3.8.1.LICENSE.txt 1183425 trunk/testlibs/junit-3.8.1.jar UNKNOWN trunk/testlibs/junit-4.10.LICENSE.txt PRE-CREATION trunk/testlibs/junit-4.10.jar UNKNOWN Diff: https://reviews.apache.org/r/2391/diff Testing --- I ran ant test and the tests continued to pass. Thanks, Kevin Update junit jar in testlibs Key: HIVE-2505 URL: https://issues.apache.org/jira/browse/HIVE-2505 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2505.1.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2504) Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory
[ https://issues.apache.org/jira/browse/HIVE-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128019#comment-13128019 ] Ashutosh Chauhan commented on HIVE-2504: No, I don't disagree with the approach. I was just interested to know whether its already possible to do so. Warehouse table subdirectories should inherit the group permissions of the warehouse parent directory - Key: HIVE-2504 URL: https://issues.apache.org/jira/browse/HIVE-2504 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach When the Hive Metastore creates a subdirectory in the Hive warehouse for a new table it does so with the default HDFS permissions. Since the default dfs.umask value is 022, this means that the new subdirectory will not inherit the group write permissions of the hive warehouse directory. We should make the umask used by Warehouse.mkdirs() configurable, and set it to use a default value of 002. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2407) File extensions not preserved in Hive.checkPaths when renaming new destination file
[ https://issues.apache.org/jira/browse/HIVE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-2407. Resolution: Fixed Fix Version/s: 0.8.0 This got fixed as part of HIVE-2296. File extensions not preserved in Hive.checkPaths when renaming new destination file --- Key: HIVE-2407 URL: https://issues.apache.org/jira/browse/HIVE-2407 Project: Hive Issue Type: Bug Affects Versions: 0.7.1 Reporter: Brian Muller Fix For: 0.8.0 In the checkPaths method of Hive.java, a new destination filename will be chosen if the source filename already exists in the destination directory. This new filename follows the simple schema of adding _copy_N to the source filename (with increasing N until a non-existing filename is found). If the file has been LZO compressed it is imperative that the extension remain .lzo so that the LZO indexer can find it and create a corresponding index file. It would be much better to use a prefix of copy_N_ or insert _copy_N somewhere else in the filename. Without this, Hive and LZO compression will not work when identical source filenames are inserted into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2503) HiveServer should provide per session configuration
[
https://issues.apache.org/jira/browse/HIVE-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13128078#comment-13128078
]
Navis commented on HIVE-2503:
-
I'm using multiple ThriftHive.Client to single HiveServer.
For example :
{code}
client1 = factory.getCilent(~~~);
client2 = factory.getCilent(~~~);
client1.execute(set hiveconf:var=value1);
client2.execute(set hiveconf:var=value2);
client1.execute(set hiveconf:var); -- returns value2
{code}
HiveServer should provide per session configuration
---
Key: HIVE-2503
URL: https://issues.apache.org/jira/browse/HIVE-2503
Project: Hive
Issue Type: Bug
Components: CLI, Server Infrastructure
Reporter: Navis
Currently ThriftHiveProcessorFactory returns same HiveConf instance to
HiveServerHandler, making impossible to use per sesssion configuration. Just
wrapping 'conf' - 'new HiveConf(conf)' seemed to solve this problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
