[jira] [Updated] (HIVE-2215) Add api for marking / querying set of partitions for events
[ https://issues.apache.org/jira/browse/HIVE-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2215: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Thanks Ashutosh! Add api for marking / querying set of partitions for events --- Key: HIVE-2215 URL: https://issues.apache.org/jira/browse/HIVE-2215 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.8.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2215_full-1.patch, hive_2215.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-trunk-h0.21 #784
See https://builds.apache.org/job/Hive-trunk-h0.21/784/changes Changes: [cws] HIVE-2215. Add api for marking / querying set of partitions for events (Ashutosh Chauhan via cws) -- [...truncated 12632 lines...] [junit] OK [junit] PREHOOK: query: select count(1) as cnt from testhivedrivertable [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2011-06-20_03-22-04_749_8495937151780152012/-mr-1 [junit] Total MapReduce jobs = 1 [junit] Launching Job 1 out of 1 [junit] Number of reduce tasks determined at compile time: 1 [junit] In order to change the average load for a reducer (in bytes): [junit] set hive.exec.reducers.bytes.per.reducer=number [junit] In order to limit the maximum number of reducers: [junit] set hive.exec.reducers.max=number [junit] In order to set a constant number of reducers: [junit] set mapred.reduce.tasks=number [junit] Job running in-process (local Hadoop) [junit] Hadoop job information for null: number of mappers: 0; number of reducers: 0 [junit] 2011-06-20 03:22:07,857 null map = 100%, reduce = 100% [junit] Ended Job = job_local_0001 [junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2011-06-20_03-22-04_749_8495937151780152012/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/service/tmp/hive_job_log_hudson_201106200322_997120075.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] PREHOOK: type: LOAD [junit] PREHOOK: Output: default@testhivedrivertable [junit] Copying data from https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt [junit] Loading data to table default.testhivedrivertable [junit] POSTHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: select * from testhivedrivertable limit 10 [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2011-06-20_03-22-09_328_6897948629714481006/-mr-1 [junit] POSTHOOK: query: select * from testhivedrivertable limit 10 [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2011-06-20_03-22-09_328_6897948629714481006/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/service/tmp/hive_job_log_hudson_201106200322_857783552.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit]
Re: Review Request: Patch for Hive-2209, extending ObjectInspectorUtils.compare with some map comparison implementations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/879/ --- (Updated 2011-06-20 12:54:09.245202) Review request for hive and Yongqiang He. Changes --- Fixed a lowercase/uppercase typo in the test classes Summary --- Patch for HIVE-2209 Diffs (updated) - serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/CrossMapEqualComparer.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/FullMapEqualComparer.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/MapEqualComparer.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2b77072 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/SimpleMapEqualComparer.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestCrossMapEqualComparer.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestFullMapEqualComparer.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestSimpleMapEqualComparer.java PRE-CREATION Diff: https://reviews.apache.org/r/879/diff Testing --- Tests added Thanks, Krishna
Re: Review Request: Add LazyBinaryColumnarSerDe
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/806/ --- (Updated 2011-06-20 12:56:38.943799) Review request for hive and Yongqiang He. Changes --- After separating out mapcomparer changes to its own patch Summary --- Add LazyBinaryColumnarSerDe This addresses bug HIVE-956. https://issues.apache.org/jira/browse/HIVE-956 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 77a6dc6 serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStruct.java e79021d serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStructBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java e927547 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java 2e2896c serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1440472 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryObject.java ea20b34 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 4285ab3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ColumnarStructObjectInspector.java 66f4f8d serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java 90561a1 serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java PRE-CREATION Diff: https://reviews.apache.org/r/806/diff Testing --- Tests added Thanks, Krishna
[jira] [Commented] (HIVE-956) Add support of columnar binary serde
[ https://issues.apache.org/jira/browse/HIVE-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051962#comment-13051962 ] jirapos...@reviews.apache.org commented on HIVE-956: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/806/ --- (Updated 2011-06-20 12:56:38.943799) Review request for hive and Yongqiang He. Changes --- After separating out mapcomparer changes to its own patch Summary --- Add LazyBinaryColumnarSerDe This addresses bug HIVE-956. https://issues.apache.org/jira/browse/HIVE-956 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 77a6dc6 serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStruct.java e79021d serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStructBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java e927547 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java 2e2896c serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1440472 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryObject.java ea20b34 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 4285ab3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ColumnarStructObjectInspector.java 66f4f8d serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java 90561a1 serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java PRE-CREATION Diff: https://reviews.apache.org/r/806/diff Testing --- Tests added Thanks, Krishna Add support of columnar binary serde Key: HIVE-956 URL: https://issues.apache.org/jira/browse/HIVE-956 Project: Hive Issue Type: New Feature Reporter: He Yongqiang Assignee: Krishna Kumar Attachments: HIVE.956.patch.0, HIVE.956.patch.1, HIVE.956.patch.2 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-956) Add support of columnar binary serde
[ https://issues.apache.org/jira/browse/HIVE-956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Kumar updated HIVE-956: --- Attachment: HIVE-956v3.patch After separating mapcomparer changes to its own jira/patch - HIVE-2209 Add support of columnar binary serde Key: HIVE-956 URL: https://issues.apache.org/jira/browse/HIVE-956 Project: Hive Issue Type: New Feature Reporter: He Yongqiang Assignee: Krishna Kumar Attachments: HIVE-956v3.patch, HIVE.956.patch.0, HIVE.956.patch.1, HIVE.956.patch.2 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-956) Add support of columnar binary serde
[ https://issues.apache.org/jira/browse/HIVE-956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Kumar updated HIVE-956: --- Status: Patch Available (was: Open) I have tested the setting of rawDataSize with a ANALYZE TABLE COMPUTE STATISTICS/DESCRIBE FORMATTED set of commands with this patch, and it works. Can you please check it again? Please note that you need to apply the patch from HIVE-2209 before applying the patch from this one, since this jira is dependent on that one. Add support of columnar binary serde Key: HIVE-956 URL: https://issues.apache.org/jira/browse/HIVE-956 Project: Hive Issue Type: New Feature Reporter: He Yongqiang Assignee: Krishna Kumar Attachments: HIVE-956v3.patch, HIVE.956.patch.0, HIVE.956.patch.1, HIVE.956.patch.2 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re Stats Publishing /Aggregation
Any reason why persistent stores such as jdbc and hbase are supported for temporary stats storage IIUC, but hadoop counters were not used for the tasks to 'publish' their stats for the aggregation task to pick it up from? Cheers, Krishna
Re: Re Stats Publishing /Aggregation
Hi Krishna, It's a good question. We should update the design doc on the wiki to include the design decisions. But here you go: Hive implemented the updating numRows stats using Hadoop counters before, but the counter was not reliable: we saw wrong stats from time to time. There are several reasons why we didn't go to the direction of fixing hadoop counter but chose JDBC/HBase as the intermediate storage for stats publishing: 1) AFAIK, hadoop counters were not designed to be super reliable/scalable to large # of updates (correct me if I'm wrong). At our production environment, we saw the peak stats publishing QPS is around 2k, which means the JT will handle a counter update every 0.5 msec. In addition aggregating counters will also add more workload to the JT, which is already heavy-loaded. This is only for 1 stats. If you have 3 stats you need to collect, you can put these 3 stats into 3 columns and do 1 insert into RDBMS/HBase in order to publish the stats. With Hadoop counter, you'll need 3 counter updates, which is not as scalable down the road. 2) even if the hadoop counter is fixed and scaled to what we expected, the turn around time is high and Hive has to add a hadoop shim for old Hadoop releases. It's a pain and not as nice as supporting the feature out-of-the-box. On Jun 20, 2011, at 6:26 AM, Krishna Kumar wrote: Any reason why persistent stores such as jdbc and hbase are supported for temporary stats storage IIUC, but hadoop counters were not used for the tasks to 'publish' their stats for the aggregation task to pick it up from? Cheers, Krishna
Re: Review Request: HIVE-2035 Use block level merge on rcfile if intermediate merge is needed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/935/ --- (Updated 2011-06-20 19:20:53.263299) Review request for hive. Changes --- Throw error at compile time for bad rcfile merge input format class rather than at runtime, remove bad test, stylistic fixes Summary --- For a table stored as RCFile, intermediate results are sometimes merged if those files are below a certain threshold. For RCFiles, we can do a block level merge that does not deserialize the blocks and is more efficient. This patch leverages the existing code used to merge for ALTER TABLE ... CONCATENATE. This addresses bug HIVE-2035. https://issues.apache.org/jira/browse/HIVE-2035 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/MergeWork.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeRecordReader.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileKeyBufferWrapper.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1136090 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1136090 trunk/ql/src/test/queries/clientpositive/rcfile_createas1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/rcfile_merge1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/rcfile_merge2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/rcfile_merge3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/rcfile_merge4.q PRE-CREATION trunk/ql/src/test/results/clientpositive/rcfile_createas1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/rcfile_merge1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/rcfile_merge2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/rcfile_merge3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/rcfile_merge4.q.out PRE-CREATION Diff: https://reviews.apache.org/r/935/diff Testing --- Thanks, Franklin
[jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052196#comment-13052196 ] John Sichi commented on HIVE-2036: -- +1. Will commit when tests pass. Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052197#comment-13052197 ] John Sichi commented on HIVE-2036: -- I mean, once the latest patch gets uploaded. Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-trunk-h0.21 #785
See https://builds.apache.org/job/Hive-trunk-h0.21/785/ -- [...truncated 30981 lines...] [junit] OK [junit] PREHOOK: query: select count(1) as cnt from testhivedrivertable [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2011-06-20_13-09-27_913_6310446770895666424/-mr-1 [junit] Total MapReduce jobs = 1 [junit] Launching Job 1 out of 1 [junit] Number of reduce tasks determined at compile time: 1 [junit] In order to change the average load for a reducer (in bytes): [junit] set hive.exec.reducers.bytes.per.reducer=number [junit] In order to limit the maximum number of reducers: [junit] set hive.exec.reducers.max=number [junit] In order to set a constant number of reducers: [junit] set mapred.reduce.tasks=number [junit] Job running in-process (local Hadoop) [junit] Hadoop job information for null: number of mappers: 0; number of reducers: 0 [junit] 2011-06-20 13:09:30,963 null map = 100%, reduce = 100% [junit] Ended Job = job_local_0001 [junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2011-06-20_13-09-27_913_6310446770895666424/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/service/tmp/hive_job_log_hudson_201106201309_629630791.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] PREHOOK: type: LOAD [junit] PREHOOK: Output: default@testhivedrivertable [junit] Copying data from https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt [junit] Loading data to table default.testhivedrivertable [junit] POSTHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: select * from testhivedrivertable limit 10 [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2011-06-20_13-09-32_485_406571490705655526/-mr-1 [junit] POSTHOOK: query: select * from testhivedrivertable limit 10 [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2011-06-20_13-09-32_485_406571490705655526/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/service/tmp/hive_job_log_hudson_201106201309_1382039106.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input:
[jira] [Updated] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed S. Albiz updated HIVE-2036: Attachment: HIVE-2036.8.patch Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch, HIVE-2036.8.patch HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed S. Albiz updated HIVE-2036: Status: Patch Available (was: Open) Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch, HIVE-2036.8.patch HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2226: Add API to metastore for table filtering based on table properties
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/910/ --- (Updated 2011-06-20 21:04:45.477266) Review request for hive and Paul Yang. Changes --- - Removed the TableQueryFilterType enum. Instead, use constant names for filter keys to allow mixing filter types. Summary (updated) --- Create a function listTableNamesByFilter that returns a list of names for tables in a database that match a certain filter. The syntax of the filter is similar to the one created by HIVE-1609. You can filter the table list based on owner, last access time, or table parameter key/values. The filtering takes place at the JDO level for efficiency/speed. To create a new kind of table filter, add a constant to thrift.if and a branch in the if statement in generateJDOFilterOverTables() in ExpressionTree. Example filter statements include: //translation: owner.matches(.*test.*) and lastAccessTime == 0 filter = Constants.HIVE_FILTER_FIELD_OWNER + like \.*test.*\ and + Constants.HIVE_FILTER_FIELD_LAST_ACCESS + = 0; //translation: owner = test_user and (parameters.get(retention) == 30 || parameters.get(retention) == 90) filter = Constants.HIVE_FILTER_FIELD_OWNER + = \test_user\ and ( + Constants.HIVE_FILTER_FIELD_PARAMS + retention = \30\ or + Constants.HIVE_FILTER_FIELD_PARAMS + retention = \90\) The filter can currently parse string or integer values, where values interpreted as strings must be in quotes. See the comments in IMetaStoreClient for more usage details/restrictions. This addresses bug HIVE-2226. https://issues.apache.org/jira/browse/HIVE-2226 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g 1136751 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1136751 Diff: https://reviews.apache.org/r/910/diff Testing --- Added test cases to TestHiveMetaStore Thanks, Sohan
[jira] [Commented] (HIVE-2226) Add API to retrieve table names by an arbitrary filter, e.g., by owner, retention, parameters, etc.
[ https://issues.apache.org/jira/browse/HIVE-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052219#comment-13052219 ] jirapos...@reviews.apache.org commented on HIVE-2226: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/910/ --- (Updated 2011-06-20 21:04:45.477266) Review request for hive and Paul Yang. Changes --- - Removed the TableQueryFilterType enum. Instead, use constant names for filter keys to allow mixing filter types. Summary (updated) --- Create a function listTableNamesByFilter that returns a list of names for tables in a database that match a certain filter. The syntax of the filter is similar to the one created by HIVE-1609. You can filter the table list based on owner, last access time, or table parameter key/values. The filtering takes place at the JDO level for efficiency/speed. To create a new kind of table filter, add a constant to thrift.if and a branch in the if statement in generateJDOFilterOverTables() in ExpressionTree. Example filter statements include: //translation: owner.matches(.*test.*) and lastAccessTime == 0 filter = Constants.HIVE_FILTER_FIELD_OWNER + like \.*test.*\ and + Constants.HIVE_FILTER_FIELD_LAST_ACCESS + = 0; //translation: owner = test_user and (parameters.get(retention) == 30 || parameters.get(retention) == 90) filter = Constants.HIVE_FILTER_FIELD_OWNER + = \test_user\ and ( + Constants.HIVE_FILTER_FIELD_PARAMS + retention = \30\ or + Constants.HIVE_FILTER_FIELD_PARAMS + retention = \90\) The filter can currently parse string or integer values, where values interpreted as strings must be in quotes. See the comments in IMetaStoreClient for more usage details/restrictions. This addresses bug HIVE-2226. https://issues.apache.org/jira/browse/HIVE-2226 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java 1136751 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g 1136751 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1136751 Diff: https://reviews.apache.org/r/910/diff Testing --- Added test cases to TestHiveMetaStore Thanks, Sohan Add API to retrieve table names by an arbitrary filter, e.g., by owner, retention, parameters, etc. --- Key: HIVE-2226 URL: https://issues.apache.org/jira/browse/HIVE-2226 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2226.1.patch Create a function called get_table_names_by_filter that returns a list of table names in a database that match a certain filter. The filter should operate similar to the one HIVE-1609. Initially, you should be able to prune the table list based on owner, retention, or table parameter key/values. The filtering should take place at the JDO level for efficiency/speed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2228) Can't use DB qualified column names in WHERE or GROUP BY clauses
Can't use DB qualified column names in WHERE or GROUP BY clauses
Key: HIVE-2228
URL: https://issues.apache.org/jira/browse/HIVE-2228
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.0
Reporter: Carl Steinbach
Hive doesn't allow you to use DB qualified column names in the WHERE or GROUP
BY clauses. The workaround is to define a table alias:
{noformat}
hive CREATE DATABASE db1;
OK
hive CREATE TABLE db1.t(a INT, b INT);
OK
hive SELECT * FROM db1.t WHERE db1.t.a 100;
FAILED: Error in semantic analysis: Line 1:26 Invalid table alias or column
reference 'db1'
hive SELECT * FROM db1.t t WHERE t.a 100;
OK
hive SELECT * FROM db1.t GROUP BY db1.t.a;
FAILED: Error in semantic analysis: Line 1:29 Invalid table alias or column
reference 'db1'
hive SELECT * FROM db1.t t GROUP BY t.a;
OK
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2228) Can't use DB qualified column names in WHERE or GROUP BY clauses
[
https://issues.apache.org/jira/browse/HIVE-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-2228:
-
Fix Version/s: 0.8.0
Can't use DB qualified column names in WHERE or GROUP BY clauses
Key: HIVE-2228
URL: https://issues.apache.org/jira/browse/HIVE-2228
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.0
Reporter: Carl Steinbach
Fix For: 0.8.0
Hive doesn't allow you to use DB qualified column names in the WHERE or GROUP
BY clauses. The workaround is to define a table alias:
{noformat}
hive CREATE DATABASE db1;
OK
hive CREATE TABLE db1.t(a INT, b INT);
OK
hive SELECT * FROM db1.t WHERE db1.t.a 100;
FAILED: Error in semantic analysis: Line 1:26 Invalid table alias or column
reference 'db1'
hive SELECT * FROM db1.t t WHERE t.a 100;
OK
hive SELECT * FROM db1.t GROUP BY db1.t.a;
FAILED: Error in semantic analysis: Line 1:29 Invalid table alias or column
reference 'db1'
hive SELECT * FROM db1.t t GROUP BY t.a;
OK
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2213) Optimize partial specification metastore functions
[ https://issues.apache.org/jira/browse/HIVE-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Yang updated HIVE-2213: Summary: Optimize partial specification metastore functions (was: Optimize get_partition_names_ps()) Optimize partial specification metastore functions -- Key: HIVE-2213 URL: https://issues.apache.org/jira/browse/HIVE-2213 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2213.1.patch, HIVE-2213.3.patch If a table has a large number of partitions, get_partition_names_ps() make take a long time to execute, because we get all of the partition names from the database. This is not very memory efficient, and the operation can be pushed down to the JDO layer without getting all of the names first. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2213) Optimize partial specification metastore functions
[ https://issues.apache.org/jira/browse/HIVE-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Yang updated HIVE-2213: Resolution: Fixed Fix Version/s: 0.8.0 Status: Resolved (was: Patch Available) Committed. Thanks Sohan! Optimize partial specification metastore functions -- Key: HIVE-2213 URL: https://issues.apache.org/jira/browse/HIVE-2213 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Fix For: 0.8.0 Attachments: HIVE-2213.1.patch, HIVE-2213.3.patch If a table has a large number of partitions, get_partition_names_ps() make take a long time to execute, because we get all of the partition names from the database. This is not very memory efficient, and the operation can be pushed down to the JDO layer without getting all of the names first. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-2036: - Resolution: Fixed Fix Version/s: 0.8.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Syed! Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Fix For: 0.8.0 Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch, HIVE-2036.8.patch HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-872) Allow type widening on COALESCE/UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed S. Albiz updated HIVE-872: --- Attachment: HIVE-872.7.patch Allow type widening on COALESCE/UNION ALL - Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch, HIVE-872.6.patch, HIVE-872.7.patch Original request: We should allow 0L to be interpreted as a bigint constant. Instead of this, we have decided that the usecases for this do not merit modifications to the ql. Instead we enable type widening on the UDF COALESCE and on the UNION ALL operator -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-872: Allow BIGINT constants
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-21 02:50:57.581129) Review request for hive and John Sichi. Changes --- Update test cases udf_coalesce.q and union2.q to check that type-widening is not applied incorrectly, use getCommonClassForComparison instead of getCommonClass to compute type widening on union operator (fixing explode_null.q) Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2462517 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ec816e9 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java f46c16c ql/src/test/queries/clientnegative/udf_coalesce.q 6d8da79 ql/src/test/queries/clientnegative/union2.q 403d19d ql/src/test/queries/clientpositive/type_widening.q PRE-CREATION ql/src/test/results/clientnegative/udf_coalesce.q.out a4c3cab ql/src/test/results/clientnegative/union2.q.out 16cfe03 ql/src/test/results/clientpositive/type_widening.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed
[jira] [Commented] (HIVE-872) Allow type widening on COALESCE/UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052334#comment-13052334 ] jirapos...@reviews.apache.org commented on HIVE-872: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-21 02:50:57.581129) Review request for hive and John Sichi. Changes --- Update test cases udf_coalesce.q and union2.q to check that type-widening is not applied incorrectly, use getCommonClassForComparison instead of getCommonClass to compute type widening on union operator (fixing explode_null.q) Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2462517 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ec816e9 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java f46c16c ql/src/test/queries/clientnegative/udf_coalesce.q 6d8da79 ql/src/test/queries/clientnegative/union2.q 403d19d ql/src/test/queries/clientpositive/type_widening.q PRE-CREATION ql/src/test/results/clientnegative/udf_coalesce.q.out a4c3cab ql/src/test/results/clientnegative/union2.q.out 16cfe03 ql/src/test/results/clientpositive/type_widening.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed Allow type widening on COALESCE/UNION ALL - Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch, HIVE-872.6.patch, HIVE-872.7.patch Original request: We should allow 0L to be interpreted as a bigint constant. Instead of this, we have decided that the usecases for this do not merit modifications to the ql. Instead we enable type widening on the UDF COALESCE and on the UNION ALL operator -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
