Hive index exceptions
I created the hive index based on hive-0.7, the stataments are as follows: create index ind_userid_txt on table users(userid)as 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED REBUILD STORED AS RCFile; ALTER INDEX ind_userid_txt ON users REBUILD; set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; insert overwrite directory /tmp/index_result select '_bucketname','_offset' from default__users_ind_userid_txt__ where userid=1; set hive.exec.index_file=/tmp/index_result; set hive.input.format=org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat; select * from users where userid=1; the exception as follows, pls explain why throw the following exceptions,thanks. 11/06/14 10:06:55 INFO exec.MapRedTask: Number of reduce tasks is set to 0 since there's no reduce operator 11/06/14 10:06:55 INFO exec.MapRedTask: Using org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat java.lang.RuntimeException: org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:548) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 11/06/14 10:06:55 ERROR exec.MapRedTask: Exception: org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask 11/06/14 10:06:55 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask hive isap-00:/home/liren/hive-0.7.0/bin #
[jira] [Commented] (HIVE-2209) Provide a way by which ObjectInspectorUtils.compare can be extended by the caller for comparing maps which are part of the object
[ https://issues.apache.org/jira/browse/HIVE-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049297#comment-13049297 ] Krishna Kumar commented on HIVE-2209: - He Yongqiang, Can you take a look at this one? Once this is in, I can upload the remainder of the patch to HIVE-956. Thanks. Provide a way by which ObjectInspectorUtils.compare can be extended by the caller for comparing maps which are part of the object - Key: HIVE-2209 URL: https://issues.apache.org/jira/browse/HIVE-2209 Project: Hive Issue Type: Improvement Reporter: Krishna Kumar Assignee: Krishna Kumar Priority: Minor Attachments: HIVE-2209v0.patch Now ObjectInspectorUtils.compare throws an exception if a map is contained (recursively) within the objects being compared. Two obvious implementations are - a simple map comparer which assumes keys of the first map can be used to fetch values from the second - a 'cross-product' comparer which compares every pair of key-value pairs in the two maps, and calls a match if and only if all pairs are matched Note that it would be difficult to provide a transitive greater-than/less-than indication with maps so that is not in scope. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2218) speedup addInputPaths
[ https://issues.apache.org/jira/browse/HIVE-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-2218: --- Attachment: HIVE-2218.1.patch speedup addInputPaths - Key: HIVE-2218 URL: https://issues.apache.org/jira/browse/HIVE-2218 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2218.1.patch Speedup the addInputPaths for combined symlink inputformat, and added some other micro optimizations which also work for normal cases. This can help reducing the start time of one query from 5 hours to less than 20 mins. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-branch-0.7.1-h0.21 #24
See https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/24/ -- [...truncated 7530 lines...] init: dynamic-serde: ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: ivy-resolve: :: loading settings :: file = https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/ivy/ivysettings.xml ivy-retrieve: compile: [echo] Compiling: hive [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/serde/build.xml:52: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds compile-test: [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build-common.xml:334: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 20 source files to https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build/serde/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/serde/src/test/org/apache/hadoop/hive/serde2/dynamic_type/TestDynamicSerDe.java uses unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build-common.xml:347: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds create-dirs: compile-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds deploy-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: init: core-compile: [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/service/build.xml:59: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds compile: compile-test: [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build-common.xml:334: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 1 source file to https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build/service/test/classes [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build-common.xml:347: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds test: test-shims: test-conditions: gen-test: create-dirs: compile-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds deploy-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: init: compile: ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: ivy-retrieve-hadoop-source: :: loading settings :: file = https://builds.apache.org/job/Hive-branch-0.7.1-h0.21/ws/hive/ivy/ivysettings.xml [ivy:retrieve] :: resolving dependencies :: org.apache.hive#hive-shims;0.7.1-SNAPSHOT [ivy:retrieve] confs: [default] [ivy:retrieve] found hadoop#core;0.20.3-CDH3-SNAPSHOT in hadoop-source [ivy:retrieve] :: resolution report :: resolve 719ms :: artifacts dl 0ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - |
[jira] [Updated] (HIVE-2218) speedup addInputPaths
[ https://issues.apache.org/jira/browse/HIVE-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-2218: --- Status: Patch Available (was: Open) speedup addInputPaths - Key: HIVE-2218 URL: https://issues.apache.org/jira/browse/HIVE-2218 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2218.1.patch Speedup the addInputPaths for combined symlink inputformat, and added some other micro optimizations which also work for normal cases. This can help reducing the start time of one query from 5 hours to less than 20 mins. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hive index exceptions
It's because you have ...index.io.HiveIndexInputFormat instead of ...index.HiveIndexInputFormat. JVS On Jun 14, 2011, at 5:41 AM, suyu xie wrote: I created the hive index based on hive-0.7, the stataments are as follows: create index ind_userid_txt on table users(userid)as 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED REBUILD STORED AS RCFile; ALTER INDEX ind_userid_txt ON users REBUILD; set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; insert overwrite directory /tmp/index_result select '_bucketname','_offset' from default__users_ind_userid_txt__ where userid=1; set hive.exec.index_file=/tmp/index_result; set hive.input.format=org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat; select * from users where userid=1; the exception as follows, pls explain why throw the following exceptions,thanks. 11/06/14 10:06:55 INFO exec.MapRedTask: Number of reduce tasks is set to 0 since there's no reduce operator 11/06/14 10:06:55 INFO exec.MapRedTask: Using org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat java.lang.RuntimeException: org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:548) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 11/06/14 10:06:55 ERROR exec.MapRedTask: Exception: org.apache.hadoop.hive.ql.index.io.HiveIndexInputFormat FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask 11/06/14 10:06:55 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask hive isap-00:/home/liren/hive-0.7.0/bin #
Re: Review Request: HIVE-2036: Update bitmap indexes for automatic usage
On 2011-06-13 22:57:46, John Sichi wrote: ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java, line 114 https://reviews.apache.org/r/857/diff/4/?file=20984#file20984line114 I don't think this should be necessary. We just want to propagate the partition column predicate (whatever it is) from the base table query to the index table query; partition pruning on the index table query will do the rest of the work. In other words, if the original query had part_key=whatever we want to preserve that on the index table query. That's what the code is already supposed to be doing before your change; was it not working? Syed Albiz wrote: This code is to prevent automatic usage from kicking in if the index has not been built on the partition specified in the partition predicate. (i.e. if the index has only been built on partition ds=foo, and the query is select key from src where ds=bar; We do not want to execute an index query in this case. It seems like adding a test for bitmaps specifically to mirror index_auto_unused.q(which is where this functionality is tested for Compact indices) would be a good idea. The logic for making sure that the necessary index partitions exist is already present in IndexWhereProcessor.checkPartitionsCoveredByIndex. If that's not working, we should fix it; it should not be necessary to change the predicate analyzer at all. - John --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/#review825 --- On 2011-06-14 04:05:43, Syed Albiz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/ --- (Updated 2011-06-14 04:05:43) Review request for hive and John Sichi. Summary --- Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. This addresses bug HIVE-2036. https://issues.apache.org/jira/browse/HIVE-2036 Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3caa4cc ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PRE-CREATION Diff: https://reviews.apache.org/r/857/diff Testing --- Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. Thanks, Syed
[jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049324#comment-13049324 ] jirapos...@reviews.apache.org commented on HIVE-2036: - bq. On 2011-06-13 22:57:46, John Sichi wrote: bq. ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java, line 114 bq. https://reviews.apache.org/r/857/diff/4/?file=20984#file20984line114 bq. bq. I don't think this should be necessary. We just want to propagate the partition column predicate (whatever it is) from the base table query to the index table query; partition pruning on the index table query will do the rest of the work. bq. bq. In other words, if the original query had bq. bq. part_key=whatever bq. bq. we want to preserve that on the index table query. That's what the code is already supposed to be doing before your change; was it not working? bq. bq. bq. Syed Albiz wrote: bq. This code is to prevent automatic usage from kicking in if the index has not been built on the partition specified in the partition predicate. (i.e. if the index has only been built on partition ds=foo, and the query is select key from src where ds=bar; We do not want to execute an index query in this case. It seems like adding a test for bitmaps specifically to mirror index_auto_unused.q(which is where this functionality is tested for Compact indices) would be a good idea. The logic for making sure that the necessary index partitions exist is already present in IndexWhereProcessor.checkPartitionsCoveredByIndex. If that's not working, we should fix it; it should not be necessary to change the predicate analyzer at all. - John --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/#review825 --- On 2011-06-14 04:05:43, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/857/ bq. --- bq. bq. (Updated 2011-06-14 04:05:43) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. bq. bq. bq. This addresses bug HIVE-2036. bq. https://issues.apache.org/jira/browse/HIVE-2036 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 bq.ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3caa4cc bq. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 bq.ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 bq.ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a bq.ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 bq.ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a bq.ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/857/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. bq. bq. bq. Thanks, bq.
[jira] [Created] (HIVE-2219) Make alter table drop partition more efficient
Make alter table drop partition more efficient Key: HIVE-2219 URL: https://issues.apache.org/jira/browse/HIVE-2219 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Sohan Jain The current function dropTable() that handles dropping multiple partitions is somewhat inefficient. For each partition you want to drop, it loops through each partition in the table to see if the partition exists. This is an _O(mn)_ operation, where _m_ is the number of partitions to drop, and _n_ is the number of partitions in the table. The running time of this function can be improved, which is useful for tables with many partitions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2219) Make alter table drop partition more efficient
[ https://issues.apache.org/jira/browse/HIVE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sohan Jain reassigned HIVE-2219: Assignee: Sohan Jain Make alter table drop partition more efficient Key: HIVE-2219 URL: https://issues.apache.org/jira/browse/HIVE-2219 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Sohan Jain Assignee: Sohan Jain The current function dropTable() that handles dropping multiple partitions is somewhat inefficient. For each partition you want to drop, it loops through each partition in the table to see if the partition exists. This is an _O(mn)_ operation, where _m_ is the number of partitions to drop, and _n_ is the number of partitions in the table. The running time of this function can be improved, which is useful for tables with many partitions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1555) JDBC Storage Handler
[ https://issues.apache.org/jira/browse/HIVE-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049329#comment-13049329 ] John Sichi commented on HIVE-1555: -- 1+2) The Hadoop jar naming convention changed in 0.21; I hit this too recently when trying out a build against 0.21. I futzed around with the Hive build and got it working quick-and-dirty, but didn't save the patch. Looks like someone has submitted one on HIVE-1612 (I haven't taken a look at it yet). If you want to help push that through, it would be a good contribution by itself. 3) Is it possible to make it work against Derby? JDBC Storage Handler Key: HIVE-1555 URL: https://issues.apache.org/jira/browse/HIVE-1555 Project: Hive Issue Type: New Feature Components: JDBC Reporter: Bob Robertson Assignee: Andrew Wilson Attachments: JDBCStorageHandler Design Doc.pdf Original Estimate: 24h Remaining Estimate: 24h With the Cassandra and HBase Storage Handlers I thought it would make sense to include a generic JDBC RDBMS Storage Handler so that you could import a standard DB table into Hive. Many people must want to perform HiveQL joins, etc against tables in other systems etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-872: Allow BIGINT constants
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-14 19:04:53.630834) Review request for hive and John Sichi. Changes --- After talking to jhsu about this, looks like the motivation for this is to provide a way to coerce int literals into bigints for UDFs that expect arguments of the same type. E.g. COALESCE(0, 1152921504606846976) will currently fail. With this updated patch, COALESCE(0L, 1152921504606846976) passes. Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed
[jira] [Commented] (HIVE-872) Allow BIGINT constants
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049343#comment-13049343 ] jirapos...@reviews.apache.org commented on HIVE-872: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-14 19:04:53.630834) Review request for hive and John Sichi. Changes --- After talking to jhsu about this, looks like the motivation for this is to provide a way to coerce int literals into bigints for UDFs that expect arguments of the same type. E.g. COALESCE(0, 1152921504606846976) will currently fail. With this updated patch, COALESCE(0L, 1152921504606846976) passes. Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed Allow BIGINT constants -- Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch We should allow 0L to be interpreted as a bigint constant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-872: Allow BIGINT constants
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/#review834 --- Wouldn't it be better to change COALESCE to just check that the two objects have the same type family (both numeric)? That's what happens in standard SQL, and then the widest type is used for the result. This is what UNION ALL should be doing also, although currently it uses the same annoying check as COALESCE. - John On 2011-06-14 19:04:53, Syed Albiz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-14 19:04:53) Review request for hive and John Sichi. Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed
[jira] [Commented] (HIVE-872) Allow BIGINT constants
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049348#comment-13049348 ] jirapos...@reviews.apache.org commented on HIVE-872: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/#review834 --- Wouldn't it be better to change COALESCE to just check that the two objects have the same type family (both numeric)? That's what happens in standard SQL, and then the widest type is used for the result. This is what UNION ALL should be doing also, although currently it uses the same annoying check as COALESCE. - John On 2011-06-14 19:04:53, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/889/ bq. --- bq. bq. (Updated 2011-06-14 19:04:53) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. bq. bq. bq. This addresses bug HIVE-872. bq. https://issues.apache.org/jira/browse/HIVE-872 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f bq.ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION bq.ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/889/diff bq. bq. bq. Testing bq. --- bq. bq. TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. bq. bq. bq. Thanks, bq. bq. Syed bq. bq. Allow BIGINT constants -- Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch We should allow 0L to be interpreted as a bigint constant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-872: Allow BIGINT constants
On 2011-06-14 19:14:04, John Sichi wrote: Wouldn't it be better to change COALESCE to just check that the two objects have the same type family (both numeric)? That's what happens in standard SQL, and then the widest type is used for the result. This is what UNION ALL should be doing also, although currently it uses the same annoying check as COALESCE. Also, CAST(0 AS BIGINT) already works the same as 0L. - John --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/#review834 --- On 2011-06-14 19:04:53, Syed Albiz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-14 19:04:53) Review request for hive and John Sichi. Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed
[jira] [Commented] (HIVE-872) Allow BIGINT constants
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049354#comment-13049354 ] jirapos...@reviews.apache.org commented on HIVE-872: bq. On 2011-06-14 19:14:04, John Sichi wrote: bq. Wouldn't it be better to change COALESCE to just check that the two objects have the same type family (both numeric)? That's what happens in standard SQL, and then the widest type is used for the result. This is what UNION ALL should be doing also, although currently it uses the same annoying check as COALESCE. bq. Also, CAST(0 AS BIGINT) already works the same as 0L. - John --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/#review834 --- On 2011-06-14 19:04:53, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/889/ bq. --- bq. bq. (Updated 2011-06-14 19:04:53) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. bq. bq. bq. This addresses bug HIVE-872. bq. https://issues.apache.org/jira/browse/HIVE-872 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f bq.ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION bq.ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/889/diff bq. bq. bq. Testing bq. --- bq. bq. TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. bq. bq. bq. Thanks, bq. bq. Syed bq. bq. Allow BIGINT constants -- Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch We should allow 0L to be interpreted as a bigint constant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-872) Allow BIGINT constants
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049362#comment-13049362 ] jirapos...@reviews.apache.org commented on HIVE-872: bq. On 2011-06-14 19:14:04, John Sichi wrote: bq. Wouldn't it be better to change COALESCE to just check that the two objects have the same type family (both numeric)? That's what happens in standard SQL, and then the widest type is used for the result. This is what UNION ALL should be doing also, although currently it uses the same annoying check as COALESCE. bq. bq. bq. John Sichi wrote: bq. Also, CAST(0 AS BIGINT) already works the same as 0L. bq. Yep, I think the request was to provide a shorthand for the explicit CAST() call. After talking more with jhsu it doesn't look like there are any major uses for that shorthand if UDFs apply type widening automatically. - Syed --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/#review834 --- On 2011-06-14 19:04:53, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/889/ bq. --- bq. bq. (Updated 2011-06-14 19:04:53) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. bq. bq. bq. This addresses bug HIVE-872. bq. https://issues.apache.org/jira/browse/HIVE-872 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 9161319 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java dfadb9f bq.ql/src/test/queries/clientpositive/bigint_const.q PRE-CREATION bq.ql/src/test/results/clientpositive/bigint_const.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/889/diff bq. bq. bq. Testing bq. --- bq. bq. TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. bq. bq. bq. Thanks, bq. bq. Syed bq. bq. Allow BIGINT constants -- Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch We should allow 0L to be interpreted as a bigint constant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2215
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ --- (Updated 2011-06-14 20:51:53.179968) Review request for hive and John Sichi. Changes --- Updated patch with Carl's comments. Carl, can you take a look? Summary --- Follow-up for HIVE-2147. This addresses bug HIVE-2215. https://issues.apache.org/jira/browse/HIVE-2215 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779 Diff: https://reviews.apache.org/r/883/diff Testing --- Added test cases for new api. Thanks, Ashutosh
[jira] [Commented] (HIVE-2215) Add api for marking / querying set of partitions for events
[ https://issues.apache.org/jira/browse/HIVE-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049405#comment-13049405 ] jirapos...@reviews.apache.org commented on HIVE-2215: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/883/ --- (Updated 2011-06-14 20:51:53.179968) Review request for hive and John Sichi. Changes --- Updated patch with Carl's comments. Carl, can you take a look? Summary --- Follow-up for HIVE-2147. This addresses bug HIVE-2215. https://issues.apache.org/jira/browse/HIVE-2215 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135779 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/LoadPartitionDoneEvent.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java PRE-CREATION trunk/metastore/src/model/package.jdo 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1135779 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1135779 Diff: https://reviews.apache.org/r/883/diff Testing --- Added test cases for new api. Thanks, Ashutosh Add api for marking / querying set of partitions for events --- Key: HIVE-2215 URL: https://issues.apache.org/jira/browse/HIVE-2215 Project: Hive Issue Type: New Feature Components: Metastore Affects Versions: 0.8.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive_2215.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049406#comment-13049406 ] jirapos...@reviews.apache.org commented on HIVE-2036: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/#review836 --- ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java https://reviews.apache.org/r/857/#comment1806 Slight rephrasing suggested: If multiple indexes are provided, it is up to handler to decide whether to use none, one, some, or all of them. The supplied predicate may reference any of the columns from any of the indexes. If the handler decides to use more than one index, then it is responsible for generating tasks to combine their search results (e.g. via a JOIN). ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java https://reviews.apache.org/r/857/#comment1805 This should be gone. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java https://reviews.apache.org/r/857/#comment1807 Delete commented-out code, or convert it into a TODO with a corresponding JIRA issue link. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/857/#comment1808 Could you explain more about what's going on here? ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/857/#comment1817 Only do indexes.get(0) once. - John On 2011-06-14 04:05:43, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/857/ bq. --- bq. bq. (Updated 2011-06-14 04:05:43) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. bq. bq. bq. This addresses bug HIVE-2036. bq. https://issues.apache.org/jira/browse/HIVE-2036 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 bq.ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3caa4cc bq. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 bq.ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 bq.ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a bq.ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 bq.ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a bq.ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/857/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. bq. bq. bq. Thanks, bq. bq. Syed bq. bq. Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement
Re: Review Request: HIVE-2215
On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/if/hive_metastore.thrift, line 46 https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line46 I think this should be changed to PartitionEventType in order to make it clear that this applies to partitions only. If in the future we need to introduce event types for tables, indexes, etc, then we should add new enums for those event types as well. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/if/hive_metastore.thrift, line 338 https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line338 This should also throw UnknownDBException and UnknownTableException. The same goes for isPartitionMarkedForEvent. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 528 https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line528 Collections aren't required to satisfy an ordering property, so we have to assume the output of this logging statement is ambiguous, e.g. [a, b] versus [b, a]. We should disambiguate this by passing in the part_vals map and logging the key/value pairs instead of just the values. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3182 https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3182 Missing exceptions: UnknownDbException and UnknownTableException. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3186 https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3186 Checking to see if the DB and Table exist should be done in the same database transaction as the rest of the operation. If you do it here there's no guarantee that the db/table will still exist when ms.markPartitionForEvent() is called. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3188 https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3188 Should we add an InvalidPartitionException and UnknownPartitionException? Seems like those are both valid exceptions in this situation. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3224 https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3224 Same issue here as before. These checks need to get pushed into ms.isPartitionMarkedForEvent(). Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java, line 82 https://reviews.apache.org/r/883/diff/1/?file=20973#file20973line82 I think the name of this method is misleading. You're marking a single partition done, not a set of partitions, right? Also, in this context being done means that the load operation on that partition has completed, so it would be good to include load in the name of the method and event class, e.g. LoadPartitionDoneEvent and onLoadPartitionDone. Done. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java, line 34 https://reviews.apache.org/r/883/diff/1/?file=20977#file20977line34 Is it possible to use org.apache.hadoop.hive.metastore.api.EventType instead of int? Another approach is to create an MPartitionEvent baseclass, and then subclass that with MPartitionLoadDoneEvent, etc, and use eventType as the internal type discriminator for JDO. No. I don't see any benefit of it. Nor I cant see how will this work. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/model/package.jdo, line 668 https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line668 You need to supply schema upgrade scripts for Derby and MySQL. Please either do that in this ticket or open a followup ticket and assign it to yourself. Will do. On 2011-06-14 01:02:20, Carl Steinbach wrote: trunk/metastore/src/model/package.jdo, line 683 https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line683 It looks like it's possible for this table to hold more than one MarkPartitionDone event for the same partition, but is that a legal state? If it is, how do you know when the load operation for a partition is still in progress? This is not for when load operation is in progress. As suggested from name its LoadPartitionDone. So, marking partition load done is idempotent. Client can mark it multiple times. So, metastore will return true if it finds one or more such partitioned marked in the table. On 2011-06-14 01:02:20,
[jira] [Commented] (HIVE-2215) Add api for marking / querying set of partitions for events
[ https://issues.apache.org/jira/browse/HIVE-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049410#comment-13049410 ] jirapos...@reviews.apache.org commented on HIVE-2215: - bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/if/hive_metastore.thrift, line 46 bq. https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line46 bq. bq. I think this should be changed to PartitionEventType in order to make it clear that this applies to partitions only. If in the future we need to introduce event types for tables, indexes, etc, then we should add new enums for those event types as well. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/if/hive_metastore.thrift, line 338 bq. https://reviews.apache.org/r/883/diff/1/?file=20969#file20969line338 bq. bq. This should also throw UnknownDBException and UnknownTableException. The same goes for isPartitionMarkedForEvent. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 528 bq. https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line528 bq. bq. Collections aren't required to satisfy an ordering property, so we have to assume the output of this logging statement is ambiguous, e.g. [a, b] versus [b, a]. We should disambiguate this by passing in the part_vals map and logging the key/value pairs instead of just the values. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3182 bq. https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3182 bq. bq. Missing exceptions: UnknownDbException and UnknownTableException. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3186 bq. https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3186 bq. bq. Checking to see if the DB and Table exist should be done in the same database transaction as the rest of the operation. If you do it here there's no guarantee that the db/table will still exist when ms.markPartitionForEvent() is called. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3188 bq. https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3188 bq. bq. Should we add an InvalidPartitionException and UnknownPartitionException? Seems like those are both valid exceptions in this situation. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3224 bq. https://reviews.apache.org/r/883/diff/1/?file=20970#file20970line3224 bq. bq. Same issue here as before. These checks need to get pushed into ms.isPartitionMarkedForEvent(). Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java, line 82 bq. https://reviews.apache.org/r/883/diff/1/?file=20973#file20973line82 bq. bq. I think the name of this method is misleading. You're marking a single partition done, not a set of partitions, right? bq. bq. Also, in this context being done means that the load operation on that partition has completed, so it would be good to include load in the name of the method and event class, e.g. LoadPartitionDoneEvent and onLoadPartitionDone. bq. Done. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionEvent.java, line 34 bq. https://reviews.apache.org/r/883/diff/1/?file=20977#file20977line34 bq. bq. Is it possible to use org.apache.hadoop.hive.metastore.api.EventType instead of int? bq. bq. Another approach is to create an MPartitionEvent baseclass, and then subclass that with MPartitionLoadDoneEvent, etc, and use eventType as the internal type discriminator for JDO. No. I don't see any benefit of it. Nor I cant see how will this work. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/model/package.jdo, line 668 bq. https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line668 bq. bq. You need to supply schema upgrade scripts for Derby and MySQL. Please either do that in this ticket or open a followup ticket and assign it to yourself. Will do. bq. On 2011-06-14 01:02:20, Carl Steinbach wrote: bq. trunk/metastore/src/model/package.jdo, line 683 bq. https://reviews.apache.org/r/883/diff/1/?file=20978#file20978line683 bq. bq. It looks like it's possible for this table to hold more than one
[jira] [Updated] (HIVE-2219) Make alter table drop partition more efficient
[ https://issues.apache.org/jira/browse/HIVE-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sohan Jain updated HIVE-2219: - Attachment: HIVE-2219.1.patch Improves the time it takes to check whether a partition to delete exists in the lists of partitions. Overall improves the complexity to _O(m + n)_ Make alter table drop partition more efficient Key: HIVE-2219 URL: https://issues.apache.org/jira/browse/HIVE-2219 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2219.1.patch The current function dropTable() that handles dropping multiple partitions is somewhat inefficient. For each partition you want to drop, it loops through each partition in the table to see if the partition exists. This is an _O(mn)_ operation, where _m_ is the number of partitions to drop, and _n_ is the number of partitions in the table. The running time of this function can be improved, which is useful for tables with many partitions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: speedup addInputPaths
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/898/ --- Review request for hive. Summary --- speedup addInputPaths This addresses bug HIVE-2218. https://issues.apache.org/jira/browse/HIVE-2218 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1135335 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1135335 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 1135335 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1135335 Diff: https://reviews.apache.org/r/898/diff Testing --- yes. Thanks, Yongqiang
[jira] [Commented] (HIVE-2218) speedup addInputPaths
[ https://issues.apache.org/jira/browse/HIVE-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049426#comment-13049426 ] jirapos...@reviews.apache.org commented on HIVE-2218: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/898/ --- Review request for hive. Summary --- speedup addInputPaths This addresses bug HIVE-2218. https://issues.apache.org/jira/browse/HIVE-2218 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1135335 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1135335 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 1135335 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1135335 Diff: https://reviews.apache.org/r/898/diff Testing --- yes. Thanks, Yongqiang speedup addInputPaths - Key: HIVE-2218 URL: https://issues.apache.org/jira/browse/HIVE-2218 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2218.1.patch Speedup the addInputPaths for combined symlink inputformat, and added some other micro optimizations which also work for normal cases. This can help reducing the start time of one query from 5 hours to less than 20 mins. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2218) speedup addInputPaths
[ https://issues.apache.org/jira/browse/HIVE-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049423#comment-13049423 ] He Yongqiang commented on HIVE-2218: https://reviews.apache.org/r/898/ speedup addInputPaths - Key: HIVE-2218 URL: https://issues.apache.org/jira/browse/HIVE-2218 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2218.1.patch Speedup the addInputPaths for combined symlink inputformat, and added some other micro optimizations which also work for normal cases. This can help reducing the start time of one query from 5 hours to less than 20 mins. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2036: Update bitmap indexes for automatic usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/ --- (Updated 2011-06-14 21:26:21.276789) Review request for hive and John Sichi. Changes --- Addressed comments, added some more commenting for why we use indexes.get(0) in IndexWhereProcessor as that seemed a bit unclear Summary --- Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. This addresses bug HIVE-2036. https://issues.apache.org/jira/browse/HIVE-2036 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3caa4cc ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PRE-CREATION Diff: https://reviews.apache.org/r/857/diff Testing --- Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. Thanks, Syed
[jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049436#comment-13049436 ] jirapos...@reviews.apache.org commented on HIVE-2036: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/ --- (Updated 2011-06-14 21:26:21.276789) Review request for hive and John Sichi. Changes --- Addressed comments, added some more commenting for why we use indexes.get(0) in IndexWhereProcessor as that seemed a bit unclear Summary --- Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. This addresses bug HIVE-2036. https://issues.apache.org/jira/browse/HIVE-2036 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3caa4cc ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PRE-CREATION Diff: https://reviews.apache.org/r/857/diff Testing --- Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. Thanks, Syed Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-trunk-h0.21 #777
See https://builds.apache.org/job/Hive-trunk-h0.21/777/ -- [...truncated 32417 lines...] [echo] Writing POM to https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/jdbc/pom.xml No ivy:settings found for the default reference 'ivy.instance'. A default instance will be used no settings file found, using default... :: loading settings :: url = jar:file:/home/hudson/.ant/lib/ivy-2.0.0-rc2.jar!/org/apache/ivy/core/settings/ivysettings.xml ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: check-ivy: create-dirs: compile-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds deploy-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: init: install-hadoopcore: install-hadoopcore-default: ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: ivy-retrieve-hadoop-source: :: loading settings :: file = https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/ivy/ivysettings.xml [ivy:retrieve] :: resolving dependencies :: org.apache.hive#hive-hwi;0.8.0-SNAPSHOT [ivy:retrieve] confs: [default] [ivy:retrieve] found hadoop#core;0.20.1 in hadoop-source [ivy:retrieve] :: resolution report :: resolve 666ms :: artifacts dl 0ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - | default | 1 | 0 | 0 | 0 || 1 | 0 | - [ivy:retrieve] :: retrieving :: org.apache.hive#hive-hwi [ivy:retrieve] confs: [default] [ivy:retrieve] 0 artifacts copied, 1 already retrieved (0kB/0ms) install-hadoopcore-internal: setup: war: compile: [echo] Compiling: hwi [javac] https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/hwi/build.xml:71: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: [echo] Jar: hwi make-pom: [echo] Writing POM to https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/hwi/pom.xml No ivy:settings found for the default reference 'ivy.instance'. A default instance will be used no settings file found, using default... :: loading settings :: url = jar:file:/home/hudson/.ant/lib/ivy-2.0.0-rc2.jar!/org/apache/ivy/core/settings/ivysettings.xml ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: check-ivy: create-dirs: compile-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds deploy-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: init: setup: compile: [echo] Compiling: hbase-handler [javac] https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build-common.xml:299: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [copy] Warning: https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/hbase-handler/src/java/conf does not exist. jar: [echo] Jar: hbase-handler make-pom: [echo] Writing POM to https://builds.apache.org/job/Hive-trunk-h0.21/ws/hive/build/hbase-handler/pom.xml No ivy:settings found for the default
[jira] [Created] (HIVE-2220) Better error messages for MetaStore configuration problems
Better error messages for MetaStore configuration problems
--
Key: HIVE-2220
URL: https://issues.apache.org/jira/browse/HIVE-2220
Project: Hive
Issue Type: Improvement
Components: Diagnosability, Metastore
Reporter: Carl Steinbach
Assignee: Carl Steinbach
We need to improve the error messages seen by users when the metastore is not
configured correctly.
MySQL JDBC JAR not on classpath:
{noformat}
hive show tables;
show tables;
FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Error creating
transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
{noformat}
JDBC URL references a MySQL DB that doesn't exist:
{noformat}
hive show tables;
FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Unknown
database 'mdb'
NestedThrowables:
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown database 'mdb'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
{noformat}
datanucleus.autoCreateSchema=false and metastore schema doesn't exist:
{noformat}
hive show tables;
FAILED: Error in metadata: javax.jdo.JDODataStoreException: Required table
missing : `DBS` in Catalog Schema .
DataNucleus requires this table to perform its persistence operations. Either
your MetaData is incorrect,
or you need to enable datanucleus.autoCreateTables
NestedThrowables:
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table
missing : `DBS` in Catalog Schema .
DataNucleus requires this table to perform its persistence operations. Either
your MetaData is incorrect,
or you need to enable datanucleus.autoCreateTables
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2176) Schema creation scripts are incomplete since they leave out tables that are specific to DataNucleus
[ https://issues.apache.org/jira/browse/HIVE-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2176. -- Resolution: Fixed Fix Version/s: 0.8.0 Hadoop Flags: [Reviewed] Committed to trunk and branch-0.7. Thanks Esteban! Schema creation scripts are incomplete since they leave out tables that are specific to DataNucleus --- Key: HIVE-2176 URL: https://issues.apache.org/jira/browse/HIVE-2176 Project: Hive Issue Type: Bug Components: Configuration, Metastore Affects Versions: 0.5.0, 0.6.0, 0.7.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: derby, mysql, postgres Fix For: 0.7.1, 0.8.0 Attachments: HIVE-2176.3.patch.txt When using the DDL SQL scripts to create the Metastore, tables like SEQUENCE_TABLE are missing and force the user to change the configuration to use Datanucleus to do all the provisioning of the Metastore tables. Adding the missing table definitions to the DDL scripts will allow to have a functional Hive Metastore without enabling additional privileges to the Metastore user and/or enabling datanucleus.autoCreateSchema property in hive-site.xml [After running the hive-schema-0.7.0.mysql.sql and revoking ALTER and CREATE privileges to the 'metastoreuser'] hive show tables; FAILED: Error in metadata: javax.jdo.JDOException: Exception thrown calling table.exists() for `SEQUENCE_TABLE` NestedThrowables: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: CREATE command denied to user 'metastoreuser'@'localhost' for table 'SEQUENCE_TABLE' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-872: Allow BIGINT constants
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-15 00:27:01.842091) Review request for hive and John Sichi. Changes --- Gut the changes to TypeProcFactory and the grammer, instead amend COALESCE and UNION ALL to allow type widening. Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2462517 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ec816e9 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java f46c16c ql/src/test/queries/clientpositive/type_widening.q PRE-CREATION ql/src/test/results/clientpositive/type_widening.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed
[jira] [Commented] (HIVE-872) Allow BIGINT constants
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049545#comment-13049545 ] jirapos...@reviews.apache.org commented on HIVE-872: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/889/ --- (Updated 2011-06-15 00:27:01.842091) Review request for hive and John Sichi. Changes --- Gut the changes to TypeProcFactory and the grammer, instead amend COALESCE and UNION ALL to allow type widening. Summary --- Added a rule to the lexical grammar to allow BIGINT constants ending with 'L', and a clause to the TypeCheckProcFactory to ensure it gets interpreted properly. This addresses bug HIVE-872. https://issues.apache.org/jira/browse/HIVE-872 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2462517 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ec816e9 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java f46c16c ql/src/test/queries/clientpositive/type_widening.q PRE-CREATION ql/src/test/results/clientpositive/type_widening.q.out PRE-CREATION Diff: https://reviews.apache.org/r/889/diff Testing --- TestCliDriver passes, previous behaviour was to accept bigint constants specified without 'L', which is also preserved, so adding additional tests for this case seems unnecessary. Thanks, Syed Allow BIGINT constants -- Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch We should allow 0L to be interpreted as a bigint constant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2158: add the HivePreparedStatement implementation based on current HIVE supported data-type
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/900/ --- Review request for hive. Summary --- Review for HIVe-2158. This addresses bug HIVE-2158. https://issues.apache.org/jira/browse/HIVE-2158 Diffs - jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java e27ec66 jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 3633326 Diff: https://reviews.apache.org/r/900/diff Testing --- Thanks, Carl
[jira] [Created] (HIVE-2221) Provide metastore upgarde script for HIVE-2215
Provide metastore upgarde script for HIVE-2215 -- Key: HIVE-2221 URL: https://issues.apache.org/jira/browse/HIVE-2221 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Fix For: 0.8.0 HIVE-2215 introduces one new table in database, upgrade scripts are needed for that. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2158) add the HivePreparedStatement implementation based on current HIVE supported data-type
[ https://issues.apache.org/jira/browse/HIVE-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049552#comment-13049552 ] jirapos...@reviews.apache.org commented on HIVE-2158: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/900/ --- Review request for hive. Summary --- Review for HIVe-2158. This addresses bug HIVE-2158. https://issues.apache.org/jira/browse/HIVE-2158 Diffs - jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java e27ec66 jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 3633326 Diff: https://reviews.apache.org/r/900/diff Testing --- Thanks, Carl add the HivePreparedStatement implementation based on current HIVE supported data-type -- Key: HIVE-2158 URL: https://issues.apache.org/jira/browse/HIVE-2158 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.6.0, 0.7.0, 0.8.0 Reporter: Yuanjun Li Assignee: Yuanjun Li Fix For: 0.7.1, 0.8.0 Attachments: HIVE-0.7.1-PreparedStatement.1.patch.txt, HIVE-0.8-PreparedStatement.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-872) Allow BIGINT constants
[ https://issues.apache.org/jira/browse/HIVE-872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049553#comment-13049553 ] John Sichi commented on HIVE-872: - The title and description for this issue need to be updated. Allow BIGINT constants -- Key: HIVE-872 URL: https://issues.apache.org/jira/browse/HIVE-872 Project: Hive Issue Type: New Feature Reporter: Zheng Shao Assignee: Syed S. Albiz Attachments: HIVE-872.1.patch We should allow 0L to be interpreted as a bigint constant. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple tables
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049555#comment-13049555 ] John Sichi commented on HIVE-2128: -- HiveInputFormat already keeps track of the mapping from path to input format. So the idea here is that instead of setting HiveIndexedInputFormat globally for the entire job, we need to be associating it only with the paths that are supposed to have index filtering applied. Automatic Indexing with multiple tables --- Key: HIVE-2128 URL: https://issues.apache.org/jira/browse/HIVE-2128 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2158) add the HivePreparedStatement implementation based on current HIVE supported data-type
[ https://issues.apache.org/jira/browse/HIVE-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049560#comment-13049560 ] Carl Steinbach commented on HIVE-2158: -- +1. add the HivePreparedStatement implementation based on current HIVE supported data-type -- Key: HIVE-2158 URL: https://issues.apache.org/jira/browse/HIVE-2158 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.6.0, 0.7.0, 0.8.0 Reporter: Yuanjun Li Assignee: Yuanjun Li Fix For: 0.7.1, 0.8.0 Attachments: HIVE-0.7.1-PreparedStatement.1.patch.txt, HIVE-0.8-PreparedStatement.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2158) add the HivePreparedStatement implementation based on current HIVE supported data-type
[ https://issues.apache.org/jira/browse/HIVE-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2158: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk and branch-0.7. Thanks Yuanjun! add the HivePreparedStatement implementation based on current HIVE supported data-type -- Key: HIVE-2158 URL: https://issues.apache.org/jira/browse/HIVE-2158 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.6.0, 0.7.0, 0.8.0 Reporter: Yuanjun Li Assignee: Yuanjun Li Fix For: 0.7.1, 0.8.0 Attachments: HIVE-0.7.1-PreparedStatement.1.patch.txt, HIVE-0.8-PreparedStatement.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2222) runnable queue in Driver and DriverContext is not thread safe
runnable queue in Driver and DriverContext is not thread safe - Key: HIVE- URL: https://issues.apache.org/jira/browse/HIVE- Project: Hive Issue Type: Bug Reporter: He Yongqiang Assignee: Namit Jain -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2140) Return correct Major / Minor version numbers for Hive Driver
[ https://issues.apache.org/jira/browse/HIVE-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2140: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk and branch-0.7. Thanks Curtis! Return correct Major / Minor version numbers for Hive Driver Key: HIVE-2140 URL: https://issues.apache.org/jira/browse/HIVE-2140 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.6.0, 0.7.0 Reporter: Curtis Boyden Assignee: Curtis Boyden Fix For: 0.7.1, 0.8.0 Attachments: hive-0.6-driver-version.patch, hive-0.7-driver-version.patch, hive-trunk-driver-version.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2158) add the HivePreparedStatement implementation based on current HIVE supported data-type
[ https://issues.apache.org/jira/browse/HIVE-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049574#comment-13049574 ] Yuanjun Li commented on HIVE-2158: -- Well, the Jasperreport can work well directly with Hive official release now. add the HivePreparedStatement implementation based on current HIVE supported data-type -- Key: HIVE-2158 URL: https://issues.apache.org/jira/browse/HIVE-2158 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.6.0, 0.7.0, 0.8.0 Reporter: Yuanjun Li Assignee: Yuanjun Li Fix For: 0.7.1, 0.8.0 Attachments: HIVE-0.7.1-PreparedStatement.1.patch.txt, HIVE-0.8-PreparedStatement.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2222) runnable queue in Driver and DriverContext is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-: - Attachment: hive..1.patch runnable queue in Driver and DriverContext is not thread safe - Key: HIVE- URL: https://issues.apache.org/jira/browse/HIVE- Project: Hive Issue Type: Bug Reporter: He Yongqiang Assignee: Namit Jain Attachments: hive..1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
