Build failed in Jenkins: Hive-trunk-h0.21 #762
See https://builds.apache.org/hudson/job/Hive-trunk-h0.21/762/ -- [...truncated 32350 lines...] [echo] Writing POM to https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/build/jdbc/pom.xml No ivy:settings found for the default reference 'ivy.instance'. A default instance will be used no settings file found, using default... :: loading settings :: url = jar:file:/home/hudson/.ant/lib/ivy-2.0.0-rc2.jar!/org/apache/ivy/core/settings/ivysettings.xml ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: check-ivy: create-dirs: compile-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds deploy-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: init: install-hadoopcore: install-hadoopcore-default: ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: ivy-retrieve-hadoop-source: :: loading settings :: file = https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/ivy/ivysettings.xml [ivy:retrieve] :: resolving dependencies :: org.apache.hive#hive-hwi;0.8.0-SNAPSHOT [ivy:retrieve] confs: [default] [ivy:retrieve] found hadoop#core;0.20.1 in hadoop-source [ivy:retrieve] :: resolution report :: resolve 663ms :: artifacts dl 1ms - | |modules|| artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| - | default | 1 | 0 | 0 | 0 || 1 | 0 | - [ivy:retrieve] :: retrieving :: org.apache.hive#hive-hwi [ivy:retrieve] confs: [default] [ivy:retrieve] 0 artifacts copied, 1 already retrieved (0kB/1ms) install-hadoopcore-internal: setup: war: compile: [echo] Compiling: hwi [javac] https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/hwi/build.xml:71: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: [echo] Jar: hwi make-pom: [echo] Writing POM to https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/build/hwi/pom.xml No ivy:settings found for the default reference 'ivy.instance'. A default instance will be used no settings file found, using default... :: loading settings :: url = jar:file:/home/hudson/.ant/lib/ivy-2.0.0-rc2.jar!/org/apache/ivy/core/settings/ivysettings.xml ivy-init-dirs: ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/build/ivy/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: check-ivy: create-dirs: compile-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds deploy-ant-tasks: create-dirs: init: compile: [echo] Compiling: anttasks [javac] https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/ant/build.xml:40: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds jar: init: setup: compile: [echo] Compiling: hbase-handler [javac] https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/build-common.xml:299: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [copy] Warning: https://builds.apache.org/hudson/job/Hive-trunk-h0.21/ws/hive/hbase-handler/src/java/conf does not exist. jar: [echo] Jar: hbase-handler make-pom: [echo] Writing POM to
[jira] [Created] (HIVE-2198) While using Hive in server mode, HiveConnection.close() is not cleaning up server side resources
While using Hive in server mode, HiveConnection.close() is not cleaning up server side resources Key: HIVE-2198 URL: https://issues.apache.org/jira/browse/HIVE-2198 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam org.apache.hadoop.hive.service.ThriftHive.Client.clean() method is called for every session end in CLI mode for the cleanup but in HiveServer mode this is not called. So this can be integrate with the HiveConnection.close() -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-956) Add support of columnar binary serde
[ https://issues.apache.org/jira/browse/HIVE-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13044968#comment-13044968 ] He Yongqiang commented on HIVE-956: --- will take a look. Add support of columnar binary serde Key: HIVE-956 URL: https://issues.apache.org/jira/browse/HIVE-956 Project: Hive Issue Type: New Feature Reporter: He Yongqiang Assignee: Krishna Kumar Attachments: HIVE.956.patch.0, HIVE.956.patch.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Travel Assistance applications now open for ApacheCon NA 2011
The Apache Software Foundation (ASF)'s Travel Assistance Committee (TAC) is now accepting applications for ApacheCon North America 2011, 7-11 November in Vancouver BC, Canada. The TAC is seeking individuals from the Apache community at-large --users, developers, educators, students, Committers, and Members-- who would like to attend ApacheCon, but need some financial support in order to be able to get there. There are limited places available, and all applicants will be scored on their individual merit. Financial assistance is available to cover flights/trains, accommodation and entrance fees either in part or in full, depending on circumstances. However, the support available for those attending only the BarCamp (7-8 November) is less than that for those attending the entire event (Conference + BarCamp 7-11 November). The Travel Assistance Committee aims to support all official ASF events, including cross-project activities; as such, it may be prudent for those in Asia and Europe to wait for an event geographically closer to them. More information can be found at http://www.apache.org/travel/index.html including a link to the online application and detailed instructions for submitting. Applications will close on 8 July 2011 at 22:00 BST (UTC/GMT +1). We wish good luck to all those who will apply, and thank you in advance for tweeting, blogging, and otherwise spreading the word. Regards, The Travel Assistance Committee
[jira] [Updated] (HIVE-2186) Dynamic Partitioning Failing because of characters not supported globStatus
[ https://issues.apache.org/jira/browse/HIVE-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklin Hu updated HIVE-2186: -- Attachment: hive-2186.5.patch fix uploaded binary files Dynamic Partitioning Failing because of characters not supported globStatus --- Key: HIVE-2186 URL: https://issues.apache.org/jira/browse/HIVE-2186 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Siying Dong Assignee: Franklin Hu Attachments: hive-2186.1.patch, hive-2186.2.patch, hive-2186.3.patch, hive-2186.4.patch, hive-2186.5.patch Some dynamic queries failed on the stage of loading partitions if dynamic partition columns contain special characters. We need to escape all of them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2188: Add a function to retrieve multiple tables on trip to the hive metastore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/ --- (Updated 2011-06-06 21:09:54.167933) Review request for hive, Paul Yang and Ashutosh Chauhan. Changes --- - Changed function name to get_multi_table - Fixed returned ordering of requested tables - Moved and added additional test cases Summary --- Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. This addresses bug HIVE-2188. https://issues.apache.org/jira/browse/HIVE-2188 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 Diff: https://reviews.apache.org/r/831/diff Testing --- Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. Thanks, Sohan
Re: Review Request: HIVE-2188: Add a function to retrieve multiple tables on trip to the hive metastore
On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: trunk/metastore/if/hive_metastore.thrift, line 249 https://reviews.apache.org/r/831/diff/1/?file=20144#file20144line249 How about calling it get_multi_table instead? multi_get_table sounds little confusing to me. Ok, get_multi_table definitely sounds better. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 521 https://reviews.apache.org/r/831/diff/1/?file=20145#file20145line521 You can write this more concisely using commons-lang utility method as: StringUtils.join(tbls,','); Thanks for the tip; I changed this. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1182 https://reviews.apache.org/r/831/diff/1/?file=20145#file20145line1182 You can get rid of tables.get(i) == null check that will never be true. I cleaned up the checks for seeing if the db had all the requested tables. Also, I realized that I did not guarantee returning tables in the same order as those requested, so I fixed that up too. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1195 https://reviews.apache.org/r/831/diff/1/?file=20145#file20145line1195 Instead of throwing RuntimeException, create MetaException and throw that. I changed it to throw a MetaException instead. However, the other metastore operations (e.g., get_table, append_partition) throw RuntimException, which is what I based get_multi_table off. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java, line 221 https://reviews.apache.org/r/831/diff/1/?file=20148#file20148line221 This test really belongs in the TestMetastore or some such in metastore dir not in HiveServer. Thanks for the pointer and for the feedback. - Sohan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review753 --- On 2011-06-06 21:09:54, Sohan Jain wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/ --- (Updated 2011-06-06 21:09:54) Review request for hive, Paul Yang and Ashutosh Chauhan. Summary --- Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. This addresses bug HIVE-2188. https://issues.apache.org/jira/browse/HIVE-2188 Diffs - trunk/metastore/if/hive_metastore.thrift 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 Diff: https://reviews.apache.org/r/831/diff Testing --- Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. Thanks, Sohan
[jira] [Commented] (HIVE-2188) Add multi_get_table function in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045121#comment-13045121 ] jirapos...@reviews.apache.org commented on HIVE-2188: - bq. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: bq. trunk/metastore/if/hive_metastore.thrift, line 249 bq. https://reviews.apache.org/r/831/diff/1/?file=20144#file20144line249 bq. bq. How about calling it get_multi_table instead? multi_get_table sounds little confusing to me. Ok, get_multi_table definitely sounds better. bq. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 521 bq. https://reviews.apache.org/r/831/diff/1/?file=20145#file20145line521 bq. bq. You can write this more concisely using commons-lang utility method as: StringUtils.join(tbls,','); Thanks for the tip; I changed this. bq. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1182 bq. https://reviews.apache.org/r/831/diff/1/?file=20145#file20145line1182 bq. bq. You can get rid of tables.get(i) == null check that will never be true. I cleaned up the checks for seeing if the db had all the requested tables. Also, I realized that I did not guarantee returning tables in the same order as those requested, so I fixed that up too. bq. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1195 bq. https://reviews.apache.org/r/831/diff/1/?file=20145#file20145line1195 bq. bq. Instead of throwing RuntimeException, create MetaException and throw that. I changed it to throw a MetaException instead. However, the other metastore operations (e.g., get_table, append_partition) throw RuntimException, which is what I based get_multi_table off. bq. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: bq. trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java, line 221 bq. https://reviews.apache.org/r/831/diff/1/?file=20148#file20148line221 bq. bq. This test really belongs in the TestMetastore or some such in metastore dir not in HiveServer. Thanks for the pointer and for the feedback. - Sohan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review753 --- On 2011-06-06 21:09:54, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/831/ bq. --- bq. bq. (Updated 2011-06-06 21:09:54) bq. bq. bq. Review request for hive, Paul Yang and Ashutosh Chauhan. bq. bq. bq. Summary bq. --- bq. bq. Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. bq. bq. bq. This addresses bug HIVE-2188. bq. https://issues.apache.org/jira/browse/HIVE-2188 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/if/hive_metastore.thrift 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 bq. bq. Diff: https://reviews.apache.org/r/831/diff bq. bq. bq. Testing bq. --- bq. bq. Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. bq. bq. bq. Thanks, bq. bq. Sohan bq. bq. Add multi_get_table function in Hive Metastore -- Key: HIVE-2188 URL: https://issues.apache.org/jira/browse/HIVE-2188 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Priority: Minor Attachments: HIVE-2188.1.patch This function would get multiple tables from the hive metastore as opposed to just one at a time, saving round trip time to the metastore. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2188) Add multi_get_table function in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sohan Jain updated HIVE-2188: - Attachment: HIVE-2188.3.patch Add multi_get_table function in Hive Metastore -- Key: HIVE-2188 URL: https://issues.apache.org/jira/browse/HIVE-2188 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Priority: Minor Attachments: HIVE-2188.1.patch, HIVE-2188.3.patch This function would get multiple tables from the hive metastore as opposed to just one at a time, saving round trip time to the metastore. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2188: Add a function to retrieve multiple tables on trip to the hive metastore
On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: trunk/metastore/if/hive_metastore.thrift, line 249 https://reviews.apache.org/r/831/diff/1/?file=20144#file20144line249 How about calling it get_multi_table instead? multi_get_table sounds little confusing to me. Sohan Jain wrote: Ok, get_multi_table definitely sounds better. get_tables is probably the ideal name, but unfortunately that's already taken by a function that should really be called get_table_names_by_pattern. In order to avoid this problem going forward, what do you think of changing the name to get_table_objects? - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review753 --- On 2011-06-06 21:09:54, Sohan Jain wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/ --- (Updated 2011-06-06 21:09:54) Review request for hive, Paul Yang and Ashutosh Chauhan. Summary --- Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. This addresses bug HIVE-2188. https://issues.apache.org/jira/browse/HIVE-2188 Diffs - trunk/metastore/if/hive_metastore.thrift 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 Diff: https://reviews.apache.org/r/831/diff Testing --- Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. Thanks, Sohan
[jira] [Commented] (HIVE-2188) Add multi_get_table function in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045128#comment-13045128 ] jirapos...@reviews.apache.org commented on HIVE-2188: - bq. On 2011-06-03 14:04:02, Ashutosh Chauhan wrote: bq. trunk/metastore/if/hive_metastore.thrift, line 249 bq. https://reviews.apache.org/r/831/diff/1/?file=20144#file20144line249 bq. bq. How about calling it get_multi_table instead? multi_get_table sounds little confusing to me. bq. bq. Sohan Jain wrote: bq. Ok, get_multi_table definitely sounds better. get_tables is probably the ideal name, but unfortunately that's already taken by a function that should really be called get_table_names_by_pattern. In order to avoid this problem going forward, what do you think of changing the name to get_table_objects? - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review753 --- On 2011-06-06 21:09:54, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/831/ bq. --- bq. bq. (Updated 2011-06-06 21:09:54) bq. bq. bq. Review request for hive, Paul Yang and Ashutosh Chauhan. bq. bq. bq. Summary bq. --- bq. bq. Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. bq. bq. bq. This addresses bug HIVE-2188. bq. https://issues.apache.org/jira/browse/HIVE-2188 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/if/hive_metastore.thrift 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 bq. bq. Diff: https://reviews.apache.org/r/831/diff bq. bq. bq. Testing bq. --- bq. bq. Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. bq. bq. bq. Thanks, bq. bq. Sohan bq. bq. Add multi_get_table function in Hive Metastore -- Key: HIVE-2188 URL: https://issues.apache.org/jira/browse/HIVE-2188 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Priority: Minor Attachments: HIVE-2188.1.patch, HIVE-2188.3.patch This function would get multiple tables from the hive metastore as opposed to just one at a time, saving round trip time to the metastore. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1595) job name for alter table T archive partition P is not correct
[ https://issues.apache.org/jira/browse/HIVE-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045134#comment-13045134 ] Paul Yang commented on HIVE-1595: - +1 Will test and commit. job name for alter table T archive partition P is not correct - Key: HIVE-1595 URL: https://issues.apache.org/jira/browse/HIVE-1595 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Namit Jain Assignee: Sohan Jain Attachments: Hive-1595.1.patch, Hive-1595.2.patch For some internal runs, I saw the job name as hadoop-0.20.1-tools.jar, which makes it difficult to identify -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: HIVE-2036: Update bitmap indexes for automatic usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/ --- Review request for hive and John Sichi. Summary --- Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. This addresses bug HIVE-2036. https://issues.apache.org/jira/browse/HIVE-2036 Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/HiveBitmapIndexInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a ql/src/test/queries/clientpositive/index_bitmap3.q 508eb94 ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION Diff: https://reviews.apache.org/r/857/diff Testing --- Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. Thanks, Syed
[jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
[ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045140#comment-13045140 ] jirapos...@reviews.apache.org commented on HIVE-2036: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/857/ --- Review request for hive and John Sichi. Summary --- Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster. This addresses bug HIVE-2036. https://issues.apache.org/jira/browse/HIVE-2036 Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1 ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/HiveBitmapIndexInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609 ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 268560d ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 0873e1a ql/src/test/queries/clientpositive/index_bitmap3.q 508eb94 ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION Diff: https://reviews.apache.org/r/857/diff Testing --- Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster. Thanks, Syed Update bitmap indexes for automatic usage - Key: HIVE-2036 URL: https://issues.apache.org/jira/browse/HIVE-2036 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support. The bitmap code will need to be extended after it is committed to enable automatic use of indexing. Most work will be focused in the BitmapIndexHandler, which needs to generate the re-entrant QL index query. There may also be significant work in the IndexPredicateAnalyzer to support predicates with OR's, instead of just AND's as it is currently. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2188) Add multi_get_table function in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045174#comment-13045174 ] jirapos...@reviews.apache.org commented on HIVE-2188: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review768 --- trunk/metastore/if/hive_metastore.thrift https://reviews.apache.org/r/831/#comment1650 Please consider changing this to get_table_objects_by_name This should also throw InvalidObjectException and UnknownDBException. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/831/#comment1648 Maybe change this to get_table_objects_by_name in order to disambiguate from cases where we're returning only tables names, or applying a filter condition, etc. etc. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/831/#comment1647 This should probably be InvalidOperationException instead of NoSuchObjectException. It might also be good to validate the dbname input parameter at this step, e.g. make sure it's not null and not an empty string. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/831/#comment1646 Failing the entire operation if a single table in the input list is not defined seems like a bad idea since we're throwing away work that will have to be repeated on the next call. Furthermore, the exception doesn't contain any information about which table(s) are not defined, so the client will have to fetch a table list again and use this to construct the list of input tables for the next get_multi_table() call. In the meantime it's possible that someone will drop a table in the list, which will invalidate the next call. I think it would be better to modify the contract to state that if a table on the input list is not found in the metastore, then the table definition will not be included in the result. This means that the function will return an empty list if none of the tables in the input list are found in the metastore. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/831/#comment1645 e.toString() actually returns a little more information than e.getMessage(). trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/831/#comment1649 Change to getTableObjectsByName? trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/831/#comment1643 It would be good to first check if the DB exists, and throw UnknownDBException if it's not found. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/831/#comment1644 Only some callers will care about having this ordering property satisfied, so instead of penalizing every caller with this performance hit, maybe it would be better to let the caller take care of this? - Carl On 2011-06-06 21:09:54, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/831/ bq. --- bq. bq. (Updated 2011-06-06 21:09:54) bq. bq. bq. Review request for hive, Paul Yang and Ashutosh Chauhan. bq. bq. bq. Summary bq. --- bq. bq. Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. bq. bq. bq. This addresses bug HIVE-2188. bq. https://issues.apache.org/jira/browse/HIVE-2188 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/if/hive_metastore.thrift 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 bq. bq. Diff: https://reviews.apache.org/r/831/diff bq. bq. bq. Testing bq. --- bq. bq. Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. bq. bq. bq. Thanks, bq. bq. Sohan bq. bq. Add multi_get_table function in Hive Metastore -- Key: HIVE-2188 URL: https://issues.apache.org/jira/browse/HIVE-2188 Project: Hive
[jira] [Created] (HIVE-2199) incorrect success flag passed to jobClose
incorrect success flag passed to jobClose - Key: HIVE-2199 URL: https://issues.apache.org/jira/browse/HIVE-2199 Project: Hive Issue Type: Bug Reporter: Franklin Hu Assignee: Franklin Hu Priority: Minor For block level merging of RCFiles, jobClose is passed the incorrect variable as the success flag -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2199) incorrect success flag passed to jobClose
[ https://issues.apache.org/jira/browse/HIVE-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklin Hu updated HIVE-2199: -- Component/s: Query Processor incorrect success flag passed to jobClose - Key: HIVE-2199 URL: https://issues.apache.org/jira/browse/HIVE-2199 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu Priority: Minor For block level merging of RCFiles, jobClose is passed the incorrect variable as the success flag -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2199) incorrect success flag passed to jobClose
[ https://issues.apache.org/jira/browse/HIVE-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklin Hu updated HIVE-2199: -- Status: Patch Available (was: Open) incorrect success flag passed to jobClose - Key: HIVE-2199 URL: https://issues.apache.org/jira/browse/HIVE-2199 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu Priority: Minor Attachments: hive-2199.1.patch For block level merging of RCFiles, jobClose is passed the incorrect variable as the success flag -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2199) incorrect success flag passed to jobClose
[ https://issues.apache.org/jira/browse/HIVE-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franklin Hu updated HIVE-2199: -- Attachment: hive-2199.1.patch fix success flag incorrect success flag passed to jobClose - Key: HIVE-2199 URL: https://issues.apache.org/jira/browse/HIVE-2199 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu Priority: Minor Attachments: hive-2199.1.patch For block level merging of RCFiles, jobClose is passed the incorrect variable as the success flag -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2194: Add actions to MetaStoreEventListener to be performed on alter table and alter partition operations.
On 2011-06-03 20:55:01, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java, line 54 https://reviews.apache.org/r/853/diff/1/?file=20442#file20442line54 This method should either succeed or throw an exception. Since it already throws checked exceptions, adding a boolean return value makes things more complicated for the caller, and signals the same type of failure as an exception but without the ability to include information about what caused the failure. I think it would be better to modify the code so that an exception is always thrown when success==FALSE. This comment also applies to the various alterTable and alterPartition methods located in other classes. Thanks for your feedback, Carl. I understand your point about the return value of those functions. Is throwing a MetaException in the case of failure appropriate? Furthermore, I noticed that a few other RawStore operations return boolean values, such as dropDatabase, alterDatabase, dropTable, addPartition, etc. The return value in each case corresponds to successfully committing the transaction. Should we adopt a uniform interface across these functions? - Sohan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/853/#review758 --- On 2011-06-03 20:20:03, Sohan Jain wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/853/ --- (Updated 2011-06-03 20:20:03) Review request for hive and Paul Yang. Summary --- HIVE-2038 introduced the MetaStoreEventListener abstract class that defines actions to be performed after particular events on a metastore. Improve upon that class by adding events to be performed on alter table and alter partition actions. Also, update the hive metastore to call the appropriate functions of the listeners when the events happen. This addresses bug HIVE-2194. https://issues.apache.org/jira/browse/HIVE-2194 Diffs - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterPartitionEvent.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1130342 Diff: https://reviews.apache.org/r/853/diff Testing --- Added test cases to TestMetaStoreEventListener. Thanks, Sohan
Re: Review Request: HIVE-2194: Add actions to MetaStoreEventListener to be performed on alter table and alter partition operations.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/853/ --- (Updated 2011-06-06 23:23:50.520460) Review request for hive and Paul Yang. Changes --- - changed interface for alterTable and alterPartition back to void return type Summary --- HIVE-2038 introduced the MetaStoreEventListener abstract class that defines actions to be performed after particular events on a metastore. Improve upon that class by adding events to be performed on alter table and alter partition actions. Also, update the hive metastore to call the appropriate functions of the listeners when the events happen. This addresses bug HIVE-2194. https://issues.apache.org/jira/browse/HIVE-2194 Diffs (updated) - trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterPartitionEvent.java PRE-CREATION Diff: https://reviews.apache.org/r/853/diff Testing --- Added test cases to TestMetaStoreEventListener. Thanks, Sohan
[jira] [Commented] (HIVE-2194) Add actions for alter table and alter partition events for metastore event listeners
[ https://issues.apache.org/jira/browse/HIVE-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045207#comment-13045207 ] jirapos...@reviews.apache.org commented on HIVE-2194: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/853/ --- (Updated 2011-06-06 23:23:50.520460) Review request for hive and Paul Yang. Changes --- - changed interface for alterTable and alterPartition back to void return type Summary --- HIVE-2038 introduced the MetaStoreEventListener abstract class that defines actions to be performed after particular events on a metastore. Improve upon that class by adding events to be performed on alter table and alter partition actions. Also, update the hive metastore to call the appropriate functions of the listeners when the events happen. This addresses bug HIVE-2194. https://issues.apache.org/jira/browse/HIVE-2194 Diffs (updated) - trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterPartitionEvent.java PRE-CREATION Diff: https://reviews.apache.org/r/853/diff Testing --- Added test cases to TestMetaStoreEventListener. Thanks, Sohan Add actions for alter table and alter partition events for metastore event listeners Key: HIVE-2194 URL: https://issues.apache.org/jira/browse/HIVE-2194 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2194.1.patch HIVE-2038 introduced the MetaStoreEventListener abstract class that defines actions to be performed after particular events on a metastore. Improve upon that class by adding events to be performed on alter table and alter partition actions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2194) Add actions for alter table and alter partition events for metastore event listeners
[ https://issues.apache.org/jira/browse/HIVE-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045208#comment-13045208 ] jirapos...@reviews.apache.org commented on HIVE-2194: - bq. On 2011-06-03 20:55:01, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java, line 54 bq. https://reviews.apache.org/r/853/diff/1/?file=20442#file20442line54 bq. bq. This method should either succeed or throw an exception. Since it already throws checked exceptions, adding a boolean return value makes things more complicated for the caller, and signals the same type of failure as an exception but without the ability to include information about what caused the failure. I think it would be better to modify the code so that an exception is always thrown when success==FALSE. bq. bq. This comment also applies to the various alterTable and alterPartition methods located in other classes. Thanks for your feedback, Carl. I understand your point about the return value of those functions. Is throwing a MetaException in the case of failure appropriate? Furthermore, I noticed that a few other RawStore operations return boolean values, such as dropDatabase, alterDatabase, dropTable, addPartition, etc. The return value in each case corresponds to successfully committing the transaction. Should we adopt a uniform interface across these functions? - Sohan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/853/#review758 --- On 2011-06-03 20:20:03, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/853/ bq. --- bq. bq. (Updated 2011-06-03 20:20:03) bq. bq. bq. Review request for hive and Paul Yang. bq. bq. bq. Summary bq. --- bq. bq. HIVE-2038 introduced the MetaStoreEventListener abstract class that defines actions to be performed after particular events on a metastore. Improve upon that class by adding events to be performed on alter table and alter partition actions. Also, update the hive metastore to call the appropriate functions of the listeners when the events happen. bq. bq. bq. This addresses bug HIVE-2194. bq. https://issues.apache.org/jira/browse/HIVE-2194 bq. bq. bq. Diffs bq. - bq. bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/AlterHandler.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterPartitionEvent.java PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterTableEvent.java PRE-CREATION bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 1130342 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1130342 bq. bq. Diff: https://reviews.apache.org/r/853/diff bq. bq. bq. Testing bq. --- bq. bq. Added test cases to TestMetaStoreEventListener. bq. bq. bq. Thanks, bq. bq. Sohan bq. bq. Add actions for alter table and alter partition events for metastore event listeners Key: HIVE-2194 URL: https://issues.apache.org/jira/browse/HIVE-2194 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2194.1.patch HIVE-2038 introduced the MetaStoreEventListener abstract class that defines actions to be performed after particular events on a metastore. Improve upon that class by adding events to be performed on alter table and alter partition actions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2100) virtual column references inside subqueries cause execution exceptions
[ https://issues.apache.org/jira/browse/HIVE-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045218#comment-13045218 ] He Yongqiang commented on HIVE-2100: authorization_part.q in negative tests failed. virtual column references inside subqueries cause execution exceptions -- Key: HIVE-2100 URL: https://issues.apache.org/jira/browse/HIVE-2100 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Attachments: HIVE-2100.txt example: create table jssarma_nilzma_bad as select a.fname, a.offset, a.val from (select hash(eventid,userid,eventtime,browsercookie,userstate,useragent,userip,serverip,clienttime,geoid,countrycode\ ,actionid,lastimpressionid,lastnavimpressionid,impressiontype,fullurl,fullreferrer,pagesection,modulesection,adsection) as val, INPUT__FILE__NAME as fname, BLOCK__OFFSET__INSIDE__FILE as offset from nectar_impression_lzma_unverified where ds='2010-07-28') a join jssarma_hc_diff b on (a.val=b.val); causes Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: cannot find field input__file__name from [org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@664310d0, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3d04fc23, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@12457d21, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@101a0ae6, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@1dc18a4c, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@d5e92d7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3bfa681c, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@34c92507, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@19e09a4, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2e8aeed0, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2344b18f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@72e5355f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@26132ae7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3465b738, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@1dfd868, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@ef894ce, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@61f1680f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2fe6e305, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@5f4275d4, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@445e228, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@802b249] at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:321) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldRef(UnionStructObjectInspector.java:96) at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:73) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at
[jira] [Updated] (HIVE-2100) virtual column references inside subqueries cause execution exceptions
[ https://issues.apache.org/jira/browse/HIVE-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-2100: --- Status: Open (was: Patch Available) virtual column references inside subqueries cause execution exceptions -- Key: HIVE-2100 URL: https://issues.apache.org/jira/browse/HIVE-2100 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Attachments: HIVE-2100.txt example: create table jssarma_nilzma_bad as select a.fname, a.offset, a.val from (select hash(eventid,userid,eventtime,browsercookie,userstate,useragent,userip,serverip,clienttime,geoid,countrycode\ ,actionid,lastimpressionid,lastnavimpressionid,impressiontype,fullurl,fullreferrer,pagesection,modulesection,adsection) as val, INPUT__FILE__NAME as fname, BLOCK__OFFSET__INSIDE__FILE as offset from nectar_impression_lzma_unverified where ds='2010-07-28') a join jssarma_hc_diff b on (a.val=b.val); causes Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: cannot find field input__file__name from [org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@664310d0, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3d04fc23, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@12457d21, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@101a0ae6, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@1dc18a4c, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@d5e92d7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3bfa681c, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@34c92507, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@19e09a4, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2e8aeed0, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2344b18f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@72e5355f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@26132ae7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3465b738, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@1dfd868, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@ef894ce, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@61f1680f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2fe6e305, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@5f4275d4, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@445e228, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@802b249] at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:321) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldRef(UnionStructObjectInspector.java:96) at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:73) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
[jira] [Assigned] (HIVE-2100) virtual column references inside subqueries cause execution exceptions
[ https://issues.apache.org/jira/browse/HIVE-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang reassigned HIVE-2100: -- Assignee: Syed S. Albiz virtual column references inside subqueries cause execution exceptions -- Key: HIVE-2100 URL: https://issues.apache.org/jira/browse/HIVE-2100 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: Syed S. Albiz Attachments: HIVE-2100.txt example: create table jssarma_nilzma_bad as select a.fname, a.offset, a.val from (select hash(eventid,userid,eventtime,browsercookie,userstate,useragent,userip,serverip,clienttime,geoid,countrycode\ ,actionid,lastimpressionid,lastnavimpressionid,impressiontype,fullurl,fullreferrer,pagesection,modulesection,adsection) as val, INPUT__FILE__NAME as fname, BLOCK__OFFSET__INSIDE__FILE as offset from nectar_impression_lzma_unverified where ds='2010-07-28') a join jssarma_hc_diff b on (a.val=b.val); causes Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121) ... 18 more Caused by: java.lang.RuntimeException: cannot find field input__file__name from [org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@664310d0, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3d04fc23, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@12457d21, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@101a0ae6, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@1dc18a4c, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@d5e92d7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3bfa681c, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@34c92507, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@19e09a4, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2e8aeed0, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2344b18f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@72e5355f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@26132ae7, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@3465b738, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@1dfd868, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@ef894ce, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@61f1680f, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@2fe6e305, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@5f4275d4, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@445e228, org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector$MyField@802b249] at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:321) at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldRef(UnionStructObjectInspector.java:96) at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878) at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:73) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444) at
[jira] [Commented] (HIVE-2199) incorrect success flag passed to jobClose
[ https://issues.apache.org/jira/browse/HIVE-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045222#comment-13045222 ] He Yongqiang commented on HIVE-2199: +1 incorrect success flag passed to jobClose - Key: HIVE-2199 URL: https://issues.apache.org/jira/browse/HIVE-2199 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu Priority: Minor Attachments: hive-2199.1.patch For block level merging of RCFiles, jobClose is passed the incorrect variable as the success flag -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2200) DROP PARTITION does not update stats
DROP PARTITION does not update stats Key: HIVE-2200 URL: https://issues.apache.org/jira/browse/HIVE-2200 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu When hive.stats.autogather=true, dropping a partition does not appropriately update the overall table stats. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2202) multi-thread hive io and execution
multi-thread hive io and execution -- Key: HIVE-2202 URL: https://issues.apache.org/jira/browse/HIVE-2202 Project: Hive Issue Type: Improvement Reporter: Namit Jain Multi-threading hive's IO part and execution part. For Hive's RCFile, the reader can be put in a separate thread, and can keep 2 blocks in memory. One block for current processing, and the other for next batch. Potentially this can be more intelligent (like determine how many threads to read depending on the execution speed), but this will depends on the HDFS client's implementation. And the writer can also put into a separate thread, this way, the execution will not need to block on writer. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2201) remove name node calls in hive by creating temporary directories
remove name node calls in hive by creating temporary directories Key: HIVE-2201 URL: https://issues.apache.org/jira/browse/HIVE-2201 Project: Hive Issue Type: Improvement Reporter: Namit Jain Currently, in Hive, when a file gets written by a FileSinkOperator, the sequence of operations is as follows: 1. In tmp directory tmp1, create a tmp file _tmp_1 2. At the end of the operator, move /tmp1/_tmp_1 to /tmp1/1 3. Move directory /tmp1 to /tmp2 4. For all files in /tmp2, remove all files starting with _tmp and duplicate files. Due to speculative execution, a lot of temporary files are created in /tmp1 (or /tmp2). This leads to a lot of name node calls, specially for large queries. The protocol above can be modified slightly: 1. In tmp directory tmp1, create a tmp file _tmp_1 2. At the end of the operator, move /tmp1/_tmp_1 to /tmp2/1 3. Move directory /tmp2 to /tmp3 4. For all files in /tmp3, remove all duplicate files. This should reduce the number of tmp files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-2188: Add a function to retrieve multiple tables on trip to the hive metastore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/ --- (Updated 2011-06-07 01:11:44.582629) Review request for hive, Paul Yang and Ashutosh Chauhan. Changes --- - included Carl's suggestions. - get_table_objects_by_name retrieves only the tables that it can, and does not fail out if it cannot retrieve a table. - removed the ordering of tables returned, since there is no guarantee every requested table is returned - modified unit tests Summary --- Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. This addresses bug HIVE-2188. https://issues.apache.org/jira/browse/HIVE-2188 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 Diff: https://reviews.apache.org/r/831/diff Testing --- Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. Thanks, Sohan
Re: Review Request: HIVE-2188: Add a function to retrieve multiple tables on trip to the hive metastore
On 2011-06-06 22:18:08, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1153 https://reviews.apache.org/r/831/diff/2/?file=20589#file20589line1153 Maybe change this to get_table_objects_by_name in order to disambiguate from cases where we're returning only tables names, or applying a filter condition, etc. etc. Sounds good. On 2011-06-06 22:18:08, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1164 https://reviews.apache.org/r/831/diff/2/?file=20589#file20589line1164 This should probably be InvalidOperationException instead of NoSuchObjectException. It might also be good to validate the dbname input parameter at this step, e.g. make sure it's not null and not an empty string. Ah ok thanks; it felt strange to throw a NoSuchObjectException there. On 2011-06-06 22:18:08, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1171 https://reviews.apache.org/r/831/diff/2/?file=20589#file20589line1171 Failing the entire operation if a single table in the input list is not defined seems like a bad idea since we're throwing away work that will have to be repeated on the next call. Furthermore, the exception doesn't contain any information about which table(s) are not defined, so the client will have to fetch a table list again and use this to construct the list of input tables for the next get_multi_table() call. In the meantime it's possible that someone will drop a table in the list, which will invalidate the next call. I think it would be better to modify the contract to state that if a table on the input list is not found in the metastore, then the table definition will not be included in the result. This means that the function will return an empty list if none of the tables in the input list are found in the metastore. I see your concern, I'll change it to retrieve only tables that are in the metastore. On 2011-06-06 22:18:08, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java, line 818 https://reviews.apache.org/r/831/diff/2/?file=20590#file20590line818 Only some callers will care about having this ordering property satisfied, so instead of penalizing every caller with this performance hit, maybe it would be better to let the caller take care of this? Sounds good. Removing the ordering especially makes sense if we cannot guarantee that we return *every* table the caller requested. On 2011-06-06 22:18:08, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java, line 802 https://reviews.apache.org/r/831/diff/2/?file=20590#file20590line802 It would be good to first check if the DB exists, and throw UnknownDBException if it's not found. Will do. As a side note, this check requires another db call, which may slow down the function a little. - Sohan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review768 --- On 2011-06-07 01:11:44, Sohan Jain wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/ --- (Updated 2011-06-07 01:11:44) Review request for hive, Paul Yang and Ashutosh Chauhan. Summary --- Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. This addresses bug HIVE-2188. https://issues.apache.org/jira/browse/HIVE-2188 Diffs - trunk/metastore/if/hive_metastore.thrift 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 Diff: https://reviews.apache.org/r/831/diff Testing --- Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. Thanks, Sohan
[jira] [Commented] (HIVE-2188) Add multi_get_table function in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045228#comment-13045228 ] jirapos...@reviews.apache.org commented on HIVE-2188: - bq. On 2011-06-06 22:18:08, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1153 bq. https://reviews.apache.org/r/831/diff/2/?file=20589#file20589line1153 bq. bq. Maybe change this to get_table_objects_by_name in order to disambiguate from cases where we're returning only tables names, or applying a filter condition, etc. etc. Sounds good. bq. On 2011-06-06 22:18:08, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1164 bq. https://reviews.apache.org/r/831/diff/2/?file=20589#file20589line1164 bq. bq. This should probably be InvalidOperationException instead of NoSuchObjectException. bq. bq. It might also be good to validate the dbname input parameter at this step, e.g. make sure it's not null and not an empty string. Ah ok thanks; it felt strange to throw a NoSuchObjectException there. bq. On 2011-06-06 22:18:08, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1171 bq. https://reviews.apache.org/r/831/diff/2/?file=20589#file20589line1171 bq. bq. Failing the entire operation if a single table in the input list is not defined seems like a bad idea since we're throwing away work that will have to be repeated on the next call. Furthermore, the exception doesn't contain any information about which table(s) are not defined, so the client will have to fetch a table list again and use this to construct the list of input tables for the next get_multi_table() call. In the meantime it's possible that someone will drop a table in the list, which will invalidate the next call. bq. bq. I think it would be better to modify the contract to state that if a table on the input list is not found in the metastore, then the table definition will not be included in the result. This means that the function will return an empty list if none of the tables in the input list are found in the metastore. bq. I see your concern, I'll change it to retrieve only tables that are in the metastore. bq. On 2011-06-06 22:18:08, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java, line 818 bq. https://reviews.apache.org/r/831/diff/2/?file=20590#file20590line818 bq. bq. Only some callers will care about having this ordering property satisfied, so instead of penalizing every caller with this performance hit, maybe it would be better to let the caller take care of this? bq. Sounds good. Removing the ordering especially makes sense if we cannot guarantee that we return *every* table the caller requested. bq. On 2011-06-06 22:18:08, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java, line 802 bq. https://reviews.apache.org/r/831/diff/2/?file=20590#file20590line802 bq. bq. It would be good to first check if the DB exists, and throw UnknownDBException if it's not found. Will do. As a side note, this check requires another db call, which may slow down the function a little. - Sohan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/#review768 --- On 2011-06-07 01:11:44, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/831/ bq. --- bq. bq. (Updated 2011-06-07 01:11:44) bq. bq. bq. Review request for hive, Paul Yang and Ashutosh Chauhan. bq. bq. bq. Summary bq. --- bq. bq. Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. bq. bq. bq. This addresses bug HIVE-2188. bq. https://issues.apache.org/jira/browse/HIVE-2188 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/if/hive_metastore.thrift 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 bq. bq. Diff: https://reviews.apache.org/r/831/diff bq. bq. bq. Testing bq. --- bq. bq. Added a test case to
[jira] [Commented] (HIVE-2188) Add multi_get_table function in Hive Metastore
[ https://issues.apache.org/jira/browse/HIVE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045229#comment-13045229 ] jirapos...@reviews.apache.org commented on HIVE-2188: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/831/ --- (Updated 2011-06-07 01:11:44.582629) Review request for hive, Paul Yang and Ashutosh Chauhan. Changes --- - included Carl's suggestions. - get_table_objects_by_name retrieves only the tables that it can, and does not fail out if it cannot retrieve a table. - removed the ordering of tables returned, since there is no guarantee every requested table is returned - modified unit tests Summary --- Created a function multi_get_table that retrieves multiple tables on one trip to the hive metastore, saving round trip time. This addresses bug HIVE-2188. https://issues.apache.org/jira/browse/HIVE-2188 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1130342 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1130342 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1130342 Diff: https://reviews.apache.org/r/831/diff Testing --- Added a test case to testMetasore() in TestHiveServer. Also tested for speed improvements in a client session. Thanks, Sohan Add multi_get_table function in Hive Metastore -- Key: HIVE-2188 URL: https://issues.apache.org/jira/browse/HIVE-2188 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Priority: Minor Attachments: HIVE-2188.1.patch, HIVE-2188.3.patch This function would get multiple tables from the hive metastore as opposed to just one at a time, saving round trip time to the metastore. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2203) Extend concat_ws() UDF to support arrays of strings
Extend concat_ws() UDF to support arrays of strings --- Key: HIVE-2203 URL: https://issues.apache.org/jira/browse/HIVE-2203 Project: Hive Issue Type: Improvement Components: UDF Reporter: Carl Steinbach Priority: Minor concat_ws() should support the following type of input parameters: concat_ws(string separator, arraystring) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2186) Dynamic Partitioning Failing because of characters not supported globStatus
[ https://issues.apache.org/jira/browse/HIVE-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045249#comment-13045249 ] Hudson commented on HIVE-2186: -- Integrated in Hive-trunk-h0.21 #764 (See [https://builds.apache.org/job/Hive-trunk-h0.21/764/]) HIVE-2186. Escaping More Characters When Generating HDFS Path Name From Table Columns (Franklin Hu via Siying Dong) sdong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1132778 Files : * /hive/trunk/data/files/escapetest.txt * /hive/trunk/ql/src/test/results/clientpositive/escape1.q.out * /hive/trunk/ql/src/test/queries/clientpositive/escape1.q * /hive/trunk/common/src/java/org/apache/hadoop/hive/common/FileUtils.java Dynamic Partitioning Failing because of characters not supported globStatus --- Key: HIVE-2186 URL: https://issues.apache.org/jira/browse/HIVE-2186 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Siying Dong Assignee: Franklin Hu Attachments: hive-2186.1.patch, hive-2186.2.patch, hive-2186.3.patch, hive-2186.4.patch, hive-2186.5.patch Some dynamic queries failed on the stage of loading partitions if dynamic partition columns contain special characters. We need to escape all of them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Jenkins build is back to normal : Hive-trunk-h0.21 #764
See https://builds.apache.org/job/Hive-trunk-h0.21/764/changes
