[jira] [Commented] (HIVE-2457) Files in Avro-backed Hive tables do not have a .avro extension
[ https://issues.apache.org/jira/browse/HIVE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113579#comment-13113579 ] jirapos...@reviews.apache.org commented on HIVE-2457: - bq. On 2011-09-21 00:26:28, Carl Steinbach wrote: bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 892 bq. https://reviews.apache.org/r/1989/diff/1/?file=2#file2line892 bq. bq. Please add this configuration property to HiveConf and hive-default.xml bq. bq. Tom White wrote: bq. Does Hive have the concept of private configuration properties? This is one that would set by SerDe's, not by users, which is why I didn't add it to HiveConf/hive-default.xml. bq. bq. Carl Steinbach wrote: bq. No, it doesn't, but it should. I'll file a JIRA. In the meantime this property should still be included in hive-default and HiveConf. OK, I added the property to hive-default and HiveConf in the latest patch. - Tom --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1989/#review1982 --- On 2011-09-20 22:28:53, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1989/ bq. --- bq. bq. (Updated 2011-09-20 22:28:53) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review for HIVE-2457 bq. bq. bq. This addresses bug HIVE-2457. bq. https://issues.apache.org/jira/browse/HIVE-2457 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1173340 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1173340 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 1173340 bq.trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1989/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Files in Avro-backed Hive tables do not have a .avro extension Key: HIVE-2457 URL: https://issues.apache.org/jira/browse/HIVE-2457 Project: Hive Issue Type: Improvement Components: Query Processor, Serializers/Deserializers Reporter: Tom White Assignee: Tom White Attachments: HIVE-2457.patch, HIVE-2457.patch When using the Avro SerDe (see HIVE-895, https://github.com/jghoman/haivvreo) the files created for an Avro table do not have a .avro extension, which causes problems for tools like Avro MapReduce or Sqoop which expect the extension. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into ZooKeeperHiveLockManager
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113769#comment-13113769 ] jirapos...@reviews.apache.org commented on HIVE-2450: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1950/ --- (Updated 2011-09-23 21:10:26.466632) Review request for hive and Ning Zhang. Changes --- update comment in hive-default.xml Summary --- move lock retry logic into ZooKeeperHiveLockManager This addresses bug HIVE-2450. https://issues.apache.org/jira/browse/HIVE-2450 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 trunk/conf/hive-default.xml 1171255 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255 Diff: https://reviews.apache.org/r/1950/diff Testing --- will run tests locally Thanks, Yongqiang move lock retry logic into ZooKeeperHiveLockManager --- Key: HIVE-2450 URL: https://issues.apache.org/jira/browse/HIVE-2450 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into ZooKeeperHiveLockManager
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113857#comment-13113857 ] jirapos...@reviews.apache.org commented on HIVE-2450: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1950/#review2061 --- Could you explain why we want the retry logic down inside of the zookeeper-specific implementation? It seems to me that having it outside is better, since then it doesn't have to be reimplemented in other lock manager implementations as they are added. trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java https://reviews.apache.org/r/1950/#comment4643 quorumServers is not used by this method...why has it been added here? trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java https://reviews.apache.org/r/1950/#comment4642 But don't we still want to rethrow eventually out of this method? Here you are squelching the exception completely. - John On 2011-09-23 21:10:26, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1950/ bq. --- bq. bq. (Updated 2011-09-23 21:10:26) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. move lock retry logic into ZooKeeperHiveLockManager bq. bq. bq. This addresses bug HIVE-2450. bq. https://issues.apache.org/jira/browse/HIVE-2450 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 bq.trunk/conf/hive-default.xml 1171255 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255 bq. bq. Diff: https://reviews.apache.org/r/1950/diff bq. bq. bq. Testing bq. --- bq. bq. will run tests locally bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. move lock retry logic into ZooKeeperHiveLockManager --- Key: HIVE-2450 URL: https://issues.apache.org/jira/browse/HIVE-2450 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2450.1.patch, HIVE-2450.2.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2181) Clean up the scratch.dir (tmp/hive-root) while restarting Hive server.
[ https://issues.apache.org/jira/browse/HIVE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112796#comment-13112796 ] jirapos...@reviews.apache.org commented on HIVE-2181: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1481/ --- (Updated 2011-09-22 18:26:19.812602) Review request for hive. Changes --- In testcases server startup is removed and added unit tests for cleanUpScratchDir(). Summary --- Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. Implemented a cleanup method in HiveServer and it will trigger based on the property value hive.start.cleanup.scrachdir This addresses bug HIVE-2181. https://issues.apache.org/jira/browse/HIVE-2181 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/common/ServerUtils.java PRE-CREATION trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1174277 trunk/conf/hive-default.xml 1174277 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1174277 trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1174277 Diff: https://reviews.apache.org/r/1481/diff Testing --- Added test case for this scenario. Thanks, chinna Clean up the scratch.dir (tmp/hive-root) while restarting Hive server. Key: HIVE-2181 URL: https://issues.apache.org/jira/browse/HIVE-2181 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Environment: Suse linux, Hadoop 20.1, Hive 0.8 Reporter: sanoj mathew Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-2181.1.patch, HIVE-2181.2.patch, HIVE-2181.3.patch, HIVE-2181.4.patch, HIVE-2181.5.patch, HIVE-2181.6.patch, HIVE-2181.patch Original Estimate: 48h Remaining Estimate: 48h Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2457) Files in Avro-backed Hive tables do not have a .avro extension
[ https://issues.apache.org/jira/browse/HIVE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112826#comment-13112826 ] jirapos...@reviews.apache.org commented on HIVE-2457: - bq. On 2011-09-21 00:26:28, Carl Steinbach wrote: bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 892 bq. https://reviews.apache.org/r/1989/diff/1/?file=2#file2line892 bq. bq. Please add this configuration property to HiveConf and hive-default.xml bq. bq. Tom White wrote: bq. Does Hive have the concept of private configuration properties? This is one that would set by SerDe's, not by users, which is why I didn't add it to HiveConf/hive-default.xml. No, it doesn't, but it should. I'll file a JIRA. In the meantime this property should still be included in hive-default and HiveConf. - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1989/#review1982 --- On 2011-09-20 22:28:53, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1989/ bq. --- bq. bq. (Updated 2011-09-20 22:28:53) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review for HIVE-2457 bq. bq. bq. This addresses bug HIVE-2457. bq. https://issues.apache.org/jira/browse/HIVE-2457 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1173340 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1173340 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 1173340 bq.trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1989/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Files in Avro-backed Hive tables do not have a .avro extension Key: HIVE-2457 URL: https://issues.apache.org/jira/browse/HIVE-2457 Project: Hive Issue Type: Improvement Components: Query Processor, Serializers/Deserializers Reporter: Tom White Assignee: Tom White Attachments: HIVE-2457.patch, HIVE-2457.patch When using the Avro SerDe (see HIVE-895, https://github.com/jghoman/haivvreo) the files created for an Avro table do not have a .avro extension, which causes problems for tools like Avro MapReduce or Sqoop which expect the extension. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2461) Add method to PerfLogger to perform cleanup/final steps.
[ https://issues.apache.org/jira/browse/HIVE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112850#comment-13112850 ] jirapos...@reviews.apache.org commented on HIVE-2461: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2023/ --- Review request for hive, Yongqiang He and Ning Zhang. Summary --- I added a method called finishPerfLogger, which is called at the end of the run method in Driver to allow PerfLogger subclasses to perform any cleanup tasks or logging that requires all the perf stats to have been calculated. It takes as parameters a Log, to be consistent with the other methods in PerfLogger, as well as the QueryPlan from Driver in order to allow information such as the query ID to be logged. This addresses bug HIVE-2461. https://issues.apache.org/jira/browse/HIVE-2461 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1173881 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1173881 Diff: https://reviews.apache.org/r/2023/diff Testing --- I created a subclass of PerfLogger which implemented the finishPerfLogger method, and verified it was successfully called and executed. I also ran some of the unit tests to verify nothing was broken. Thanks, Kevin Add method to PerfLogger to perform cleanup/final steps. Key: HIVE-2461 URL: https://issues.apache.org/jira/browse/HIVE-2461 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2461.1.patch.txt I think a method added to PerfLogger to perform cleanup/final steps would be very useful. For example, it could be used to close any database connections created as part of a PerfLogger subclass, or to perform logging that requires all perf values to first be calculated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2462) make INNER a non-reserved keyword
[ https://issues.apache.org/jira/browse/HIVE-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112860#comment-13112860 ] jirapos...@reviews.apache.org commented on HIVE-2462: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2024/ --- Review request for hive. Summary --- HIVE-2462 This addresses bug HIVE-2462. https://issues.apache.org/jira/browse/HIVE-2462 Diffs - http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1174316 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/innerjoin.q 1174316 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/innerjoin.q.out 1174316 Diff: https://reviews.apache.org/r/2024/diff Testing --- Thanks, John make INNER a non-reserved keyword - Key: HIVE-2462 URL: https://issues.apache.org/jira/browse/HIVE-2462 Project: Hive Issue Type: Improvement Reporter: John Sichi Assignee: John Sichi Fix For: 0.9.0 Attachments: HIVE-2462.1.patch HIVE-2191 introduced the INNER keyword as reserved, which breaks backwards compatibility for queries which were using it as an identifier. This patch addresses that. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2442) Metastore upgrade script and schema DDL for Hive 0.8.0
[ https://issues.apache.org/jira/browse/HIVE-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13112981#comment-13112981 ] jirapos...@reviews.apache.org commented on HIVE-2442: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2026/ --- Review request for hive. Summary --- Metastore upgrade scripts and schema DDL for Hive 0.8.0 and 0.9.0-dev. The 0.8.0 version of this patch does not include the 0.9.0 schema DDL and 0.8.0-to-0.9.0 upgrade script. This addresses bug HIVE-2442. https://issues.apache.org/jira/browse/HIVE-2442 Diffs - metastore/scripts/upgrade/derby/hive-schema-0.8.0.derby.sql PRE-CREATION metastore/scripts/upgrade/derby/hive-schema-0.9.0.derby.sql PRE-CREATION metastore/scripts/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql PRE-CREATION metastore/scripts/upgrade/derby/upgrade-0.8.0-to-0.9.0.derby.sql PRE-CREATION metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql 2e3821d metastore/scripts/upgrade/mysql/hive-schema-0.8.0.mysql.sql PRE-CREATION metastore/scripts/upgrade/mysql/hive-schema-0.9.0.mysql.sql PRE-CREATION metastore/scripts/upgrade/mysql/upgrade-0.7.0-to-0.8.0.mysql.sql PRE-CREATION metastore/scripts/upgrade/mysql/upgrade-0.8.0-to-0.9.0.mysql.sql PRE-CREATION Diff: https://reviews.apache.org/r/2026/diff Testing --- Thanks, Carl Metastore upgrade script and schema DDL for Hive 0.8.0 -- Key: HIVE-2442 URL: https://issues.apache.org/jira/browse/HIVE-2442 Project: Hive Issue Type: Task Components: Metastore Reporter: Carl Steinbach Assignee: Carl Steinbach Priority: Blocker Fix For: 0.8.0 Attachments: HIVE-2442-trunk.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2461) Add method to PerfLogger to perform cleanup/final steps.
[ https://issues.apache.org/jira/browse/HIVE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113042#comment-13113042 ] jirapos...@reviews.apache.org commented on HIVE-2461: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2023/ --- (Updated 2011-09-22 23:54:37.554390) Review request for hive, Yongqiang He and Ning Zhang. Summary (updated) --- I added a method called finishPerfLogger, which is called at the end of the run method in Driver to allow PerfLogger subclasses to perform any cleanup tasks or logging that requires all the perf stats to have been calculated. It takes as parameters a Log, to be consistent with the other methods in PerfLogger, as well as the QueryPlan from Driver in order to allow information such as the query ID to be logged. I should note that finishPerfLogger is not guaranteed to run. This was my intention. This addresses bug HIVE-2461. https://issues.apache.org/jira/browse/HIVE-2461 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1173881 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1173881 Diff: https://reviews.apache.org/r/2023/diff Testing --- I created a subclass of PerfLogger which implemented the finishPerfLogger method, and verified it was successfully called and executed. I also ran some of the unit tests to verify nothing was broken. Thanks, Kevin Add method to PerfLogger to perform cleanup/final steps. Key: HIVE-2461 URL: https://issues.apache.org/jira/browse/HIVE-2461 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2461.1.patch.txt I think a method added to PerfLogger to perform cleanup/final steps would be very useful. For example, it could be used to close any database connections created as part of a PerfLogger subclass, or to perform logging that requires all perf values to first be calculated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2461) Add method to PerfLogger to perform cleanup/final steps.
[ https://issues.apache.org/jira/browse/HIVE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113047#comment-13113047 ] jirapos...@reviews.apache.org commented on HIVE-2461: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2023/#review2030 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java https://reviews.apache.org/r/2023/#comment4579 one nitpick is that do you think call this function a 'close' will be more clear? we already have a PerfLogEnd(). - Yongqiang On 2011-09-22 23:54:37, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2023/ bq. --- bq. bq. (Updated 2011-09-22 23:54:37) bq. bq. bq. Review request for hive, Yongqiang He and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. I added a method called finishPerfLogger, which is called at the end of the run method in Driver to allow PerfLogger subclasses to perform any cleanup tasks or logging that requires all the perf stats to have been calculated. It takes as parameters a Log, to be consistent with the other methods in PerfLogger, as well as the QueryPlan from Driver in order to allow information such as the query ID to be logged. bq. bq. I should note that finishPerfLogger is not guaranteed to run. This was my intention. bq. bq. bq. This addresses bug HIVE-2461. bq. https://issues.apache.org/jira/browse/HIVE-2461 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1173881 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1173881 bq. bq. Diff: https://reviews.apache.org/r/2023/diff bq. bq. bq. Testing bq. --- bq. bq. I created a subclass of PerfLogger which implemented the finishPerfLogger method, and verified it was successfully called and executed. bq. bq. I also ran some of the unit tests to verify nothing was broken. bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. Add method to PerfLogger to perform cleanup/final steps. Key: HIVE-2461 URL: https://issues.apache.org/jira/browse/HIVE-2461 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2461.1.patch.txt I think a method added to PerfLogger to perform cleanup/final steps would be very useful. For example, it could be used to close any database connections created as part of a PerfLogger subclass, or to perform logging that requires all perf values to first be calculated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2461) Add method to PerfLogger to perform cleanup/final steps.
[ https://issues.apache.org/jira/browse/HIVE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113048#comment-13113048 ] jirapos...@reviews.apache.org commented on HIVE-2461: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2023/ --- (Updated 2011-09-23 00:08:56.319090) Review request for hive, Yongqiang He and Ning Zhang. Changes --- Thanks Yongqiang. I didn't like the name either, but the only other one I could come up with was finalize which I didn't want to use for obvious reasons. Summary --- I added a method called finishPerfLogger, which is called at the end of the run method in Driver to allow PerfLogger subclasses to perform any cleanup tasks or logging that requires all the perf stats to have been calculated. It takes as parameters a Log, to be consistent with the other methods in PerfLogger, as well as the QueryPlan from Driver in order to allow information such as the query ID to be logged. I should note that finishPerfLogger is not guaranteed to run. This was my intention. This addresses bug HIVE-2461. https://issues.apache.org/jira/browse/HIVE-2461 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1173881 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1173881 Diff: https://reviews.apache.org/r/2023/diff Testing --- I created a subclass of PerfLogger which implemented the finishPerfLogger method, and verified it was successfully called and executed. I also ran some of the unit tests to verify nothing was broken. Thanks, Kevin Add method to PerfLogger to perform cleanup/final steps. Key: HIVE-2461 URL: https://issues.apache.org/jira/browse/HIVE-2461 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2461.1.patch.txt, HIVE-2461.2.patch.txt I think a method added to PerfLogger to perform cleanup/final steps would be very useful. For example, it could be used to close any database connections created as part of a PerfLogger subclass, or to perform logging that requires all perf values to first be calculated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2181) Clean up the scratch.dir (tmp/hive-root) while restarting Hive server.
[ https://issues.apache.org/jira/browse/HIVE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13109494#comment-13109494 ] jirapos...@reviews.apache.org commented on HIVE-2181: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1481/ --- (Updated 2011-09-21 13:30:22.283181) Review request for hive. Changes --- spelling mistake corrected Summary --- Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. Implemented a cleanup method in HiveServer and it will trigger based on the property value hive.start.cleanup.scrachdir This addresses bug HIVE-2181. https://issues.apache.org/jira/browse/HIVE-2181 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1173539 trunk/conf/hive-default.xml 1173539 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1173539 trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1173539 Diff: https://reviews.apache.org/r/1481/diff Testing --- Added test case for this scenario. Thanks, chinna Clean up the scratch.dir (tmp/hive-root) while restarting Hive server. Key: HIVE-2181 URL: https://issues.apache.org/jira/browse/HIVE-2181 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Environment: Suse linux, Hadoop 20.1, Hive 0.8 Reporter: sanoj mathew Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-2181.1.patch, HIVE-2181.2.patch, HIVE-2181.3.patch, HIVE-2181.4.patch, HIVE-2181.5.patch, HIVE-2181.patch Original Estimate: 48h Remaining Estimate: 48h Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2178) Log related Check style Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13109498#comment-13109498 ] jirapos...@reviews.apache.org commented on HIVE-2178: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1978/ --- (Updated 2011-09-21 13:36:23.594198) Review request for hive, Carl Steinbach and John Sichi. Changes --- Thrift generated code changes are excluded from the patch. Thrift generated code changes i will follow up. Summary --- Fixed Log related Check style Comments like including root exception and adding messages. This addresses bug HIVE-2178. https://issues.apache.org/jira/browse/HIVE-2178 Diffs (updated) - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1173507 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java 1173507 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1173507 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1173507 trunk/ql/src/test/results/clientnegative/describe_xpath1.q.out 1173507 trunk/ql/src/test/results/clientnegative/describe_xpath2.q.out 1173507 trunk/ql/src/test/results/clientnegative/describe_xpath3.q.out 1173507 trunk/ql/src/test/results/clientnegative/describe_xpath4.q.out 1173507 trunk/ql/src/test/results/clientnegative/lockneg4.q.out 1173507 trunk/ql/src/test/results/clientnegative/lockneg5.q.out 1173507 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/SimpleCharStream.java 1173507 trunk/serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 1173507 Diff: https://reviews.apache.org/r/1978/diff Testing --- Ran all unit tests Thanks, chinna Log related Check style Comments fixes -- Key: HIVE-2178 URL: https://issues.apache.org/jira/browse/HIVE-2178 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2178.1.patch, HIVE-2178.2.patch, HIVE-2178.3.patch, HIVE-2178.patch Fix Log related Check style Comments -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2178) Log related Check style Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13109501#comment-13109501 ] jirapos...@reviews.apache.org commented on HIVE-2178: - bq. On 2011-09-20 18:12:16, Ashutosh Chauhan wrote: bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/MetaException.java, lines 118-121 bq. https://reviews.apache.org/r/1978/diff/1/?file=43792#file43792line118 bq. bq. Same as above. Thrift generated code changes are excluded from the patch. Thrift generated code changes i will follow up. - chinna --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1978/#review1975 --- On 2011-09-21 13:36:23, chinna wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1978/ bq. --- bq. bq. (Updated 2011-09-21 13:36:23) bq. bq. bq. Review request for hive, Carl Steinbach and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Fixed Log related Check style Comments like including root exception and adding messages. bq. bq. bq. This addresses bug HIVE-2178. bq. https://issues.apache.org/jira/browse/HIVE-2178 bq. bq. bq. Diffs bq. - bq. bq.trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1173507 bq.trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java 1173507 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1173507 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1173507 bq.trunk/ql/src/test/results/clientnegative/describe_xpath1.q.out 1173507 bq.trunk/ql/src/test/results/clientnegative/describe_xpath2.q.out 1173507 bq.trunk/ql/src/test/results/clientnegative/describe_xpath3.q.out 1173507 bq.trunk/ql/src/test/results/clientnegative/describe_xpath4.q.out 1173507 bq.trunk/ql/src/test/results/clientnegative/lockneg4.q.out 1173507 bq.trunk/ql/src/test/results/clientnegative/lockneg5.q.out 1173507 bq. trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/SimpleCharStream.java 1173507 bq. trunk/serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 1173507 bq. bq. Diff: https://reviews.apache.org/r/1978/diff bq. bq. bq. Testing bq. --- bq. bq. Ran all unit tests bq. bq. bq. Thanks, bq. bq. chinna bq. bq. Log related Check style Comments fixes -- Key: HIVE-2178 URL: https://issues.apache.org/jira/browse/HIVE-2178 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2178.1.patch, HIVE-2178.2.patch, HIVE-2178.3.patch, HIVE-2178.patch Fix Log related Check style Comments -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2206) add a new optimizer for query correlation discovery and optimization
[ https://issues.apache.org/jira/browse/HIVE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13109753#comment-13109753 ] jirapos...@reviews.apache.org commented on HIVE-2206: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2001/ --- Review request for hive. Summary --- This optimizer exploits intra-query correlations and merges multiple correlated MapReduce jobs into one jobs. This addresses bug HIVE-2206. https://issues.apache.org/jira/browse/HIVE-2206 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1173271 trunk/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationCompositeOperator.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationDispatchOperator.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationFakeReduceSinkOperator.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationManualForwardOperator.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationReduceSinkOperator.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecReducer.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationGenMRRedSink1.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizer.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizerUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationCompositeDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationDispatchDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationFakeReduceSinkDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationManualForwardDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationReduceSinkDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1173271 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCountDistinct.java PRE-CREATION trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 1173271 trunk/ql/src/test/results/clientpositive/show_functions.q.out 1173271 trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1173271 trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1173271 trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1173271 trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1173271 Diff: https://reviews.apache.org/r/2001/diff Testing --- Ran all unit tests Thanks, Yin add a new optimizer for query correlation discovery and optimization Key: HIVE-2206 URL: https://issues.apache.org/jira/browse/HIVE-2206 Project: Hive Issue Type: New Feature Reporter: He Yongqiang Assignee: Yin Huai Attachments: HIVE-2206.1.patch.txt, HIVE-2206.2.patch.txt, HIVE-2206.3.patch.txt, HIVE-2206.4.patch.txt, HIVE-2206.5.patch.txt, Queries, YSmartPatchForHive.patch, testQueries.q reference: http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-7.pdf -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2178) Log related Check style Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108714#comment-13108714 ] jirapos...@reviews.apache.org commented on HIVE-2178: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1978/ --- Review request for hive, Carl Steinbach and John Sichi. Summary --- Fixed Log related Check style Comments like including root exception and adding messages. This addresses bug HIVE-2178. https://issues.apache.org/jira/browse/HIVE-2178 Diffs - trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1172979 trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java 1172979 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidObjectException.java 1172979 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/MetaException.java 1172979 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1172979 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1172979 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1172979 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1172979 trunk/ql/src/test/results/clientnegative/describe_xpath1.q.out 1172979 trunk/ql/src/test/results/clientnegative/describe_xpath2.q.out 1172979 trunk/ql/src/test/results/clientnegative/describe_xpath3.q.out 1172979 trunk/ql/src/test/results/clientnegative/describe_xpath4.q.out 1172979 trunk/ql/src/test/results/clientnegative/lockneg4.q.out 1172979 trunk/ql/src/test/results/clientnegative/lockneg5.q.out 1172979 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/SimpleCharStream.java 1172979 trunk/serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 1172979 Diff: https://reviews.apache.org/r/1978/diff Testing --- Ran all unit tests Thanks, chinna Log related Check style Comments fixes -- Key: HIVE-2178 URL: https://issues.apache.org/jira/browse/HIVE-2178 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2178.1.patch, HIVE-2178.2.patch, HIVE-2178.patch Fix Log related Check style Comments -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2178) Log related Check style Comments fixes
[ https://issues.apache.org/jira/browse/HIVE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108879#comment-13108879 ] jirapos...@reviews.apache.org commented on HIVE-2178: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1978/#review1975 --- trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidObjectException.java https://reviews.apache.org/r/1978/#comment4465 This is a generated code. You should not hand edit it. If you find a need for editing it, its probably a bug in Thrift , you can open a jira on that project. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/MetaException.java https://reviews.apache.org/r/1978/#comment4466 Same as above. - Ashutosh On 2011-09-20 13:48:02, chinna wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1978/ bq. --- bq. bq. (Updated 2011-09-20 13:48:02) bq. bq. bq. Review request for hive, Carl Steinbach and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Fixed Log related Check style Comments like including root exception and adding messages. bq. bq. bq. This addresses bug HIVE-2178. bq. https://issues.apache.org/jira/browse/HIVE-2178 bq. bq. bq. Diffs bq. - bq. bq.trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1172979 bq.trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java 1172979 bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidObjectException.java 1172979 bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/MetaException.java 1172979 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1172979 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1172979 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1172979 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1172979 bq.trunk/ql/src/test/results/clientnegative/describe_xpath1.q.out 1172979 bq.trunk/ql/src/test/results/clientnegative/describe_xpath2.q.out 1172979 bq.trunk/ql/src/test/results/clientnegative/describe_xpath3.q.out 1172979 bq.trunk/ql/src/test/results/clientnegative/describe_xpath4.q.out 1172979 bq.trunk/ql/src/test/results/clientnegative/lockneg4.q.out 1172979 bq.trunk/ql/src/test/results/clientnegative/lockneg5.q.out 1172979 bq. trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/SimpleCharStream.java 1172979 bq. trunk/serde/src/java/org/apache/hadoop/hive/serde2/thrift/TBinarySortableProtocol.java 1172979 bq. bq. Diff: https://reviews.apache.org/r/1978/diff bq. bq. bq. Testing bq. --- bq. bq. Ran all unit tests bq. bq. bq. Thanks, bq. bq. chinna bq. bq. Log related Check style Comments fixes -- Key: HIVE-2178 URL: https://issues.apache.org/jira/browse/HIVE-2178 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2178.1.patch, HIVE-2178.2.patch, HIVE-2178.patch Fix Log related Check style Comments -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2181) Clean up the scratch.dir (tmp/hive-root) while restarting Hive server.
[ https://issues.apache.org/jira/browse/HIVE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108956#comment-13108956 ] jirapos...@reviews.apache.org commented on HIVE-2181: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1481/#review1977 --- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/1481/#comment4467 SCRATCH needs a T in it (everywhere). - John On 2011-09-16 17:30:40, chinna wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1481/ bq. --- bq. bq. (Updated 2011-09-16 17:30:40) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. bq. bq. Implemented a cleanup method in HiveServer and it will trigger based on the property value hive.start.cleanup.scrachdir bq. bq. bq. This addresses bug HIVE-2181. bq. https://issues.apache.org/jira/browse/HIVE-2181 bq. bq. bq. Diffs bq. - bq. bq.trunk/conf/hive-default.xml 1171488 bq.trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1171488 bq. trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1171488 bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171488 bq. bq. Diff: https://reviews.apache.org/r/1481/diff bq. bq. bq. Testing bq. --- bq. bq. Added test case for this scenario. bq. bq. bq. Thanks, bq. bq. chinna bq. bq. Clean up the scratch.dir (tmp/hive-root) while restarting Hive server. Key: HIVE-2181 URL: https://issues.apache.org/jira/browse/HIVE-2181 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Environment: Suse linux, Hadoop 20.1, Hive 0.8 Reporter: sanoj mathew Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-2181.1.patch, HIVE-2181.2.patch, HIVE-2181.3.patch, HIVE-2181.4.patch, HIVE-2181.patch Original Estimate: 48h Remaining Estimate: 48h Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2457) Files in Avro-backed Hive tables do not have a .avro extension
[ https://issues.apache.org/jira/browse/HIVE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13109036#comment-13109036 ] jirapos...@reviews.apache.org commented on HIVE-2457: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1989/ --- Review request for hive. Summary --- Review for HIVE-2457 This addresses bug HIVE-2457. https://issues.apache.org/jira/browse/HIVE-2457 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1173340 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1173340 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 1173340 trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java PRE-CREATION Diff: https://reviews.apache.org/r/1989/diff Testing --- Thanks, Carl Files in Avro-backed Hive tables do not have a .avro extension Key: HIVE-2457 URL: https://issues.apache.org/jira/browse/HIVE-2457 Project: Hive Issue Type: Improvement Components: Query Processor, Serializers/Deserializers Reporter: Tom White Attachments: HIVE-2457.patch When using the Avro SerDe (see HIVE-895, https://github.com/jghoman/haivvreo) the files created for an Avro table do not have a .avro extension, which causes problems for tools like Avro MapReduce or Sqoop which expect the extension. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2457) Files in Avro-backed Hive tables do not have a .avro extension
[
https://issues.apache.org/jira/browse/HIVE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13109130#comment-13109130
]
jirapos...@reviews.apache.org commented on HIVE-2457:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1989/#review1982
---
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
https://reviews.apache.org/r/1989/#comment4468
Please add this configuration property to HiveConf and hive-default.xml
trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java
https://reviews.apache.org/r/1989/#comment4479
In addition to the unit test it would also be nice to test this via
TestCliDriver.
It should be possible to verify this from the CLI by doing something like
this:
-- Set the filename suffix property. Then create a new table and stream
-- data into it. Then use the dfs cat command to dump the contents of
-- the raw files in the warehouse to stdout
hive dfs -cat ${hiveconf:hive.metastore.warehouse.dir}/tablename/*.avro;
- Carl
On 2011-09-20 22:28:53, Carl Steinbach wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1989/
bq. ---
bq.
bq. (Updated 2011-09-20 22:28:53)
bq.
bq.
bq. Review request for hive.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review for HIVE-2457
bq.
bq.
bq. This addresses bug HIVE-2457.
bq. https://issues.apache.org/jira/browse/HIVE-2457
bq.
bq.
bq. Diffs
bq. -
bq.
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
1173340
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1173340
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
1173340
bq.trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1989/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq.
bq. Thanks,
bq.
bq. Carl
bq.
bq.
Files in Avro-backed Hive tables do not have a .avro extension
Key: HIVE-2457
URL: https://issues.apache.org/jira/browse/HIVE-2457
Project: Hive
Issue Type: Improvement
Components: Query Processor, Serializers/Deserializers
Reporter: Tom White
Assignee: Tom White
Attachments: HIVE-2457.patch
When using the Avro SerDe (see HIVE-895, https://github.com/jghoman/haivvreo)
the files created for an Avro table do not have a .avro extension, which
causes problems for tools like Avro MapReduce or Sqoop which expect the
extension.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2453) Need a way to categorize queries in hooks for improved logging
[ https://issues.apache.org/jira/browse/HIVE-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107989#comment-13107989 ] jirapos...@reviews.apache.org commented on HIVE-2453: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1933/ --- (Updated 2011-09-19 17:09:57.838587) Review request for hive and Ning Zhang. Changes --- QueryProperties now captures distribute by as Ning requested, and cluster by as it seemed like a logical addition. I added test cases for these as well. Summary --- The information that would be useful for categorizing queries is clearest in the Semantic Analyzer, when the data from the Parser is interpreted. I added a new class which is designed to collect that data here, and place it ultimately in the QueryPlan where it will be available to hooks. The information I collect is whether or not the query has the following clauses: Join Group By Order By Sort By Group By after a Join clause Also, I store whether or not a script is used for mapping or reducing. This addresses bug HIVE-2453. https://issues.apache.org/jira/browse/HIVE-2453 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1170719 trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1170719 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1170719 trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/CheckQueryPropertiesHook.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/query_properties.q PRE-CREATION trunk/ql/src/test/results/clientpositive/query_properties.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1933/diff Testing --- I added a new test, which runs a variety of queries, such that each of the flags in QueryProperties is set by at least one query, and also some are set in combinations. I also added a hook which prints the contents of QueryProperties to error on the console. I checked the output in the results file and verified it matched what I expected. Thanks, Kevin Need a way to categorize queries in hooks for improved logging -- Key: HIVE-2453 URL: https://issues.apache.org/jira/browse/HIVE-2453 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2453.1.patch.txt, HIVE-2453.2.patch.txt We need a way to categorize queries, such as whether or not the include a join clause, a group by clause, etc., in the hooks. This will allow for better performance logging. Currently the only way I can find is to go through the operators in the tasks, but which operators are used for the different types of queries may change over time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2456) JDBCStatsAggregator DELETE STATEMENT should escape _ and %
[ https://issues.apache.org/jira/browse/HIVE-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108001#comment-13108001 ] jirapos...@reviews.apache.org commented on HIVE-2456: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1954/ --- (Updated 2011-09-19 17:18:39.180498) Review request for hive. Changes --- escape '%' and '_' in DELETE statement Summary --- JDBCStatsAggregator first aggregates stats from all publishers, and then delete these intermediate results. The delete is using LIKE operator, so it needs to escape '_' and '%'. This addresses bug HIVE-2456. https://issues.apache.org/jira/browse/HIVE-2456 Diffs - Diff: https://reviews.apache.org/r/1954/diff Testing --- running unit tests. Thanks, Ning JDBCStatsAggregator DELETE STATEMENT should escape _ and % -- Key: HIVE-2456 URL: https://issues.apache.org/jira/browse/HIVE-2456 Project: Hive Issue Type: Improvement Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-2456.patch JDBCStatsAggregator first aggregates stats from all publishers, and then delete these intermediate results. The delete is using LIKE operator, so it needs to escape '_' and '%'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2456) JDBCStatsAggregator DELETE STATEMENT should escape _ and %
[ https://issues.apache.org/jira/browse/HIVE-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108011#comment-13108011 ] jirapos...@reviews.apache.org commented on HIVE-2456: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1957/ --- Review request for hive. Summary --- JDBCStatsAggregator first aggregates stats from all publishers, and then delete these intermediate results. The delete is using LIKE operator, so it needs to escape '_' and '%'. This addresses bug HIVE-2456. https://issues.apache.org/jira/browse/HIVE-2456 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1172720 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.java 1172720 Diff: https://reviews.apache.org/r/1957/diff Testing --- Running unit tests. Thanks, Ning JDBCStatsAggregator DELETE STATEMENT should escape _ and % -- Key: HIVE-2456 URL: https://issues.apache.org/jira/browse/HIVE-2456 Project: Hive Issue Type: Improvement Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-2456.patch JDBCStatsAggregator first aggregates stats from all publishers, and then delete these intermediate results. The delete is using LIKE operator, so it needs to escape '_' and '%'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2450) move lock retry logic into ZooKeeperHiveLockManager
[ https://issues.apache.org/jira/browse/HIVE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107636#comment-13107636 ] jirapos...@reviews.apache.org commented on HIVE-2450: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1950/ --- Review request for hive and Ning Zhang. Summary --- move lock retry logic into ZooKeeperHiveLockManager This addresses bug HIVE-2450. https://issues.apache.org/jira/browse/HIVE-2450 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171255 trunk/conf/hive-default.xml 1171255 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1171255 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java 1171255 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1171255 Diff: https://reviews.apache.org/r/1950/diff Testing --- will run tests locally Thanks, Yongqiang move lock retry logic into ZooKeeperHiveLockManager --- Key: HIVE-2450 URL: https://issues.apache.org/jira/browse/HIVE-2450 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2450.1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2380) Add ByteArray Datatype
[ https://issues.apache.org/jira/browse/HIVE-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107357#comment-13107357 ] jirapos...@reviews.apache.org commented on HIVE-2380: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ --- (Updated 2011-09-18 05:52:41.197110) Review request for hive. Changes --- Updated patch incorporating feedback Summary --- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380. https://issues.apache.org/jira/browse/HIVE-2380 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/serde/if/serde.thrift 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java
[jira] [Commented] (HIVE-2181) Clean up the scratch.dir (tmp/hive-root) while restarting Hive server.
[ https://issues.apache.org/jira/browse/HIVE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106615#comment-13106615 ] jirapos...@reviews.apache.org commented on HIVE-2181: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1481/ --- (Updated 2011-09-16 17:30:40.980271) Review request for hive. Changes --- Rebased the patch on latest trunk Summary --- Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. Implemented a cleanup method in HiveServer and it will trigger based on the property value hive.start.cleanup.scrachdir This addresses bug HIVE-2181. https://issues.apache.org/jira/browse/HIVE-2181 Diffs (updated) - trunk/conf/hive-default.xml 1171488 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1171488 trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1171488 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171488 Diff: https://reviews.apache.org/r/1481/diff Testing --- Added test case for this scenario. Thanks, chinna Clean up the scratch.dir (tmp/hive-root) while restarting Hive server. Key: HIVE-2181 URL: https://issues.apache.org/jira/browse/HIVE-2181 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.8.0 Environment: Suse linux, Hadoop 20.1, Hive 0.8 Reporter: sanoj mathew Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-2181.1.patch, HIVE-2181.2.patch, HIVE-2181.3.patch, HIVE-2181.patch Original Estimate: 48h Remaining Estimate: 48h Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2017) Driver.execute() should maintaining SessionState in case of runtime errors
[
https://issues.apache.org/jira/browse/HIVE-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106619#comment-13106619
]
jirapos...@reviews.apache.org commented on HIVE-2017:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1609/
---
(Updated 2011-09-16 17:33:41.236186)
Review request for hive, John Sichi and Ning Zhang.
Changes
---
Rebased the patch on the latest trunk
Summary
---
while processing two parallel tasks if one of the task fails the
Driver.taskCleanup() will call system.exit() this will shutdown the jvm so it
is replaced with the logic to stop the remaining tasks.. Here need to cleanup 2
kinds of tasks one is non-mr tasks and mr tasks. For stopping non-mr tasks
used the thread.interrupt() because every non-mr task will be executed as a
thread and for mr tasks maintained a variable called jobKillUri's this variable
will track the spawned job kill uri, in taskCleanup() if it is a mr task using
this variable kill the job.
This addresses bug HIVE-2017.
https://issues.apache.org/jira/browse/HIVE-2017
Diffs (updated)
-
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
1170977
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1170977
trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
1170977
trunk/ql/src/test/queries/clientnegative/alter_exit.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/alter_exit.q.out PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1170977
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1170977
Diff: https://reviews.apache.org/r/1609/diff
Testing
---
Added test case for this scenario. Ran existing test cases
Thanks,
chinna
Driver.execute() should maintaining SessionState in case of runtime errors
--
Key: HIVE-2017
URL: https://issues.apache.org/jira/browse/HIVE-2017
Project: Hive
Issue Type: Bug
Reporter: Ning Zhang
Assignee: Chinna Rao Lalam
Attachments: HIVE-2017.1.patch, HIVE-2017.2.patch
Here's a snippet from Driver.execute():
{code}
// TODO: This error messaging is not very informative. Fix that.
errorMessage = FAILED: Execution Error, return code + exitVal
+ from
+ tsk.getClass().getName();
SQLState = 08S01;
console.printError(errorMessage);
if (running.size() != 0) {
taskCleanup();
}
return 9;
{code}
I simply returned in case of runtime errors without maintaining SessionState.
It could cause resource leak mentioned in HIVE-1959.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics publishers possibility
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106620#comment-13106620 ] jirapos...@reviews.apache.org commented on HIVE-2446: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1909/ --- (Updated 2011-09-16 17:35:58.909415) Review request for hive and Ning Zhang. Changes --- Changed interface of ClientStatsPublisher, made instantiation of Client Stats Publishers more efficient, this diff no longer removes some unused imports Summary (updated) --- The purpose of this change is to allow publication or storage of counters while the job is running. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. Made cosmetic changes in one other class This addresses bug HIVE-2446. https://issues.apache.org/jira/browse/HIVE-2446 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171283 trunk/conf/hive-default.xml 1171283 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1171283 trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1171283 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION Diff: https://reviews.apache.org/r/1909/diff Testing --- Run some random tests, and still running the unit tests. Thanks, Robert Introduction of client statistics publishers possibility Key: HIVE-2446 URL: https://issues.apache.org/jira/browse/HIVE-2446 Project: Hive Issue Type: Improvement Components: Clients, Statistics Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch Original Estimate: 1h Remaining Estimate: 1h The purpose of this change is to allow publication or storage of counters while the job is running. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. Made cosmetic changes in two other classes -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics publishers possibility
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106625#comment-13106625 ] jirapos...@reviews.apache.org commented on HIVE-2446: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1909/#review1935 --- Ship it! - Ning On 2011-09-16 17:35:58, Robert Surówka wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1909/ bq. --- bq. bq. (Updated 2011-09-16 17:35:58) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. The purpose of this change is to allow publication or storage of counters while the job is running. bq. bq. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. bq. bq. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. bq. bq. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. bq. bq. Made cosmetic changes in one other class bq. bq. bq. This addresses bug HIVE-2446. bq. https://issues.apache.org/jira/browse/HIVE-2446 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171283 bq.trunk/conf/hive-default.xml 1171283 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1171283 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1171283 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1909/diff bq. bq. bq. Testing bq. --- bq. bq. Run some random tests, and still running the unit tests. bq. bq. bq. Thanks, bq. bq. Robert bq. bq. Introduction of client statistics publishers possibility Key: HIVE-2446 URL: https://issues.apache.org/jira/browse/HIVE-2446 Project: Hive Issue Type: Improvement Components: Clients, Statistics Reporter: Robert Surówka Assignee: Robert Surówka Priority: Minor Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch, HIVE-2446.2.patch Original Estimate: 1h Remaining Estimate: 1h The purpose of this change is to allow publication or storage of counters while the job is running. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. Made cosmetic changes in two other classes -- This message is automatically generated by JIRA. For more information on JIRA, see:
[jira] [Commented] (HIVE-2453) Need a way to categorize queries in hooks for improved logging
[ https://issues.apache.org/jira/browse/HIVE-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106720#comment-13106720 ] jirapos...@reviews.apache.org commented on HIVE-2453: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1933/ --- Review request for hive and Ning Zhang. Summary --- The information that would be useful for categorizing queries is clearest in the Semantic Analyzer, when the data from the Parser is interpreted. I added a new class which is designed to collect that data here, and place it ultimately in the QueryPlan where it will be available to hooks. The information I collect is whether or not the query has the following clauses: Join Group By Order By Sort By Group By after a Join clause Also, I store whether or not a script is used for mapping or reducing. This addresses bug HIVE-2453. https://issues.apache.org/jira/browse/HIVE-2453 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1170719 trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1170719 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1170719 trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/CheckQueryPropertiesHook.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/query_properties.q PRE-CREATION trunk/ql/src/test/results/clientpositive/query_properties.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1933/diff Testing --- I added a new test, which runs a variety of queries, such that each of the flags in QueryProperties is set by at least one query, and also some are set in combinations. I also added a hook which prints the contents of QueryProperties to error on the console. I checked the output in the results file and verified it matched what I expected. Thanks, Kevin Need a way to categorize queries in hooks for improved logging -- Key: HIVE-2453 URL: https://issues.apache.org/jira/browse/HIVE-2453 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2453.1.patch.txt We need a way to categorize queries, such as whether or not the include a join clause, a group by clause, etc., in the hooks. This will allow for better performance logging. Currently the only way I can find is to go through the operators in the tasks, but which operators are used for the different types of queries may change over time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2453) Need a way to categorize queries in hooks for improved logging
[ https://issues.apache.org/jira/browse/HIVE-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106830#comment-13106830 ] jirapos...@reviews.apache.org commented on HIVE-2453: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1933/#review1946 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java https://reviews.apache.org/r/1933/#comment4427 can you split it into 2 parts: useScriptInMapper and useScriptInReducer? - Ning On 2011-09-16 19:04:32, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1933/ bq. --- bq. bq. (Updated 2011-09-16 19:04:32) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. The information that would be useful for categorizing queries is clearest in the Semantic Analyzer, when the data from the Parser is interpreted. I added a new class which is designed to collect that data here, and place it ultimately in the QueryPlan where it will be available to hooks. bq. bq. The information I collect is whether or not the query has the following clauses: bq.Join bq.Group By bq.Order By bq.Sort By bq.Group By after a Join clause bq. bq. Also, I store whether or not a script is used for mapping or reducing. bq. bq. bq. This addresses bug HIVE-2453. bq. https://issues.apache.org/jira/browse/HIVE-2453 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1170719 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java PRE-CREATION bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1170719 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1170719 bq. trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/CheckQueryPropertiesHook.java PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/query_properties.q PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/query_properties.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1933/diff bq. bq. bq. Testing bq. --- bq. bq. I added a new test, which runs a variety of queries, such that each of the flags in QueryProperties is set by at least one query, and also some are set in combinations. bq. I also added a hook which prints the contents of QueryProperties to error on the console. bq. bq. I checked the output in the results file and verified it matched what I expected. bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. Need a way to categorize queries in hooks for improved logging -- Key: HIVE-2453 URL: https://issues.apache.org/jira/browse/HIVE-2453 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2453.1.patch.txt We need a way to categorize queries, such as whether or not the include a join clause, a group by clause, etc., in the hooks. This will allow for better performance logging. Currently the only way I can find is to go through the operators in the tasks, but which operators are used for the different types of queries may change over time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2453) Need a way to categorize queries in hooks for improved logging
[ https://issues.apache.org/jira/browse/HIVE-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106941#comment-13106941 ] jirapos...@reviews.apache.org commented on HIVE-2453: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1933/ --- (Updated 2011-09-17 00:14:50.529819) Review request for hive and Ning Zhang. Summary --- The information that would be useful for categorizing queries is clearest in the Semantic Analyzer, when the data from the Parser is interpreted. I added a new class which is designed to collect that data here, and place it ultimately in the QueryPlan where it will be available to hooks. The information I collect is whether or not the query has the following clauses: Join Group By Order By Sort By Group By after a Join clause Also, I store whether or not a script is used for mapping or reducing. This addresses bug HIVE-2453. https://issues.apache.org/jira/browse/HIVE-2453 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1170719 trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1170719 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1170719 trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/CheckQueryPropertiesHook.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/query_properties.q PRE-CREATION trunk/ql/src/test/results/clientpositive/query_properties.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1933/diff Testing --- I added a new test, which runs a variety of queries, such that each of the flags in QueryProperties is set by at least one query, and also some are set in combinations. I also added a hook which prints the contents of QueryProperties to error on the console. I checked the output in the results file and verified it matched what I expected. Thanks, Kevin Need a way to categorize queries in hooks for improved logging -- Key: HIVE-2453 URL: https://issues.apache.org/jira/browse/HIVE-2453 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2453.1.patch.txt We need a way to categorize queries, such as whether or not the include a join clause, a group by clause, etc., in the hooks. This will allow for better performance logging. Currently the only way I can find is to go through the operators in the tasks, but which operators are used for the different types of queries may change over time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2453) Need a way to categorize queries in hooks for improved logging
[ https://issues.apache.org/jira/browse/HIVE-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107017#comment-13107017 ] jirapos...@reviews.apache.org commented on HIVE-2453: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1933/#review1956 --- Ship it! - Ning On 2011-09-17 00:14:50, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1933/ bq. --- bq. bq. (Updated 2011-09-17 00:14:50) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. The information that would be useful for categorizing queries is clearest in the Semantic Analyzer, when the data from the Parser is interpreted. I added a new class which is designed to collect that data here, and place it ultimately in the QueryPlan where it will be available to hooks. bq. bq. The information I collect is whether or not the query has the following clauses: bq.Join bq.Group By bq.Order By bq.Sort By bq.Group By after a Join clause bq. bq. Also, I store whether or not a script is used for mapping or reducing. bq. bq. bq. This addresses bug HIVE-2453. bq. https://issues.apache.org/jira/browse/HIVE-2453 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1170719 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java PRE-CREATION bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1170719 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1170719 bq. trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/CheckQueryPropertiesHook.java PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/query_properties.q PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/query_properties.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1933/diff bq. bq. bq. Testing bq. --- bq. bq. I added a new test, which runs a variety of queries, such that each of the flags in QueryProperties is set by at least one query, and also some are set in combinations. bq. I also added a hook which prints the contents of QueryProperties to error on the console. bq. bq. I checked the output in the results file and verified it matched what I expected. bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. Need a way to categorize queries in hooks for improved logging -- Key: HIVE-2453 URL: https://issues.apache.org/jira/browse/HIVE-2453 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2453.1.patch.txt We need a way to categorize queries, such as whether or not the include a join clause, a group by clause, etc., in the hooks. This will allow for better performance logging. Currently the only way I can find is to go through the operators in the tasks, but which operators are used for the different types of queries may change over time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-198) Parse errors report incorrectly.
[ https://issues.apache.org/jira/browse/HIVE-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105662#comment-13105662 ] jirapos...@reviews.apache.org commented on HIVE-198: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1818/ --- (Updated 2011-09-15 20:25:10.729894) Review request for hive. Changes --- Fixed missing tests create_or_replace_view6, TestJdbcDriver.java. Summary --- Modified the ParseException message to include the actual token it fails on (When available). It should make it easier to grok the actual problem from the error. This addresses bug HIVE-198. https://issues.apache.org/jira/browse/HIVE-198 Diffs (updated) - trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 1164307 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1164307 trunk/ql/src/test/results/clientnegative/column_rename3.q.out 1164307 trunk/ql/src/test/results/clientnegative/create_or_replace_view6.q.out 1164307 trunk/ql/src/test/results/clientnegative/invalid_select_expression.q.out 1164307 trunk/ql/src/test/results/clientnegative/invalid_tbl_name.q.out 1164307 trunk/ql/src/test/results/compiler/errors/garbage.q.out 1164307 trunk/ql/src/test/results/compiler/errors/missing_overwrite.q.out 1164307 trunk/ql/src/test/results/compiler/errors/wrong_distinct2.q.out 1164307 Diff: https://reviews.apache.org/r/1818/diff Testing --- ant test -Dtestcase=TestParseNegative Thanks, avivey Parse errors report incorrectly. Key: HIVE-198 URL: https://issues.apache.org/jira/browse/HIVE-198 Project: Hive Issue Type: Bug Components: Query Processor Reporter: S. Alex Smith Assignee: Aviv Eyal Labels: parse Attachments: HIVE-198.2.patch.txt, PraseErrorMessage.patch The following two queries fail: CREATE TABLE output_table(userid, bigint); CREATE TABLE output_table(userid bigint, age int, sex string, location string); each giving the error message FAILED: Parse Error: line 1:16 mismatched input 'TABLE' expecting KW_TEMPORARY Although one might not catch it from the error message, the problem with the first is that there is a comma between userid and bigint, and the problem with the second is that location is a reserved keyword. Reported errors should more accurately describe the nature of the error, such as no type given for column 'userid' or 'location' is not a valid column name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2447) Add job ID to MapRedStats
[ https://issues.apache.org/jira/browse/HIVE-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105681#comment-13105681 ] jirapos...@reviews.apache.org commented on HIVE-2447: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1918/#review1917 --- Ship it! - Ning On 2011-09-15 17:39:49, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1918/ bq. --- bq. bq. (Updated 2011-09-15 17:39:49) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. I added the job ID to the MapRedStats class. It is populated as part of the constructor. bq. bq. bq. This addresses bug HIVE-2447. bq. https://issues.apache.org/jira/browse/HIVE-2447 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1171172 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1171172 bq. bq. Diff: https://reviews.apache.org/r/1918/diff bq. bq. bq. Testing bq. --- bq. bq. Verified the job ID was being populated. bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. Add job ID to MapRedStats - Key: HIVE-2447 URL: https://issues.apache.org/jira/browse/HIVE-2447 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2447.1.patch.txt Currently, we collect a lot of stats per job in MapRedStats, but there is no way to associate these stats with a specific job. Adding the job ID to the class would help to make the association clear. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics publishers possibility
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105710#comment-13105710 ] jirapos...@reviews.apache.org commented on HIVE-2446: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1909/#review1918 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java https://reviews.apache.org/r/1909/#comment4399 should -1 be replaced by 0 here? trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java https://reviews.apache.org/r/1909/#comment4400 I've thought it over again and feel it's better to use the counter name here rather than display name. Display name has the benefit that the users see the same name at the JT page and the metrics page, but display name could be too long (sometimes may contain special characters) and it could be changed over time. I think name should be a better ID here. The only caveat is that the person who's looking at the metrics need to know how it is translated to the display name in JT page. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java https://reviews.apache.org/r/1909/#comment4398 Is this line too long? Hive's line length shouldn't exceed 100 chars. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java https://reviews.apache.org/r/1909/#comment4401 Rather than passing JobID, it would be more flexible to pass a String converted from jobID. This is the approach that StatsPublisher interface is using. - Ning On 2011-09-14 22:19:21, Robert Surówka wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1909/ bq. --- bq. bq. (Updated 2011-09-14 22:19:21) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. The purpose of this change is to allow publication or storage of counters while the job is running. bq. bq. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. bq. bq. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. bq. bq. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. bq. bq. Made cosmetic changes in two other classes bq. bq. bq. This addresses bug HIVE-2446. bq. https://issues.apache.org/jira/browse/HIVE-2446 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1170867 bq.trunk/conf/hive-default.xml 1170867 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1170867 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1170867 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1170867 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1909/diff bq. bq. bq. Testing bq. --- bq. bq. Run some random tests, and still running the unit tests. bq. bq. bq. Thanks, bq. bq. Robert bq. bq. Introduction of client statistics publishers possibility Key: HIVE-2446 URL: https://issues.apache.org/jira/browse/HIVE-2446 Project: Hive Issue Type: Improvement Components: Clients, Statistics Reporter: Robert Surówka Priority: Minor Attachments: HIVE-2446.1.patch, HIVE-2446.1.patch Original Estimate: 1h Remaining Estimate: 1h The purpose of this change is to allow publication or storage of counters while the job is
[jira] [Commented] (HIVE-2445) The PerfLogger should log the full name of hooks, not just the simple name.
[ https://issues.apache.org/jira/browse/HIVE-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104663#comment-13104663 ] jirapos...@reviews.apache.org commented on HIVE-2445: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1882/ --- Review request for hive and Ning Zhang. Summary --- I change the method called when logging pre, failure, and post hooks to the PerfLogger to getName instead of getSimpleName. This addresses bug HIVE-2445. https://issues.apache.org/jira/browse/HIVE-2445 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1170268 Diff: https://reviews.apache.org/r/1882/diff Testing --- Ran the unit test queries, and verified the intended change in the logs. Thanks, Kevin The PerfLogger should log the full name of hooks, not just the simple name. --- Key: HIVE-2445 URL: https://issues.apache.org/jira/browse/HIVE-2445 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Sometimes the simple name of a hook is not enough to identify it, so the PerfLogger should log the full name instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2445) The PerfLogger should log the full name of hooks, not just the simple name.
[ https://issues.apache.org/jira/browse/HIVE-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104681#comment-13104681 ] jirapos...@reviews.apache.org commented on HIVE-2445: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1882/#review1893 --- Ship it! - Ning On 2011-09-14 16:53:49, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1882/ bq. --- bq. bq. (Updated 2011-09-14 16:53:49) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. I change the method called when logging pre, failure, and post hooks to the PerfLogger to getName instead of getSimpleName. bq. bq. bq. This addresses bug HIVE-2445. bq. https://issues.apache.org/jira/browse/HIVE-2445 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1170268 bq. bq. Diff: https://reviews.apache.org/r/1882/diff bq. bq. bq. Testing bq. --- bq. bq. Ran the unit test queries, and verified the intended change in the logs. bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. The PerfLogger should log the full name of hooks, not just the simple name. --- Key: HIVE-2445 URL: https://issues.apache.org/jira/browse/HIVE-2445 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Sometimes the simple name of a hook is not enough to identify it, so the PerfLogger should log the full name instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2398) Hive server doesn't return schema for 'set' command
[ https://issues.apache.org/jira/browse/HIVE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104738#comment-13104738 ] jirapos...@reviews.apache.org commented on HIVE-2398: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1605/#review1895 --- jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java https://reviews.apache.org/r/1605/#comment4376 Need to assert that the size of the resultset is 0. Might also be good to log the contents of the resultset. - Carl On 2011-09-13 20:03:48, Prasad Mujumdar wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1605/ bq. --- bq. bq. (Updated 2011-09-13 20:03:48) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. The Hive server currently has the framework to process the Cli commands like 'set'. But it returns an empty schema definition for these resultsets. The patch here is to add a getSchema() to the CommandResponse interface. The SetProcessor is modified to build a schema object which is returned with getSchema. bq. HiveServer is modified to call the getSchema for command cases as well. bq. bq. bq. This addresses bug HIVE-2398. bq. https://issues.apache.org/jira/browse/HIVE-2398 bq. bq. bq. Diffs bq. - bq. bq.jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java e744d11 bq. ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorResponse.java 4f7f472 bq.ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java 9ca5ca4 bq.service/src/java/org/apache/hadoop/hive/service/HiveServer.java 95f322a bq. bq. Diff: https://reviews.apache.org/r/1605/diff bq. bq. bq. Testing bq. --- bq. bq. Tested with ODBC program to execute the set command. Ran Hive unit tests. bq. bq. bq. Thanks, bq. bq. Prasad bq. bq. Hive server doesn't return schema for 'set' command --- Key: HIVE-2398 URL: https://issues.apache.org/jira/browse/HIVE-2398 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.7.1, 0.8.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-2398.patch, HIVE-2398.patch The Hive server does process the CLI commands like 'set', 'set -v' sent by ODBC or JDBC clients. But currently only the data is returned to client but not schema for that resultset. This makes it unusable for a ODBC or JDBC client to use this option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2398) Hive server doesn't return schema for 'set' command
[ https://issues.apache.org/jira/browse/HIVE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104817#comment-13104817 ] jirapos...@reviews.apache.org commented on HIVE-2398: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1605/ --- (Updated 2011-09-14 19:41:13.647029) Review request for hive and Carl Steinbach. Changes --- Added check for resultset non empty. The full set -v output would be too long, also most of the rows will different in each test setup. Hence I am suggesting not to do that. Let me know if you disagree. Summary --- The Hive server currently has the framework to process the Cli commands like 'set'. But it returns an empty schema definition for these resultsets. The patch here is to add a getSchema() to the CommandResponse interface. The SetProcessor is modified to build a schema object which is returned with getSchema. HiveServer is modified to call the getSchema for command cases as well. This addresses bug HIVE-2398. https://issues.apache.org/jira/browse/HIVE-2398 Diffs (updated) - jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java e744d11 ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorResponse.java 4f7f472 ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java 9ca5ca4 service/src/java/org/apache/hadoop/hive/service/HiveServer.java 95f322a Diff: https://reviews.apache.org/r/1605/diff Testing --- Tested with ODBC program to execute the set command. Ran Hive unit tests. Thanks, Prasad Hive server doesn't return schema for 'set' command --- Key: HIVE-2398 URL: https://issues.apache.org/jira/browse/HIVE-2398 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.7.1, 0.8.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-2398.patch, HIVE-2398.patch The Hive server does process the CLI commands like 'set', 'set -v' sent by ODBC or JDBC clients. But currently only the data is returned to client but not schema for that resultset. This makes it unusable for a ODBC or JDBC client to use this option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2446) Introduction of client statistics publishers possibility
[ https://issues.apache.org/jira/browse/HIVE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104944#comment-13104944 ] jirapos...@reviews.apache.org commented on HIVE-2446: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1909/ --- Review request for hive and Ning Zhang. Summary --- The purpose of this change is to allow publication or storage of counters while the job is running. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. Made cosmetic changes in two other classes This addresses bug HIVE-2446. https://issues.apache.org/jira/browse/HIVE-2446 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1170867 trunk/conf/hive-default.xml 1170867 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartition.java 1170867 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1170867 trunk/ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 1170867 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/ClientStatsPublisher.java PRE-CREATION Diff: https://reviews.apache.org/r/1909/diff Testing --- Run some random tests, and still running the unit tests. Thanks, Robert Introduction of client statistics publishers possibility Key: HIVE-2446 URL: https://issues.apache.org/jira/browse/HIVE-2446 Project: Hive Issue Type: Improvement Components: Clients, Statistics Reporter: Robert Surówka Priority: Minor Attachments: HIVE-2446.1.patch Original Estimate: 1h Remaining Estimate: 1h The purpose of this change is to allow publication or storage of counters while the job is running. Introduced two new variables to hive-default.xml and HiveConf.java: hive.client.stats.publishers and hive.client.stats.counters. First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters. Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers. Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence. Made cosmetic changes in two other classes -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2398) Hive server doesn't return schema for 'set' command
[ https://issues.apache.org/jira/browse/HIVE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104986#comment-13104986 ] jirapos...@reviews.apache.org commented on HIVE-2398: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1605/#review1903 --- Ship it! +1. Will commit if tests pass. - Carl On 2011-09-14 19:41:13, Prasad Mujumdar wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1605/ bq. --- bq. bq. (Updated 2011-09-14 19:41:13) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. The Hive server currently has the framework to process the Cli commands like 'set'. But it returns an empty schema definition for these resultsets. The patch here is to add a getSchema() to the CommandResponse interface. The SetProcessor is modified to build a schema object which is returned with getSchema. bq. HiveServer is modified to call the getSchema for command cases as well. bq. bq. bq. This addresses bug HIVE-2398. bq. https://issues.apache.org/jira/browse/HIVE-2398 bq. bq. bq. Diffs bq. - bq. bq.jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java e744d11 bq. ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorResponse.java 4f7f472 bq.ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java 9ca5ca4 bq.service/src/java/org/apache/hadoop/hive/service/HiveServer.java 95f322a bq. bq. Diff: https://reviews.apache.org/r/1605/diff bq. bq. bq. Testing bq. --- bq. bq. Tested with ODBC program to execute the set command. Ran Hive unit tests. bq. bq. bq. Thanks, bq. bq. Prasad bq. bq. Hive server doesn't return schema for 'set' command --- Key: HIVE-2398 URL: https://issues.apache.org/jira/browse/HIVE-2398 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.7.1, 0.8.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-2398.patch, HIVE-2398.patch, HIVE-2398.patch.1 The Hive server does process the CLI commands like 'set', 'set -v' sent by ODBC or JDBC clients. But currently only the data is returned to client but not schema for that resultset. This makes it unusable for a ODBC or JDBC client to use this option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1975) insert overwrite directory Not able to insert data with multi level directory path
[
https://issues.apache.org/jira/browse/HIVE-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103799#comment-13103799
]
jirapos...@reviews.apache.org commented on HIVE-1975:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1829/
---
Review request for hive and Yongqiang He.
Summary
---
If insert into multilevel directory like '/HIVEFT25686/chinna/'
(insert overwrite directory '/HIVEFT25686/chinna/' select * from dept_j) is
failing because fs.rename(sourcePath, targetPath) is failing rename to
multilevel directories, so first created the target path.
This addresses bug HIVE-1975.
https://issues.apache.org/jira/browse/HIVE-1975
Diffs
-
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1170044
Diff: https://reviews.apache.org/r/1829/diff
Testing
---
Ran all testcases
Thanks,
chinna
insert overwrite directory Not able to insert data with multi level
directory path
Key: HIVE-1975
URL: https://issues.apache.org/jira/browse/HIVE-1975
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0
Environment: Hadoop 0.20.1, Hive0.5.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Attachments: HIVE-1975.1.patch, HIVE-1975.patch
Below query execution is failed
Ex:
{noformat}
insert overwrite directory '/HIVEFT25686/chinna/' select * from dept_j;
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[
https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103803#comment-13103803
]
jirapos...@reviews.apache.org commented on HIVE-2182:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1759/
---
(Updated 2011-09-13 17:32:55.124065)
Review request for hive and John Sichi.
Changes
---
Patch rebased on trunk
Summary
---
while executing UDF if the implementation jar is not present in MR class path
it is throwing nullpointer exception instead of throwing nullpointer exception
throwing meaning full exception with the required details.
This addresses bug HIVE-2182.
https://issues.apache.org/jira/browse/HIVE-2182
Diffs (updated)
-
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java
1170044
trunk/ql/src/test/queries/clientnegative/udfnull.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/udfnull.q.out PRE-CREATION
trunk/ql/src/test/results/compiler/plan/cast1.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/groupby4.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/groupby6.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/input20.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/input8.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/join2.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1170044
trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1170044
Diff: https://reviews.apache.org/r/1759/diff
Testing
---
Added unit tests
Thanks,
chinna
Avoid null pointer exception when executing UDF
---
Key: HIVE-2182
URL: https://issues.apache.org/jira/browse/HIVE-2182
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0, 0.8.0
Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5)
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch,
HIVE-2182.4.patch, HIVE-2182.5.patch, HIVE-2182.patch
For using UDF's executed following steps
{noformat}
add jar /home/udf/udf.jar;
create temporary function grade as 'udf.Grade';
select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m;
{noformat}
But from the above steps if we miss the first step (add jar) and execute
remaining steps
{noformat}
create temporary function grade as 'udf.Grade';
select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m;
{noformat}
In tasktracker it is throwing this exception
{noformat}
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 18 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
[jira] [Commented] (HIVE-2145) NPE during parsing order-by expression
[ https://issues.apache.org/jira/browse/HIVE-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102832#comment-13102832 ] jirapos...@reviews.apache.org commented on HIVE-2145: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1802/ --- Review request for hive and Ning Zhang. Summary --- While compiling the query if order by contains function throwing an exception.. This addresses bug HIVE-2145. https://issues.apache.org/jira/browse/HIVE-2145 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1167000 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1167000 trunk/ql/src/test/queries/clientnegative/orderby_function.q PRE-CREATION trunk/ql/src/test/results/clientnegative/orderby_function.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1802/diff Testing --- Added unit test case and ran all existing test cases Thanks, chinna NPE during parsing order-by expression -- Key: HIVE-2145 URL: https://issues.apache.org/jira/browse/HIVE-2145 Project: Hive Issue Type: Bug Reporter: Ning Zhang Assignee: Chinna Rao Lalam Attachments: HIVE-2145.patch The following query throws NPE, where it should have throw parsing exception. hive select key, count(1) cnt from src group by key order by count(1) limit 10; select key, count(1) cnt from src group by key order by count(1) limit 10; FAILED: Hive Internal Error: java.lang.NullPointerException(null) java.lang.NullPointerException at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:153) at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:640) at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:761) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102) at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:156) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:6830) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:6788) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genReduceSinkPlan(SemanticAnalyzer.java:4303) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:5461) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:6022) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6607) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:790) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:209) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:286) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:514) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Correct syntax should be: hive select key, count(1) cnt from src group by key order by cnt limit 10; -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2398) Hive server doesn't return schema for 'set' command
[ https://issues.apache.org/jira/browse/HIVE-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102909#comment-13102909 ] jirapos...@reviews.apache.org commented on HIVE-2398: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1605/#review1857 --- Please add a testcase. Also, I tried running 'set -v' from the CLI after setting hive.cli.print.header=true, and I don't see the column name in the output. This seems like a bug? ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java https://reviews.apache.org/r/1605/#comment4245 Checkstyle: static final variable names should appear in ALL CAPS ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java https://reviews.apache.org/r/1605/#comment4248 bld? Might be good to change the name to getSchema. Also, since setColName is a constant there's no need for this method to have an input parameter. ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java https://reviews.apache.org/r/1605/#comment4247 Please add static imports for STRING_TYPE_NAME, SERIALIZATION_NULL_FORMAT, and defaultNullString. service/src/java/org/apache/hadoop/hive/service/HiveServer.java https://reviews.apache.org/r/1605/#comment4249 This comment is no longer valid. Also, it might be cleaner to remove the else clause. - Carl On 2011-08-19 22:37:45, Prasad Mujumdar wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1605/ bq. --- bq. bq. (Updated 2011-08-19 22:37:45) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. The Hive server currently has the framework to process the Cli commands like 'set'. But it returns an empty schema definition for these resultsets. The patch here is to add a getSchema() to the CommandResponse interface. The SetProcessor is modified to build a schema object which is returned with getSchema. bq. HiveServer is modified to call the getSchema for command cases as well. bq. bq. bq. This addresses bug HIVE-2398. bq. https://issues.apache.org/jira/browse/HIVE-2398 bq. bq. bq. Diffs bq. - bq. bq. ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorResponse.java 4f7f472 bq.ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java 97fa1ab bq.service/src/java/org/apache/hadoop/hive/service/HiveServer.java c4b041f bq. bq. Diff: https://reviews.apache.org/r/1605/diff bq. bq. bq. Testing bq. --- bq. bq. Tested with ODBC program to execute the set command. Ran Hive unit tests. bq. bq. bq. Thanks, bq. bq. Prasad bq. bq. Hive server doesn't return schema for 'set' command --- Key: HIVE-2398 URL: https://issues.apache.org/jira/browse/HIVE-2398 Project: Hive Issue Type: Bug Components: Server Infrastructure Affects Versions: 0.7.1, 0.8.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Attachments: HIVE-2398.patch The Hive server does process the CLI commands like 'set', 'set -v' sent by ODBC or JDBC clients. But currently only the data is returned to client but not schema for that resultset. This makes it unusable for a ODBC or JDBC client to use this option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2440) make hive mapper initialize faster when having tons of input files
[ https://issues.apache.org/jira/browse/HIVE-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102935#comment-13102935 ] jirapos...@reviews.apache.org commented on HIVE-2440: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1813/ --- Review request for hive and Ning Zhang. Summary --- make hive mapper initialize faster when having tons of input files This addresses bug hive-2440. https://issues.apache.org/jira/browse/hive-2440 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1167289 Diff: https://reviews.apache.org/r/1813/diff Testing --- Thanks, Yongqiang make hive mapper initialize faster when having tons of input files -- Key: HIVE-2440 URL: https://issues.apache.org/jira/browse/HIVE-2440 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2440.1.patch, HIVE-2440.2.patch when one hive job has tons of input files, a lot of mappers may fail because of slow initialization. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2440) make hive mapper initialize faster when having tons of input files
[ https://issues.apache.org/jira/browse/HIVE-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102950#comment-13102950 ] jirapos...@reviews.apache.org commented on HIVE-2440: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1813/#review1859 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java https://reviews.apache.org/r/1813/#comment4250 do you need childrenPaths? It's only used for adding paths, but nobody is reading it. - Ning On 2011-09-12 19:15:54, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1813/ bq. --- bq. bq. (Updated 2011-09-12 19:15:54) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. make hive mapper initialize faster when having tons of input files bq. bq. bq. This addresses bug hive-2440. bq. https://issues.apache.org/jira/browse/hive-2440 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1167289 bq. bq. Diff: https://reviews.apache.org/r/1813/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. make hive mapper initialize faster when having tons of input files -- Key: HIVE-2440 URL: https://issues.apache.org/jira/browse/HIVE-2440 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2440.1.patch, HIVE-2440.2.patch when one hive job has tons of input files, a lot of mappers may fail because of slow initialization. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-198) Parse errors report incorrectly.
[ https://issues.apache.org/jira/browse/HIVE-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103261#comment-13103261 ] jirapos...@reviews.apache.org commented on HIVE-198: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1818/ --- Review request for hive. Summary --- Modified the message to include the actual token it fails on (When available). It should make it easier to grok the actual problem from the error. This addresses bug HIVE-198. https://issues.apache.org/jira/browse/HIVE-198 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1164307 trunk/ql/src/test/results/clientnegative/column_rename3.q.out 1164307 trunk/ql/src/test/results/clientnegative/invalid_select_expression.q.out 1164307 trunk/ql/src/test/results/clientnegative/invalid_tbl_name.q.out 1164307 trunk/ql/src/test/results/compiler/errors/garbage.q.out 1164307 trunk/ql/src/test/results/compiler/errors/missing_overwrite.q.out 1164307 trunk/ql/src/test/results/compiler/errors/wrong_distinct2.q.out 1164307 Diff: https://reviews.apache.org/r/1818/diff Testing --- ant test -Dtestcase=TestParseNegative Thanks, avivey Parse errors report incorrectly. Key: HIVE-198 URL: https://issues.apache.org/jira/browse/HIVE-198 Project: Hive Issue Type: Bug Components: Query Processor Reporter: S. Alex Smith Assignee: Aviv Eyal Labels: parse Attachments: HIVE-198.2.patch.txt, PraseErrorMessage.patch The following two queries fail: CREATE TABLE output_table(userid, bigint); CREATE TABLE output_table(userid bigint, age int, sex string, location string); each giving the error message FAILED: Parse Error: line 1:16 mismatched input 'TABLE' expecting KW_TEMPORARY Although one might not catch it from the error message, the problem with the first is that there is a comma between userid and bigint, and the problem with the second is that location is a reserved keyword. Reported errors should more accurately describe the nature of the error, such as no type given for column 'userid' or 'location' is not a valid column name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-198) Parse errors report incorrectly.
[ https://issues.apache.org/jira/browse/HIVE-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13103264#comment-13103264 ] jirapos...@reviews.apache.org commented on HIVE-198: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1818/ --- (Updated 2011-09-13 01:39:58.571166) Review request for hive. Summary (updated) --- Modified the ParseException message to include the actual token it fails on (When available). It should make it easier to grok the actual problem from the error. This addresses bug HIVE-198. https://issues.apache.org/jira/browse/HIVE-198 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1164307 trunk/ql/src/test/results/clientnegative/column_rename3.q.out 1164307 trunk/ql/src/test/results/clientnegative/invalid_select_expression.q.out 1164307 trunk/ql/src/test/results/clientnegative/invalid_tbl_name.q.out 1164307 trunk/ql/src/test/results/compiler/errors/garbage.q.out 1164307 trunk/ql/src/test/results/compiler/errors/missing_overwrite.q.out 1164307 trunk/ql/src/test/results/compiler/errors/wrong_distinct2.q.out 1164307 Diff: https://reviews.apache.org/r/1818/diff Testing --- ant test -Dtestcase=TestParseNegative Thanks, avivey Parse errors report incorrectly. Key: HIVE-198 URL: https://issues.apache.org/jira/browse/HIVE-198 Project: Hive Issue Type: Bug Components: Query Processor Reporter: S. Alex Smith Assignee: Aviv Eyal Labels: parse Attachments: HIVE-198.2.patch.txt, PraseErrorMessage.patch The following two queries fail: CREATE TABLE output_table(userid, bigint); CREATE TABLE output_table(userid bigint, age int, sex string, location string); each giving the error message FAILED: Parse Error: line 1:16 mismatched input 'TABLE' expecting KW_TEMPORARY Although one might not catch it from the error message, the problem with the first is that there is a comma between userid and bigint, and the problem with the second is that location is a reserved keyword. Reported errors should more accurately describe the nature of the error, such as no type given for column 'userid' or 'location' is not a valid column name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13102149#comment-13102149 ] jirapos...@reviews.apache.org commented on HIVE-1694: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/ --- (Updated 2011-09-10 21:10:06.178279) Review request for hive and John Sichi. Changes --- Added order-by to queries for test determinism. Summary --- This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. This addresses bug HIVE-1694. https://issues.apache.org/jira/browse/HIVE-1694 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 66ee0be data/files/lineitem.txt PRE-CREATION data/files/tbl.txt PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 5053576 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7a00c00 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java bec8787 ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dcdfb9e ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 699519b ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q PRE-CREATION ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1194/diff Testing --- Thanks, Prajakta Accelerate GROUP BY execution using indexes --- Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Assignee: Prajakta Kalmegh Attachments: HIVE-1694.1.patch.txt, HIVE-1694.2.patch.txt, HIVE-1694.3.patch.txt, HIVE-1694.4.patch, HIVE-1694.5.patch, HIVE-1694.6.patch, HIVE-1694.7.patch, HIVE-1694_2010-10-28.diff, demo_q1.hql, demo_q2.hql The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 on creating separate JIRA issue for tracking index usage in optimizer query execution. The aim of this effort is to use indexes to accelerate query execution (for certain class of queries). E.g. - Filters and range scans (already being worked on by He Yongqiang as part of HIVE-417?) - Joins (index based joins) - Group By, Order By and other misc cases The proposal is multi-step: 1. Building index based operators, compiler and execution engine changes 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose between index scans, full table scans etc.) This JIRA initially focuses on the first step. This JIRA is expected to hold the information about index based plans operator implementations for above mentioned cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2223) support grouping on complex types in Hive
[ https://issues.apache.org/jira/browse/HIVE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101212#comment-13101212 ] jirapos...@reviews.apache.org commented on HIVE-2223: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1745/ --- (Updated 2011-09-09 13:50:30.594056) Review request for hive. Summary --- Adds hash codes for List and Map object inspectors. This addresses bug HIVE-2223. https://issues.apache.org/jira/browse/HIVE-2223 Diffs - Diff: https://reviews.apache.org/r/1745/diff Testing --- Added unittest. Thanks, Jonathan support grouping on complex types in Hive - Key: HIVE-2223 URL: https://issues.apache.org/jira/browse/HIVE-2223 Project: Hive Issue Type: New Feature Reporter: Kate Ting Assignee: Jonathan Chang Priority: Minor Attachments: HIVE-2223.patch Creating a query with a GROUP BY statement when an array type column is part of the column list is not yet supported: CREATE TABLE test_group_by ( key INT, group INT, terms ARRAYSTRING); SELECT key, terms, count(group) FROM test_group_by GROUP BY key, terms; ... Hash code on complex types not supported yet. java.lang.RuntimeException: Error while closing operators at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:232) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:799) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:462) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:470) at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:211) ... 4 more Caused by: java.lang.RuntimeException: Hash code on complex types not supported yet. at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hashCode(ObjectInspectorUtils.java:348) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:187) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598) at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:746) at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:780) ... 9 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[
https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101405#comment-13101405
]
jirapos...@reviews.apache.org commented on HIVE-2182:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1759/
---
(Updated 2011-09-09 18:08:07.528830)
Review request for hive and John Sichi.
Summary
---
while executing UDF if the implementation jar is not present in MR class path
it is throwing nullpointer exception instead of throwing nullpointer exception
throwing meaning full exception with the required details.
This addresses bug HIVE-2182.
https://issues.apache.org/jira/browse/HIVE-2182
Diffs (updated)
-
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java
1167000
trunk/ql/src/test/queries/clientnegative/udfnull.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/udfnull.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1759/diff
Testing
---
Added unit tests
Thanks,
chinna
Avoid null pointer exception when executing UDF
---
Key: HIVE-2182
URL: https://issues.apache.org/jira/browse/HIVE-2182
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0, 0.8.0
Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5)
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.3.patch,
HIVE-2182.patch
For using UDF's executed following steps
{noformat}
add jar /home/udf/udf.jar;
create temporary function grade as 'udf.Grade';
select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m;
{noformat}
But from the above steps if we miss the first step (add jar) and execute
remaining steps
{noformat}
create temporary function grade as 'udf.Grade';
select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m;
{noformat}
In tasktracker it is throwing this exception
{noformat}
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 18 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 18 more
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
... 31 more
{noformat}
Instead of null pointer exception it should throw meaning full exception
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2440) make hive mapper initialize faster when having tons of input files
[ https://issues.apache.org/jira/browse/HIVE-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101563#comment-13101563 ] jirapos...@reviews.apache.org commented on HIVE-2440: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1783/ --- Review request for hive and Ning Zhang. Summary --- when one hive job has tons of input files, a lot of mappers may fail because of slow initialization. This addresses bug HIVE-2440. https://issues.apache.org/jira/browse/HIVE-2440 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1167289 Diff: https://reviews.apache.org/r/1783/diff Testing --- Thanks, Yongqiang make hive mapper initialize faster when having tons of input files -- Key: HIVE-2440 URL: https://issues.apache.org/jira/browse/HIVE-2440 Project: Hive Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2440.1.patch when one hive job has tons of input files, a lot of mappers may fail because of slow initialization. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2402) Function like with empty string is throwing null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100495#comment-13100495 ] jirapos...@reviews.apache.org commented on HIVE-2402: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1757/ --- Review request for hive and John Sichi. Summary --- By default patterntype is complex so it is expecting Pattern object but in this scenario pattern object is null so it is throwing nullpointer exception. Default patterntype can be NONE and in parseSimplePattern() it will be assigned with right type. This addresses bug HIVE-2402. https://issues.apache.org/jira/browse/HIVE-2402 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java 1165244 trunk/ql/src/test/queries/clientpositive/udf_like.q 1165244 trunk/ql/src/test/results/clientpositive/udf_like.q.out 1165244 Diff: https://reviews.apache.org/r/1757/diff Testing --- Added unit testcase. Thanks, chinna Function like with empty string is throwing null pointer exception -- Key: HIVE-2402 URL: https://issues.apache.org/jira/browse/HIVE-2402 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2402.1.patch, HIVE-2402.patch select emp.ename from emp where ename like '' This query is throwing null pointer exception -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2182) Avoid null pointer exception when executing UDF
[
https://issues.apache.org/jira/browse/HIVE-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100631#comment-13100631
]
jirapos...@reviews.apache.org commented on HIVE-2182:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1759/
---
Review request for hive and John Sichi.
Summary
---
while executing UDF if the implementation jar is not present in MR class path
it is throwing nullpointer exception instead of throwing nullpointer exception
throwing meaning full exception with the required details.
This addresses bug HIVE-2182.
https://issues.apache.org/jira/browse/HIVE-2182
Diffs
-
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java
1166865
trunk/ql/src/test/queries/clientnegative/udfnull.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/udfnull.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1759/diff
Testing
---
Added unit tests
Thanks,
chinna
Avoid null pointer exception when executing UDF
---
Key: HIVE-2182
URL: https://issues.apache.org/jira/browse/HIVE-2182
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0, 0.8.0
Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5)
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Attachments: HIVE-2182.1.patch, HIVE-2182.2.patch, HIVE-2182.patch
For using UDF's executed following steps
{noformat}
add jar /home/udf/udf.jar;
create temporary function grade as 'udf.Grade';
select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m;
{noformat}
But from the above steps if we miss the first step (add jar) and execute
remaining steps
{noformat}
create temporary function grade as 'udf.Grade';
select m.userid,m.name,grade(m.maths,m.physics,m.chemistry) from marks m;
{noformat}
In tasktracker it is throwing this exception
{noformat}
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 18 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:133)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:878)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:904)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 18 more
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
... 31 more
{noformat}
Instead of null pointer exception it should throw meaning full exception
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100839#comment-13100839 ] jirapos...@reviews.apache.org commented on HIVE-1694: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/ --- (Updated 2011-09-09 01:14:16.218940) Review request for hive and John Sichi. Summary --- This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. This addresses bug HIVE-1694. https://issues.apache.org/jira/browse/HIVE-1694 Diffs (updated) - ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out PRE-CREATION ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 699519b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dcdfb9e ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 66ee0be data/files/lineitem.txt PRE-CREATION data/files/tbl.txt PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 5053576 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7a00c00 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java bec8787 ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION Diff: https://reviews.apache.org/r/1194/diff Testing --- Thanks, Prajakta Accelerate GROUP BY execution using indexes --- Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Assignee: Prajakta Kalmegh Attachments: HIVE-1694.1.patch.txt, HIVE-1694.2.patch.txt, HIVE-1694.3.patch.txt, HIVE-1694.4.patch, HIVE-1694.5.patch, HIVE-1694_2010-10-28.diff, demo_q1.hql, demo_q2.hql The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 on creating separate JIRA issue for tracking index usage in optimizer query execution. The aim of this effort is to use indexes to accelerate query execution (for certain class of queries). E.g. - Filters and range scans (already being worked on by He Yongqiang as part of HIVE-417?) - Joins (index based joins) - Group By, Order By and other misc cases The proposal is multi-step: 1. Building index based operators, compiler and execution engine changes 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose between index scans, full table scans etc.) This JIRA initially focuses on the first step. This JIRA is expected to hold the information about index based plans operator implementations for above mentioned cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2412) Update Eclipse configuration to include Mockito dependency
[ https://issues.apache.org/jira/browse/HIVE-2412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098750#comment-13098750 ] jirapos...@reviews.apache.org commented on HIVE-2412: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1665/#review1782 --- Ship it! +1 - Amareshwari On 2011-08-26 02:12:54, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1665/ bq. --- bq. bq. (Updated 2011-08-26 02:12:54) bq. bq. bq. Review request for hive and Amareshwari Sriramadasu. bq. bq. bq. Summary bq. --- bq. bq. This patch does two things: bq. bq. * Fixes the eclipse project generation code by including mockito bq.on the classpath bq. bq. * Updates the HiveCli launch configuration template so that the bq.metastore JAR is no longer on the classpath, and also moves the bq.hive-exec JAR to the end of the classpath. This was done in bq.order to give preference on the classpath to local changes, bq.which up to this point have been obscured by hive-exec and bq.hive-metastore JAR files located in the build directory. Note bq.that in order to run the HiveCLI launchtemplate you now need to bq.have the Datanucleus Eclipse Plugin installed, and bq.must enhance the project. bq. bq. bq. This addresses bug HIVE-2412. bq. https://issues.apache.org/jira/browse/HIVE-2412 bq. bq. bq. Diffs bq. - bq. bq.eclipse-templates/.classpath b59b343 bq.eclipse-templates/HiveCLI.launchtemplate 0fa86cb bq. bq. Diff: https://reviews.apache.org/r/1665/diff bq. bq. bq. Testing bq. --- bq. bq. Verified that project generation and import works. Also verified that HiveCLI launch template works. bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Update Eclipse configuration to include Mockito dependency -- Key: HIVE-2412 URL: https://issues.apache.org/jira/browse/HIVE-2412 Project: Hive Issue Type: Bug Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.9.0 Attachments: HIVE-2412.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098986#comment-13098986 ] jirapos...@reviews.apache.org commented on HIVE-1694: - bq. On 2011-08-05 21:20:21, John Sichi wrote: bq. ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java, line 172 bq. https://reviews.apache.org/r/1194/diff/2/?file=30443#file30443line172 bq. bq. See recent changes in corresponding CompactIndexHandler code for HIVEOPTINDEXFILTER; need the same here (or better, factor out common code here and elsewhere). bq. bq. On a related note, you may be able to use the same technique instead of isQueryInsertToTable; this would be preferable since it's nice to be able to use the index rewrite in cases where it's a normal INSERT table with index being used for GROUP BY on SELECT from some other table. bq. I have factored out the common code in all Index handler classes and placed it in IndexUtils file. I also removed the code for isQueryInsertToTable and am setting the HIVEOPTINDEXFILTER to false instead. bq. On 2011-08-05 21:20:21, John Sichi wrote: bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java, line 153 bq. https://reviews.apache.org/r/1194/diff/2/?file=30449#file30449line153 bq. bq. Shouldn't this be the same as COUNT(*)? bq. Yes it is. I missed to change this part from the previous code. - Prajakta --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/#review1303 --- On 2011-08-03 10:31:42, Prajakta Kalmegh wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1194/ bq. --- bq. bq. (Updated 2011-08-03 10:31:42) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. bq. bq. bq. This addresses bug HIVE-1694. bq. https://issues.apache.org/jira/browse/HIVE-1694 bq. bq. bq. Diffs bq. - bq. bq.common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f bq.ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff bq.ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a57f9cf bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 8295687 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 699519b bq.ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q PRE-CREATION bq.ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1194/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Prajakta bq. bq. Accelerate GROUP BY execution using indexes --- Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Assignee: Prajakta Kalmegh Attachments: HIVE-1694.1.patch.txt, HIVE-1694.2.patch.txt, HIVE-1694.3.patch.txt, HIVE-1694.4.patch, HIVE-1694.5.patch, HIVE-1694_2010-10-28.diff, demo_q1.hql, demo_q2.hql The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at
[jira] [Commented] (HIVE-2430) Performance degradation in stats DB after JIRA HIVE-2144 (https://issues.apache.org/jira/browse/HIVE-2144)
[ https://issues.apache.org/jira/browse/HIVE-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099178#comment-13099178 ] jirapos...@reviews.apache.org commented on HIVE-2430: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1737/ --- Review request for hive and Ning Zhang. Summary --- I considered two different strategies for handling duplicates after removing the primary key from the table. 1) Go back to performing a select, and then updating if a row for the file exists, or inserting a new record otherwise. 2) Always insert records and then during aggregation get the max value for each statistic with a group by on the file name, and then aggregate those statistics. This diff contains the code for option 2. I determined this to be the better option by adding a couple stress tests to TestStatsPublisherEnhanced, and then comparing the run times for the two implementations using derby and MySQL. The two tests checked the performance when inserting a couple hundred rows for each of two files, and inserting several hundred rows, each for a different file. In each case, when i ran the tests on my machine there wasn't much difference for derby, but for MySQL I was seeing both tests run about 100 ms faster for MySQL. I ran both tests several times, to confirm what I was seeing. Note that previously, if statistics were added for a file, and then statistics were added again for that same file, but missing some number of values, those missing values were erased from the row. With this new implementation the old values for those missing statistics will be used. This case will probably never happen in the field. This addresses bug HIVE-2430. https://issues.apache.org/jira/browse/HIVE-2430 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 1165899 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsSetupConstants.java 1165899 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.java 1165899 trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisherEnhanced.java 1165899 Diff: https://reviews.apache.org/r/1737/diff Testing --- I added two new stress tests to TestStatsPublisherEnhanced. I also modified one of the tests to reflect the modified behavior described in the Description. I ran the unit test queries as well. Thanks, Kevin Performance degradation in stats DB after JIRA HIVE-2144 (https://issues.apache.org/jira/browse/HIVE-2144) -- Key: HIVE-2430 URL: https://issues.apache.org/jira/browse/HIVE-2430 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2430.1.patch.txt After JIRA HIVE-2144 (https://issues.apache.org/jira/browse/HIVE-2144), the performance in stats DB degraded significantly due to MySQL's inefficient index maintenance. We should remove the primary index introduced in that JIRA and resolve duplicates in the stats aggregation phase. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2429) skip corruption bug that cause data not decompressed
[ https://issues.apache.org/jira/browse/HIVE-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099255#comment-13099255 ] jirapos...@reviews.apache.org commented on HIVE-2429: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1738/ --- Review request for Yongqiang He. Summary --- Fix for lazy decompression This addresses bug HIVE-2429. https://issues.apache.org/jira/browse/HIVE-2429 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java 1166283 trunk/ql/src/test/queries/clientpositive/rcfile_toleratecorruptions.q PRE-CREATION trunk/ql/src/test/results/clientpositive/rcfile_toleratecorruptions.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1738/diff Testing --- Added unit test Thanks, Ramkumar skip corruption bug that cause data not decompressed Key: HIVE-2429 URL: https://issues.apache.org/jira/browse/HIVE-2429 Project: Hive Issue Type: Bug Reporter: He Yongqiang Assignee: Ramkumar Vadali Attachments: HIVE-2429.patch This is a regression of https://issues.apache.org/jira/browse/HIVE-2404 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2397) Support with rollup option for group by
[ https://issues.apache.org/jira/browse/HIVE-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099593#comment-13099593 ] jirapos...@reviews.apache.org commented on HIVE-2397: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1637/ --- (Updated 2011-09-07 21:47:48.620408) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Changes --- Two changes: 1) Added groupby_withrollup.q.out, but forgot to add groupby_withrollup.q 2) I modified the method modifyAggregations in GroupbyOperator to take a list of AggregationBuffer[]'s. This makes it so that we only need to evaluate each row of input once, and then we can update all the rollup aggregations at once, instead of reevaluating the same row of input for each rollup row. Summary --- If a user runs a query that includes group by ... with rollup, the behavior is like that of MySQL, see the task. I had to implement 4 different ways of providing this behavior to fit in with the 4 different ways of implementing group by depending on whether map aggregation is allowed, and whether the data is known to be skewed. If map aggregation is allowed, it is a simple matter of adding new keys with an increasing number of NULLs to the hash map to collect the data for the new rows as part of the map side hash aggregation. If map aggregation is not allowed and the data is not skewed, I perform a reduce job which performs a hash aggregation very similar to the way it is performed on the map side. I then perform a mergepartial reduce job to perform a final aggregation on the hash aggregation. If map aggregation is not allowed and the data is skewed and there are no distinct aggregations for the group by, I aggregate data for the new rows with NULLs as part of the non-hash aggregation. This was as simple as adding the new functionality to the map side hash aggregation. If map aggregation is not allowed and the data is skewed and there are distinct aggregations fro the group by, I perform a reduce job which performs a hash aggregation, and then use the same implementation that is used when there is no rollup option set. I have done my best not to detract from any optimizations that were made for each of the four different implementations of group by, but, r for the ones where I add a new reduce job, I am not sure how successful I was. Currently, the optimizations for multiple group bys is not supported for queries with the rollup option set, but I am continuing to look into this. This addresses bug HIVE-2397. https://issues.apache.org/jira/browse/HIVE-2397 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1166376 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java 1166376 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 1166376 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1166376 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1166376 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 1166376 trunk/ql/src/test/queries/clientpositive/groupby10_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby11_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_nomap_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_withrollup.q PRE-CREATION
[jira] [Commented] (HIVE-2431) upgrading thrift version didn't upgrade libthrift.jar symlink correctly
[ https://issues.apache.org/jira/browse/HIVE-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099678#comment-13099678 ] jirapos...@reviews.apache.org commented on HIVE-2431: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1740/ --- Review request for hive and Carl Steinbach. Summary --- simple change to fix the libfb303.jar and libthrift.jar symlinks. This addresses bug HIVE-2431. https://issues.apache.org/jira/browse/HIVE-2431 Diffs - trunk/build.xml 1166476 Diff: https://reviews.apache.org/r/1740/diff Testing --- ant clean package then check build/dist/lib for the correctness. Thanks, Ning upgrading thrift version didn't upgrade libthrift.jar symlink correctly --- Key: HIVE-2431 URL: https://issues.apache.org/jira/browse/HIVE-2431 Project: Hive Issue Type: Bug Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-2431.patch libthrift.jar and libfb303.jar are symlinks to the current thrift version. With the upgrade to 0.7, there's a bug in the symlink creation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2416) Multiple distinct function to support hive.groupby.skewindata optimization
[
https://issues.apache.org/jira/browse/HIVE-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099914#comment-13099914
]
jirapos...@reviews.apache.org commented on HIVE-2416:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1741/
---
(Updated 2011-09-08 04:52:29.083703)
Review request for hive and Amareshwari Sriramadasu.
Changes
---
Update BUGs field to link JIRA
Summary
---
Currently when multiple distinct function is used, hive.groupby.skewindata
optimization parameter shall be set false, or else an exception is raised:
Error in semantic analysis: DISTINCT on different columns not supported with
skew in data
Skew groupby should support multiple distinct function
This addresses bug HIVE-2416.
https://issues.apache.org/jira/browse/HIVE-2416
Diffs
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/groupby2_map_skew_multi_distinct.q
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/groupby3_multi_distinct.q
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/groupby2_map_skew_multi_distinct.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/groupby2_map_skew_multi_distinct.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/groupby3_map_skew_multi_distinct.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/groupby3_multi_distinct.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby2.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby2_map_skew.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby2_map_skew_multi_distinct.q.out
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby3.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby3_map_skew.q.out
1162620
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/nullgroup4.q.out
1162620
Diff: https://reviews.apache.org/r/1741/diff
Testing
---
All UT passed
Thanks,
Ted
Multiple distinct function to support hive.groupby.skewindata optimization
--
Key: HIVE-2416
URL: https://issues.apache.org/jira/browse/HIVE-2416
Project: Hive
Issue Type: Improvement
Reporter: Ted Xu
Assignee: Ted Xu
Attachments: multi_distinct_skew.patch
Currently when multiple distinct function is used, hive.groupby.skewindata
optimization parameter shall be set false, or else an exception is raised:
{code}
Error in semantic analysis: DISTINCT on different columns not supported with
skew in data
{code}
Skew groupby should support multiple distinct function
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2431) upgrading thrift version didn't upgrade libthrift.jar symlink correctly
[ https://issues.apache.org/jira/browse/HIVE-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100078#comment-13100078 ] jirapos...@reviews.apache.org commented on HIVE-2431: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1740/#review1811 --- Ship it! +1 - Amareshwari On 2011-09-08 00:25:14, Ning Zhang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1740/ bq. --- bq. bq. (Updated 2011-09-08 00:25:14) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. simple change to fix the libfb303.jar and libthrift.jar symlinks. bq. bq. bq. This addresses bug HIVE-2431. bq. https://issues.apache.org/jira/browse/HIVE-2431 bq. bq. bq. Diffs bq. - bq. bq.trunk/build.xml 1166476 bq. bq. Diff: https://reviews.apache.org/r/1740/diff bq. bq. bq. Testing bq. --- bq. bq. ant clean package then check build/dist/lib for the correctness. bq. bq. bq. Thanks, bq. bq. Ning bq. bq. upgrading thrift version didn't upgrade libthrift.jar symlink correctly --- Key: HIVE-2431 URL: https://issues.apache.org/jira/browse/HIVE-2431 Project: Hive Issue Type: Bug Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-2431.patch libthrift.jar and libfb303.jar are symlinks to the current thrift version. With the upgrade to 0.7, there's a bug in the symlink creation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2196) Ensure HiveConf includes all properties defined in hive-default.xml
[ https://issues.apache.org/jira/browse/HIVE-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13097758#comment-13097758 ] jirapos...@reviews.apache.org commented on HIVE-2196: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1423/ --- (Updated 2011-09-06 06:16:26.733090) Review request for hive, Carl Steinbach and John Sichi. Changes --- Reworked on the patch and latest patch taken care the below issues 1) Configuration property values verified and made it sync with the hive-default.xml 2) Compilation failure corrected. Summary --- The below configurations are not used in the code base so removed from the hive-default.xml hive.mapjoin.hashtable.initialCapacity hive.mapjoin.hashtable.loadfactor hive.mapjoin.smalltable.filesize hive.optimize.pruner hive.stats.jdbc.atomic hive.concurrency.manager The below configurations are not used in the code base but these configurations are required for the system so need to maintain in the hive-default.xml and no need to add in the HiveConf.java javax.jdo.option.ConnectionDriverName javax.jdo.PersistenceManagerFactoryClass javax.jdo.option.DetachAllOnCommit javax.jdo.option.NonTransactionalRead javax.jdo.option.ConnectionUserName fs.har.impl This addresses bug HIVE-2196. https://issues.apache.org/jira/browse/HIVE-2196 Diffs (updated) - trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestRemoteHiveMetaStore.java 1165244 trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java 1165244 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMarkPartitionRemote.java 1165244 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreAuthorization.java 1165244 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java 1165244 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1165244 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1165244 trunk/conf/hive-default.xml 1165244 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1165244 Diff: https://reviews.apache.org/r/1423/diff Testing --- All unit tests passed Thanks, chinna Ensure HiveConf includes all properties defined in hive-default.xml --- Key: HIVE-2196 URL: https://issues.apache.org/jira/browse/HIVE-2196 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0 Reporter: Carl Steinbach Assignee: Chinna Rao Lalam Attachments: HIVE-2196.1.patch, HIVE-2196.2.patch, HIVE-2196.3.patch, HIVE-2196.build.log, HIVE-2196.patch There are a bunch of properties that are defined in hive-default.xml but not in HiveConf. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2427) Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.
[ https://issues.apache.org/jira/browse/HIVE-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098211#comment-13098211 ] jirapos...@reviews.apache.org commented on HIVE-2427: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ --- (Updated 2011-09-06 17:41:24.308161) Review request for hive and Siying Dong. Changes --- Added the new property to hive-default.xml Summary --- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427. https://issues.apache.org/jira/browse/HIVE-2427 Diffs (updated) - trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293 trunk/conf/hive-default.xml 1164293 Diff: https://reviews.apache.org/r/1705/diff Testing --- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin Warn user that precision is lost when bigint is implicitly cast to double in joins and unions. -- Key: HIVE-2427 URL: https://issues.apache.org/jira/browse/HIVE-2427 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2427.1.patch.txt, HIVE-2427.2.patch.txt When a bigint is implicitly cast to a double (when a bigint is involved in an equality expression with a string or double as part of a join condition, or when a bigint column is to be combined with a string or double column as part of a union) precision may be lost, resulting in unexpected behavior. Until we fix the underlying issue we should throw an error in strict mode, and a warning in nonstrict mode alerting the user about this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2405) get_privilege does not get user level privilege
[
https://issues.apache.org/jira/browse/HIVE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098409#comment-13098409
]
jirapos...@reviews.apache.org commented on HIVE-2405:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1646/#review1774
---
trunk/ql/src/test/queries/clientpositive/authorization_7.q
https://reviews.apache.org/r/1646/#comment4051
yes. what we want to do is to test the global privilege. The first command
is to grant a global privilege, and after that do test with that.
- Yongqiang
On 2011-08-24 22:23:39, Yongqiang He wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1646/
bq. ---
bq.
bq. (Updated 2011-08-24 22:23:39)
bq.
bq.
bq. Review request for hive and Ning Zhang.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. get_privilege does not get user level privilege and should return
emptylist instead of null
bq.
bq.
bq. This addresses bug HIVE-2405.
bq. https://issues.apache.org/jira/browse/HIVE-2405
bq.
bq.
bq. Diffs
bq. -
bq.
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
1161232
bq.trunk/ql/src/test/queries/clientpositive/authorization_7.q PRE-CREATION
bq.trunk/ql/src/test/results/clientpositive/authorization_7.q.out
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1646/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq.
bq. Thanks,
bq.
bq. Yongqiang
bq.
bq.
get_privilege does not get user level privilege
---
Key: HIVE-2405
URL: https://issues.apache.org/jira/browse/HIVE-2405
Project: Hive
Issue Type: Bug
Reporter: He Yongqiang
Assignee: He Yongqiang
Attachments: HIVE-2405.1.patch, HIVE-2405.2.patch
hive set hive.security.authorization.enabled=true;
hive grant all to user heyongqiang;
hive show grant user heyongqiang;
principalName heyongqiang
principalType USER
privilege All
grantTime Wed Aug 24 11:51:54 PDT 2011
grantor heyongqiang
Time taken: 0.032 seconds
hive CREATE TABLE src (foo INT, bar STRING);
Authorization failed:No privilege 'Create' found for outputs {
database:default}. Use show grant to get more details.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2428) ADD ARCHIVE has no effect when query runs in local mode
[ https://issues.apache.org/jira/browse/HIVE-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098488#comment-13098488 ] jirapos...@reviews.apache.org commented on HIVE-2428: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1731/ --- Review request for hive, Scott Chen and Ning Zhang. Summary --- Added the -archives flag to the command run when a map reduce task is run locally. It runs essentially exactly as it does for the -files flag. This addresses bug HIVE-2428. https://issues.apache.org/jira/browse/HIVE-2428 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 1164711 Diff: https://reviews.apache.org/r/1731/diff Testing --- I run the unit test queries to verify nothing was broken. I also ran the query specified in the issues description and verified that information was returned for both files and archives in both local and non-local mode. Thanks, Kevin ADD ARCHIVE has no effect when query runs in local mode --- Key: HIVE-2428 URL: https://issues.apache.org/jira/browse/HIVE-2428 Project: Hive Issue Type: Bug Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2428.1.patch.txt When you ADD ARCHIVE a file and the query runs in local mode, the file is not available at all. For example running the commands echo '#!/bin/bash\nls -l 12;read' foo.sh hive -e 'ADD FILE foo.sh; ADD ARCHIVE /tmp/mapreduce_test.zip; SELECT TRANSFORM(*) USING sh foo.sh FROM src' when the query is allowed to run in local mode, only information for foo.sh is returned. However, if the query is not allowed to run in local mode, information for both foo.sh and mapreduce_test.zip is returned. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2426) Test that views with joins work properly
[
https://issues.apache.org/jira/browse/HIVE-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098500#comment-13098500
]
jirapos...@reviews.apache.org commented on HIVE-2426:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1732/
---
Review request for hive.
Summary
---
HIVE-2426 patch 3
This addresses bug HIVE-2426.
https://issues.apache.org/jira/browse/HIVE-2426
Diffs
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java
1165909
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/join_view.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join_view.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1732/diff
Testing
---
Thanks,
John
Test that views with joins work properly
Key: HIVE-2426
URL: https://issues.apache.org/jira/browse/HIVE-2426
Project: Hive
Issue Type: Test
Reporter: Charles Chen
Assignee: Charles Chen
Fix For: 0.9.0
Attachments: HIVE-2426.3.patch, HIVE-2426v2.patch
With the testcase
{noformat}
drop table invites;
drop table invites2;
create table invites (foo int, bar string) partitioned by (ds string);
create table invites2 (foo int, bar string) partitioned by (ds string);
set hive.mapred.mode=strict;
-- test join views: see HIVE-1989
create view v as select invites.bar, invites2.foo, invites2.ds from invites
join invites2 on invites.ds=invites2.ds;
explain select * from v where ds='2011-09-01';
drop view v;
drop table invites;
drop table invites2;
{noformat}
We should not have the partition pruner complain about invites.ds not having
a predicate because the predicate invites2.ds='2011-09-01' will be inferred
with the ppd transitivity optimization
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2427) Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.
[ https://issues.apache.org/jira/browse/HIVE-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098526#comment-13098526 ] jirapos...@reviews.apache.org commented on HIVE-2427: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ --- (Updated 2011-09-07 01:33:36.605379) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Summary --- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427. https://issues.apache.org/jira/browse/HIVE-2427 Diffs - trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293 trunk/conf/hive-default.xml 1164293 Diff: https://reviews.apache.org/r/1705/diff Testing --- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin Warn user that precision is lost when bigint is implicitly cast to double in joins and unions. -- Key: HIVE-2427 URL: https://issues.apache.org/jira/browse/HIVE-2427 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2427.1.patch.txt, HIVE-2427.2.patch.txt When a bigint is implicitly cast to a double (when a bigint is involved in an equality expression with a string or double as part of a join condition, or when a bigint column is to be combined with a string or double column as part of a union) precision may be lost, resulting in unexpected behavior. Until we fix the underlying issue we should throw an error in strict mode, and a warning in nonstrict mode alerting the user about this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2374) Make compression used between map reduce tasks configurable.
[ https://issues.apache.org/jira/browse/HIVE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098525#comment-13098525 ] jirapos...@reviews.apache.org commented on HIVE-2374: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1516/ --- (Updated 2011-09-07 01:34:05.392697) Review request for hive, Yongqiang He and Ning Zhang. Summary --- I added a field to MapredWork and MapredLocalWork which indicates whether it is intermediate or not. By intermediate, I mean that if the query is an insert, there is at least one other map reduce task that is guaranteed to happen before the move. If the query is not an insert, intermediate applies to them all. I determine this by defaulting the flag to true, and setting it to false when the tasks to move the data into a table or file are generated. If the work for a map reduce task (local or otherwise) is intermediate, then we set the compression to be used on the output of the reduce to some configured value, the default is LZO. This addresses bug HIVE-2374. https://issues.apache.org/jira/browse/HIVE-2374 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164667 trunk/conf/hive-default.xml 1164667 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1164667 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java 1164667 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1164667 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 1164667 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 1164667 trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 1164667 trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyIsIntermediateHook.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/intermediate_compression.q PRE-CREATION trunk/ql/src/test/results/clientpositive/auto_join0.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join10.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join11.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join12.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join13.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join15.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join18.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join2.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join20.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join21.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join22.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join23.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join24.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join26.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join27.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join28.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join29.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join30.q.out 1164667 trunk/ql/src/test/results/clientpositive/auto_join31.q.out 1164667 trunk/ql/src/test/results/clientpositive/cluster.q.out 1164667 trunk/ql/src/test/results/clientpositive/ctas.q.out 1164667 trunk/ql/src/test/results/clientpositive/filter_join_breaktask.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby1.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby10.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby11.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby1_limit.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby1_map_skew.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby2_map_skew.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby3.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby3_map_skew.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby4.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby5.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby6.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby6_map_skew.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby8.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby8_map.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby8_map_skew.q.out 1164667 trunk/ql/src/test/results/clientpositive/groupby8_noskew.q.out 1164667
[jira] [Commented] (HIVE-2397) Support with rollup option for group by
[ https://issues.apache.org/jira/browse/HIVE-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098530#comment-13098530 ] jirapos...@reviews.apache.org commented on HIVE-2397: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1637/ --- (Updated 2011-09-07 01:34:25.870854) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Summary --- If a user runs a query that includes group by ... with rollup, the behavior is like that of MySQL, see the task. I had to implement 4 different ways of providing this behavior to fit in with the 4 different ways of implementing group by depending on whether map aggregation is allowed, and whether the data is known to be skewed. If map aggregation is allowed, it is a simple matter of adding new keys with an increasing number of NULLs to the hash map to collect the data for the new rows as part of the map side hash aggregation. If map aggregation is not allowed and the data is not skewed, I perform a reduce job which performs a hash aggregation very similar to the way it is performed on the map side. I then perform a mergepartial reduce job to perform a final aggregation on the hash aggregation. If map aggregation is not allowed and the data is skewed and there are no distinct aggregations for the group by, I aggregate data for the new rows with NULLs as part of the non-hash aggregation. This was as simple as adding the new functionality to the map side hash aggregation. If map aggregation is not allowed and the data is skewed and there are distinct aggregations fro the group by, I perform a reduce job which performs a hash aggregation, and then use the same implementation that is used when there is no rollup option set. I have done my best not to detract from any optimizations that were made for each of the four different implementations of group by, but, r for the ones where I add a new reduce job, I am not sure how successful I was. Currently, the optimizations for multiple group bys is not supported for queries with the rollup option set, but I am continuing to look into this. This addresses bug HIVE-2397. https://issues.apache.org/jira/browse/HIVE-2397 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1160895 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapper.java 1160895 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/KeyWrapperFactory.java 1160895 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1160895 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1160895 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 1160895 trunk/ql/src/test/queries/clientpositive/groupby10_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby11_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_nomap_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby1_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_limit_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_multi_distinct_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby2_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby7_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_map_skew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_map_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_noskew_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby8_withrollup.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/groupby9_withrollup.q PRE-CREATION
[jira] [Commented] (HIVE-2376) Upgrade Hive's Thrift dependency to version 0.7.0
[ https://issues.apache.org/jira/browse/HIVE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096997#comment-13096997 ] jirapos...@reviews.apache.org commented on HIVE-2376: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1695/#review1745 --- Ship it! +1 Running tests - Amareshwari On 2011-09-01 22:08:47, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1695/ bq. --- bq. bq. (Updated 2011-09-01 22:08:47) bq. bq. bq. Review request for hive and Amareshwari Sriramadasu. bq. bq. bq. Summary bq. --- bq. bq. * Upgrade Hive's Thrift dependency to version 0.7.0 bq. * Resolve Thrift dependency using Ivy bq. bq. bq. This addresses bug HIVE-2376. bq. https://issues.apache.org/jira/browse/HIVE-2376 bq. bq. bq. Diffs bq. - bq. bq.service/src/java/org/apache/hadoop/hive/service/HiveServer.java c4b041f bq.shims/ivy.xml 88fae44 bq.metastore/ivy.xml 7647b37 bq.metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 285908e bq.ql/build.xml 15a58ef bq.ql/ivy.xml 99f3786 bq.serde/ivy.xml 01c4d0c bq.service/ivy.xml 7254a09 bq.build.xml f20cbb6 bq.cli/ivy.xml abe7232 bq.eclipse-templates/.classpath 74e5542 bq.ivy/libraries.properties 4236df1 bq.jdbc/ivy.xml ec26e74 bq.lib/README 6cc2fba bq.lib/thrift-fb303.LICENSE 4eacb64 bq.lib/thrift.LICENSE 9d189ef bq. bq. Diff: https://reviews.apache.org/r/1695/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Upgrade Hive's Thrift dependency to version 0.7.0 - Key: HIVE-2376 URL: https://issues.apache.org/jira/browse/HIVE-2376 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.8.0 Attachments: HIVE-2376-complete.1.patch.txt, HIVE-2376-complete.2.patch.txt, HIVE-2376-no-gen.1.patch.txt, HIVE-2376-no-gen.2.patch.txt, libfb303-0.7.0.jar, libthrift-0.7.0.jar -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1989) recognize transitivity of predicates on join keys
[ https://issues.apache.org/jira/browse/HIVE-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095797#comment-13095797 ] jirapos...@reviews.apache.org commented on HIVE-1989: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-09-02 06:26:31.707450) Review request for hive. Changes --- Rebase to trunk Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164353 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join8.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1164353 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1164353
[jira] [Commented] (HIVE-1989) recognize transitivity of predicates on join keys
[ https://issues.apache.org/jira/browse/HIVE-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096118#comment-13096118 ] jirapos...@reviews.apache.org commented on HIVE-1989: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-09-02 16:49:23.015066) Review request for hive. Changes --- Fixed unit tests Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164358 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join8.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/outer_join_ppr.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join3.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_union.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1164358 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1164358
[jira] [Commented] (HIVE-2427) Warn user that precision is lost when bigint is implicitly cast to double in joins and unions.
[ https://issues.apache.org/jira/browse/HIVE-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096127#comment-13096127 ] jirapos...@reviews.apache.org commented on HIVE-2427: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1705/ --- Review request for hive and Siying Dong. Summary --- I added checks before the type conversions for both joins and unions to either throw an error or a warning depending if a bigint was going to be converted to a double. I also added a new variable hive.mapred.bigint.comparison.mode, which is set to either strict or nonstrict to indicate whether an error or a warning should be given when a bigint is converted to a double. This is instead of the original implementation which used the variable hive.mapred.mode. This addresses bug HIVE-2427. https://issues.apache.org/jira/browse/HIVE-2427 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1164293 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_double_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_double_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint.q 1164293 trunk/ql/src/test/queries/clientnegative/compare_string_bigint_join.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/compare_string_bigint_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/filter_join_breaktask2.q 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_double_bigint_join.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_double_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint.q.out 1164293 trunk/ql/src/test/results/clientnegative/compare_string_bigint_join.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/compare_string_bigint_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask2.q.out 1164293 Diff: https://reviews.apache.org/r/1705/diff Testing --- I added several more tests to demonstrate the errors. I also modified a unit test in which a join was being done which involved a comparison between a string and a bigint. I set hive.mapred.bigint.comparison.mode to nonstrict, so the only change was the new warning appeared in the output. Thanks, Kevin Warn user that precision is lost when bigint is implicitly cast to double in joins and unions. -- Key: HIVE-2427 URL: https://issues.apache.org/jira/browse/HIVE-2427 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2427.1.patch.txt When a bigint is implicitly cast to a double (when a bigint is involved in an equality expression with a string or double as part of a join condition, or when a bigint column is to be combined with a string or double column as part of a union) precision may be lost, resulting in unexpected behavior. Until we fix the underlying issue we should throw an error in strict mode, and a warning in nonstrict mode alerting the user about this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1989) recognize transitivity of predicates on join keys
[ https://issues.apache.org/jira/browse/HIVE-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095473#comment-13095473 ] jirapos...@reviews.apache.org commented on HIVE-1989: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-09-01 17:58:25.969663) Review request for hive. Changes --- Fix unit tests Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1163909 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_transitivity.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join8.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_transitivity.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/input4.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1163909 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1163909
[jira] [Commented] (HIVE-1989) recognize transitivity of predicates on join keys
[ https://issues.apache.org/jira/browse/HIVE-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095619#comment-13095619 ] jirapos...@reviews.apache.org commented on HIVE-1989: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-09-01 21:31:00.970058) Review request for hive. Changes --- Changed extractFinalCandidates() to decompose AND expression into parts so the parts can be replicated over transitively even if the whole thing can't. Added comments. Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/union.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/udf_when.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/udf_case.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/subq.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/input4.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_transitivity.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out 1163905
[jira] [Commented] (HIVE-2376) Upgrade Hive's Thrift dependency to version 0.6.1
[ https://issues.apache.org/jira/browse/HIVE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095639#comment-13095639 ] jirapos...@reviews.apache.org commented on HIVE-2376: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1695/ --- Review request for hive and Amareshwari Sriramadasu. Summary --- * Upgrade Hive's Thrift dependency to version 0.7.0 * Resolve Thrift dependency using Ivy This addresses bug HIVE-2376. https://issues.apache.org/jira/browse/HIVE-2376 Diffs - service/src/java/org/apache/hadoop/hive/service/HiveServer.java c4b041f shims/ivy.xml 88fae44 metastore/ivy.xml 7647b37 metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 285908e ql/build.xml 15a58ef ql/ivy.xml 99f3786 serde/ivy.xml 01c4d0c service/ivy.xml 7254a09 build.xml f20cbb6 cli/ivy.xml abe7232 eclipse-templates/.classpath 74e5542 ivy/libraries.properties 4236df1 jdbc/ivy.xml ec26e74 lib/README 6cc2fba lib/thrift-fb303.LICENSE 4eacb64 lib/thrift.LICENSE 9d189ef Diff: https://reviews.apache.org/r/1695/diff Testing --- Thanks, Carl Upgrade Hive's Thrift dependency to version 0.6.1 - Key: HIVE-2376 URL: https://issues.apache.org/jira/browse/HIVE-2376 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.8.0 Attachments: HIVE-2376-complete.1.patch.txt, HIVE-2376-complete.2.patch.txt, HIVE-2376-no-gen.1.patch.txt, HIVE-2376-no-gen.2.patch.txt, libfb303-0.7.0.jar, libthrift-0.7.0.jar -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2425) Print the counters SLOTS_MILLIS_MAPS and SLOTS_MILLIS_REDUCES to the Hive CLI output.
[ https://issues.apache.org/jira/browse/HIVE-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095707#comment-13095707 ] jirapos...@reviews.apache.org commented on HIVE-2425: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1697/ --- Review request for hive and Siying Dong. Summary --- I print the new stats out where we are already printing out the other map reduce stats for each job. This addresses bug HIVE-2425. https://issues.apache.org/jira/browse/HIVE-2425 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 1164320 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1164320 Diff: https://reviews.apache.org/r/1697/diff Testing --- I ran a query in local mode and verified the stats were not printed. I ran a map only query and verified only the map time was printed. I ran a query involving multiple jobs involving both map only and map and reduce, and verified that the appropriate stats were printed for each. In all cases, I verified the query completed successfully. Thanks, Kevin Print the counters SLOTS_MILLIS_MAPS and SLOTS_MILLIS_REDUCES to the Hive CLI output. - Key: HIVE-2425 URL: https://issues.apache.org/jira/browse/HIVE-2425 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2425.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2304) Support PreparedStatement.setObject
[ https://issues.apache.org/jira/browse/HIVE-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095709#comment-13095709 ] jirapos...@reviews.apache.org commented on HIVE-2304: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1700/ --- Review request for hive. Summary --- Review for HIVE-2304 (Ido Hadanny) This addresses bug HIVE-2304. https://issues.apache.org/jira/browse/HIVE-2304 Diffs - jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 679792c jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java e744d11 Diff: https://reviews.apache.org/r/1700/diff Testing --- Thanks, Carl Support PreparedStatement.setObject --- Key: HIVE-2304 URL: https://issues.apache.org/jira/browse/HIVE-2304 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.7.1 Reporter: Ido Hadanny Assignee: Ido Hadanny Priority: Minor Fix For: 0.8.0 Attachments: HIVE-0.8-SetObject.1.patch.txt Original Estimate: 1h Remaining Estimate: 1h PreparedStatement.setObject is important for spring's jdbcTemplate support -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2304) Support PreparedStatement.setObject
[ https://issues.apache.org/jira/browse/HIVE-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095713#comment-13095713 ] jirapos...@reviews.apache.org commented on HIVE-2304: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1700/#review1731 --- jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java https://reviews.apache.org/r/1700/#comment3948 Please Add a negative testcase that exercises this exception. jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java https://reviews.apache.org/r/1700/#comment3947 Please add a new testcase or add to an existing testcase instead of modifying a pre-existing testcase. You have provided test coverage for PreparedStatement.setObject() at the expense of removing test coverage for PreparedStatement.setBoolean()/setInt(),etc. - Carl On 2011-09-02 01:35:33, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1700/ bq. --- bq. bq. (Updated 2011-09-02 01:35:33) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review for HIVE-2304 (Ido Hadanny) bq. bq. bq. This addresses bug HIVE-2304. bq. https://issues.apache.org/jira/browse/HIVE-2304 bq. bq. bq. Diffs bq. - bq. bq.jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 679792c bq.jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java e744d11 bq. bq. Diff: https://reviews.apache.org/r/1700/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Support PreparedStatement.setObject --- Key: HIVE-2304 URL: https://issues.apache.org/jira/browse/HIVE-2304 Project: Hive Issue Type: Sub-task Components: JDBC Affects Versions: 0.7.1 Reporter: Ido Hadanny Assignee: Ido Hadanny Priority: Minor Attachments: HIVE-0.8-SetObject.1.patch.txt Original Estimate: 1h Remaining Estimate: 1h PreparedStatement.setObject is important for spring's jdbcTemplate support -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1989) recognize transitivity of predicates on join keys
[ https://issues.apache.org/jira/browse/HIVE-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095784#comment-13095784 ] jirapos...@reviews.apache.org commented on HIVE-1989: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-09-02 05:47:55.711939) Review request for hive. Changes --- Added unparseIdentifier for ASTNode handling Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/input4.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_transitivity.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_repeated_alias2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_transitivity.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join8.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1163905
[jira] [Commented] (HIVE-2426) Test that views with joins work properly
[
https://issues.apache.org/jira/browse/HIVE-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095791#comment-13095791
]
jirapos...@reviews.apache.org commented on HIVE-2426:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1702/
---
Review request for hive.
Summary
---
https://issues.apache.org/jira/browse/HIVE-2426
This addresses bug HIVE-2426.
https://issues.apache.org/jira/browse/HIVE-2426
Diffs
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java
1163905
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/join_view.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join_view.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1702/diff
Testing
---
Thanks,
Charles
Test that views with joins work properly
Key: HIVE-2426
URL: https://issues.apache.org/jira/browse/HIVE-2426
Project: Hive
Issue Type: Test
Reporter: Charles Chen
Assignee: Charles Chen
Attachments: HIVE-2426v2.patch
With the testcase
{noformat}
drop table invites;
drop table invites2;
create table invites (foo int, bar string) partitioned by (ds string);
create table invites2 (foo int, bar string) partitioned by (ds string);
set hive.mapred.mode=strict;
-- test join views: see HIVE-1989
create view v as select invites.bar, invites2.foo, invites2.ds from invites
join invites2 on invites.ds=invites2.ds;
explain select * from v where ds='2011-09-01';
drop view v;
drop table invites;
drop table invites2;
{noformat}
We should not have the partition pruner complain about invites.ds not having
a predicate because the predicate invites2.ds='2011-09-01' will be inferred
with the ppd transitivity optimization
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095013#comment-13095013
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-09-01 00:08:37.474019)
Review request for hive.
Changes
---
Fixed ppd_outer_join4.q.out
Summary
---
https://issues.apache.org/jira/browse/HIVE-2337
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163856
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_outer_join5.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out
1163856
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Unit tests passed
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch, HIVE-2337v3.patch,
HIVE-2337v4.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 1
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095022#comment-13095022
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-09-01 00:19:17.176704)
Review request for hive.
Changes
---
Rebased to current trunk
Summary
---
https://issues.apache.org/jira/browse/HIVE-2337
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163875
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out
1163875
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Unit tests passed
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Fix For: 0.9.0
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch, HIVE-2337v3.patch,
HIVE-2337v4.patch, HIVE-2337v5.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 1
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t3
TableScan
alias: t3
Reduce Output Operator
key expressions:
expr: id
[jira] [Commented] (HIVE-2247) ALTER TABLE RENAME PARTITION
[ https://issues.apache.org/jira/browse/HIVE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095082#comment-13095082 ] jirapos...@reviews.apache.org commented on HIVE-2247: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1105/ --- (Updated 2011-09-01 02:23:59.714244) Review request for Siying Dong. Changes --- +work.getInputs().add(new ReadEntity(oldPart)); +work.getOutputs().add(new WriteEntity(newPart)); Summary --- Implement ALTER TABLE PARTITION RENAME function to rename a partition. Add HiveQL syntax ALTER TABLE bar PARTITION (k1='v1', k2='v2') RENAME TO PARTITION (k1='v3', k2='v4'); This is my first Hive diff, I just learn everything from existing codebase and may not have a good understanding on it. Feel free to inform me if I make something wrong. Thanks This addresses bug HIVE-2247. https://issues.apache.org/jira/browse/HIVE-2247 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1145366 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1145366 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1145366 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1145366 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1145366 trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1145366 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_rename_partition.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_rename_partition_authorization.q PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1105/diff Testing --- Add a partition A in the table Rename partition A to partition B Show the partitions in the table, it returns partition B. SELECT the data from partition A, it returns no results SELECT the data from partition B, it returns the data originally stored in partition A Thanks, Weiyan ALTER TABLE RENAME PARTITION Key: HIVE-2247 URL: https://issues.apache.org/jira/browse/HIVE-2247 Project: Hive Issue Type: New Feature Reporter: Siying Dong Assignee: Weiyan Wang Attachments: HIVE-2247.3.patch.txt,
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095104#comment-13095104
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/#review1710
---
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
https://reviews.apache.org/r/1275/#comment3884
There is a weird non-ASCII character on this line.
- John
On 2011-09-01 00:19:17, Charles Chen wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1275/
bq. ---
bq.
bq. (Updated 2011-09-01 00:19:17)
bq.
bq.
bq. Review request for hive.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. https://issues.apache.org/jira/browse/HIVE-2337
bq.
bq.
bq. This addresses bug HIVE-2337.
bq. https://issues.apache.org/jira/browse/HIVE-2337
bq.
bq.
bq. Diffs
bq. -
bq.
bq.
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163875
bq.
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out
1163875
bq.
bq. Diff: https://reviews.apache.org/r/1275/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Unit tests passed
bq.
bq.
bq. Thanks,
bq.
bq. Charles
bq.
bq.
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Fix For: 0.9.0
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch, HIVE-2337v3.patch,
HIVE-2337v4.patch, HIVE-2337v5.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095110#comment-13095110
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-09-01 04:26:59.076177)
Review request for hive.
Changes
---
Oops fixed dropped unit test, javadoc character
Summary
---
https://issues.apache.org/jira/browse/HIVE-2337
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163875
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_outer_join5.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out
1163875
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Unit tests passed
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Fix For: 0.9.0
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch, HIVE-2337v3.patch,
HIVE-2337v4.patch, HIVE-2337v5.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 1
value expressions:
expr: id
type: int
expr: key
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095116#comment-13095116
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-09-01 04:42:25.815081)
Review request for hive.
Changes
---
Added TestParse changes ??
Summary
---
https://issues.apache.org/jira/browse/HIVE-2337
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163875
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_outer_join5.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out
1163875
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/input4.q.xml
1163875
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml
1163875
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Unit tests passed
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Fix For: 0.9.0
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch, HIVE-2337v3.patch,
HIVE-2337v4.patch, HIVE-2337v5.patch, HIVE-2337v6.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr:
[jira] [Commented] (HIVE-1989) recognize transitivity of predicates on join keys
[ https://issues.apache.org/jira/browse/HIVE-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095134#comment-13095134 ] jirapos...@reviews.apache.org commented on HIVE-1989: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1228/ --- (Updated 2011-09-01 05:43:51.993258) Review request for hive. Changes --- Rebase to trunk Summary --- https://issues.apache.org/jira/browse/HIVE-1989 This addresses bug HIVE-1989. https://issues.apache.org/jira/browse/HIVE-1989 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1163905 http://svn.apache.org/repos/asf/hive/trunk/conf/hive-default.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_transitivity.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/cluster.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join16.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join38.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_clusterby.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_gby_join.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join2.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_join3.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join1.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join2.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join4.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_transitivity.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_udf_case.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin9.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/smb_mapjoin_6.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_named_struct.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/union22.q.out 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join2.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join3.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1163905 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/subq.q.xml 1163905
[jira] [Commented] (HIVE-2415) disallow partition column names when doing replace columns
[ https://issues.apache.org/jira/browse/HIVE-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13093498#comment-13093498 ] jirapos...@reviews.apache.org commented on HIVE-2415: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1672/#review1686 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java https://reviews.apache.org/r/1672/#comment3842 remove TAB trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/1672/#comment3844 tab trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/1672/#comment3843 tab trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/1672/#comment3845 does this metastore object change causes the mapped DB table change? it looks dangerous here. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/1672/#comment3846 why we need to clear deserializer here? Is it to make sure column names are not from deserializer? - Ning On 2011-08-28 23:14:12, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1672/ bq. --- bq. bq. (Updated 2011-08-28 23:14:12) bq. bq. bq. Review request for hive and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. move validation of renaming/replacing columns from runtime to parser bq. bq. bq. This addresses bug HIVE-2415. bq. https://issues.apache.org/jira/browse/HIVE-2415 bq. bq. bq. Diffs bq. - bq. bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1162190 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1162190 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1162190 bq.trunk/ql/src/test/queries/clientnegative/replace_columns.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/replace_columns_2.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/replace_columns_3.q PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/altern1.q.out 1162190 bq.trunk/ql/src/test/results/clientnegative/column_rename1.q.out 1162190 bq.trunk/ql/src/test/results/clientnegative/column_rename2.q.out 1162190 bq.trunk/ql/src/test/results/clientnegative/column_rename4.q.out 1162190 bq.trunk/ql/src/test/results/clientnegative/replace_columns.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/replace_columns_2.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/replace_columns_3.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/replace_columns_4.q.out PRE-CREATION bq.trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1162190 bq. bq. Diff: https://reviews.apache.org/r/1672/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. disallow partition column names when doing replace columns -- Key: HIVE-2415 URL: https://issues.apache.org/jira/browse/HIVE-2415 Project: Hive Issue Type: Bug Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2415.1.patch alter table replace columns allows to add a column with the same name as partition column, which introduced inconsistency. We should disallow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2382) Invalid predicate pushdown from incorrect column expression map for select operator generated by GROUP BY operation
[
https://issues.apache.org/jira/browse/HIVE-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094102#comment-13094102
]
jirapos...@reviews.apache.org commented on HIVE-2382:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1566/
---
(Updated 2011-08-30 21:07:18.651251)
Review request for hive.
Changes
---
TestParse test output different
Summary
---
https://issues.apache.org/jira/browse/HIVE-2382
This addresses bug HIVE-2382.
https://issues.apache.org/jira/browse/HIVE-2382
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
1163362
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/groupby_ppd.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/groupby_ppd.q.out
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
1163362
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
1163362
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
1163362
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/groupby4.q.xml
1163362
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
1163362
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/groupby6.q.xml
1163362
Diff: https://reviews.apache.org/r/1566/diff
Testing
---
Unit tests passed
Thanks,
Charles
Invalid predicate pushdown from incorrect column expression map for select
operator generated by GROUP BY operation
---
Key: HIVE-2382
URL: https://issues.apache.org/jira/browse/HIVE-2382
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.6.0
Reporter: Charles Chen
Assignee: Charles Chen
Priority: Critical
Fix For: 0.8.0
Attachments: HIVE-2382v1.patch
When a GROUP BY is specified, a select operator is added before the GROUP BY
in SemanticAnalyzer.insertSelectAllPlanForGroupBy. Currently, the column
expression map for this is set to the column expression map for the parent
operator. This behavior is incorrect as, for example, the parent operator
could rearrange the order of the columns (_col0 = _col0, _col1 = _col2,
_col2 = _col1) and the new operator should not repeat this.
The predicate pushdown optimization uses the column expression map to track
which columns a filter expression refers to at different operators. This
results in a filter on incorrect columns.
Here is a simple case of this going wrong: Using
{noformat}
create table invites (id int, foo int, bar int);
{noformat}
executing the query
{noformat}
explain select * from (select foo, bar from (select bar, foo from invites c
union all select bar, foo from invites d) b) a group by bar, foo having bar=1;
{noformat}
results in
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
a-subquery1:b-subquery1:c
TableScan
alias: c
Filter Operator
predicate:
expr: (foo = 1)
type: boolean
Select Operator
expressions:
expr: bar
type: int
expr: foo
type: int
outputColumnNames: _col0, _col1
Union
Select Operator
expressions:
expr: _col1
type: int
expr: _col0
type: int
outputColumnNames: _col0, _col1
Select Operator
expressions:
expr: _col0
type: int
expr: _col1
type: int
outputColumnNames: _col0, _col1
Group By Operator
bucketGroup: false
keys:
expr: _col1
type: int
expr: _col0
type: int
mode: hash
[jira] [Commented] (HIVE-2383) Incorrect alias filtering for predicate pushdown
[
https://issues.apache.org/jira/browse/HIVE-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094171#comment-13094171
]
jirapos...@reviews.apache.org commented on HIVE-2383:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1568/
---
(Updated 2011-08-30 22:46:54.310929)
Review request for hive.
Summary
---
https://issues.apache.org/jira/browse/HIVE-2383
This addresses bug HIVE-2383.
https://issues.apache.org/jira/browse/HIVE-2383
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_repeated_alias.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1568/diff
Testing
---
Thanks,
Charles
Incorrect alias filtering for predicate pushdown
Key: HIVE-2383
URL: https://issues.apache.org/jira/browse/HIVE-2383
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.6.0
Reporter: Charles Chen
Assignee: Charles Chen
Priority: Critical
Fix For: 0.8.0
Attachments: HIVE-2383v1.patch, HIVE-2383v2.patch
The predicate pushdown optimizer starts at the topmost operators traverses
the operator tree, at each stage collecting predicates to be pushed down. At
each operator, ive.ql.ppd.OpProcFactory.DefaultPPD.mergeWithChildrenPred is
called, which merges the predicates of the children nodes into the current
node. The predicates are stored in hive.ql.ppd.ExprWalkerInfo.pushdownPreds
as a map from the alias a predicate refers to (a predicate may only refer to
one alias at a time as only such predicates can be pushed) to a list of such
predicates. Since at each stage the alias the predicate refers to may change
(subqueries may change aliases), this is updated for each operator
(hive.ql.ppd.ExprWalkerProcFactory.extractPushdownPreds is called which walks
the ExprNodeDesc for each predicate). When a JoinOperator is encountered,
mergeWithChildrenPred is passed an optional parameter aliases which
contains a set of aliases that can be pushed per ansi semantics (see
hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases). The part that is
incorrect is that aliases are filtered in mergeWithChildrenPred before
extractPushdownPreds is called, which associates the predicates with the
correct alias in the current operator's context while the filtering should
happen after.
In test case Q2 below, when the predicate a.bar=3 comes into the
JoinOperator, the alias is a coming in so it is accepted for pushdown.
When brought into the JoinOperator's context, however, since the predicate
refers to b.foo in the inner scope, we should not actually accept this for
pushdown.
With the test cases
{noformat}
-- Q1: predicate should not be pushed on the right side of a left outer join
(this is correct in trunk)
explain
SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo
WHERE b.bar=3;
-- Q2: predicate should not be pushed on the right side of a left outer join
(this is broken in trunk)
explain
SELECT * FROM
(SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo) a
WHERE a.bar=3;
-- Q3: predicate should be pushed (this is correct in trunk)
explain
SELECT * FROM
(SELECT a.foo as foo1, b.foo as foo2, a.bar
FROM pokes a JOIN pokes2 b
ON a.foo=b.foo) a
WHERE a.bar=3;
{noformat}
The current output is
{noformat}
hive
-- Q1: predicate should not be pushed on the right side of a left outer
join
explain
SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo
WHERE b.bar=3;
OK
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF (TOK_TABNAME pokes) a)
(TOK_TABREF (TOK_TABNAME pokes2) b) (= (. (TOK_TABLE_OR_COL a) foo) (.
(TOK_TABLE_OR_COL b) foo (TOK_INSERT (TOK_DESTINATION (TOK_DIR
TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) foo) foo1)
(TOK_SELEXPR (. (TOK_TABLE_OR_COL b) foo) foo2) (TOK_SELEXPR (.
(TOK_TABLE_OR_COL b) bar)))
[jira] [Commented] (HIVE-2383) Incorrect alias filtering for predicate pushdown
[
https://issues.apache.org/jira/browse/HIVE-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094183#comment-13094183
]
jirapos...@reviews.apache.org commented on HIVE-2383:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1568/
---
(Updated 2011-08-30 23:30:03.671952)
Review request for hive.
Changes
---
fixed nullpointer
Summary
---
https://issues.apache.org/jira/browse/HIVE-2383
This addresses bug HIVE-2383.
https://issues.apache.org/jira/browse/HIVE-2383
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_repeated_alias.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1568/diff
Testing
---
Thanks,
Charles
Incorrect alias filtering for predicate pushdown
Key: HIVE-2383
URL: https://issues.apache.org/jira/browse/HIVE-2383
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.6.0
Reporter: Charles Chen
Assignee: Charles Chen
Priority: Critical
Fix For: 0.8.0
Attachments: HIVE-2383v1.patch, HIVE-2383v2.patch
The predicate pushdown optimizer starts at the topmost operators traverses
the operator tree, at each stage collecting predicates to be pushed down. At
each operator, ive.ql.ppd.OpProcFactory.DefaultPPD.mergeWithChildrenPred is
called, which merges the predicates of the children nodes into the current
node. The predicates are stored in hive.ql.ppd.ExprWalkerInfo.pushdownPreds
as a map from the alias a predicate refers to (a predicate may only refer to
one alias at a time as only such predicates can be pushed) to a list of such
predicates. Since at each stage the alias the predicate refers to may change
(subqueries may change aliases), this is updated for each operator
(hive.ql.ppd.ExprWalkerProcFactory.extractPushdownPreds is called which walks
the ExprNodeDesc for each predicate). When a JoinOperator is encountered,
mergeWithChildrenPred is passed an optional parameter aliases which
contains a set of aliases that can be pushed per ansi semantics (see
hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases). The part that is
incorrect is that aliases are filtered in mergeWithChildrenPred before
extractPushdownPreds is called, which associates the predicates with the
correct alias in the current operator's context while the filtering should
happen after.
In test case Q2 below, when the predicate a.bar=3 comes into the
JoinOperator, the alias is a coming in so it is accepted for pushdown.
When brought into the JoinOperator's context, however, since the predicate
refers to b.foo in the inner scope, we should not actually accept this for
pushdown.
With the test cases
{noformat}
-- Q1: predicate should not be pushed on the right side of a left outer join
(this is correct in trunk)
explain
SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo
WHERE b.bar=3;
-- Q2: predicate should not be pushed on the right side of a left outer join
(this is broken in trunk)
explain
SELECT * FROM
(SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo) a
WHERE a.bar=3;
-- Q3: predicate should be pushed (this is correct in trunk)
explain
SELECT * FROM
(SELECT a.foo as foo1, b.foo as foo2, a.bar
FROM pokes a JOIN pokes2 b
ON a.foo=b.foo) a
WHERE a.bar=3;
{noformat}
The current output is
{noformat}
hive
-- Q1: predicate should not be pushed on the right side of a left outer
join
explain
SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo
WHERE b.bar=3;
OK
ABSTRACT SYNTAX TREE:
(TOK_QUERY (TOK_FROM (TOK_LEFTOUTERJOIN (TOK_TABREF (TOK_TABNAME pokes) a)
(TOK_TABREF (TOK_TABNAME pokes2) b) (= (. (TOK_TABLE_OR_COL a) foo) (.
(TOK_TABLE_OR_COL b) foo (TOK_INSERT (TOK_DESTINATION (TOK_DIR
TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) foo) foo1)
(TOK_SELEXPR (. (TOK_TABLE_OR_COL b) foo) foo2) (TOK_SELEXPR (.
[jira] [Commented] (HIVE-2383) Incorrect alias filtering for predicate pushdown
[
https://issues.apache.org/jira/browse/HIVE-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094307#comment-13094307
]
jirapos...@reviews.apache.org commented on HIVE-2383:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1568/
---
(Updated 2011-08-31 05:01:19.947004)
Review request for hive.
Changes
---
Fix filter creation when not pushed down
Summary
---
https://issues.apache.org/jira/browse/HIVE-2383
This addresses bug HIVE-2383.
https://issues.apache.org/jira/browse/HIVE-2383
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_repeated_alias.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join8.q.out
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias.q.out
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_bitmap_empty.q.out
1163438
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml
1163438
Diff: https://reviews.apache.org/r/1568/diff
Testing
---
Thanks,
Charles
Incorrect alias filtering for predicate pushdown
Key: HIVE-2383
URL: https://issues.apache.org/jira/browse/HIVE-2383
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.6.0
Reporter: Charles Chen
Assignee: Charles Chen
Priority: Critical
Fix For: 0.8.0
Attachments: HIVE-2383v1.patch, HIVE-2383v2.patch, HIVE-2383v5.patch
The predicate pushdown optimizer starts at the topmost operators traverses
the operator tree, at each stage collecting predicates to be pushed down. At
each operator, ive.ql.ppd.OpProcFactory.DefaultPPD.mergeWithChildrenPred is
called, which merges the predicates of the children nodes into the current
node. The predicates are stored in hive.ql.ppd.ExprWalkerInfo.pushdownPreds
as a map from the alias a predicate refers to (a predicate may only refer to
one alias at a time as only such predicates can be pushed) to a list of such
predicates. Since at each stage the alias the predicate refers to may change
(subqueries may change aliases), this is updated for each operator
(hive.ql.ppd.ExprWalkerProcFactory.extractPushdownPreds is called which walks
the ExprNodeDesc for each predicate). When a JoinOperator is encountered,
mergeWithChildrenPred is passed an optional parameter aliases which
contains a set of aliases that can be pushed per ansi semantics (see
hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases). The part that is
incorrect is that aliases are filtered in mergeWithChildrenPred before
extractPushdownPreds is called, which associates the predicates with the
correct alias in the current operator's context while the filtering should
happen after.
In test case Q2 below, when the predicate a.bar=3 comes into the
JoinOperator, the alias is a coming in so it is accepted for pushdown.
When brought into the JoinOperator's context, however, since the predicate
refers to b.foo in the inner scope, we should not actually accept this for
pushdown.
With the test cases
{noformat}
-- Q1: predicate should not be pushed on the right side of a left outer join
(this is correct in trunk)
explain
SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo
WHERE b.bar=3;
-- Q2: predicate should not be pushed on the right side of a left outer join
(this is broken in trunk)
explain
SELECT * FROM
(SELECT a.foo as foo1, b.foo as foo2, b.bar
FROM pokes a LEFT OUTER JOIN pokes2 b
ON a.foo=b.foo) a
WHERE a.bar=3;
-- Q3: predicate should be pushed (this is correct in trunk)
explain
SELECT * FROM
(SELECT a.foo as foo1, b.foo as foo2, a.bar
FROM pokes a JOIN pokes2 b
ON a.foo=b.foo) a
WHERE a.bar=3;
{noformat}
The current output is
{noformat}
hive
-- Q1: predicate should not
[jira] [Commented] (HIVE-2383) Incorrect alias filtering for predicate pushdown
[ https://issues.apache.org/jira/browse/HIVE-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094320#comment-13094320 ] jirapos...@reviews.apache.org commented on HIVE-2383: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1568/#review1700 --- http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java https://reviews.apache.org/r/1568/#comment3855 The predicates are already cloned in ExprWalkerProcFactory.extractPushdownPreds() and this behavior is inconsistent with addPushdowns() which doesn't clone the predicates. Moreover, not cloning the expressions is convenient for the newToOldExprMap so that now the ExprNodeDesc's can be compared with == instead of a deep comparison of the tree. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java https://reviews.apache.org/r/1568/#comment3856 From what I understand, currently, when the PPD encounters a JoinOperator, mergeChildrenPred is used to find the filters that are not being pushed down past that operator based on the alias and these should be the same filters that are stored in ExprWalkerInfo.nonFinalPreds (except the filters we want filter on the output columns of the JoinOperator while the nonFinalPreds filter on its input). The bug in HIVE-2383 is that alias filtering should be done in the context of the JoinOperator, not on its children, so now mergeChildrenPred doesn't return the correct filters to create. I changed the JoinPPD to use the nonFinalPreds to figure out which filters to create and added a map in ExprWalkerInfo.newToOldExprMap to store the mapping from each filter in the operator's context to the filter that it came from in the children's context (this is populated during ExprWalkerProcFactory.extractPushdownPreds()). - Charles On 2011-08-31 05:01:19, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1568/ bq. --- bq. bq. (Updated 2011-08-31 05:01:19) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-2383 bq. bq. bq. This addresses bug HIVE-2383. bq. https://issues.apache.org/jira/browse/HIVE-2383 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerProcFactory.java 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_repeated_alias.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/auto_join8.q.out 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables.q.out 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/join8.q.out 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_repeated_alias.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/udf_bitmap_empty.q.out 1163438 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/compiler/plan/join8.q.xml 1163438 bq. bq. Diff: https://reviews.apache.org/r/1568/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Charles bq. bq. Incorrect alias filtering for predicate pushdown Key: HIVE-2383 URL: https://issues.apache.org/jira/browse/HIVE-2383 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.6.0 Reporter: Charles Chen Assignee: Charles Chen Priority: Critical Fix For: 0.8.0 Attachments: HIVE-2383v1.patch, HIVE-2383v2.patch, HIVE-2383v5.patch The predicate pushdown optimizer starts at the topmost operators traverses the operator tree, at each stage collecting predicates to be pushed down. At each operator, ive.ql.ppd.OpProcFactory.DefaultPPD.mergeWithChildrenPred is called, which merges the predicates of the children nodes into the current node. The predicates are stored in hive.ql.ppd.ExprWalkerInfo.pushdownPreds
[jira] [Commented] (HIVE-2415) disallow partition column names when doing replace columns
[ https://issues.apache.org/jira/browse/HIVE-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092565#comment-13092565 ] jirapos...@reviews.apache.org commented on HIVE-2415: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1672/ --- Review request for hive and Ning Zhang. Summary --- move validation of renaming/replacing columns from runtime to parser This addresses bug HIVE-2415. https://issues.apache.org/jira/browse/HIVE-2415 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1162190 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1162190 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1162190 trunk/ql/src/test/queries/clientnegative/replace_columns.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/replace_columns_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/replace_columns_3.q PRE-CREATION trunk/ql/src/test/results/clientnegative/altern1.q.out 1162190 trunk/ql/src/test/results/clientnegative/column_rename1.q.out 1162190 trunk/ql/src/test/results/clientnegative/column_rename2.q.out 1162190 trunk/ql/src/test/results/clientnegative/column_rename4.q.out 1162190 trunk/ql/src/test/results/clientnegative/replace_columns.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/replace_columns_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/replace_columns_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/replace_columns_4.q.out PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1162190 Diff: https://reviews.apache.org/r/1672/diff Testing --- Thanks, Yongqiang disallow partition column names when doing replace columns -- Key: HIVE-2415 URL: https://issues.apache.org/jira/browse/HIVE-2415 Project: Hive Issue Type: Bug Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HIVE-2415.1.patch alter table replace columns allows to add a column with the same name as partition column, which introduced inconsistency. We should disallow this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2404) Allow RCFile Reader to tolerate corruptions
[ https://issues.apache.org/jira/browse/HIVE-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092390#comment-13092390 ] jirapos...@reviews.apache.org commented on HIVE-2404: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1660/ --- (Updated 2011-08-27 23:13:24.160233) Review request for Yongqiang He and Paul Yang. Changes --- Added unit-test, also handled ChecksumException Summary --- Sometimes it is useful to tolerate corruptions during a query and return results based on the files that can be processed. A single corrupt block of data should not prevent reading the rest of the data. We need a way to gracefully ignore errors while reading a RC File This addresses bug HIVE-2404. https://issues.apache.org/jira/browse/HIVE-2404 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java 1161660 trunk/ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java 1161660 Diff: https://reviews.apache.org/r/1660/diff Testing --- Manual testing with a corrupt RC file Thanks, Ramkumar Allow RCFile Reader to tolerate corruptions --- Key: HIVE-2404 URL: https://issues.apache.org/jira/browse/HIVE-2404 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.7.1 Reporter: Ramkumar Vadali Assignee: Ramkumar Vadali Priority: Minor Attachments: toleratecorruptions.patch Sometimes it is useful to tolerate corruptions during a query and return results based on the files that can be processed. A single corrupt block of data should not prevent reading the rest of the data. We need a way to gracefully ignore errors while reading a RC File -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
