Re: Review Request: HIVE-1644 Use filter pushdown for automatically accessing indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/ --- (Updated 2011-04-16 06:04:26.681814) Review request for hive. Changes --- HIVE-1644.14.patch Summary --- Review request for HIVE-1644.12.patch This addresses bug HIVE-1644. https://issues.apache.org/jira/browse/HIVE-1644 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589 conf/hive-default.xml c42197f ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385 ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION Diff: https://reviews.apache.org/r/558/diff Testing --- Thanks, Russell
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020560#comment-13020560 ] jirapos...@reviews.apache.org commented on HIVE-1644: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/ --- (Updated 2011-04-16 06:04:26.681814) Review request for hive. Changes --- HIVE-1644.14.patch Summary --- Review request for HIVE-1644.12.patch This addresses bug HIVE-1644. https://issues.apache.org/jira/browse/HIVE-1644 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589 conf/hive-default.xml c42197f ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385 ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION Diff: https://reviews.apache.org/r/558/diff Testing --- Thanks, Russell use filter pushdown for automatically accessing indexes --- Key: HIVE-1644 URL: https://issues.apache.org/jira/browse/HIVE-1644 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: John Sichi Assignee: Russell Melick Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch, HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch, HIVE-1644.2.patch, HIVE-1644.3.patch, HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch, HIVE-1644.8.patch, HIVE-1644.9.patch HIVE-1226 provides utilities for analyzing filters which have been pushed down to a table scan. The next step is to use these for selecting available indexes and generating access plans for those indexes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1644) use filter pushdown for automatically accessing indexes
[ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Melick updated HIVE-1644: - Attachment: HIVE-1644.14.patch Include new unit tests. Also asked a few more questions on the review board. use filter pushdown for automatically accessing indexes --- Key: HIVE-1644 URL: https://issues.apache.org/jira/browse/HIVE-1644 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: John Sichi Assignee: Russell Melick Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch, HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch, HIVE-1644.14.patch, HIVE-1644.2.patch, HIVE-1644.3.patch, HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch, HIVE-1644.8.patch, HIVE-1644.9.patch HIVE-1226 provides utilities for analyzing filters which have been pushed down to a table scan. The next step is to use these for selecting available indexes and generating access plans for those indexes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020562#comment-13020562 ] jirapos...@reviews.apache.org commented on HIVE-1644: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/#review492 --- ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java https://reviews.apache.org/r/558/#comment998 Still need to change hive.index.compact.file to hive.index.blockfilter.file , but hopefully bitmap gets committed soon. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java https://reviews.apache.org/r/558/#comment997 I'm not sure the way I'm doing it currently will work with partitions. I don't take them into account when generating the index query. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment1000 see later comment about why this abort needs to be skipped for anything to run. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment999 This doesn't seem to work (it always returns false here). This checks whether the partitions equal each other, which I don't think can happen since they're on different tables. What information in a partition do I need to be checking? ql/src/test/queries/clientpositive/index_auto_multiple.q https://reviews.apache.org/r/558/#comment995 Is there a multiple column table? Or, what's the best way to create a multi-column table and populate it with data? I can't figure out a good way to query the value column, so the src table seems less than ideal. ql/src/test/queries/clientpositive/index_auto_unused.q https://reviews.apache.org/r/558/#comment996 How do unbuilt partitions work? I didn't see any way to delay the building, so I don't know how to have an index with unbuilt partitions. - Russell On 2011-04-16 06:04:26, Russell Melick wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/558/ bq. --- bq. bq. (Updated 2011-04-16 06:04:26) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review request for HIVE-1644.12.patch bq. bq. bq. This addresses bug HIVE-1644. bq. https://issues.apache.org/jira/browse/HIVE-1644 bq. bq. bq. Diffs bq. - bq. bq.common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589 bq.conf/hive-default.xml c42197f bq.ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0 bq.ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385 bq.ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b bq.ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 bq.ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84 bq.ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 bq.ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION bq.
Re: Review Request: HIVE-1644 Use filter pushdown for automatically accessing indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/#review492 --- ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java https://reviews.apache.org/r/558/#comment998 Still need to change hive.index.compact.file to hive.index.blockfilter.file , but hopefully bitmap gets committed soon. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java https://reviews.apache.org/r/558/#comment997 I'm not sure the way I'm doing it currently will work with partitions. I don't take them into account when generating the index query. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment1000 see later comment about why this abort needs to be skipped for anything to run. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment999 This doesn't seem to work (it always returns false here). This checks whether the partitions equal each other, which I don't think can happen since they're on different tables. What information in a partition do I need to be checking? ql/src/test/queries/clientpositive/index_auto_multiple.q https://reviews.apache.org/r/558/#comment995 Is there a multiple column table? Or, what's the best way to create a multi-column table and populate it with data? I can't figure out a good way to query the value column, so the src table seems less than ideal. ql/src/test/queries/clientpositive/index_auto_unused.q https://reviews.apache.org/r/558/#comment996 How do unbuilt partitions work? I didn't see any way to delay the building, so I don't know how to have an index with unbuilt partitions. - Russell On 2011-04-16 06:04:26, Russell Melick wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/ --- (Updated 2011-04-16 06:04:26) Review request for hive. Summary --- Review request for HIVE-1644.12.patch This addresses bug HIVE-1644. https://issues.apache.org/jira/browse/HIVE-1644 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589 conf/hive-default.xml c42197f ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385 ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION Diff:
Build failed in Jenkins: Hive-0.7.0-h0.20 #78
See https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/78/ -- [...truncated 26904 lines...] [junit] Loading data to table default.srcbucket2 [junit] POSTHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/srcbucket23.txt' INTO TABLE srcbucket2 [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@srcbucket2 [junit] OK [junit] Copying file: https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.txt [junit] PREHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.txt' INTO TABLE src [junit] PREHOOK: type: LOAD [junit] Copying data from https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.txt [junit] Loading data to table default.src [junit] POSTHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.txt' INTO TABLE src [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@src [junit] OK [junit] PREHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv3.txt' INTO TABLE src1 [junit] PREHOOK: type: LOAD [junit] Copying data from https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv3.txt [junit] Copying file: https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv3.txt [junit] Loading data to table default.src1 [junit] POSTHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv3.txt' INTO TABLE src1 [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@src1 [junit] OK [junit] Copying file: https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.seq [junit] PREHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.seq' INTO TABLE src_sequencefile [junit] PREHOOK: type: LOAD [junit] Copying data from https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.seq [junit] Loading data to table default.src_sequencefile [junit] POSTHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.seq' INTO TABLE src_sequencefile [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@src_sequencefile [junit] OK [junit] Copying file: https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/complex.seq [junit] PREHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/complex.seq' INTO TABLE src_thrift [junit] PREHOOK: type: LOAD [junit] Copying data from https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/complex.seq [junit] Loading data to table default.src_thrift [junit] POSTHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/complex.seq' INTO TABLE src_thrift [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@src_thrift [junit] OK [junit] PREHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/json.txt' INTO TABLE src_json [junit] PREHOOK: type: LOAD [junit] Copying data from https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/json.txt [junit] Copying file: https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/json.txt [junit] Loading data to table default.src_json [junit] POSTHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/json.txt' INTO TABLE src_json [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@src_json [junit] OK [junit] diff https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/build/ql/test/logs/negative/wrong_distinct1.q.out https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/ql/src/test/results/compiler/errors/wrong_distinct1.q.out [junit] Done query: wrong_distinct1.q [junit] Begin query: wrong_distinct2.q [junit] Hive history file=https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/build/ql/tmp/hive_job_log_hudson_201104161208_609330292.txt [junit] Hive history file=https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/build/ql/tmp/hive_job_log_hudson_201104161208_742570138.txt [junit] PREHOOK: query: LOAD DATA LOCAL INPATH 'https://hudson.apache.org/hudson/job/Hive-0.7.0-h0.20/ws/hive/data/files/kv1.txt' OVERWRITE INTO TABLE srcpart PARTITION (ds='2008-04-08',hr='11') [junit] PREHOOK: type: LOAD [junit] Copying data from
Build failed in Jenkins: Hive-trunk-h0.20 #675
See https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/675/ -- [...truncated 29832 lines...] [junit] OK [junit] PREHOOK: query: select count(1) as cnt from testhivedrivertable [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2011-04-16_13-09-15_867_65479708827511508/-mr-1 [junit] Total MapReduce jobs = 1 [junit] Launching Job 1 out of 1 [junit] Number of reduce tasks determined at compile time: 1 [junit] In order to change the average load for a reducer (in bytes): [junit] set hive.exec.reducers.bytes.per.reducer=number [junit] In order to limit the maximum number of reducers: [junit] set hive.exec.reducers.max=number [junit] In order to set a constant number of reducers: [junit] set mapred.reduce.tasks=number [junit] Job running in-process (local Hadoop) [junit] Hadoop job information for null: number of mappers: 0; number of reducers: 0 [junit] 2011-04-16 13:09:18,963 null map = 100%, reduce = 100% [junit] Ended Job = job_local_0001 [junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2011-04-16_13-09-15_867_65479708827511508/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/ws/hive/build/service/tmp/hive_job_log_hudson_201104161309_1508639088.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: load data local inpath 'https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] PREHOOK: type: LOAD [junit] PREHOOK: Output: default@testhivedrivertable [junit] Copying data from https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/ws/hive/data/files/kv1.txt [junit] Loading data to table default.testhivedrivertable [junit] POSTHOOK: query: load data local inpath 'https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] POSTHOOK: type: LOAD [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: select * from testhivedrivertable limit 10 [junit] PREHOOK: type: QUERY [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2011-04-16_13-09-20_521_7616609718971135008/-mr-1 [junit] POSTHOOK: query: select * from testhivedrivertable limit 10 [junit] POSTHOOK: type: QUERY [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2011-04-16_13-09-20_521_7616609718971135008/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://hudson.apache.org/hudson/job/Hive-trunk-h0.20/ws/hive/build/service/tmp/hive_job_log_hudson_201104161309_1425843023.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: CREATETABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: CREATETABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE
[jira] [Created] (HIVE-2116) Optimize map-side scans for right-side of join
Optimize map-side scans for right-side of join -- Key: HIVE-2116 URL: https://issues.apache.org/jira/browse/HIVE-2116 Project: Hive Issue Type: Improvement Reporter: Ron Bodkin I had a large query like select * from a join b on a.key=b.key where...; Table b was too large, so I attempted to optimize by adding constraints on b to the where clause, e.g., where b.size=mn and b.size=mx and ...; However, the Hive 0.8.0 optimizer pushed the constraint on b into the reduce phase (defeating its purpose). I was able to force Hive to run the optimization map-side by this workaround: join (select * from b where size=mn and size=mx) b on a.key=b.key where ...; But it would be nice for Hive to pull filters on joined records into the map phase where possible. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2116) Optimize map-side scans for right-side of join
[ https://issues.apache.org/jira/browse/HIVE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020701#comment-13020701 ] Ron Bodkin commented on HIVE-2116: -- Table b was an HBase table, rather than a traditional HDFS file, if that is relevant to the issue. Optimize map-side scans for right-side of join -- Key: HIVE-2116 URL: https://issues.apache.org/jira/browse/HIVE-2116 Project: Hive Issue Type: Improvement Reporter: Ron Bodkin I had a large query like select * from a join b on a.key=b.key where...; Table b was too large, so I attempted to optimize by adding constraints on b to the where clause, e.g., where b.size=mn and b.size=mx and ...; However, the Hive 0.8.0 optimizer pushed the constraint on b into the reduce phase (defeating its purpose). I was able to force Hive to run the optimization map-side by this workaround: join (select * from b where size=mn and size=mx) b on a.key=b.key where ...; But it would be nice for Hive to pull filters on joined records into the map phase where possible. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
