[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726373#comment-13726373
]
Hudson commented on HIVE-4827:
------------------------------
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #39 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/39/])
HIVE-4827: Merge a Map-only task to its child task (Yin Huai via Gunther
Hagleitner) (gunther:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1509082)
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/QueryPlanTreeTransformation.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
* /hive/trunk/ql/src/test/queries/clientpositive/auto_join33.q
* /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer1.q
* /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer3.q
* /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer4.q
* /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer5.q
* /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer6.q
* /hive/trunk/ql/src/test/queries/clientpositive/correlationoptimizer7.q
* /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin1.q
* /hive/trunk/ql/src/test/queries/clientpositive/multiMapJoin2.q
* /hive/trunk/ql/src/test/queries/clientpositive/union34.q
* /hive/trunk/ql/src/test/results/clientpositive/auto_join0.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join10.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join13.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join15.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join16.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join20.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join21.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join22.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join23.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join24.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join26.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join28.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join29.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join32.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_join33.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out
* /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/correlationoptimizer7.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join28.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join32.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join33.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join_star.q.out
*
/hive/trunk/ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out
* /hive/trunk/ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out
* /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery.q.out
* /hive/trunk/ql/src/test/results/clientpositive/mapjoin_subquery2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/mapjoin_test_outer.q.out
* /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/multiMapJoin2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/multi_join_union.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union34.q.out
> Merge a Map-only task to its child task
> ---------------------------------------
>
> Key: HIVE-4827
> URL: https://issues.apache.org/jira/browse/HIVE-4827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.12.0
> Reporter: Yin Huai
> Assignee: Yin Huai
> Fix For: 0.12.0
>
> Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch, HIVE-4827.3.patch,
> HIVE-4827.4.patch, HIVE-4827.5.patch, HIVE-4827.6.patch, HIVE-4827.7.patch,
> HIVE-4827.8.patch
>
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
> Map-only job (MapJoin) to its following MapReduce job. But this merge only
> happens when the MapReduce job has a single input. With Correlation Optimizer
> (HIVE-2206), it is possible that the MapReduce job can have multiple inputs
> (for multiple operation paths). It is desired to improve CommonJoinResolver
> to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
> GROUP BY x2.key2) tmp2
> ON (tmp1.key = tmp2.key)
> GROUP BY tmp1.key;
> {\code}
> In this query, join operations inside tmp1 and tmp2 will be converted to two
> MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
> tmp1 and tmp2, and the last aggregation will be executed in the same
> MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
> now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
> MapReduce job.
> Another example:
> {code:sql}
> SELECT tmp1.key
> FROM (SELECT x1.key2 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> UNION ALL
> SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
> {\code}
> For this case, we will have three Map-only jobs (two for MapJoins and one for
> Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
