[
https://issues.apache.org/jira/browse/HIVE-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681730#comment-13681730
]
Eric Hanson commented on HIVE-4660:
-----------------------------------
If the following use case can be handled in Tez, that would be fantastic.
With vectorized query execution (HIVE-4160) on ORC data I am seeing the
following. A simple single-table group-by/aggregate query against 218 million
rows on a 6 core machine, with one map followed by one reduce, takes about 40
seconds to run. But if you look at the Windows task manager, you see that there
is about a 20 second setup period with only 5-20% CPU consumption, then there
is a 5 second burst of 100% CPU consumption where all the mappers are running
full steam, and then it takes another 15 seconds or so to finish the query,
also with only about 5-20% CPU consumption. The query is not I/O bound.
If the CPU slack could be eliminated, i.e. the CPU cores could run near 100%
from start to finish, the query could probably run in 7 seconds.
If you could include a discussion of this use case in the spec and how Tez will
help now and/or in later Tez versions, or if other work beyond the scope of Tez
is needed, that would be great.
> Let there be Tez (aka mrr ftw)
> ------------------------------
>
> Key: HIVE-4660
> URL: https://issues.apache.org/jira/browse/HIVE-4660
> Project: Hive
> Issue Type: New Feature
> Reporter: Gunther Hagleitner
> Assignee: Gunther Hagleitner
> Attachments: HiveonTez.pdf
>
>
> Tez is a new application framework built on Hadoop Yarn that can execute
> complex directed acyclic graphs of general data processing tasks. Here's the
> project's page: http://incubator.apache.org/projects/tez.html
> The interesting thing about Tez from Hive's perspective is that it will over
> time allow us to overcome inefficiencies in query processing due to having to
> express every algorithm in the map-reduce paradigm.
> The barrier to entry is pretty low as well: Tez can actually run unmodified
> MR jobs; But as a first step we can without much trouble start using more of
> Tez' features by taking advantage of the MRR pattern.
> MRR simply means that there can be any number of reduce stages following a
> single map stage - without having to write intermediate results to HDFS and
> re-read them in a new job. This is common when queries require multiple
> shuffles on keys without correlation (e.g.: join - grp by - window function -
> order by)
> For more details see the attached design doc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
