[
https://issues.apache.org/jira/browse/HADOOP-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624278#action_12624278
]
Ashish Thusoo commented on HADOOP-3601:
---------------------------------------
Hi Tenaali,
By default we do the group by in 2 stages. In the first stage we generate
partial aggregates and then in the second stage we generate the final
aggregates. If there is a query of the form
select t.c1, count(DISTINCT t.c2) from t group by t.c1
We would first run a map reduce job with the key as c1, c2 to generate the
partial aggregates c1, count(DISTINCT c2).
We would then follow this up with a second stage map reduce with key as c1 on
the output of the previous stage and we would generate the final aggregate as
c1, sum(partial aggregates).
In case distinct is absent e.g.
select t.c1, count(t.c2) from t group by t.c1
we would run the first map reduce by randomly distributing the rows to the
reducers in order to generate partial aggregates c1, count(c2)
and then we would generate the final aggregates similar to the case mentioned
above.
We only support DISTINCT only one column right now.
For join, every join is done in a map-reduce job and the result from one stage
is fed into the next join etc. The join keys are used as the map keys and the
join is done as a cartesian product on the values that arrive for the different
tables in the reducer. We do optimizations so that we can join multiple tables
in the same map reduce task (in case the join key for a table is same e.g. a.c1
= b.c1 and b.c1 = c.c1)
We will be doing many more optimizations for both of these and we will be
putting out all this information on the wiki very soon.
> Hive as a contrib project
> -------------------------
>
> Key: HADOOP-3601
> URL: https://issues.apache.org/jira/browse/HADOOP-3601
> Project: Hadoop Core
> Issue Type: Wish
> Affects Versions: 0.17.0
> Environment: N/A
> Reporter: Joydeep Sen Sarma
> Priority: Minor
> Attachments: HiveTutorial.pdf
>
> Original Estimate: 1080h
> Remaining Estimate: 1080h
>
> Hive is a data warehouse built on top of flat files (stored primarily in
> HDFS). It includes:
> - Data Organization into Tables with logical and hash partitioning
> - A Metastore to store metadata about Tables/Partitions etc
> - A SQL like query language over object data stored in Tables
> - DDL commands to define and load external data into tables
> Hive's query language is executed using Hadoop map-reduce as the execution
> engine. Queries can use either single stage or multi-stage map-reduce. Hive
> has a native format for tables - but can handle any data set (for example
> json/thrift/xml) using an IO library framework.
> Hive uses Antlr for query parsing, Apache JEXL for expression evaluation and
> may use Apache Derby as an embedded database for MetaStore. Antlr has a BSD
> license and should be compatible with Apache license.
> We are currently thinking of contributing to the 0.17 branch as a contrib
> project (since that is the version under which it will get tested internally)
> - but looking for advice on the best release path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
