[
https://issues.apache.org/jira/browse/HIVE-537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715878#action_12715878
]
Zheng Shao commented on HIVE-537:
---------------------------------
An example usage is for multiple distinct. Min Zhou talked with me offline and
has shown that doing multiple distinct in a single map-reduce job can be much
faster than doing them separately and then join the results.
{code}
Query:
select a, count(distinct b), count(distinct c), sum(d)
Plan:
Map side:
Emit: distribution_key: a, sort_key: a, 0, b, value: d
Emit: distribution_key: a, sort_key: a, 1, c, value: nothing
Reduce side:
Group By:
a, 0, count(distinct b), sum(d)
a, 1, count(distinct c)
Flatten:
a, count(distinct b), sum(d), count(distinct c)
{code}
> Hive TypeInfo/ObjectInspector to support union (besides struct, array, and
> map)
> -------------------------------------------------------------------------------
>
> Key: HIVE-537
> URL: https://issues.apache.org/jira/browse/HIVE-537
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
>
> There are already some cases inside the code that we use heterogeneous data:
> JoinOperator, and UnionOperator (in the sense that different parents can pass
> in records with different ObjectInspectors).
> We currently use Operator's parentID to distinguish that. However that
> approach does not extend to more complex plans that might be needed in the
> future.
> We will support the union type like this:
> {code}
> TypeDefinition:
> type: primitivetype | structtype | arraytype | maptype | uniontype
> uniontype: "union" "<" tag ":" type ("," tag ":" type)* ">"
> Example:
> union<0:int,1:double,2:array<string>,3:struct<a:int,b:string>>
> Example of serialized data format:
> We will first store the tag byte before we serialize the object. On
> deserialization, we will first read out the tag byte, then we know what is
> the current type of the following object, so we can deserialize it
> successfully.
> Interface for ObjectInspector:
> interface UnionObjectInspector {
> /** Returns the array of OIs that are for each of the tags
> */
> ObjectInspector[] getObjectInspectors();
> /** Return the tag of the object.
> */
> byte getTag(Object o);
> /** Return the field based on the tag value associated with the Object.
> */
> Object getField(Object o);
> };
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
