[
https://issues.apache.org/jira/browse/HIVE-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang resolved HIVE-8118.
-------------------------------
Resolution: Fixed
Fix Version/s: spark-branch
All sub tasks are completed. Thus, this JIRA is closed as fixed as well.
> Support work that have multiple child works to work around SPARK-3622 [Spark
> Branch]
> -------------------------------------------------------------------------------------
>
> Key: HIVE-8118
> URL: https://issues.apache.org/jira/browse/HIVE-8118
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Chao
> Labels: Spark-M1
> Fix For: spark-branch
>
> Attachments: HIVE-8118.pdf
>
>
> In the current implementation, both SparkMapRecordHandler and
> SparkReduceRecorderHandler takes only one result collector, which limits that
> the corresponding map or reduce task can have only one child. It's very
> comment in multi-insert queries where a map/reduce task has more than one
> children. A query like the following has two map tasks as parents:
> {code}
> select name, sum(value) from dec group by name union all select name, value
> from dec order by name
> {code}
> It's possible in the future an optimation may be implemented so that a map
> work is followed by two reduce works and then connected to a union work.
> Thus, we should take this as a general case. Tez is currently providing a
> collector for each child operator in the map-side or reduce side operator
> tree. We can take Tez as a reference.
> Spark currently doesn't have a tranformation that supports mutliple output
> datasets from a single input dataset (SPARK-3622). This is a workaround for
> this gap.
> Likely this is a big change and subtasks are possible.
> With this, we can have a simpler and clean multi-insert implementation. This
> is also the problem observed in HIVE-7731 and HIVE-7503.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
