[
https://issues.apache.org/jira/browse/SPARK-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287901#comment-14287901
]
Xuefu Zhang commented on SPARK-3622:
------------------------------------
Unfortunately no. In Hive the problem is more general than just wring to
different files. What Hive needs is some transformation that can take multiple
map functions and output multiple RDDs from an input RDD. These result RDDs may
be further processed than just simply being written to files.
> Provide a custom transformation that can output multiple RDDs
> -------------------------------------------------------------
>
> Key: SPARK-3622
> URL: https://issues.apache.org/jira/browse/SPARK-3622
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Xuefu Zhang
>
> All existing transformations return just one RDD at most, even for those
> which takes user-supplied functions such as mapPartitions() . However,
> sometimes a user provided function may need to output multiple RDDs. For
> instance, a filter function that divides the input RDD into serveral RDDs.
> While it's possible to get multiple RDDs by transforming the same RDD
> multiple times, it may be more efficient to do this concurrently in one shot.
> Especially user's existing function is already generating different data sets.
> This the case in Hive on Spark, where Hive's map function and reduce function
> can output different data sets to be consumed by subsequent stages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
