[
https://issues.apache.org/jira/browse/SPARK-32341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166129#comment-17166129
]
Hyukjin Kwon commented on SPARK-32341:
--------------------------------------
You can do it via DataFrame and SQL APIs. I think such optimization was the
main reason of DataFrame APIs.
> add mutiple filter in rdd function
> ----------------------------------
>
> Key: SPARK-32341
> URL: https://issues.apache.org/jira/browse/SPARK-32341
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 2.4.6, 3.0.0
> Reporter: gaokui
> Priority: Major
>
> when i use spark rdd . i often use to read kafka data.And kafka data has lots
> of kinds data set.
> I filter these rdd by kafka key , then i can use Array[rdd] to fill every
> topic rdd.
> But at that , i use rdd.filter,that will generate more than one stage.Data
> will process by many task, that consume too many time. And it is not
> necessary.
> i hope add multiple filter function not rdd.filter ,that will return
> Array[RDD] in one stage by dividing all mixture data RDD to single data set
> RDD .
> function like Array[RDD]=rdd.multiplefilter(setcondition).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
