[
https://issues.apache.org/jira/browse/SPARK-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-9983:
-------------------------------
Description:
In distributed query execution, there are two kinds of operators:
(1) operators that exchange data between different executors or threads:
examples include broadcast, shuffle.
(2) operators that process data in a single thread: examples include project,
filter, group by, etc.
This ticket proposes clearly differentiating them and create local operators in
Spark. This leads to a lot of benefits: easier to test, easier to optimize data
exchange, better design (single responsibility), and potentially even having a
hyper-optimized single-node version of DataFrame.
was:
In distributed query execution, there are two kinds of operators:
(1) operators that exchange data between different executors or threads:
examples include broadcast, shuffle.
(2) operators that process data in a single thread: examples include project,
filter, group by, etc.
This ticket proposes clearly differentiating them and create local operators in
Spark. This leads to a lot of benefits: easier to test, easier to optimize data
exchange, and better design (single responsibility).
> Local physical operators for query execution
> --------------------------------------------
>
> Key: SPARK-9983
> URL: https://issues.apache.org/jira/browse/SPARK-9983
> Project: Spark
> Issue Type: Story
> Components: SQL
> Reporter: Reynold Xin
> Assignee: Shixiong Zhu
>
> In distributed query execution, there are two kinds of operators:
> (1) operators that exchange data between different executors or threads:
> examples include broadcast, shuffle.
> (2) operators that process data in a single thread: examples include project,
> filter, group by, etc.
> This ticket proposes clearly differentiating them and create local operators
> in Spark. This leads to a lot of benefits: easier to test, easier to optimize
> data exchange, better design (single responsibility), and potentially even
> having a hyper-optimized single-node version of DataFrame.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
