[
https://issues.apache.org/jira/browse/SPARK-49914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Milicevic updated SPARK-49914:
------------------------------------
Summary: [M3] Spark Connect improvements (was: [M1] Spark Connect
improvements)
> [M3] Spark Connect improvements
> -------------------------------
>
> Key: SPARK-49914
> URL: https://issues.apache.org/jira/browse/SPARK-49914
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: David Milicevic
> Priority: Major
>
> In SPARK-48452 we have investigated the approach for Spark Connect and came
> up with some initial/basic solution - i.e. we are wrapping results in the
> LocalRelation with a isSqlScript param set to true. Our LogicalPlan extends
> the Command hierarchy and this way we didn't have to alter a lot on the
> SparkConnect end.
> However, this is not the ideal/proper design and we should improve it:
> * Introduce proper LogicalPlan hierarchy for eager execution.
> * Remove isSqlScript flag from LocalRelation.
> * Investigate other options to return results through Spark Connect (i.e.
> result streaming; anything else?).
> * Anything else?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
