[
https://issues.apache.org/jira/browse/SPARK-61116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61116:
------------------------------------
Assignee: Hyukjin Kwon
> Port the Read, Range and SQL plans, Column.to_plan, and the type and
> conversion helpers of the connect import path, in the Python client as
> PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61116
> URL: https://issues.apache.org/jira/browse/SPARK-61116
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> With the generated messages of {{pyspark.sql.connect.proto}} and
> {{DataFrame(plan, session)}} (SPARK-61111), the rest of the connect import
> path that builds or reads messages can be PySpark's:
> * In {{pyspark.sql.connect.plan}}, {{Read}}, {{Range}} and {{SQL}} are
> placeholders. Their constructors take any arguments and build no relation, so
> {{DataFrame(Range(0, 3, 1), spark)}} fails. PySpark's build the {{read}},
> {{range}} and {{sql}} relations, which
> {{pyspark.testing.connectutils.PlanOnlyTestFixture}} uses.
> * {{Column.to_plan(session)}}, the {{spark.connect.Expression}} of a column,
> is missing.
> * {{pyspark.sql.connect.types}} lacks {{parse_attr_name}},
> {{verify_col_name}} and {{verify_numeric_col_name}}.
> * {{pyspark.sql.connect.conversion}} ({{storage_level_to_proto}},
> {{proto_to_storage_level}} and {{proto_to_remote_cached_dataframe}}) is
> missing.
> The fix is to port them from Apache Spark v4.3.0, over the messages of the
> core.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
