[ 
https://issues.apache.org/jira/browse/SPARK-61862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61862:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send the requests of toArrow, toPandas, toLocalIterator and the map and group 
> UDFs in PySpark's order, and yield calendar intervals as PySpark's iterator 
> does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61862
>                 URL: https://issues.apache.org/jira/browse/SPARK-61862
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61772 (createDataFrame reads its settings at PySpark's 
> point): other conversions of the Python drop-in send their requests, or run 
> their checks, in another order than PySpark 4.3.0, or give other values:
> - toArrow executes the query before it analyzes the schema and checks that 
> Arrow can convert it (PySpark computes to_arrow_schema(self.schema) first, of 
> the cached schema), so a duplicated nested field name or an interval type 
> runs the query before the client error.
> - toPandas and toLocalIterator read their settings before they build the 
> plan, so a plan that cannot be built sends Config requests where PySpark 
> sends nothing.
> - toLocalIterator raises the year-month interval error before the server's 
> error, and yields a calendar interval as a str where PySpark yields pyarrow's 
> MonthDayNano.
> - mapInPandas, mapInArrow, applyInPandas and applyInArrow analyze the input 
> before they parse the DDL of the output schema; cogroup sends an AnalyzePlan 
> of each side and binds their columns to the UDF, where PySpark's function has 
> no arguments; applyInPandasWithState parses its state schema, which PySpark 
> sends as given.
> - toLocalIterator releases the GIL for every row, so a CPU-busy Python thread 
> slows the iteration (300k rows: 15-38 s here, 2.4 s on PySpark).
> - spark.udf.register("f", fn, "long") parses the DDL once, where PySpark 
> parses it twice.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to