[
https://issues.apache.org/jira/browse/SPARK-61485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61485:
------------------------------------
Assignee: Hyukjin Kwon
> Pickle the methods of Row, VariantVal, Geometry and Geography by reference to
> pyspark.sql.types, and read an Arrow stream of one schema in the
> invalid-UTF-8 scan
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61485
> URL: https://issues.apache.org/jira/browse/SPARK-61485
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Follow-ups of a post-hoc review of SPARK-61402 and SPARK-61405, checked
> against PySpark 4.3.0:
> - A UDF that is, or captures, a method of Row, VariantVal, Geometry or
> Geography (F.udf(Geography.getSrid, "int"), F.udf(VariantVal.toJson,
> "string"), F.udf(variant.toJson, "string"), a UDF capturing Row.asDict)
> failed on the server with PythonException ModuleNotFoundError: No module
> named 'pyspark._values' (or 'pyspark._row'). The extension loads these
> classes from the embedded modules pyspark._values and pyspark._row, which set
> the classes' __module__ to pyspark.sql.types but leave their methods'
> __module__ as the embedded module, so cloudpickle pickled the methods by
> reference to a module that exists on the client only. PySpark's methods are
> of pyspark.sql.types, which the server's Python workers import. The methods
> now report pyspark.sql.types.
> - The invalid-UTF-8 scan of a result (first_invalid_utf8) took the column
> count from the first batch and the field types from the last schema message,
> so a stream with a second, narrower schema (which a valid Arrow IPC stream
> never has) panicked with an index out of bounds. A stream of more than one
> schema now does not read there, and the decoding error is reported as is.
> - docs/compatibility.md did not describe SPARK-61405's changes: the strata
> that sample_by rejects, the seed that it draws on the client, and the
> Value::Map fill of fillna_value.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]