[
https://issues.apache.org/jira/browse/SPARK-62066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62066.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 879
[https://github.com/apache/spark-connect-rust/pull/879]
> [Connect][Python] Drop-in: take local rows apart by the cached names of a
> StructType changed in place, raise PySpark's errors of the types that
> proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED
> conditions of no parameters
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62066
> URL: https://issues.apache.org/jira/browse/SPARK-62066
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> 1. A StructType whose fields list was changed in place. PySpark's StructType
> caches names as it is made (python/pyspark/sql/types.py:1471) and as add adds
> a field (types.py:1541, 1557), and fieldNames() is list(self.names)
> (types.py:1761), so st.fields.append(StructField("b", StringType())),
> st.fields.pop() or st.fields[0] = ... leaves the names as they were. Its
> conversion of local rows takes each row apart by schema.fieldNames()
> (python/pyspark/sql/conversion.py:941-975, to_row), and the converter of a
> struct below the top level its values (conversion.py:605-660), and converts
> the values by position with the fields as they are (conversion.py:978-1003):
> the arity of a row and the keys of a dict are of the cached names, a field
> past them has no value (IndexError of row[i]), and below the top level a
> value past the fields has no converter and fewer names than fields fail
> _deduplicate_field_names (python/pyspark/sql/pandas/types.py:1630-1653), both
> IndexError. createDataFrame of a pandas DataFrame names it with
> toDF(*schema.fieldNames()) (python/pyspark/sql/connect/session.py:608, 793).
> The drop-in took the rows apart by the names of the fields as they are.
> st = StructType([StructField("a", LongType())])
> st.fields.append(StructField("b", StringType()))
> spark.createDataFrame([(1, "q")], st)
> # PySpark: PySparkValueError [AXIS_LENGTH_MISMATCH] Length mismatch:
> Expected axis has 1 elements, new values have 2 elements.
> # drop-in: [Row(a=1, b='q')]
> spark.createDataFrame([(1,)], st)
> # PySpark: IndexError: tuple index out of range; drop-in:
> AXIS_LENGTH_MISMATCH (2, 1)
> st = StructType([StructField("a", LongType())])
> st.fields[0] = StructField("z", StringType())
> spark.createDataFrame([{"a": 1}], st).collect()
> # PySpark: [Row(z='1')]; drop-in: [Row(z=None)]
> An element that is not a StructField, such as a str appended, raises the
> drop-in's TypeError ("StructType fields must be StructField objects")
> wherever the native type lowers its fields, where PySpark's Python methods
> raise the AttributeError of the first attribute that they read; this is
> inherent to the native type and documented.
> 2. proto_schema_to_pyspark_data_type of a type that it does not convert.
> PySpark's last branch raises PySparkValueError(UNSUPPORTED_OPERATION,
> {"operation": f"data type {schema}"}) of the proto text of the type
> (python/pyspark/sql/connect/types.py:218-333), for a type of no kind, such as
> pb2.DataType() or the element type that an array does not set, and of a kind
> that it has no branch of, such as an unparsed type, in the order of its
> conversion. It makes a user-defined type of its Python class, of no
> conversion of its SQL type.
> proto_schema_to_pyspark_data_type(pb2.DataType())
> # PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type is not
> supported.
> # drop-in: PySparkRuntimeError DataType kind not set
> dt = pb2.DataType(); dt.unparsed.data_type_string = "int"
> proto_schema_to_pyspark_data_type(dt)
> # PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type unparsed {
> data_type_string: "int" } is not supported.
> # drop-in: UnparsedDataType('int')
> 3. The MALFORMED conditions of the conversion of local rows. PySpark's
> converters raise PySparkValueError(errorClass="MALFORMED_VARIANT")
> (conversion.py:889), MALFORMED_GEOGRAPHY (903) and MALFORMED_GEOMETRY (917)
> without messageParameters, so getMessageParameters() is None. The drop-in's
> are {}.
> spark.createDataFrame([("x",)], "v variant") # getMessageParameters():
> PySpark None, drop-in {}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]