[ 
https://issues.apache.org/jira/browse/SPARK-62066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62066:
-----------------------------------
    Labels: pull-request-available  (was: )

> [Connect][Python] Drop-in: take local rows apart by the cached names of a 
> StructType changed in place, raise PySpark's errors of the types that 
> proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED 
> conditions of no parameters
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62066
>                 URL: https://issues.apache.org/jira/browse/SPARK-62066
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> 1. A StructType whose fields list was changed in place. PySpark's StructType 
> caches names as it is made (python/pyspark/sql/types.py:1471) and as add adds 
> a field (types.py:1541, 1557), and fieldNames() is list(self.names) 
> (types.py:1761), so st.fields.append(StructField("b", StringType())), 
> st.fields.pop() or st.fields[0] = ... leaves the names as they were. Its 
> conversion of local rows takes each row apart by schema.fieldNames() 
> (python/pyspark/sql/conversion.py:941-975, to_row), and the converter of a 
> struct below the top level its values (conversion.py:605-660), and converts 
> the values by position with the fields as they are (conversion.py:978-1003): 
> the arity of a row and the keys of a dict are of the cached names, a field 
> past them has no value (IndexError of row[i]), and below the top level a 
> value past the fields has no converter and fewer names than fields fail 
> _deduplicate_field_names (python/pyspark/sql/pandas/types.py:1630-1653), both 
> IndexError. createDataFrame of a pandas DataFrame names it with 
> toDF(*schema.fieldNames()) (python/pyspark/sql/connect/session.py:608, 793). 
> The drop-in took the rows apart by the names of the fields as they are.
>     st = StructType([StructField("a", LongType())])
>     st.fields.append(StructField("b", StringType()))
>     spark.createDataFrame([(1, "q")], st)
>     # PySpark: PySparkValueError [AXIS_LENGTH_MISMATCH] Length mismatch: 
> Expected axis has 1 elements, new values have 2 elements.
>     # drop-in: [Row(a=1, b='q')]
>     spark.createDataFrame([(1,)], st)
>     # PySpark: IndexError: tuple index out of range; drop-in: 
> AXIS_LENGTH_MISMATCH (2, 1)
>     st = StructType([StructField("a", LongType())])
>     st.fields[0] = StructField("z", StringType())
>     spark.createDataFrame([{"a": 1}], st).collect()
>     # PySpark: [Row(z='1')]; drop-in: [Row(z=None)]
> An element that is not a StructField, such as a str appended, raises the 
> drop-in's TypeError ("StructType fields must be StructField objects") 
> wherever the native type lowers its fields, where PySpark's Python methods 
> raise the AttributeError of the first attribute that they read; this is 
> inherent to the native type and documented.
> 2. proto_schema_to_pyspark_data_type of a type that it does not convert. 
> PySpark's last branch raises PySparkValueError(UNSUPPORTED_OPERATION, 
> {"operation": f"data type {schema}"}) of the proto text of the type 
> (python/pyspark/sql/connect/types.py:218-333), for a type of no kind, such as 
> pb2.DataType() or the element type that an array does not set, and of a kind 
> that it has no branch of, such as an unparsed type, in the order of its 
> conversion. It makes a user-defined type of its Python class, of no 
> conversion of its SQL type.
>     proto_schema_to_pyspark_data_type(pb2.DataType())
>     # PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type  is not 
> supported.
>     # drop-in: PySparkRuntimeError DataType kind not set
>     dt = pb2.DataType(); dt.unparsed.data_type_string = "int"
>     proto_schema_to_pyspark_data_type(dt)
>     # PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type unparsed { 
> data_type_string: "int" } is not supported.
>     # drop-in: UnparsedDataType('int')
> 3. The MALFORMED conditions of the conversion of local rows. PySpark's 
> converters raise PySparkValueError(errorClass="MALFORMED_VARIANT") 
> (conversion.py:889), MALFORMED_GEOGRAPHY (903) and MALFORMED_GEOMETRY (917) 
> without messageParameters, so getMessageParameters() is None. The drop-in's 
> are {}.
>     spark.createDataFrame([("x",)], "v variant")  # getMessageParameters(): 
> PySpark None, drop-in {}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to