Hyukjin Kwon created SPARK-62066:
------------------------------------

             Summary: [Connect][Python] Drop-in: take local rows apart by the 
cached names of a StructType changed in place, raise PySpark's errors of the 
types that proto_schema_to_pyspark_data_type does not convert, and raise the 
MALFORMED conditions of no parameters
                 Key: SPARK-62066
                 URL: https://issues.apache.org/jira/browse/SPARK-62066
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


1. A StructType whose fields list was changed in place. PySpark's StructType 
caches names as it is made (python/pyspark/sql/types.py:1471) and as add adds a 
field (types.py:1541, 1557), and fieldNames() is list(self.names) 
(types.py:1761), so st.fields.append(StructField("b", StringType())), 
st.fields.pop() or st.fields[0] = ... leaves the names as they were. Its 
conversion of local rows takes each row apart by schema.fieldNames() 
(python/pyspark/sql/conversion.py:941-975, to_row), and the converter of a 
struct below the top level its values (conversion.py:605-660), and converts the 
values by position with the fields as they are (conversion.py:978-1003): the 
arity of a row and the keys of a dict are of the cached names, a field past 
them has no value (IndexError of row[i]), and below the top level a value past 
the fields has no converter and fewer names than fields fail 
_deduplicate_field_names (python/pyspark/sql/pandas/types.py:1630-1653), both 
IndexError. createDataFrame of a pandas DataFrame names it with 
toDF(*schema.fieldNames()) (python/pyspark/sql/connect/session.py:608, 793). 
The drop-in took the rows apart by the names of the fields as they are.

    st = StructType([StructField("a", LongType())])
    st.fields.append(StructField("b", StringType()))
    spark.createDataFrame([(1, "q")], st)
    # PySpark: PySparkValueError [AXIS_LENGTH_MISMATCH] Length mismatch: 
Expected axis has 1 elements, new values have 2 elements.
    # drop-in: [Row(a=1, b='q')]
    spark.createDataFrame([(1,)], st)
    # PySpark: IndexError: tuple index out of range; drop-in: 
AXIS_LENGTH_MISMATCH (2, 1)

    st = StructType([StructField("a", LongType())])
    st.fields[0] = StructField("z", StringType())
    spark.createDataFrame([{"a": 1}], st).collect()
    # PySpark: [Row(z='1')]; drop-in: [Row(z=None)]

An element that is not a StructField, such as a str appended, raises the 
drop-in's TypeError ("StructType fields must be StructField objects") wherever 
the native type lowers its fields, where PySpark's Python methods raise the 
AttributeError of the first attribute that they read; this is inherent to the 
native type and documented.

2. proto_schema_to_pyspark_data_type of a type that it does not convert. 
PySpark's last branch raises PySparkValueError(UNSUPPORTED_OPERATION, 
{"operation": f"data type {schema}"}) of the proto text of the type 
(python/pyspark/sql/connect/types.py:218-333), for a type of no kind, such as 
pb2.DataType() or the element type that an array does not set, and of a kind 
that it has no branch of, such as an unparsed type, in the order of its 
conversion. It makes a user-defined type of its Python class, of no conversion 
of its SQL type.

    proto_schema_to_pyspark_data_type(pb2.DataType())
    # PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type  is not 
supported.
    # drop-in: PySparkRuntimeError DataType kind not set
    dt = pb2.DataType(); dt.unparsed.data_type_string = "int"
    proto_schema_to_pyspark_data_type(dt)
    # PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type unparsed { 
data_type_string: "int" } is not supported.
    # drop-in: UnparsedDataType('int')

3. The MALFORMED conditions of the conversion of local rows. PySpark's 
converters raise PySparkValueError(errorClass="MALFORMED_VARIANT") 
(conversion.py:889), MALFORMED_GEOGRAPHY (903) and MALFORMED_GEOMETRY (917) 
without messageParameters, so getMessageParameters() is None. The drop-in's are 
{}.

    spark.createDataFrame([("x",)], "v variant")  # getMessageParameters(): 
PySpark None, drop-in {}




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to