Hyukjin Kwon created SPARK-61999:
------------------------------------

             Summary: Keep the metadata of data types in their copies and 
pickles, warn of PySpark's modules, and take PySpark's order of the errors of 
repartition and the schema of a failed plan
                 Key: SPARK-61999
                 URL: https://issues.apache.org/jira/browse/SPARK-61999
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the documented differences of the data types against PySpark 
>4.3.0:
- A regression of SPARK-61960 (#832): copy.copy, copy.deepcopy, pickles and 
toNullable of a StructType (and of an ArrayType or MapType of one) whose fields 
hold metadata that JSON does not keep, such as {'k': (1, 2)} or {1: 'a'}, give 
a type that is not equal to the original, as they go through the JSON of the 
type; toNullable of {'k': 10**30} gives 1e+30.
- The warnings of the extension (registerTempTable, replace of a dict and a 
value, catalog.registerFunction, createExternalTable, the deprecated dbName 
arguments, getItem and getField of a Column, the samplingRatio and verifySchema 
of createDataFrame, the conf's set and unset, the listener bus, and the window 
of no partitions) name the user's file and module, where PySpark's name 
pyspark/sql/connect/..., so filterwarnings("ignore", module="pyspark") does not 
silence them.
- pyspark.sql.types has no DataTypeSingleton, so PySpark's cloudpickles of a 
subclass of a singleton type made in __main__ fail to load.
- The pickles and copies of the drop-in's own types lose user attributes of the 
instance's __dict__.
- repartition(2**40, col) raises the ValueError of the count before the errors 
of the partition expressions and of the input, where PySpark builds those first.
- schema, columns and dtypes of a DataFrame whose plan raises an AttributeError 
(a join on a tuple) raise, where PySpark's properties read 
__getattr__("_schema").




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to