Hyukjin Kwon created SPARK-61999:
------------------------------------
Summary: Keep the metadata of data types in their copies and
pickles, warn of PySpark's modules, and take PySpark's order of the errors of
repartition and the schema of a failed plan
Key: SPARK-61999
URL: https://issues.apache.org/jira/browse/SPARK-61999
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the documented differences of the data types against PySpark
>4.3.0:
- A regression of SPARK-61960 (#832): copy.copy, copy.deepcopy, pickles and
toNullable of a StructType (and of an ArrayType or MapType of one) whose fields
hold metadata that JSON does not keep, such as {'k': (1, 2)} or {1: 'a'}, give
a type that is not equal to the original, as they go through the JSON of the
type; toNullable of {'k': 10**30} gives 1e+30.
- The warnings of the extension (registerTempTable, replace of a dict and a
value, catalog.registerFunction, createExternalTable, the deprecated dbName
arguments, getItem and getField of a Column, the samplingRatio and verifySchema
of createDataFrame, the conf's set and unset, the listener bus, and the window
of no partitions) name the user's file and module, where PySpark's name
pyspark/sql/connect/..., so filterwarnings("ignore", module="pyspark") does not
silence them.
- pyspark.sql.types has no DataTypeSingleton, so PySpark's cloudpickles of a
subclass of a singleton type made in __main__ fail to load.
- The pickles and copies of the drop-in's own types lose user attributes of the
instance's __dict__.
- repartition(2**40, col) raises the ValueError of the count before the errors
of the partition expressions and of the input, where PySpark builds those first.
- schema, columns and dtypes of a DataFrame whose plan raises an AttributeError
(a join on a tuple) raise, where PySpark's properties read
__getattr__("_schema").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]