Hyukjin Kwon created SPARK-62013:
------------------------------------

             Summary: Make the result objects of a DataFrame PySpark's: the 
bytes WKB of spatial values, the selfDestruct copy of toPandas, one 
executionInfo object and the Arrow metadata of _to_table
                 Key: SPARK-62013
                 URL: https://issues.apache.org/jira/browse/SPARK-62013
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in for pyspark-client 4.3.0 gives four results of a DataFrame that 
differ from PySpark's, each against a Spark 4.3.0 server:

1. Spatial values under spark.sql.execution.pyspark.binaryAsBytes=false. 
PySpark's ArrowTableToRowsConversion makes a Geometry or a Geography of the 
`wkb` bytes of its Arrow struct (python/pyspark/sql/conversion.py:1387-1419), 
whatever the setting, which makes only the binary values bytearray 
(conversion.py:1313-1322). The drop-in's rows give the struct's binary as 
bytearray, of which pyspark.sql.types._udt_converter made the spatial value.

     spark.conf.set("spark.sql.execution.pyspark.binaryAsBytes", "false")
     wkb = "x'0101000000000000000000f03f0000000000000040'"
     row = spark.sql(f"select st_geomfromwkb({wkb}) g, st_geogfromwkb({wkb}) h, 
array(st_geomfromwkb({wkb})) a").collect()[0]
     print([type(v.getBytes()).__name__ for v in (row.g, row.h, row.a[0])])

   pyspark-client 4.3.0: ['bytes', 'bytes', 'bytes']
   drop-in: ['bytearray', 'bytearray', 'bytearray'] (also of toLocalIterator)

2. toPandas with spark.sql.execution.arrow.pyspark.selfDestruct.enabled=true. 
PySpark's SparkConnectClient._execute_and_fetch copies each column of each 
batch into its own allocation with pa.concat_arrays before the conversion 
(python/pyspark/sql/connect/client/core.py:1994-2013). The drop-in's 
DataFrame.toPandas did not.

     spark.conf.set("spark.sql.execution.arrow.pyspark.selfDestruct.enabled", 
"true")
     spark.sql("select interval '1-2' year to month a").toPandas()
     spark.range(3).select().toPandas().shape

   pyspark-client 4.3.0: KeyError: 21 (the copy wraps the month_interval array, 
of which PyArrow has no Python class; also of the empty result of `... where 
false`), and (0, 0), as a copied batch without columns has no rows.
   drop-in: pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas 
block for Arrow data of type month_interval is known., and (3, 0).

3. DataFrame.executionInfo. PySpark's property returns the `_execution_info` 
that the last execution of the DataFrame set 
(python/pyspark/sql/connect/dataframe.py:2434-2436). The drop-in made a new 
ExecutionInfo at each access.

     df = spark.range(3); df.collect()
     a = df.executionInfo; a.custom = 1
     print(a is df.executionInfo, getattr(df.executionInfo, "custom", 
"missing"), a.metrics is df.executionInfo.metrics)

   pyspark-client 4.3.0: True 1 True (also after df.take(1), which runs a 
derived DataFrame)
   drop-in: False missing False

4. DataFrame._to_table(). PySpark's reads the Arrow IPC stream of the server 
with PyArrow (core.py:1889, dataframe.py:1980-1986). The server writes the 
metadata of the schema and of every field, an empty one where there is none, 
which PyArrow reads as {}, and the keys of a field's metadata in its order. The 
drop-in's table, of the batches that arrow-rs decodes, had None for each, and 
the keys in any order.

     wkb = "x'0101000000000000000000f03f0000000000000040'"
     t, _ = spark.sql(f"select st_geomfromwkb({wkb}) g, 1 i")._to_table()
     print(t.schema.metadata, t.schema.field("i").metadata, 
t.schema.field("g").type.field(0).metadata, 
t.schema.field("g").type.field(1).metadata)

   pyspark-client 4.3.0: {} {} {} {b'geometry': b'true', b'srid': b'0'}
   drop-in: None None None and the keys of the last in either order, from run 
to run.

Expected: the four as PySpark's.

Two docs are corrected too:
- dev/parity/known-differences.md: a timestamp whose wall clock in the session 
time zone is outside Python's datetime range (years 1 to 9999), and in the 
local time zone is not, raises OverflowError ("date value out of range") in 
PySpark's collect, toLocalIterator and first, which localize it in the session 
time zone; the drop-in returns the instant in the local time zone, such as 
datetime(1, 1, 1, 0, 0) of timestamp_seconds(-62135596800) with the session 
time zone America/Los_Angeles on a host in UTC. This is intentional, as for the 
session time zones that Python does not know, and is documented, not changed.
- AGENTS.md said that crates/pyspark-rs/src/row.py embeds PySpark's Row class. 
It is a port without PySpark's annotations and the @overloads of __new__ 
(inspect.signature(Row.asDict) is (self, recursive=False), PySpark's (self, 
recursive: bool = False) -> Dict[str, Any]).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to