Hyukjin Kwon created SPARK-62013:
------------------------------------
Summary: Make the result objects of a DataFrame PySpark's: the
bytes WKB of spatial values, the selfDestruct copy of toPandas, one
executionInfo object and the Arrow metadata of _to_table
Key: SPARK-62013
URL: https://issues.apache.org/jira/browse/SPARK-62013
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in for pyspark-client 4.3.0 gives four results of a DataFrame that
differ from PySpark's, each against a Spark 4.3.0 server:
1. Spatial values under spark.sql.execution.pyspark.binaryAsBytes=false.
PySpark's ArrowTableToRowsConversion makes a Geometry or a Geography of the
`wkb` bytes of its Arrow struct (python/pyspark/sql/conversion.py:1387-1419),
whatever the setting, which makes only the binary values bytearray
(conversion.py:1313-1322). The drop-in's rows give the struct's binary as
bytearray, of which pyspark.sql.types._udt_converter made the spatial value.
spark.conf.set("spark.sql.execution.pyspark.binaryAsBytes", "false")
wkb = "x'0101000000000000000000f03f0000000000000040'"
row = spark.sql(f"select st_geomfromwkb({wkb}) g, st_geogfromwkb({wkb}) h,
array(st_geomfromwkb({wkb})) a").collect()[0]
print([type(v.getBytes()).__name__ for v in (row.g, row.h, row.a[0])])
pyspark-client 4.3.0: ['bytes', 'bytes', 'bytes']
drop-in: ['bytearray', 'bytearray', 'bytearray'] (also of toLocalIterator)
2. toPandas with spark.sql.execution.arrow.pyspark.selfDestruct.enabled=true.
PySpark's SparkConnectClient._execute_and_fetch copies each column of each
batch into its own allocation with pa.concat_arrays before the conversion
(python/pyspark/sql/connect/client/core.py:1994-2013). The drop-in's
DataFrame.toPandas did not.
spark.conf.set("spark.sql.execution.arrow.pyspark.selfDestruct.enabled",
"true")
spark.sql("select interval '1-2' year to month a").toPandas()
spark.range(3).select().toPandas().shape
pyspark-client 4.3.0: KeyError: 21 (the copy wraps the month_interval array,
of which PyArrow has no Python class; also of the empty result of `... where
false`), and (0, 0), as a copied batch without columns has no rows.
drop-in: pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas
block for Arrow data of type month_interval is known., and (3, 0).
3. DataFrame.executionInfo. PySpark's property returns the `_execution_info`
that the last execution of the DataFrame set
(python/pyspark/sql/connect/dataframe.py:2434-2436). The drop-in made a new
ExecutionInfo at each access.
df = spark.range(3); df.collect()
a = df.executionInfo; a.custom = 1
print(a is df.executionInfo, getattr(df.executionInfo, "custom",
"missing"), a.metrics is df.executionInfo.metrics)
pyspark-client 4.3.0: True 1 True (also after df.take(1), which runs a
derived DataFrame)
drop-in: False missing False
4. DataFrame._to_table(). PySpark's reads the Arrow IPC stream of the server
with PyArrow (core.py:1889, dataframe.py:1980-1986). The server writes the
metadata of the schema and of every field, an empty one where there is none,
which PyArrow reads as {}, and the keys of a field's metadata in its order. The
drop-in's table, of the batches that arrow-rs decodes, had None for each, and
the keys in any order.
wkb = "x'0101000000000000000000f03f0000000000000040'"
t, _ = spark.sql(f"select st_geomfromwkb({wkb}) g, 1 i")._to_table()
print(t.schema.metadata, t.schema.field("i").metadata,
t.schema.field("g").type.field(0).metadata,
t.schema.field("g").type.field(1).metadata)
pyspark-client 4.3.0: {} {} {} {b'geometry': b'true', b'srid': b'0'}
drop-in: None None None and the keys of the last in either order, from run
to run.
Expected: the four as PySpark's.
Two docs are corrected too:
- dev/parity/known-differences.md: a timestamp whose wall clock in the session
time zone is outside Python's datetime range (years 1 to 9999), and in the
local time zone is not, raises OverflowError ("date value out of range") in
PySpark's collect, toLocalIterator and first, which localize it in the session
time zone; the drop-in returns the instant in the local time zone, such as
datetime(1, 1, 1, 0, 0) of timestamp_seconds(-62135596800) with the session
time zone America/Los_Angeles on a host in UTC. This is intentional, as for the
session time zones that Python does not know, and is documented, not changed.
- AGENTS.md said that crates/pyspark-rs/src/row.py embeds PySpark's Row class.
It is a port without PySpark's annotations and the @overloads of __new__
(inspect.signature(Row.asDict) is (self, recursive=False), PySpark's (self,
recursive: bool = False) -> Dict[str, Any]).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]