[
https://issues.apache.org/jira/browse/SPARK-62013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62013:
------------------------------------
Assignee: Hyukjin Kwon
> Make the result objects of a DataFrame PySpark's: the bytes WKB of spatial
> values, the selfDestruct copy of toPandas, one executionInfo object and the
> Arrow metadata of _to_table
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62013
> URL: https://issues.apache.org/jira/browse/SPARK-62013
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> The drop-in for pyspark-client 4.3.0 gives four results of a DataFrame that
> differ from PySpark's, each against a Spark 4.3.0 server:
> 1. Spatial values under spark.sql.execution.pyspark.binaryAsBytes=false.
> PySpark's ArrowTableToRowsConversion makes a Geometry or a Geography of the
> `wkb` bytes of its Arrow struct (python/pyspark/sql/conversion.py:1387-1419),
> whatever the setting, which makes only the binary values bytearray
> (conversion.py:1313-1322). The drop-in's rows give the struct's binary as
> bytearray, of which pyspark.sql.types._udt_converter made the spatial value.
> spark.conf.set("spark.sql.execution.pyspark.binaryAsBytes", "false")
> wkb = "x'0101000000000000000000f03f0000000000000040'"
> row = spark.sql(f"select st_geomfromwkb({wkb}) g, st_geogfromwkb({wkb})
> h, array(st_geomfromwkb({wkb})) a").collect()[0]
> print([type(v.getBytes()).__name__ for v in (row.g, row.h, row.a[0])])
> pyspark-client 4.3.0: ['bytes', 'bytes', 'bytes']
> drop-in: ['bytearray', 'bytearray', 'bytearray'] (also of toLocalIterator)
> 2. toPandas with spark.sql.execution.arrow.pyspark.selfDestruct.enabled=true.
> PySpark's SparkConnectClient._execute_and_fetch copies each column of each
> batch into its own allocation with pa.concat_arrays before the conversion
> (python/pyspark/sql/connect/client/core.py:1994-2013). The drop-in's
> DataFrame.toPandas did not.
> spark.conf.set("spark.sql.execution.arrow.pyspark.selfDestruct.enabled",
> "true")
> spark.sql("select interval '1-2' year to month a").toPandas()
> spark.range(3).select().toPandas().shape
> pyspark-client 4.3.0: KeyError: 21 (the copy wraps the month_interval
> array, of which PyArrow has no Python class; also of the empty result of `...
> where false`), and (0, 0), as a copied batch without columns has no rows.
> drop-in: pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas
> block for Arrow data of type month_interval is known., and (3, 0).
> 3. DataFrame.executionInfo. PySpark's property returns the `_execution_info`
> that the last execution of the DataFrame set
> (python/pyspark/sql/connect/dataframe.py:2434-2436). The drop-in made a new
> ExecutionInfo at each access.
> df = spark.range(3); df.collect()
> a = df.executionInfo; a.custom = 1
> print(a is df.executionInfo, getattr(df.executionInfo, "custom",
> "missing"), a.metrics is df.executionInfo.metrics)
> pyspark-client 4.3.0: True 1 True (also after df.take(1), which runs a
> derived DataFrame)
> drop-in: False missing False
> 4. DataFrame._to_table(). PySpark's reads the Arrow IPC stream of the server
> with PyArrow (core.py:1889, dataframe.py:1980-1986). The server writes the
> metadata of the schema and of every field, an empty one where there is none,
> which PyArrow reads as {}, and the keys of a field's metadata in its order.
> The drop-in's table, of the batches that arrow-rs decodes, had None for each,
> and the keys in any order.
> wkb = "x'0101000000000000000000f03f0000000000000040'"
> t, _ = spark.sql(f"select st_geomfromwkb({wkb}) g, 1 i")._to_table()
> print(t.schema.metadata, t.schema.field("i").metadata,
> t.schema.field("g").type.field(0).metadata,
> t.schema.field("g").type.field(1).metadata)
> pyspark-client 4.3.0: {} {} {} {b'geometry': b'true', b'srid': b'0'}
> drop-in: None None None and the keys of the last in either order, from run
> to run.
> Expected: the four as PySpark's.
> Two docs are corrected too:
> - dev/parity/known-differences.md: a timestamp whose wall clock in the
> session time zone is outside Python's datetime range (years 1 to 9999), and
> in the local time zone is not, raises OverflowError ("date value out of
> range") in PySpark's collect, toLocalIterator and first, which localize it in
> the session time zone; the drop-in returns the instant in the local time
> zone, such as datetime(1, 1, 1, 0, 0) of timestamp_seconds(-62135596800) with
> the session time zone America/Los_Angeles on a host in UTC. This is
> intentional, as for the session time zones that Python does not know, and is
> documented, not changed.
> - AGENTS.md said that crates/pyspark-rs/src/row.py embeds PySpark's Row
> class. It is a port without PySpark's annotations and the @overloads of
> __new__ (inspect.signature(Row.asDict) is (self, recursive=False), PySpark's
> (self, recursive: bool = False) -> Dict[str, Any]).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]