[
https://issues.apache.org/jira/browse/SPARK-60279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60279:
-----------------------------------
Labels: pull-request-available (was: )
> Spark Connect GroupedData.pivot rejects None, date and Decimal pivot values
> that classic accepts
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-60279
> URL: https://issues.apache.org/jira/browse/SPARK-60279
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 5.0.0
> Reporter: Shuheng Ma
> Priority: Minor
> Labels: pull-request-available
>
> h3. Description
> In Spark Connect, {{GroupedData.pivot(pivot_col, values)}} only accepts
> values of type bool, float, int or str. Values that classic PySpark accepts,
> such as {{None}}, {{datetime.date}}, {{datetime.datetime}} and
> {{decimal.Decimal}}, raise {{NOT_EXPECTED_TYPE}}. A common way to hit this is
> to collect the pivot values with {{distinct()}}: if the column has a NULL,
> the list contains {{None}}.
> h3. Reproducer
> {code:python}
> # spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
> .master("local[2]") for classic
> import datetime, decimal
> df = spark.createDataFrame(
> [(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
> (2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
> "k int, s string, d date, dec decimal(10,1), v int")
> vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
> df.groupBy("k").pivot("s", vals).sum("v").columns
> # classic: ['k', 'a', 'null']
> # Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
> bool, float, int or str, got NoneType.
> df.groupBy("k").pivot("d", [datetime.date(2020, 1, 1)]).sum("v").columns
> # classic: ['k', '2020-01-01']
> # Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
> bool, float, int or str, got date.
> df.groupBy("k").pivot("dec", [decimal.Decimal("1.5")]).sum("v").columns
> # classic: ['k', '1.5']
> # Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
> bool, float, int or str, got Decimal.
> {code}
> h3. Cause
> {{GroupedData.pivot}} in {{python/pyspark/sql/connect/group.py}} checks each
> value with {{isinstance(v, (bool, float, int, str))}}. The allowed types
> match the {{LiteralType}} type hint rather than what {{F.lit}} supports. The
> values are later turned into literals with {{F.lit}}, which handles None,
> date, datetime and Decimal, so the check rejects values that would work. The
> check has been there since the Connect pivot was added (SPARK-41693, 3.4.0).
> Code:
> https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257
> h3. Suggested fix
> Also accept None, datetime.date (which covers datetime.datetime) and
> decimal.Decimal in that check. Nested lists and dicts keep the current error;
> classic rejects them too. Classic also accepts bytes, bytearray,
> datetime.time, datetime.timedelta, NumPy scalars such as np.int64, np.bool_
> and np.float32, and a tuple of values; this fix does not cover them. Another
> option is to drop the per-value check and let {{F.lit}} validate, as classic
> leaves it to the JVM: that also covers bytes, time, timedelta and NumPy
> scalars, but a Column or a nested list would then fail on the server with
> {{CONNECT_INVALID_PLAN.UNSUPPORTED_LITERAL_TYPE}} instead of on the client.
> h3. Who hits it
> A feature pipeline on Databricks serverless (Spark Connect) hit the None case
> and worked around it by dropping None from the values, which loses the "null"
> column that classic would produce:
> https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3
> Tested on PySpark 4.2.0 from PyPI and on master (5.0.0-SNAPSHOT), Python
> 3.12, with a local Spark Connect server (first line of the reproducer). I
> have a fix with tests ready and will open a PR shortly.
> Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
> reproducer and this description.
> Generated-by: Claude Code (Claude Opus 5.5)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
