[
https://issues.apache.org/jira/browse/SPARK-60279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuheng Ma updated SPARK-60279:
-------------------------------
Description:
h3. Description
In Spark Connect, {{GroupedData.pivot(pivot_col, values)}} only accepts values
of type bool, float, int or str. Values that classic PySpark accepts, such as
{{None}}, {{datetime.date}}, {{datetime.datetime}} and {{decimal.Decimal}},
raise {{NOT_EXPECTED_TYPE}}. A common way to hit this is to collect the pivot
values with {{distinct()}}: if the column has a NULL, the list contains
{{None}}.
h3. Reproducer
{code:python}
# spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
.master("local[2]") for classic
import datetime, decimal
df = spark.createDataFrame(
[(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
(2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
"k int, s string, d date, dec decimal(10,1), v int")
vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
df.groupBy("k").pivot("s", vals).sum("v").columns
# classic: ['k', 'a', 'null']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got NoneType.
df.groupBy("k").pivot("d", [datetime.date(2020, 1, 1)]).sum("v").columns
# classic: ['k', '2020-01-01']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got date.
df.groupBy("k").pivot("dec", [decimal.Decimal("1.5")]).sum("v").columns
# classic: ['k', '1.5']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got Decimal.
{code}
h3. Cause
{{GroupedData.pivot}} in {{python/pyspark/sql/connect/group.py}} checks each
value with {{isinstance(v, (bool, float, int, str))}}. The allowed types match
the {{LiteralType}} type hint rather than what {{F.lit}} supports. The values
are later turned into literals with {{F.lit}}, which handles None, date,
datetime and Decimal, so the check rejects values that would work. The check
has been there since the Connect pivot was added (SPARK-41693, 3.4.0).
Code:
https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257
h3. Suggested fix
Also accept None, datetime.date (which covers datetime.datetime) and
decimal.Decimal in that check. Nested lists and dicts keep the current error;
classic rejects them too. Classic also accepts bytes, bytearray, datetime.time,
datetime.timedelta, NumPy scalars such as np.int64, np.bool_ and np.float32,
and a tuple of values; this fix does not cover them. Another option is to drop
the per-value check and let {{F.lit}} validate, as classic leaves it to the
JVM: that also covers bytes, time, timedelta and NumPy scalars, but a Column or
a nested list would then fail on the server with
{{CONNECT_INVALID_PLAN.UNSUPPORTED_LITERAL_TYPE}} instead of on the client.
h3. Who hits it
A feature pipeline on Databricks serverless (Spark Connect) hit the None case
and worked around it by dropping None from the values, which loses the "null"
column that classic would produce:
https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3
Tested on PySpark 4.2.0 from PyPI and on master (5.0.0-SNAPSHOT), Python 3.12,
with a local Spark Connect server (first line of the reproducer). I have a fix
with tests ready and will open a PR shortly.
Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
reproducer and this description.
Generated-by: Claude Code (Claude Opus 5.5)
was:
h3. Description
In Spark Connect, {{GroupedData.pivot(pivot_col, values)}} only accepts values
of type bool, float, int or str. Values that classic PySpark accepts, such as
{{None}}, {{datetime.date}}, {{datetime.datetime}} and {{decimal.Decimal}},
raise {{NOT_EXPECTED_TYPE}}. A common way to hit this is to collect the pivot
values with {{distinct()}}: if the column has a NULL, the list contains
{{None}}.
h3. Reproducer
{code:python}
# spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
.master("local[2]") for classic
import datetime, decimal
df = spark.createDataFrame(
[(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
(2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
"k int, s string, d date, dec decimal(10,1), v int")
vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
df.groupBy("k").pivot("s", vals).sum("v").columns
# classic: ['k', 'a', 'null']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got NoneType.
df.groupBy("k").pivot("d", [datetime.date(2020, 1, 1)]).sum("v").columns
# classic: ['k', '2020-01-01']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got date.
df.groupBy("k").pivot("dec", [decimal.Decimal("1.5")]).sum("v").columns
# classic: ['k', '1.5']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got Decimal.
{code}
h3. Cause
{{GroupedData.pivot}} in {{python/pyspark/sql/connect/group.py}} checks each
value with {{isinstance(v, (bool, float, int, str))}}. The allowed types match
the {{LiteralType}} type hint rather than what {{F.lit}} supports. The values
are later turned into literals with {{F.lit}}, which handles None, date,
datetime and Decimal, so the check rejects values that would work. The check
has been there since the Connect pivot was added (SPARK-41693, 3.4.0).
Code:
https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257
h3. Suggested fix
Also accept None, datetime.date (which covers datetime.datetime) and
decimal.Decimal in that check. Nested lists and dicts keep the current error;
classic rejects them too. Classic also accepts bytes, bytearray, datetime.time,
datetime.timedelta, NumPy scalars such as np.int64, np.bool_ and np.float32,
and a tuple of values; this fix does not cover them. Another option is to drop
the per-value check and let {{F.lit}} validate, as classic leaves it to the
JVM: that also covers bytes, time, timedelta and NumPy scalars, but a Column or
a nested list would then fail on the server with
{{CONNECT_INVALID_PLAN.UNSUPPORTED_LITERAL_TYPE}} instead of on the client.
h3. Who hits it
A feature pipeline on Databricks serverless (Spark Connect) hit the None case
and worked around it by dropping None from the values, which loses the "null"
column that classic would produce:
https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3
Tested on PySpark 4.2.0 from PyPI and on master (5.0.0-SNAPSHOT), Python 3.12,
with a local Spark Connect server (first line of the reproducer). I have a fix
with tests ready and will open a PR shortly.
Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
reproducer and this description.
Generated-by: Claude Code (Claude Opus 5.5)
> Spark Connect GroupedData.pivot rejects None, date and Decimal pivot values
> that classic accepts
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-60279
> URL: https://issues.apache.org/jira/browse/SPARK-60279
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 5.0.0
> Reporter: Shuheng Ma
> Priority: Minor
>
> h3. Description
> In Spark Connect, {{GroupedData.pivot(pivot_col, values)}} only accepts
> values of type bool, float, int or str. Values that classic PySpark accepts,
> such as {{None}}, {{datetime.date}}, {{datetime.datetime}} and
> {{decimal.Decimal}}, raise {{NOT_EXPECTED_TYPE}}. A common way to hit this is
> to collect the pivot values with {{distinct()}}: if the column has a NULL,
> the list contains {{None}}.
> h3. Reproducer
> {code:python}
> # spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
> .master("local[2]") for classic
> import datetime, decimal
> df = spark.createDataFrame(
> [(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
> (2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
> "k int, s string, d date, dec decimal(10,1), v int")
> vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
> df.groupBy("k").pivot("s", vals).sum("v").columns
> # classic: ['k', 'a', 'null']
> # Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
> bool, float, int or str, got NoneType.
> df.groupBy("k").pivot("d", [datetime.date(2020, 1, 1)]).sum("v").columns
> # classic: ['k', '2020-01-01']
> # Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
> bool, float, int or str, got date.
> df.groupBy("k").pivot("dec", [decimal.Decimal("1.5")]).sum("v").columns
> # classic: ['k', '1.5']
> # Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
> bool, float, int or str, got Decimal.
> {code}
> h3. Cause
> {{GroupedData.pivot}} in {{python/pyspark/sql/connect/group.py}} checks each
> value with {{isinstance(v, (bool, float, int, str))}}. The allowed types
> match the {{LiteralType}} type hint rather than what {{F.lit}} supports. The
> values are later turned into literals with {{F.lit}}, which handles None,
> date, datetime and Decimal, so the check rejects values that would work. The
> check has been there since the Connect pivot was added (SPARK-41693, 3.4.0).
> Code:
> https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257
> h3. Suggested fix
> Also accept None, datetime.date (which covers datetime.datetime) and
> decimal.Decimal in that check. Nested lists and dicts keep the current error;
> classic rejects them too. Classic also accepts bytes, bytearray,
> datetime.time, datetime.timedelta, NumPy scalars such as np.int64, np.bool_
> and np.float32, and a tuple of values; this fix does not cover them. Another
> option is to drop the per-value check and let {{F.lit}} validate, as classic
> leaves it to the JVM: that also covers bytes, time, timedelta and NumPy
> scalars, but a Column or a nested list would then fail on the server with
> {{CONNECT_INVALID_PLAN.UNSUPPORTED_LITERAL_TYPE}} instead of on the client.
> h3. Who hits it
> A feature pipeline on Databricks serverless (Spark Connect) hit the None case
> and worked around it by dropping None from the values, which loses the "null"
> column that classic would produce:
> https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3
> Tested on PySpark 4.2.0 from PyPI and on master (5.0.0-SNAPSHOT), Python
> 3.12, with a local Spark Connect server (first line of the reproducer). I
> have a fix with tests ready and will open a PR shortly.
> Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
> reproducer and this description.
> Generated-by: Claude Code (Claude Opus 5.5)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
