hzmsh opened a new pull request, #59541: URL: https://github.com/apache/spark/pull/59541
### What changes were proposed in this pull request? Fixes [SPARK-60279](https://issues.apache.org/jira/browse/SPARK-60279). Fix: - `GroupedData.pivot` in Spark Connect checks each pivot value with `isinstance(v, (bool, float, int, str))`. This PR also accepts `None`, `datetime.date` (which covers `datetime.datetime`) and `decimal.Decimal` there. - The values are then passed to `F.lit` (in `GroupedData.__init__`), and `F.lit` already supports these types, so nothing else needs to change. The `expected_type` in the error message is updated to match. - Classic is not changed: it passes the values to the JVM, which already accepts them. ### Why are the changes needed? The same code works in classic and fails in Connect. A common way to hit it is to collect the pivot values with `distinct()`, which gives `None` when the column has a NULL. ```python # spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect, .master("local[2]") for classic import datetime, decimal df = spark.createDataFrame( [(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10), (2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)], "k int, s string, d date, dec decimal(10,1), v int", ) vals = [r[0] for r in df.select("s").distinct().collect()] # contains None df.groupBy("k").pivot("s", vals).sum("v").columns # classic: ['k', 'a', 'null'] # Connect on master and 4.2.0: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be bool, float, int or str, got NoneType. ``` `datetime.date` and `decimal.Decimal` values fail the same way (`got date`, `got Decimal`), while classic gives columns `2020-01-01` and `1.5`. When it started: - The check has been there since the Connect pivot was added in [SPARK-41693](https://issues.apache.org/jira/browse/SPARK-41693) (3.4.0), so every Connect version is affected. The allowed types match the `LiteralType` type hint (`Union[bool, float, int, str]`); `F.lit` already handled `None`, `date` and `Decimal` at that time. Reports from users: - A feature pipeline on Databricks serverless (Spark Connect) hit the `None` case: the first error in its error catalog is `NOT_EXPECTED_TYPE: got NoneType` in `.pivot()`. The workaround drops `None` from the values, which loses the `null` column that classic would produce ([WenjieAtBayer/kerendia_nba_3_1_develop#3](https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3)). What goes wrong: - In [`connect/group.py`](https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257), `pivot` raises `NOT_EXPECTED_TYPE` for any value that is not a bool, float, int or str, before the values reach `F.lit`. ### Does this PR introduce _any_ user-facing change? Yes, relative to 4.2.0 as well as master. Behavior change: - In Spark Connect, `pivot` now accepts `None`, `date`, `datetime` and `Decimal` values and returns the same columns and rows as classic. Before, it raised `NOT_EXPECTED_TYPE`. - The error message for other values changes from "should be bool, float, int or str" to "should be bool, date, Decimal, float, int, None or str". - `pivot` without `values` and values of the types accepted before behave as before. Not covered: - A tuple of values (`values=("a",)`) is still rejected in Connect, though classic accepts it at runtime. The `values` type hint does not allow tuples in classic either; [SPARK-37116](https://issues.apache.org/jira/browse/SPARK-37116) (allow tuples as pivot values in PySpark) has been open since 2021. - `bytes`, `bytearray`, `datetime.time`, `datetime.timedelta` and NumPy scalars such as `np.int64(1)`, `np.bool_(True)` and `np.float32(1.5)` are still rejected in Connect, though classic accepts them. Only `np.float64` and `np.str_` values already work, because they subclass `float` and `str`. They are left out to keep this change small; `SparkConnectStatTests.test_grouped_data` still uses `bytes` as its invalid example. - The `values` type hint (`List[LiteralType]`) is unchanged in both classic and Connect, so type checkers still flag `None`, `date` and `Decimal` values. ### How was this patch tested? Tests: - Added `GroupTestsMixin.test_pivot_values_with_null_date_and_decimal`, which runs in classic (`test_group`) and Connect (`test_parity_group`). It checks the column names and rows for pivot values from `distinct()` that include `None` (columns `k`, `null`, `a`), a `date` value (column `2020-01-01`) and a `Decimal` value (column `10.0`). - Updated the expected message in `SparkConnectStatTests.test_grouped_data`. - Without the fix, the new test passes in classic and fails in Connect with `NOT_EXPECTED_TYPE ... got NoneType`. - With the fix, all tests in the three modules pass (9 in `test_group`, 9 in `test_parity_group`, 18 in `test_connect_stat`). - The repro above was also run against PySpark 4.2.0 from PyPI. - Commands: ``` python/run-tests --python-executables=python3 --testnames 'pyspark.sql.tests.test_group,pyspark.sql.tests.connect.test_parity_group,pyspark.sql.tests.connect.test_connect_stat' dev/lint-python ``` ### Was this patch authored or co-authored using generative AI tooling? Co-developed with Claude Code (Claude Opus 5.5): the fix, the tests and this description. The checks above were run locally. Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
