ybapat opened a new pull request, #57568:
URL: https://github.com/apache/spark/pull/57568
### What changes were proposed in this pull request?
The SQL `decode` function in Spark supports two forms:
1. **Binary decode** (2-arg): `decode(binary_col, charset)` — decodes binary
data to a string.
2. **Oracle-style CASE WHEN** (3+ args): `decode(expr, search1, result1,
..., [default])` — returns the result for the first matching search value.
The CASE WHEN form is implemented in Scala (`object Decode { def
createExpr(...) }`) and available via `spark.sql("SELECT decode(...)")`, but
PySpark's `decode()` function only exposed the 2-argument binary form. Issue
#57534 tracks this gap.
This PR updates `pyspark.sql.functions.decode` (both classic and Connect
paths) to accept variadic `*args` and dispatch to the correct form:
- 1 arg → binary decode with charset (existing behavior)
- 3+ args → Oracle-style CASE WHEN (new)
- 0 args → `PySparkTypeError` with `NOT_ENOUGH_ARGS`
Changes:
- `python/pyspark/sql/functions/builtin.py`: Change signature to
`decode(col, *args)`, add 3-branch dispatch, update docstring with examples.
- `python/pyspark/sql/connect/functions/builtin.py`: Mirror changes for
Spark Connect path.
- `python/pyspark/errors/error-conditions.json`: Add `NOT_ENOUGH_ARGS` error
class.
### Why are the changes needed?
PySpark users cannot use the Oracle-style `decode` via the Python API
without resorting to `spark.sql()`. This change closes the gap between the SQL
and Python APIs.
### Does this PR introduce _any_ user-facing change?
Yes — new functionality. The existing 2-argument binary decode is fully
backward compatible. Users can now call:
```python
sf.decode(col("status"), lit(1), lit("active"), lit(0), lit("inactive"),
lit("unknown"))
```
### How was this patch tested?
Manual verification of both dispatch paths. The existing doctest for binary
decode continues to pass. The CASE WHEN examples are marked `# doctest: +SKIP`
as they require a live Spark session.
Closes #57534
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
