Vivek1106-04 opened a new pull request, #58242:
URL: https://github.com/apache/spark/pull/58242
### What changes were proposed in this pull request?
This PR adds two new built-in math functions, `gcd` and `lcm`:
```sql
gcd(expr1, expr2) -- returns BIGINT
lcm(expr1, expr2) -- returns BIGINT
```
Both arguments are implicitly cast to BIGINT and the result is BIGINT,
matching the existing
`factorial` expression, which likewise takes an integral argument and
returns BIGINT.
Semantics:
| Case | `gcd` | `lcm` |
|---|---|---|
| either argument is NULL | NULL | NULL |
| both arguments are 0 | 0 | 0 |
| one argument is 0 | `abs(other)` | 0 |
| negative arguments | result is non-negative | result is non-negative |
| result not representable as BIGINT | `ARITHMETIC_OVERFLOW` under ANSI
mode, NULL otherwise | `ARITHMETIC_OVERFLOW` under ANSI mode, NULL otherwise |
The greatest common divisor is computed with the Euclidean algorithm. The
least common multiple
divides by the greatest common divisor before multiplying, so a
representable result never
overflows on the way there — for example `lcm(4611686018427387904, 2)`
returns `4611686018427387904`
even though the naive product `4611686018427387904 * 2` would not fit.
Overflow arises in exactly two places, and in both PostgreSQL raises as well:
* `gcd` where the result would be `-Long.MinValue`, which is not
representable — that is, the
input pairs `(0, x)`, `(x, 0)` and `(x, x)` for `x = Long.MinValue`.
* `lcm` where `abs(a) / gcd(a, b) * abs(b)` exceeds `Long.MaxValue`.
Both are reported with the existing `ARITHMETIC_OVERFLOW` error condition,
so no new error
condition is introduced. Overflow is gated on ANSI mode as it is elsewhere
in Spark — `conv` in the
same file takes the same approach — raising under ANSI mode and returning
NULL otherwise.
Both helpers live in `MathUtils` and are shared by the interpreted and
codegen paths so the two
cannot diverge.
The functions are exposed through SQL, the Scala/Java `functions` API,
PySpark, and Spark Connect.
### Why are the changes needed?
Spark SQL currently has no way to compute either value. There is no
expression for it, and no
combination of existing built-ins produces the result, so users have to fall
back to a UDF — which
for PySpark means a Python round trip per row and no whole-stage codegen.
Both functions are standard in comparable engines:
* PostgreSQL 13+: `gcd(a, b)`, `lcm(a, b)` for `integer`, `bigint` and
`numeric`
* DuckDB: `gcd(a, b)`, `lcm(a, b)` (aliases `greatest_common_divisor`,
`least_common_multiple`)
Common uses include reducing fractions and ratios to lowest terms, aligning
batch or partition
sizes, computing the repeat period of overlapping schedules, and normalizing
denominators before
aggregation.
### Does this PR introduce _any_ user-facing change?
Yes. Two new built-in functions are available in SQL, the Scala/Java
`functions` API, PySpark and
Spark Connect. Previously `gcd` and `lcm` were unresolved function names:
```sql
-- Before
spark-sql> SELECT gcd(24, 36);
[UNRESOLVED_ROUTINE] Cannot resolve routine `gcd` on search path
[`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].
SQLSTATE: 42883
-- After
spark-sql> SELECT gcd(24, 36);
12
spark-sql> SELECT lcm(4, 6);
12
```
No existing behavior changes; the change is purely additive.
### How was this patch tested?
New tests:
* `MathExpressionsSuite` — unit tests for both expressions covering ordinary
values, negative
inputs, zero, NULL, the boundary values around `Long.MinValue` /
`Long.MaxValue`, and overflow in
both ANSI and non-ANSI mode, plus interpreted-vs-codegen consistency
checks.
* `math.sql` golden-file tests, regenerated for the ANSI and non-ANSI
results and the analyzer
results.
* `PlanGenerationTestSuite` function tests, with the regenerated Spark
Connect plan and explain
golden files.
* PySpark doctests for `gcd` and `lcm`.
Existing suites re-run and passing: `MathExpressionsSuite`,
`SQLQueryTestSuite` (`math.sql`),
`ExpressionsSchemaSuite`, `PlanGenerationTestSuite`,
`ProtoToParsedPlanTestSuite`.
`dev/lint-scala` and `dev/lint-python` both pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
