Shiv-61 commented on issue #59826:
URL: https://github.com/apache/spark/issues/59826#issuecomment-6539581650
## Non-deterministic SQL function documentation generation on duplicate
`ExpressionInfo` names
I observed that `sql/gen-sql-functions-docs.py` produces non-deterministic
documentation between clean JVM runs when multiple `ExpressionInfo` entries
share the same function name (e.g., `explode`, `inline`, `posexplode` in both
`FunctionRegistry` and `TableFunctionRegistry`).
**Root Cause:** In `PythonSQLUtils.listBuiltinFunctionInfos()`:
```scala
(FunctionRegistry.functionSet.flatMap(...) ++
TableFunctionRegistry.functionSet.flatMap(...))
.groupBy(_.getName).map(v => v._2.head).toArray
```
Iteration order over `functionSet` and `groupBy` is not guaranteed to be
stable across JVM runs, so `.head` may select either scalar or table function
info (e.g., `SELECT explode(...)` vs. `SELECT * FROM explode(...)`).
Additionally, `gen-sql-functions-docs.py` sorts only by name, leaving ties
unresolved.
**Proposed Fix:**
- In `PythonSQLUtils.scala`, sort candidate `ExpressionInfo` entries by a
stable tie-breaker (e.g., class name, usage, or registry source) before
selecting `.head`, or return them deterministically.
- In `gen-sql-functions-docs.py`, use a composite sort key `(x.name,
x.usage, x.examples)`.
- Add a regression test verifying stable selection across runs.
Happy to open a PR if this approach looks good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
