MicroMilo opened a new issue, #59826: URL: https://github.com/apache/spark/issues/59826
### What is the problem? The generated built-in SQL function documentation is nondeterministic when multiple `ExpressionInfo` entries share the same function name. I reproduced this on commit `5b714349fb89a307c247ad7160cae546da909e7e` using the documented generator. Two clean runs in separate JVMs, both with `TZ=UTC` and `spark.sql.session.timeZone=UTC`, completed successfully and each executed 1,381 SQL examples. Both produced 53 generated files, but the generator-function examples differed. For example: Run A: ```sql -- explode SELECT explode(array(10, 20)); ``` Run B: ```sql -- explode SELECT * FROM explode(array(10, 20)); ``` The same instability occurred for `explode_outer`, `inline`, `inline_outer`, `posexplode`, and `posexplode_outer`. ### Root cause `PythonSQLUtils.listBuiltinFunctionInfos()` currently does: ```scala groupBy(_.getName).map(v => v._2.head).toArray ``` `groupBy(...).map(...)` does not preserve a stable iteration order. For duplicate names, `.head` therefore selects an arbitrary `ExpressionInfo` on a fresh JVM. `sql/gen-sql-functions-docs.py` then sorts only by `x.name`, so equal-name entries retain that arbitrary order. Relevant files: - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala#L72-L80 - https://github.com/apache/spark/blob/master/sql/gen-sql-functions-docs.py#L70-L73 ### Suggested fix Sort each duplicate-name group by a stable tie-breaker before selecting an entry, or return all relevant entries in a deterministic order. The Python generator should also use a stable key such as `(name, usage, examples)` rather than only `name`. A regression test can launch two fresh JVMs and compare the selected `(name, usage, examples, since)` tuples, or run the generator twice while ignoring separately documented random/time-dependent example outputs. Scope note: this report is specifically about duplicate-name selection. Separately, examples such as `rand()`, `uuid()`, `current_time()`, and AES-GCM also change output between runs; those are not needed to reproduce this particular bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
