HyukjinKwon opened a new pull request, #57079:
URL: https://github.com/apache/spark/pull/57079
### What changes were proposed in this pull request?
This adds `FunctionAcceptedTypesSuite` and a golden file recording, for
every built-in
**scalar** function and every argument position, which input `DataType`s the
analyzer
accepts, rejects, or cannot classify.
For each function + argument position it probes every candidate type (all
atomic types plus
`NULL`, `CalendarInterval`, `Variant`, and representative complex types) by
building
`function(args…)` as an `UnresolvedFunction` with typed-`null` literals and
running it through
the analyzer. Each type is bucketed as:
- **AcceptedDeclared** — accepted, and declared natively by the resolved
expression via
`ExpectsInputTypes` (empty means the expression validates ad hoc — *not*
that it rejects
everything).
- **AcceptedViaCast** — all types the analyzer accepts, including via
implicit cast.
- **Rejected** — `DATATYPE_MISMATCH`.
- **Inconclusive** — other analysis errors (e.g. a foldable-only or value
constraint) that are
not a pure type mismatch.
Results are written to
`sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md`,
regenerated with
`SPARK_GENERATE_GOLDEN_FILES=1` (same convention as
`ExpressionsSchemaSuite`). The current
table covers 433 functions / 653 function-position rows. Functions with no
analyzable baseline
argument list (e.g. many aggregate/window/generator functions, or functions
requiring specific
foldable/enum arguments) are omitted for now and can be added in follow-ups.
The suite is tagged `@ExtendedSQLTest`, so it is **skipped in the default
test run** (it probes
every function against every candidate type) and runs only in the dedicated
"extended tests" CI
leg — no build or CI configuration change is required.
### Why are the changes needed?
Spark's built-in functions do not consistently document which input types
each argument accepts.
The authoritative constraint lives only on the catalyst expression
(`inputTypes` /
`checkInputDataTypes`), while `functions.scala` and `pyspark.sql.functions`
are name-based
facades over it. Following the principle that we should not document what is
not tested, this
suite establishes a **test-backed source of truth** for accepted types.
Planned follow-ups use
this golden table to add and verify accepted-type documentation in SQL
`@ExpressionDescription`,
Scala Scaladoc, and PySpark docstrings.
The `AcceptedDeclared` vs `AcceptedViaCast` split is deliberate: e.g.
`upper` declares only
`STRING` but the analyzer accepts almost any atomic type via implicit cast;
documenting the
declared set is the honest answer, with the cast behaviour available
alongside.
### Does this PR introduce _any_ user-facing change?
No. Test-only (adds a test suite and a test-resource golden file).
### How was this patch tested?
```
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly
*FunctionAcceptedTypesSuite"
-Dtest.include.tags=org.apache.spark.tags.ExtendedSQLTest
```
then re-ran without regeneration to confirm the golden comparison passes
deterministically, and
confirmed the suite does not run in the default (non-extended) test
configuration. Spot-checked
the golden rows for `abs`, `upper`, `date_add`, `array_contains`.
### Was this patch authored or co-authored using generative AI tooling?
Yes, drafted with assistance from Claude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
