[
https://issues.apache.org/jira/browse/SPARK-57999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-57999:
---------------------------------
Issue Type: Umbrella (was: Test)
Summary: Document accepted input and return types for built-in functions
(was: Document accepted input types for built-in functions across SQL, Scala
and PySpark, backed by a generated test)
> Document accepted input and return types for built-in functions
> ---------------------------------------------------------------
>
> Key: SPARK-57999
> URL: https://issues.apache.org/jira/browse/SPARK-57999
> Project: Spark
> Issue Type: Umbrella
> Components: SQL, Tests
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Minor
> Labels: pull-request-available
>
> Spark's built-in functions do not consistently document which input DataTypes
> each argument accepts. The authoritative constraint lives only on the
> catalyst expression (inputTypes / checkInputDataTypes), and the public
> functions APIs (functions.scala, pyspark.sql.functions) are name-based
> facades over it.
> Following the principle that we should not document what is not tested, this
> issue tracks documenting the accepted input types of built-in functions
> across all three surfaces -- SQL @ExpressionDescription, Scala Scaladoc, and
> PySpark docstrings -- with every documented type backed by a test.
> The work is staged in a single PR:
> 1. FunctionAcceptedTypesSuite: an exhaustive, generated suite that probes
> every built-in scalar function and argument position against every candidate
> DataType through the analyzer, recording whether each type is accepted
> (declared natively via ExpectsInputTypes vs. only via implicit cast),
> rejected (DATATYPE_MISMATCH), or inconclusive. Results are written to a
> golden file,
> sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md
> (regenerated with SPARK_GENERATE_GOLDEN_FILES=1), and the suite is tagged
> @ExtendedSQLTest so it is skipped in the default test run.
> 2. A structured, machine-parseable 'Accepted types' convention added to each
> surface, with a consistency test per surface that fails if the documented
> types disagree with the golden table: SQL @ExpressionDescription arguments
> (verified in ExpressionInfoSuite), PySpark Parameters docstrings, and Scala
> functions.scala @param Scaladoc.
> Coverage is filled and enforced group-by-group (math_funcs, string_funcs,
> ...).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
