[
https://issues.apache.org/jira/browse/SPARK-57999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-57999:
---------------------------------
Description:
Spark's built-in functions do not consistently document which input DataTypes
each argument accepts. The authoritative constraint lives only on the catalyst
expression (inputTypes / checkInputDataTypes), and the public functions APIs
(functions.scala, pyspark.sql.functions) are name-based facades over it.
Following the principle that we should not document what is not tested, this
issue tracks documenting the accepted input types of built-in functions across
all three surfaces -- SQL @ExpressionDescription, Scala Scaladoc, and PySpark
docstrings -- with every documented type backed by a test.
The work is staged in a single PR:
1. FunctionAcceptedTypesSuite: an exhaustive, generated suite that probes every
built-in scalar function and argument position against every candidate DataType
through the analyzer, recording whether each type is accepted (declared
natively via ExpectsInputTypes vs. only via implicit cast), rejected
(DATATYPE_MISMATCH), or inconclusive. Results are written to a golden file,
sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md
(regenerated with SPARK_GENERATE_GOLDEN_FILES=1), and the suite is tagged
@ExtendedSQLTest so it is skipped in the default test run.
2. A structured, machine-parseable 'Accepted types' convention added to each
surface, with a consistency test per surface that fails if the documented types
disagree with the golden table: SQL @ExpressionDescription arguments (verified
in ExpressionInfoSuite), PySpark Parameters docstrings, and Scala
functions.scala @param Scaladoc.
Coverage is filled and enforced group-by-group (math_funcs, string_funcs, ...).
was:
Spark's built-in functions do not consistently document which input DataTypes
each argument accepts. The authoritative constraint lives only on the catalyst
expression (inputTypes / checkInputDataTypes), and the public functions APIs
(functions.scala, pyspark.sql.functions) are name-based facades over it.
As a foundation for documenting accepted types (and per the principle that we
should not document what is not tested), this adds FunctionAcceptedTypesSuite:
an exhaustive, generated suite that, for every built-in scalar function and
every argument position, probes each candidate DataType through the analyzer
and records whether it is accepted (declared natively via ExpectsInputTypes vs.
only via implicit cast), rejected (DATATYPE_MISMATCH), or inconclusive (other
analysis errors). Results are written to a golden file,
sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md,
regenerated with SPARK_GENERATE_GOLDEN_FILES=1.
The suite is tagged @ExtendedSQLTest so it is skipped in the default test run
(it probes every function against every type) and runs only in the dedicated
extended-tests CI leg.
This is the first step; follow-ups will use the golden table to drive and
verify accepted-type documentation in SQL @ExpressionDescription, Scala
Scaladoc, and PySpark docstrings.
Summary: Document accepted input types for built-in functions across
SQL, Scala and PySpark, backed by a generated test (was: Add a generated,
test-backed table of accepted input types for built-in scalar functions)
> Document accepted input types for built-in functions across SQL, Scala and
> PySpark, backed by a generated test
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57999
> URL: https://issues.apache.org/jira/browse/SPARK-57999
> Project: Spark
> Issue Type: Test
> Components: SQL, Tests
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Minor
> Labels: pull-request-available
>
> Spark's built-in functions do not consistently document which input DataTypes
> each argument accepts. The authoritative constraint lives only on the
> catalyst expression (inputTypes / checkInputDataTypes), and the public
> functions APIs (functions.scala, pyspark.sql.functions) are name-based
> facades over it.
> Following the principle that we should not document what is not tested, this
> issue tracks documenting the accepted input types of built-in functions
> across all three surfaces -- SQL @ExpressionDescription, Scala Scaladoc, and
> PySpark docstrings -- with every documented type backed by a test.
> The work is staged in a single PR:
> 1. FunctionAcceptedTypesSuite: an exhaustive, generated suite that probes
> every built-in scalar function and argument position against every candidate
> DataType through the analyzer, recording whether each type is accepted
> (declared natively via ExpectsInputTypes vs. only via implicit cast),
> rejected (DATATYPE_MISMATCH), or inconclusive. Results are written to a
> golden file,
> sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md
> (regenerated with SPARK_GENERATE_GOLDEN_FILES=1), and the suite is tagged
> @ExtendedSQLTest so it is skipped in the default test run.
> 2. A structured, machine-parseable 'Accepted types' convention added to each
> surface, with a consistency test per surface that fails if the documented
> types disagree with the golden table: SQL @ExpressionDescription arguments
> (verified in ExpressionInfoSuite), PySpark Parameters docstrings, and Scala
> functions.scala @param Scaladoc.
> Coverage is filled and enforced group-by-group (math_funcs, string_funcs,
> ...).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
