aray commented on code in PR #37303:
URL: https://github.com/apache/spark/pull/37303#discussion_r950158722
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala:
##########
@@ -41,6 +42,98 @@ import org.apache.spark.sql.types._
class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession {
import testImplicits._
+ test("DataFrame function and SQL functon parity") {
+ // This test compares the available list of DataFrame functions in
+ // org.apache.spark.sql.functions with the SQL function registry. This
attempts to verify that
+ // the DataFrame functions are a subset of the functions in the SQL
function registry (subject
+ // to exclusions and expectations). It also produces a list of the
differences between the two.
+ // See also test_function_parity in test_functions.py.
+ //
+ // NOTE FOR DEVELOPERS:
+ // If this test fails one of the following needs to happen
+ // * If a function was added to org.apache.spark.sql.functions but not the
function registry
+ // add it to the below expectedOnlyDataFrameFunctions set.
Review Comment:
To clarify this test only checks that `org.apachep.spark.sql.functions._` is
a subset of functions defined in the SQL function registry (subject to
exclusions and expectations), not the other way around. In the other direction
we simply provide a report of functions that are only available in the SQL
function registry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
