Hyukjin Kwon created SPARK-21914:
------------------------------------
Summary: Running examples as tests in SQL builtin function
documentation
Key: SPARK-21914
URL: https://issues.apache.org/jira/browse/SPARK-21914
Project: Spark
Issue Type: Test
Components: SQL
Affects Versions: 2.3.0
Reporter: Hyukjin Kwon
It looks we have added many examples in {{ExpressionDescription}} for builtin
functions.
Actually, if I have seen correctly, we have fixed many examples so far in some
minor PRs and sometimes require to add the examples as tests sql and golden
files.
As we have formatted examples in {{ExpressionDescription.examples}} -
https://github.com/apache/spark/blob/ba327ee54c32b11107793604895bd38559804858/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java#L44-L50,
and we have `SQLQueryTestSuite`, I think we could run the examples as tests
like Python's doctests.
Rough way I am thinking:
1. Loads the example in {{ExpressionDescription}}.
2. identify queries by {{>}}.
3. identify the rest of them as the results.
4. run the examples by reusing {{SQLQueryTestSuite}} if possible.
5. compare the output by reusing {{SQLQueryTestSuite}} if possible.
Advantages of doing this I could think for now:
- Reduce the number of PRs to fix the examples
- De-duplicate the test cases that should be added into sql and golden files.
- Correct documentation with correct examples.
- Reduce reviewing costs for documentation fix PRs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
