dtenedor opened a new pull request, #57691:
URL: https://github.com/apache/spark/pull/57691
### What changes were proposed in this pull request?
This PR follows up on #57122 and #57639 by replacing the per-case Python
snippet files of the DataFrame golden test framework with a single test class
per golden file.
The files under `python/pyspark/sql/tests/df_golden/scripts/group_by/` are
replaced by one module, `python/pyspark/sql/tests/df_golden/test_group_by.py`,
holding one class whose `_test_<case>` methods each build and return the
DataFrame under test:
```python
class GroupByGoldenTests(DFGoldenTestMixin, ReusedConnectTestCase):
golden_file = "group_by.test"
@unordered
def _test_group_by_count(self, spark):
"""Aggregate with non-empty GroupBy expressions."""
return
spark.table("testData").groupBy(col("a")).agg(count(col("b")))
```
This is ordinary Python: imports are at the top of the module, cases can
share helpers, and `spark` is a parameter rather than a name injected into an
`exec()` namespace.
Cases are ordinary unit tests: `DFGoldenTestMixin` (a mixin, following the
existing `pyspark.testing.goldenutils.GoldenFileTestMixin` pattern) registers a
real `test_<case>` method for every `_test_<case>` method, carrying its
docstring over. Each case is reported individually and can be run on its own:
```
python/run-tests --testnames \
"pyspark.sql.tests.df_golden.test_group_by
GroupByGoldenTests.test_group_by_count"
```
### Why are the changes needed?
The framework as merged put each case's DataFrame program in its own file,
which drew several objections in the review of #57122:
1. **File count.** One file per case does not scale.
2. **The snippets were not real Python.** They were `exec()`'d with `spark`
injected into their namespace, which is why they needed an `F821` lint
exemption and a RAT exclusion, and why they could not be imported or run
outside the framework.
3. **Reproducing a failure was hard** A whole `.test` file used to run as
one unittest method, so a failure named a case but gave no way to run just that
case. Now every case is a test method that can be selected by name, and the
case body is an importable method.
4. **Structure.** Organizing cases as methods of a class is the conventional
Python shape, and it lets cases share helpers.
### Does this PR introduce _any_ user-facing change?
No. This is test-only: the framework and its test corpus.
### How was this patch tested?
This PR is test-only. The migration was verified as follows.
- Golden outputs are unchanged.
- The regeneration writer reproduces the file byte for byte.
- The class and the golden file are in sync.
- Framework unit tests:
`pyspark.sql.tests.df_golden.test_df_golden_framework` passes (51 tests).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
