zhengruifeng opened a new pull request, #56106:
URL: https://github.com/apache/spark/pull/56106
### What changes were proposed in this pull request?
Adds a `with_sql_conf({K: V, ...})` class decorator on `SQLTestUtils`
in `pyspark.testing.sqlutils` (also exposed as a module-level alias)
that wraps `setUpClass` / `tearDownClass` to apply and revert Spark
confs around the inherited fixtures. Migrates three parity files as
proof:
- `test_parity_column.py` (2 classes)
- `test_parity_udf_combinations.py` (1 class — also makes the
previously asymmetric set-without-unset pair symmetric)
- `test_parity_arrow_python_udf.py` (5 classes)
Usage:
```python
from pyspark.testing.sqlutils import with_sql_conf
@with_sql_conf({"spark.sql.execution.pythonUDF.arrow.enabled": "true"})
class MyParityTests(SomeMixin, ReusedConnectTestCase):
pass
```
### Why are the changes needed?
Many parity tests override `setUpClass` / `tearDownClass` solely to
set one or two Spark confs and unset them on teardown. Each such
override is ~10 lines of identical-shape boilerplate, and the
hand-written `set` / `unset` pairs are easy to drift apart (one such
asymmetric pair is fixed by this PR). Consolidating to a single
declarative decorator removes the duplication and provides a common
seam for migrating the remaining callsites in a follow-up.
### Does this PR introduce _any_ user-facing change?
No — test infrastructure only.
### How was this patch tested?
- New `WithSqlConfTests` in `python/pyspark/sql/tests/test_utils.py`
exercises the decorator end-to-end against `ReusedSQLTestCase`:
asserts both decorator-set keys land via `self.spark.conf.get(...)`.
- The three migrated `ReusedConnectTestCase`-based parity files run
locally; in particular `ColumnParityTests.test_df_col_resolution_mode`
and its `WithNonStrictDFColResolution` subclass read the
decorator-set conf back through `self.spark.conf.get(...)` and pass,
confirming the decorator works on both classic and Connect sessions
and composes correctly through layered subclassing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
