GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/20863
[SPARK-23691][PYTHON][BRANCH-2.3] Use sql_conf util in PySpark tests where
possible
## What changes were proposed in this pull request?
This PR backports https://github.com/apache/spark/pull/20830 to reduce the
diff against master and restore the default value back in PySpark tests.
https://github.com/apache/spark/commit/d6632d185e147fcbe6724545488ad80dce20277e
added an useful util. This backport extracts and brings this util:
```python
@contextmanager
def sql_conf(self, pairs):
...
```
to allow configuration set/unset within a block:
```python
with self.sql_conf({"spark.blah.blah.blah", "blah"})
# test codes
```
This PR proposes to use this util where possible in PySpark tests.
Note that there look already few places affecting tests without restoring
the original value back in unittest classes.
## How was this patch tested?
Likewise, manually tested via:
```
./run-tests --modules=pyspark-sql --python-executables=python2
./run-tests --modules=pyspark-sql --python-executables=python3
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark backport-20830
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20863.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20863
----
commit 4e8045cbddc39a5b8f3488b832a1ac092da68de9
Author: hyukjinkwon <gurwls223@...>
Date: 2018-03-20T04:25:37Z
[SPARK-23691][PYTHON] Use sql_conf util in PySpark tests where possible
https://github.com/apache/spark/commit/d6632d185e147fcbe6724545488ad80dce20277e
added an useful util
```python
contextmanager
def sql_conf(self, pairs):
...
```
to allow configuration set/unset within a block:
```python
with self.sql_conf({"spark.blah.blah.blah", "blah"})
# test codes
```
This PR proposes to use this util where possible in PySpark tests.
Note that there look already few places affecting tests without restoring
the original value back in unittest classes.
Manually tested via:
```
./run-tests --modules=pyspark-sql --python-executables=python2
./run-tests --modules=pyspark-sql --python-executables=python3
```
Author: hyukjinkwon <[email protected]>
Closes #20830 from HyukjinKwon/cleanup-sql-conf.
commit 3056e3c469209d72c97046f9668e30e0dbc5818d
Author: hyukjinkwon <gurwls223@...>
Date: 2018-03-20T05:27:26Z
Extracts and brings sql_conf util
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
