This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 2e382c8 [SPARK-38073][PYTHON] Update atexit function to avoid issues
with late binding
2e382c8 is described below
commit 2e382c8bff2d0c3733b9b525168254971ca1175e
Author: zero323 <[email protected]>
AuthorDate: Fri Feb 4 20:21:02 2022 -0800
[SPARK-38073][PYTHON] Update atexit function to avoid issues with late
binding
### What changes were proposed in this pull request?
This PR updates function registered in PySpark shell `atexit` to capture
`SparkContext` instead of depending on the surrounding context.
**Note**
A simpler approach
```python
atexit.register(sc.stop)
```
is possible, but won't work properly in case of contexts with monkey
patched `stop` methods (for example like
[pyspark-asyncactions](https://github.com/zero323/pyspark-asyncactions))
I also consider using `_active_spark_context`
```python
atexit.register(lambda: (
SparkContext._active_spark_context.stop()
if SparkContext._active_spark_context
else None
))
```
but `SparkContext` is also out of scope, so that doesn't work without
introducing a standard function within the scope.
### Why are the changes needed?
When using `ipython` as a driver with Python 3.8, `sc` goes out of scope
before `atexit` function is called. This leads to `NameError` on exit. This is
a mild annoyance and likely a bug in ipython (there are quite a few of these
with similar behavior), but it is easy to address on our side, without causing
regressions for users of earlier Python versions.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual testing to confirm that:
- Named error is no longer thrown on exit with ipython and Python 3.8 or
later.
- `stop` is indeed invoked on exit with both plain interpreter and ipython
shells.
Closes #35396 from zero323/SPARK-38073.
Authored-by: zero323 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3e0d4899dcb3be226a120cbeec8df78ff7fb00ba)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/shell.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index 25aadb1..0c6a608 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -45,7 +45,7 @@ except Exception:
sc = spark.sparkContext
sql = spark.sql
-atexit.register(lambda: sc.stop())
+atexit.register((lambda sc: lambda: sc.stop())(sc))
# for compatibility
sqlContext = spark._wrapped
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
