[spark] branch branch-3.2 updated: [SPARK-38073][PYTHON] Update atexit function to avoid issues with late binding

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new 2e382c8  [SPARK-38073][PYTHON] Update atexit function to avoid issues 
with late binding
2e382c8 is described below

commit 2e382c8bff2d0c3733b9b525168254971ca1175e
Author: zero323 <mszymkiew...@gmail.com>
AuthorDate: Fri Feb 4 20:21:02 2022 -0800

    [SPARK-38073][PYTHON] Update atexit function to avoid issues with late 
binding
    
    ### What changes were proposed in this pull request?
    
    This PR updates function registered in PySpark shell `atexit` to capture 
`SparkContext` instead of depending on the surrounding context.
    
    **Note**
    
    A simpler approach
    
    ```python
    atexit.register(sc.stop)
    ```
    
    is possible, but won't work properly in case of contexts with monkey 
patched `stop` methods (for example like 
[pyspark-asyncactions](https://github.com/zero323/pyspark-asyncactions))
    
    I also consider using `_active_spark_context`
    
    ```python
    atexit.register(lambda: (
        SparkContext._active_spark_context.stop()
        if SparkContext._active_spark_context
        else None
    ))
    ```
    
    but `SparkContext` is also out of scope, so that doesn't work without 
introducing a standard function within the scope.
    
    ### Why are the changes needed?
    
    When using `ipython` as a driver with Python 3.8, `sc` goes out of scope 
before `atexit` function is called. This leads to `NameError` on exit. This is 
a mild annoyance and likely a bug in ipython (there are quite a few of these 
with similar behavior), but it is easy to address on our side, without causing 
regressions for users of earlier Python versions.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manual testing to confirm that:
    
    - Named error is no longer thrown on exit with ipython and Python 3.8 or 
later.
    - `stop` is indeed invoked on exit with both plain interpreter and ipython 
shells.
    
    Closes #35396 from zero323/SPARK-38073.
    
    Authored-by: zero323 <mszymkiew...@gmail.com>
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
    (cherry picked from commit 3e0d4899dcb3be226a120cbeec8df78ff7fb00ba)
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
 python/pyspark/shell.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index 25aadb1..0c6a608 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -45,7 +45,7 @@ except Exception:
 
 sc = spark.sparkContext
 sql = spark.sql
-atexit.register(lambda: sc.stop())
+atexit.register((lambda sc: lambda: sc.stop())(sc))
 
 # for compatibility
 sqlContext = spark._wrapped

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org