wbo4958 opened a new pull request, #49615:
URL: https://github.com/apache/spark/pull/49615
### What changes were proposed in this pull request?
``` python
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.linalg import Vectors
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").getOrCreate()
dataset = spark.createDataFrame(
[(Vectors.dense([0.0]), 0.0),
(Vectors.dense([0.4]), 1.0),
(Vectors.dense([0.5]), 0.0),
(Vectors.dense([0.6]), 1.0),
(Vectors.dense([1.0]), 1.0)] * 10,
["features", "label"])
lr = LogisticRegression()
model = lr.fit(dataset)
```
The above code will raise exception at the end. Even I remove the
`@try_remote_del` from
https://github.com/apache/spark/blob/master/python/pyspark/ml/wrapper.py#L58 ,
the issue is still there.
``` console
Exception ignored in: <function JavaWrapper.del_ at 0x70b8caf2f920>_
Traceback (most recent call last):
File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/util.py", line
254, in wrapped
File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/wrapper.py",
line 60, in __del__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <function JavaWrapper.del_ at 0x70b8caf2f920>_
Traceback (most recent call last):
File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/util.py", line
254, in wrapped
File "/home/xxx/work.d/spark/spark-master/python/pyspark/ml/wrapper.py",
line 60, in __del__
ImportError: sys.meta_path is None, Python is likely shutting down
```
Looks like the `__del__` function of JavaWrapper is called after python
shutting down and the python related modules is not functional.
### Why are the changes needed?
Fix the bug, good user experience.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passes
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
