This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new bda1ecd [SPARK-37253][PYTHON] `try_simplify_traceback` should not
fail when `tb_frame.f_lineno` is None
bda1ecd is described below
commit bda1ecd2477d28a2f9205fb8e22d298a83412ec9
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 9 20:03:34 2021 +0900
[SPARK-37253][PYTHON] `try_simplify_traceback` should not fail when
`tb_frame.f_lineno` is None
### What changes were proposed in this pull request?
This PR aims to handle the corner case when `tb_frame.f_lineno` is `None`
in `try_simplify_traceback` which was added by
https://github.com/apache/spark/pull/30309 at Apache Spark 3.1.0.
### Why are the changes needed?
This will handle the following corner case.
```python
Traceback (most recent call last):
File
"/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/worker.py",
line 630, in main
tb = try_simplify_traceback(sys.exc_info()[-1])
File
"/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/util.py",
line 217, in try_simplify_traceback
new_tb = types.TracebackType(
TypeError: 'NoneType' object cannot be interpreted as an integer
```
Python GitHub Repo also has the test case for this corner case.
-
https://github.com/python/cpython/blob/main/Lib/test/test_exceptions.py#L2373
```python
None if frame.f_lineno is None else
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
N/A
Closes #34530 from dongjoon-hyun/SPARK-37253.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 8ae88d01b46d581367d0047b50fcfb65078ab972)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/util.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index e075b04..e0933f1 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -214,7 +214,7 @@ def try_simplify_traceback(tb):
tb_next=tb_next,
tb_frame=cur_tb.tb_frame,
tb_lasti=cur_tb.tb_frame.f_lasti,
- tb_lineno=cur_tb.tb_frame.f_lineno)
+ tb_lineno=cur_tb.tb_frame.f_lineno if cur_tb.tb_frame.f_lineno is
not None else -1)
tb_next = new_tb
return new_tb
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
