viirya commented on a change in pull request #28661:
URL: https://github.com/apache/spark/pull/28661#discussion_r431603460
##########
File path: python/pyspark/sql/utils.py
##########
@@ -75,21 +96,29 @@ class UnknownException(CapturedException):
def convert_exception(e):
s = e.toString()
- stackTrace = '\n\t at '.join(map(lambda x: x.toString(),
e.getStackTrace()))
c = e.getCause()
+
+ jvm = SparkContext._jvm
+ jwriter = jvm.java.io.StringWriter()
+ e.printStackTrace(jvm.java.io.PrintWriter(jwriter))
+ stacktrace = jwriter.toString()
Review comment:
Not sure about this part, why need the change?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1784,6 +1784,15 @@ object SQLConf {
.version("3.0.0")
.fallbackConf(ARROW_EXECUTION_ENABLED)
+ val PYSPARK_JVM_STACKTRACE_ENABLED =
+ buildConf("spark.sql.pyspark.jvmStacktrace.enabled")
+ .doc("When true, it shows the JVM stacktrace in the user-facing PySpark
exception " +
+ "together with Python stacktrace. By default, it is disabled and hides
JVM stacktrace " +
+ "and shows a Python-friendly exception only.")
+ .version("3.0.0")
Review comment:
Is this targeting 3.0.0?
##########
File path: python/pyspark/sql/utils.py
##########
@@ -99,7 +128,9 @@ def deco(*a, **kw):
except py4j.protocol.Py4JJavaError as e:
converted = convert_exception(e.java_exception)
if not isinstance(converted, UnknownException):
- raise converted
+ # Hide where the exception came from that shows a non-Pythonic
+ # JVM exception message.
+ raise_from(converted)
Review comment:
So `raise_from` is used to cut the exception chain from JVM?
##########
File path: python/pyspark/sql/utils.py
##########
@@ -18,8 +18,19 @@
import py4j
import sys
+from pyspark import SparkContext
+
if sys.version_info.major >= 3:
unicode = str
+ # Disable exception chaining (PEP 3134) in captured exceptions
+ # in order to hide JVM stacktace.
+ exec("""
+def raise_from(e):
+ raise e from None
+""")
+else:
+ def raise_from(e):
+ raise e
Review comment:
So seems PEP 3134 is only for 3.0+, we don't cut exception chaining in
Python 2.7 with this `raise_from`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
