Repository: spark
Updated Branches:
refs/heads/master 6308c65f0 -> 7c7266208
[SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
## What changes were proposed in this pull request?
This PR proposes to improve error message from:
```
>>> sc.show_profiles()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1000, in show_profiles
self.profiler_collector.show_profiles()
AttributeError: 'NoneType' object has no attribute 'show_profiles'
>>> sc.dump_profiles("/tmp/abc")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles
self.profiler_collector.dump_profiles(path)
AttributeError: 'NoneType' object has no attribute 'dump_profiles'
```
to
```
>>> sc.show_profiles()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1003, in show_profiles
raise RuntimeError("'spark.python.profile' configuration must be set "
RuntimeError: 'spark.python.profile' configuration must be set to 'true' to
enable Python profile.
>>> sc.dump_profiles("/tmp/abc")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles
raise RuntimeError("'spark.python.profile' configuration must be set "
RuntimeError: 'spark.python.profile' configuration must be set to 'true' to
enable Python profile.
```
## How was this patch tested?
Unit tests added in `python/pyspark/tests.py` and manual tests.
Author: hyukjinkwon <[email protected]>
Closes #19260 from HyukjinKwon/profile-errors.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7c726620
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7c726620
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7c726620
Branch: refs/heads/master
Commit: 7c7266208a3be984ac1ce53747dc0c3640f4ecac
Parents: 6308c65
Author: hyukjinkwon <[email protected]>
Authored: Mon Sep 18 13:20:11 2017 +0900
Committer: hyukjinkwon <[email protected]>
Committed: Mon Sep 18 13:20:11 2017 +0900
----------------------------------------------------------------------
python/pyspark/context.py | 12 ++++++++++--
python/pyspark/tests.py | 16 ++++++++++++++++
2 files changed, 26 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/7c726620/python/pyspark/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index a704604..a33f6dc 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -997,12 +997,20 @@ class SparkContext(object):
def show_profiles(self):
""" Print the profile stats to stdout """
- self.profiler_collector.show_profiles()
+ if self.profiler_collector is not None:
+ self.profiler_collector.show_profiles()
+ else:
+ raise RuntimeError("'spark.python.profile' configuration must be
set "
+ "to 'true' to enable Python profile.")
def dump_profiles(self, path):
""" Dump the profile stats into directory `path`
"""
- self.profiler_collector.dump_profiles(path)
+ if self.profiler_collector is not None:
+ self.profiler_collector.dump_profiles(path)
+ else:
+ raise RuntimeError("'spark.python.profile' configuration must be
set "
+ "to 'true' to enable Python profile.")
def getConf(self):
conf = SparkConf()
http://git-wip-us.apache.org/repos/asf/spark/blob/7c726620/python/pyspark/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/tests.py b/python/pyspark/tests.py
index 3c108ec..da99872 100644
--- a/python/pyspark/tests.py
+++ b/python/pyspark/tests.py
@@ -1296,6 +1296,22 @@ class ProfilerTests(PySparkTestCase):
rdd.foreach(heavy_foo)
+class ProfilerTests2(unittest.TestCase):
+ def test_profiler_disabled(self):
+ sc = SparkContext(conf=SparkConf().set("spark.python.profile",
"false"))
+ try:
+ self.assertRaisesRegexp(
+ RuntimeError,
+ "'spark.python.profile' configuration must be set",
+ lambda: sc.show_profiles())
+ self.assertRaisesRegexp(
+ RuntimeError,
+ "'spark.python.profile' configuration must be set",
+ lambda: sc.dump_profiles("/tmp/abc"))
+ finally:
+ sc.stop()
+
+
class InputFormatTests(ReusedPySparkTestCase):
@classmethod
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
