[
https://issues.apache.org/jira/browse/SPARK-50041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-50041.
----------------------------------
Resolution: Invalid
> Show proper error message in memory profiler when it's not installed
> --------------------------------------------------------------------
>
> Key: SPARK-50041
> URL: https://issues.apache.org/jira/browse/SPARK-50041
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.0.0
> Reporter: Hyukjin Kwon
> Priority: Minor
> Labels: pull-request-available
>
> Running
> {code}
> from pyspark.sql.functions import pandas_udf
> df = spark.range(10)
> @pandas_udf("long")
> def add1(x):
> return x + 1
> spark.conf.set("spark.sql.pyspark.udf.profiler", "memory")
> added = df.select(add1("id"))
> added.show()
> spark.profile.show(type="memory")
> {code}
> shows an error message like:
> {code}
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/.../python/pyspark/sql/profiler.py", line 320, in show
> self.profiler_collector.show_memory_profiles(id)
> File "/.../python/pyspark/sql/profiler.py", line 145, in
> show_memory_profiles
> show(id)
> File "/.../python/pyspark/sql/profiler.py", line 139, in show
> MemoryProfiler._show_results(cm)
> File "/.../python/pyspark/profiler.py", line 497, in _show_results
> tmp = template.format(lineno, total_mem, inc, occurrences,
> all_lines[lineno - 1])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
