Hyukjin Kwon created SPARK-50041:
------------------------------------
Summary: Show proper error message in memory profiler
Key: SPARK-50041
URL: https://issues.apache.org/jira/browse/SPARK-50041
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.0.0
Reporter: Hyukjin Kwon
Running
{code}
from pyspark.sql.functions import pandas_udf
df = spark.range(10)
@pandas_udf("long")
def add1(x):
return x + 1
spark.conf.set("spark.sql.pyspark.udf.profiler", "memory")
added = df.select(add1("id"))
added.show()
spark.profile.show(type="memory")
{code}
shows an error message like:
{code}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../python/pyspark/sql/profiler.py", line 320, in show
self.profiler_collector.show_memory_profiles(id)
File "/.../python/pyspark/sql/profiler.py", line 145, in show_memory_profiles
show(id)
File "/.../python/pyspark/sql/profiler.py", line 139, in show
MemoryProfiler._show_results(cm)
File "/.../python/pyspark/profiler.py", line 497, in _show_results
tmp = template.format(lineno, total_mem, inc, occurrences, all_lines[lineno
- 1])
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
