[ https://issues.apache.org/jira/browse/SPARK-28590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-28590: ---------------------------------- Affects Version/s: (was: 2.4.0) 3.0.0 > Add sort_stats Setter for Custom Profiler > ----------------------------------------- > > Key: SPARK-28590 > URL: https://issues.apache.org/jira/browse/SPARK-28590 > Project: Spark > Issue Type: Improvement > Components: PySpark > Affects Versions: 3.0.0 > Reporter: Albertus Kelvin > Priority: Minor > > When I want to use BasicProfiler with different sorters in sort_stats, I > sometimes need to create a custom profiler and implement the show() method > only to replace the following line: stats.sort_stats("time", > "cumulative").print_stats(). > I think it'd be better if the users are able to specify the sorters without > creating a custom profiler. > I implemented the changes in PySpark only. > To apply the setter and getter methods, one can use the following way: > {code:python} > conf = SparkConf().set("spark.python.profile", "true") > # use BasicProfiler > sc = SparkContext('local', 'test', conf=conf) > sc.profiler_collector.profiler_cls.set_sort_stats_sorters(BasicProfiler, > ['ncalls', 'tottime', 'name'] > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org