[ https://issues.apache.org/jira/browse/SPARK-24013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-24013. ----------------------------- Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > ApproximatePercentile grinds to a halt on sorted input. > ------------------------------------------------------- > > Key: SPARK-24013 > URL: https://issues.apache.org/jira/browse/SPARK-24013 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Juliusz Sompolski > Assignee: Marco Gaido > Priority: Major > Fix For: 2.4.0 > > Attachments: screenshot-1.png > > > Running > {code} > sql("select approx_percentile(rid, array(0.1)) from (select rand() as rid > from range(10000000))").collect() > {code} > takes 7 seconds, while > {code} > sql("select approx_percentile(id, array(0.1)) from range(10000000)").collect() > {code} > grinds to a halt - processes the first million rows quickly, and then slows > down to a few thousands rows / second (4m rows processed after 20 minutes). > Thread dumps show that it spends time in QuantileSummary.compress. > Seems it hits some edge case inefficiency when dealing with sorted data? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org