[GitHub] [spark] srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer

2019-03-16 Thread GitBox
srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer URL: https://github.com/apache/spark/pull/23986#issuecomment-473578962 @fitermay I guess it has to be more explicitly constructed as an `Ordering`? ``` [error] /home/jenkins/workspace/spa

[GitHub] [spark] srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer

2019-03-14 Thread GitBox
srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer URL: https://github.com/apache/spark/pull/23986#issuecomment-473122926 Merged to master This is an automated message from the Ap

[GitHub] [spark] srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer

2019-03-08 Thread GitBox
srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer URL: https://github.com/apache/spark/pull/23986#issuecomment-471026686 Hm! That's surprising. Looking at min vs minBy, it even seems like min has more indirection (calls foldLeft). The implicit

[GitHub] [spark] srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer

2019-03-07 Thread GitBox
srowen commented on issue #23986: [SPARK-27070] Fix performance bug in DefaultPartitionCoalescer URL: https://github.com/apache/spark/pull/23986#issuecomment-470613691 There's a little detail in the JIRA that's pretty suggestive that this is the bottleneck; if there's a stack trace or more