[
https://issues.apache.org/jira/browse/SPARK-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313619#comment-15313619
]
Apache Spark commented on SPARK-15748:
--------------------------------------
User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/13491
> Replace inefficient foldLeft() call in PartitionStatistics
> ----------------------------------------------------------
>
> Key: SPARK-15748
> URL: https://issues.apache.org/jira/browse/SPARK-15748
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> PartitionStatistics uses foldLeft and list concatenation to flatten an
> iterator of lists, but this is extremely inefficient compared to simply doing
> flatMap/flatten because it performs many unnecessary object allocations.
> Simply replacing this foldLeft by a flatMap results in fair performance gains
> when constructing PartitionStatistics instances for tables with many columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
