[
https://issues.apache.org/jira/browse/SPARK-15877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-15877:
---------------------------------
Labels: bulk-closed (was: )
> DataSource executed twice when using ORDER BY
> ---------------------------------------------
>
> Key: SPARK-15877
> URL: https://issues.apache.org/jira/browse/SPARK-15877
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.6.1
> Reporter: Matthew Livesey
> Priority: Major
> Labels: bulk-closed
>
> When executing using a custom DataSource, I observed that if using an Order
> By, the underlying DataSource is executed twice. A small example
> demonstrating this is here:
> https://github.com/mattinbits/spark-sql-sort-double-execution
> From debugging, I find that when there is a "Sort" in the Logical plan (and
> therefore the resulting Physical plan) an "Exchange" object is inserted into
> the plan by the method "EnsureRequirements.ensureDistributionAndOrdering()".
> this Exchange has a RangePartitioner. At the point that the dataframe is
> converted to an RDD, (which is before computation of that RDD should occur).
> The RangePartitioner causes execution of the RDD for the purpose of sampling
> to get statistics to guide partitioning. This is done by
> "Exchange.prepareShuffleDependency()" which in turn calls
> "SamplingUtils.reservoirSampleAndCount()".
> In some cases, this causes a significant performance degradation, anywhere
> that the cost of computing the RDD is high. The RDD gets executed twice, once
> during the conversion of Dataframe to RDD, and then again when the RDD is
> eventually computed, e.g. by a call to "collect()". There doesn't appear to
> be any configuration setting to control whether an RDD should be executed for
> sampling, and I haven't been able to determine whether the sampling is
> necessary to get the correct results or if it is just aimed at improving
> performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
