Repository: spark Updated Branches: refs/heads/master 0c2aee69b -> f7f4e9c2d
[SPARK-22412][SQL] Fix incorrect comment in DataSourceScanExec ## What changes were proposed in this pull request? Next fit decreasing bin packing algorithm is used to combine splits in DataSourceScanExec but the comment incorrectly states that first fit decreasing algorithm is used. The current implementation doesn't go back to a previously used bin other than the bin that the last element was put into. Author: Vinitha Gankidi <[email protected]> Closes #19634 from vgankidi/SPARK-22412. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f7f4e9c2 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f7f4e9c2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f7f4e9c2 Branch: refs/heads/master Commit: f7f4e9c2db405b887832fcb592cd4522795d00ca Parents: 0c2aee6 Author: Vinitha Gankidi <[email protected]> Authored: Sat Nov 4 11:09:47 2017 -0700 Committer: gatorsmile <[email protected]> Committed: Sat Nov 4 11:09:47 2017 -0700 ---------------------------------------------------------------------- .../scala/org/apache/spark/sql/execution/DataSourceScanExec.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/f7f4e9c2/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala index e9f6503..a607ec0 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala @@ -469,7 +469,7 @@ case class FileSourceScanExec( currentSize = 0 } - // Assign files to partitions using "First Fit Decreasing" (FFD) + // Assign files to partitions using "Next Fit Decreasing" splitFiles.foreach { file => if (currentSize + file.length > maxSplitBytes) { closePartition() --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
