GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/8831
[SPARK-10710] Remove ability to disable spilling in core and SQL
It does not make much sense to set `spark.shuffle.spill` or
`spark.sql.planner.externalSort` to false: I believe that these configurations
were initially added as "escape hatches" to guard against bugs in the external
operators, but these operators are now mature and well-tested. In addition,
these configurations are not handled in a consistent way anymore: SQL's
Tungsten codepath ignores these configurations and will continue to use
spilling operators. Similarly, Spark Core's `tungsten-sort` shuffle manager
does not respect `spark.shuffle.spill=false`.
This pull request removes these configurations, adds warnings at the
appropriate places, and deletes a large amount of code which was only used in
code paths that did not support spilling.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
remove-ability-to-disable-spilling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8831.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8831
----
commit d81ef04e98565ac3fe6a97e97df7ac95fe4895a6
Author: Josh Rosen <[email protected]>
Date: 2015-09-18T21:55:13Z
Remove ability to set spark.shuffle.spill=false.
commit 4bce5f2e8e90b5c2e953f057b16f7ccc64df52a0
Author: Josh Rosen <[email protected]>
Date: 2015-09-18T22:15:15Z
Remove ability to set spark.sql.planner.externalSort=false.
commit bb053c780438848cd7fa02ab3dfb0fece1afe866
Author: Josh Rosen <[email protected]>
Date: 2015-09-18T22:19:54Z
Make similar changes in PySpark.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
