Repository: spark Updated Branches: refs/heads/master 1662e9311 -> 6942aeeb0
[SPARK-21603][SQL][FOLLOW-UP] Change the default value of maxLinesPerFunction into 4000 ## What changes were proposed in this pull request? This pr changed the default value of `maxLinesPerFunction` into `4000`. In #18810, we had this new option to disable code generation for too long functions and I found this option only affected `Q17` and `Q66` in TPC-DS. But, `Q66` had some performance regression: ``` Q17 w/o #18810, 3224ms --> q17 w/#18810, 2627ms (improvement) Q66 w/o #18810, 1712ms --> q66 w/#18810, 3032ms (regression) ``` To keep the previous performance in TPC-DS, we better set higher value at `maxLinesPerFunction` by default. ## How was this patch tested? Existing tests. Author: Takeshi Yamamuro <[email protected]> Closes #19021 from maropu/SPARK-21603-FOLLOWUP-1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6942aeeb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6942aeeb Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6942aeeb Branch: refs/heads/master Commit: 6942aeeb0a0095a1ba85a817eb9e0edc410e5624 Parents: 1662e93 Author: Takeshi Yamamuro <[email protected]> Authored: Wed Aug 23 12:02:24 2017 -0700 Committer: gatorsmile <[email protected]> Committed: Wed Aug 23 12:02:24 2017 -0700 ---------------------------------------------------------------------- .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/6942aeeb/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---------------------------------------------------------------------- diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 2c7397c..a685099 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -577,10 +577,10 @@ object SQLConf { .doc("The maximum lines of a single Java function generated by whole-stage codegen. " + "When the generated function exceeds this threshold, " + "the whole-stage codegen is deactivated for this subtree of the current query plan. " + - "The default value 2667 is the max length of byte code JIT supported " + - "for a single function(8000) divided by 3.") + "The default value 4000 is the max length of byte code JIT supported " + + "for a single function(8000) divided by 2.") .intConf - .createWithDefault(2667) + .createWithDefault(4000) val FILES_MAX_PARTITION_BYTES = buildConf("spark.sql.files.maxPartitionBytes") .doc("The maximum number of bytes to pack into a single partition when reading files.") --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
