deepujain commented on code in PR #59047: URL: https://github.com/apache/spark/pull/59047#discussion_r4127210558
########## docs/sql-performance-tuning.md: ########## @@ -400,6 +400,29 @@ spark.sql.optimizer.excludedRules=org.apache.spark.sql.execution.planmerging.Mer Use exactly that name: the rule was called `MergeScalarSubqueries` before Spark 4.2 and sat in a different package before Spark 4.3, and unknown names in `spark.sql.optimizer.excludedRules` are silently ignored, so an older name carried over from a previous version does not turn the rule off. See the [SQL migration guide](sql-migration-guide.html) for the old names. +## Whole-Stage Code Generation + +Whole-stage code generation combines operators into generated Java methods. On HotSpot JVMs with +`-XX:+DontCompileHugeMethods` enabled (the default), methods larger than 8000 bytes are not JIT +compiled. If a generated method is too large, consider lowering the following limit to `8000` so +Spark runs the affected subtree without whole-stage code generation. Compare performance on your +workload before retaining the setting; disabling whole-stage code generation can also reduce +performance. + +<table class="spark-config"> +<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr></thead> +<tr> + <td><code>spark.sql.codegen.hugeMethodLimit</code></td> + <td>65535</td> + <td> + Maximum bytecode size of a compiled Java method generated by whole-stage code generation. + If a compiled method exceeds this limit, Spark disables whole-stage code generation for that Review Comment: Clarified in b0be6d5c: the default 65535 cannot activate the post-compilation check; only a lower limit can. The guide now distinguishes this from compilation failures handled by spark.sql.codegen.fallback. ########## docs/sql-performance-tuning.md: ########## @@ -400,6 +400,29 @@ spark.sql.optimizer.excludedRules=org.apache.spark.sql.execution.planmerging.Mer Use exactly that name: the rule was called `MergeScalarSubqueries` before Spark 4.2 and sat in a different package before Spark 4.3, and unknown names in `spark.sql.optimizer.excludedRules` are silently ignored, so an older name carried over from a previous version does not turn the rule off. See the [SQL migration guide](sql-migration-guide.html) for the old names. +## Whole-Stage Code Generation + +Whole-stage code generation combines operators into generated Java methods. On HotSpot JVMs with +`-XX:+DontCompileHugeMethods` enabled (the default), methods larger than 8000 bytes are not JIT +compiled. If a generated method is too large, consider lowering the following limit to `8000` so +Spark runs the affected subtree without whole-stage code generation. Compare performance on your Review Comment: Updated in b0be6d5c to describe one codegen stage, with stages above and below still able to use generated code. Added the diagnostic log text and kept the workload-performance warning. The non-API Jekyll build passes and the rendered section was checked. ########## docs/sql-performance-tuning.md: ########## @@ -400,6 +400,29 @@ spark.sql.optimizer.excludedRules=org.apache.spark.sql.execution.planmerging.Mer Use exactly that name: the rule was called `MergeScalarSubqueries` before Spark 4.2 and sat in a different package before Spark 4.3, and unknown names in `spark.sql.optimizer.excludedRules` are silently ignored, so an older name carried over from a previous version does not turn the rule off. See the [SQL migration guide](sql-migration-guide.html) for the old names. +## Whole-Stage Code Generation + +Whole-stage code generation combines operators into generated Java methods. On HotSpot JVMs with +`-XX:+DontCompileHugeMethods` enabled (the default), methods larger than 8000 bytes are not JIT +compiled. If a generated method is too large, consider lowering the following limit to `8000` so +Spark runs the affected subtree without whole-stage code generation. Compare performance on your +workload before retaining the setting; disabling whole-stage code generation can also reduce +performance. + +<table class="spark-config"> +<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr></thead> +<tr> + <td><code>spark.sql.codegen.hugeMethodLimit</code></td> Review Comment: I kept the setting internal and removed its configuration-table row in b0be6d5c. It now appears only as an advanced option in prose, explicitly stating that it has no compatibility guarantee and may change in future releases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
