GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/19447
[SPARK-22215][SQL] Add configuration to set the threshold for generated
class
## What changes were proposed in this pull request?
SPARK-18016 introduced an arbitrary threshold for the size of a generated
class
(https://github.com/apache/spark/blob/83488cc3180ca18f829516f550766efb3095881e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L286).
This value is hardcoded.
In some cases making it smaller can help avoiding the error of exceeding
the maximum number of entries in the Constant Pool.
This PR introduces a new configuration parameter, which defaults to the
previous value, but it allows to set this to a smaller one if needed.
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark SPARK-22215
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19447.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19447
----
commit c69be31314d9aa96c3920073beaf7cca46d507fa
Author: Marco Gaido <[email protected]>
Date: 2017-10-06T12:58:36Z
[SPARK-22215][SQL] Add configuration to set the threshold for generated
class
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
