GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19767
[WIP][SPARK-22543][SQL] fix java 64kb compile error for deeply nested
expressions
## What changes were proposed in this pull request?
A frequently reported issue of Spark is the Java 64kb compile error. This
is because Spark generates a very big method and it's usually caused by 2
reasons:
1. a deep expression tree, e.g. a very complex filter condition
2. many individual expressions, e.g. expressions can have many children,
operators can have many expressions.
This PR focuses on 1. There are already several patches(#15620 #18972
#18641) trying to fix this issue and some of them are already merged. However
this is an endless job as every non-leaf expression has this issue.
This PR proposes to fix this issue in `Expression.genCode`, to make sure
the code for a single expression won't grow too big.
TODO: check TPCDS to make sure no performance regression.
## How was this patch tested?
existing test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark codegen
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19767.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19767
----
commit aaa5b6f452034660880d8175f450ba8c8cfad23b
Author: Wenchen Fan <[email protected]>
Date: 2017-11-16T16:55:43Z
fix java 64kb compile error for deeply nested expressions
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
