GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/7101
[SPARK-8713] Make codegen thread safe
Codegen takes three steps:
1. Take a list of expressions, convert them into Java source code and a
list of expressions that don't not support codegen (fallback to interpret mode).
2. Compile the Java source into Java class (bytecode)
3. Using the Java class and the list of expression to build a Projection.
Currently, we cache the whole three steps, the key is a list of expression,
result is projection. Because some of expressions (which may not thread-safe,
for example, Random) will be hold by the Projection, the projection maybe not
thread safe.
This PR change to only cache the second step, then we can build projection
using codegen even some expressions are not thread-safe, because the cache will
not hold any expression anymore.
cc @marmbrus @rxin @JoshRosen
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark codegen_safe
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7101.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7101
----
commit 1793cf1d27101862d6c95f33d2701847e4c69ea7
Author: Davies Liu <[email protected]>
Date: 2015-06-29T22:41:36Z
make codegen thread safe
commit 4ddaaed59701c6bf3a44f9d85ef063031ed4015f
Author: Davies Liu <[email protected]>
Date: 2015-06-29T22:43:03Z
Merge branch 'master' of github.com:apache/spark into codegen_safe
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
