GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/12322
[SPARK-14554][SQL] Dataset.map may generate wrong java code for wide table
## What changes were proposed in this pull request?
In
https://github.com/apache/spark/pull/12047/files#diff-94a1f59bcc9b6758c4ca874652437634R529,
we may split field expressions codes in `CreateExternalRow` to support wide
table. However, after we support whole stage codegen for typed operations, the
`CreateExternalRow` may also appear in whole stage codegen.
The problem is, in whole stage codegen, the input for expressions is not
always the input row, but can be `CodeGenContext.currentVars`, which doesn't
work well with `CodeGenContext.splitExpressions`.
This PR stops splitting field expressions codes in `CreateExternalRow` if
it's being whole stage codegened, but we need a better way to fix this problem
instead of avoiding it.
## How was this patch tested?
new test in DatasetSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark codegen
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12322.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12322
----
commit 0ad119490ac5fba06d6cdeda1d4a9e9e0f8c0c8e
Author: Wenchen Fan <[email protected]>
Date: 2016-04-12T03:43:08Z
Dataset.map may generate wrong java code for wide table
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
