GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/12364
[SPARK-14139][SQL] RowEncoder should preserve schema nullability
## What changes were proposed in this pull request?
The problem is: In `RowEncoder`, we use `Invoke` to get the field of an
external row, which lose the nullability information. This PR creates a
`GetExternalRowField` expression, so that we can preserve the nullability info.
TODO: simplify the null handling logic in `RowEncoder`, to remove so many
if branches.
## How was this patch tested?
new tests in `RowEncoderSuite`
Note that, This PR takes over https://github.com/apache/spark/pull/11980,
with a little simplification, so all credits should go to @koertkuipers
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark nullable
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12364.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12364
----
commit 847d7c7cdfe6626ea1f73656f9eaf868d641ae1c
Author: Koert Kuipers <[email protected]>
Date: 2016-03-26T20:14:18Z
change RowEncoder to respect nullable for struct fields when generating
extractors
commit b500c8bb1d3d8f1ea1d88a2f761056b946598871
Author: Koert Kuipers <[email protected]>
Date: 2016-03-26T20:46:54Z
merge from master
commit 3e32b6aa4dbc0adcdd892ee838ccaed77a67dc58
Author: Koert Kuipers <[email protected]>
Date: 2016-03-26T21:05:11Z
make scalastyle happy
commit 15adc9c6f798b9b08d20229b14d83e6804d29668
Author: Koert Kuipers <[email protected]>
Date: 2016-03-28T15:24:41Z
Merge branch 'master' into feat-rowencoder-nullable
commit 2600a2e28e56f580db512bc7474cdfd1fd17df58
Author: Koert Kuipers <[email protected]>
Date: 2016-03-29T19:09:58Z
first try at using GetExternalRowField instead of Invoke. fails unit test
RowEncoderSuite:encode/decode:Product
commit 50c00054f601d3b59f23493ba4ab46e6d2c1ac03
Author: Koert Kuipers <[email protected]>
Date: 2016-03-29T19:10:07Z
Merge branch 'master' into feat-rowencoder-nullable
commit 05e5f19b3394882c33b49676b587f1890c72e1a3
Author: Koert Kuipers <[email protected]>
Date: 2016-04-03T18:23:10Z
merge from master
commit 4b140e5d2f3efa1d4866ec41032d5f1704b6f332
Author: Koert Kuipers <[email protected]>
Date: 2016-04-03T20:03:52Z
fix for extractsFor renamed to serializerFor
commit e9a9a30e1804785d3534bea78cf2ce588f7fc51b
Author: Koert Kuipers <[email protected]>
Date: 2016-04-03T20:05:20Z
fix pattern match in GetExternalRowField where the data types are external
commit 8231a155a089b941a912841dba42eba43f69a9c5
Author: Wenchen Fan <[email protected]>
Date: 2016-04-13T16:20:53Z
Merge remote-tracking branch 'origin/master' into nullable
commit f8c0bfecf37111b73c87bf42870994d5362c84ec
Author: Wenchen Fan <[email protected]>
Date: 2016-04-13T16:56:37Z
some simplification
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
