GitHub user yangw1234 opened a pull request:
https://github.com/apache/spark/pull/15416
[SPARK-17849] Fix NPE problem when using grouping sets
## What changes were proposed in this pull request?
Prior this pr, the following code would cause an NPE:
`case class point(a:String, b:String, c:String, d: Int)`
`val data = Seq(
point("1","2","3", 1),
point("4","5","6", 1),
point("7","8","9", 1)
)`
`sc.parallelize(data).toDF().registerTempTable("table")`
`spark.sql("select a, b, c, count(d) from table group by a, b, c GROUPING
SETS ((a)) ").show()`
The reason is that when the grouping_id() behavior was changed in #10677,
some code (which should be changed) was left out.
Take the above code for example, prior #10677, the bit mask for set "(a)"
was `001`, while after #10677 the bit mask was changed to `011`. However, the
`nonNullBitmask` was not changed accordingly.
This pr will fix this problem.
## How was this patch tested?
add integration tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yangw1234/spark groupingid
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15416.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15416
----
commit a6d4651d63c5d31dcd7b1ca6d48be9707316e33b
Author: wangyang <[email protected]>
Date: 2016-10-10T04:19:31Z
fix grouping id bug
commit ab82b211afc9c9d2030c25b4f953db6f4b1bb62e
Author: wangyang <[email protected]>
Date: 2016-10-10T07:03:35Z
add test
commit c849b6a02ee65e3a818abc03f301d82b36ec7f43
Author: wangyang <[email protected]>
Date: 2016-10-10T07:31:51Z
add test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
