GitHub user sbcd90 opened a pull request:
https://github.com/apache/spark/pull/13012
[SPARK-15204][SQL] Nullable is not correct for Aggregator
## What changes were proposed in this pull request?
```
object SimpleSum extends Aggregator[Row, Int, Int] {
def zero: Int = 0
def reduce(b: Int, a: Row) = b + a.getInt(1)
def merge(b1: Int, b2: Int) = b1 + b2
def finish(b: Int) = b
def bufferEncoder: Encoder[Int] = Encoders.scalaInt
def outputEncoder: Encoder[Int] = Encoders.scalaInt
}
val df = List(("a", 1), ("a", 2), ("a", 3)).toDF("k", "v")
val df1 = df.groupBy("k").agg(SimpleSum.toColumn as "v1")
df1.printSchema
df1.show
root
|-- k: string (nullable = true)
|-- v1: integer (nullable = true)
+---+---+
| k| v1|
+---+---+
| a| 6|
+---+---+
```
The above example shows `v1` as `nullable=true` which is incorrect.
The aggregate fields should be non-nullable in the schema.
## How was this patch tested?
unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sbcd90/spark aggregatorSchemaIssue
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13012.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13012
----
commit c66d36c3ce956ad7c14ed18c3877bd0c6c257d2d
Author: Subhobrata Dey <[email protected]>
Date: 2016-05-09T19:39:50Z
[SPARK-15204][SQL] Nullable is not correct for Aggregator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
