Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21611
This affects the dataset with `Aggregator` of output type `Option`.
For normal dataset schema, it does affect the following case:
```scala
val ds = Seq(Some(1), None, null).toDS()
ds.show
ds.printSchema
```
Before:
```
[info] If the schema is inferred from a Scala tuple/case class, or a Java
bean, please try to use scala.Option[_] or other nullable types (
e.g. java.lang.Integer instead of int/scala.Int).
```
After:
```scala
+-----+
|value|
+-----+
| 1|
| null|
| null|
+-----+
root
|-- value: integer (nullable = true)
```
This dataset's schema is a row with a nullable integer. I feel that it
should be allowed to have null.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
