Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21611
There are 2 problems:
1. `Option` is mistakenly mapped to a struct type. It's not only for
aggregate, but join too
```
scala> val a = Seq(Some(1)).toDS
a: org.apache.spark.sql.Dataset[Some[Int]] = [value: int]
scala> val b = Seq(Some(1.2)).toDS
b: org.apache.spark.sql.Dataset[Some[Double]] = [value: double]
scala> a.joinWith(b, lit(true))
res2: org.apache.spark.sql.Dataset[(Some[Int], Some[Double])] = [_1:
struct<value: int>, _2: struct<value: double>]
```
2. `Aggregator` can't use option of product.
I think these 2 problems can be solved separately, what do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
