Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
> For Product, normally it should be a struct type column, the special case
I can think of is: 1) at root level, flatten it into multiple columns, and the
Product can't be null.
> For Option[Product], normally it should also be a struct type column,
seems there is no special case.
This doesn't make any change to `Product`.
For `Product`, 1. at root level, the schema is all fields are flatten it
into multiple columns. The Product can't be null.
```scala
val df = Seq((1 -> "a"), (2 -> "b")).toDF()
df.printSchema()
root
|-- _1: integer (nullable = false)
|-- _2: string (nullable = true)
```
2. At non-root level, `Product` is a struct type column.
```scala
val df = Seq((1, (1 -> "a")), (2, (2 -> "b")), (3, null)).toDF()
df.printSchema()
root
|-- _1: integer (nullable = false)
|-- _2: struct (nullable = true)
| |-- _1: integer (nullable = false)
| |-- _2: string (nullable = true)
```
For `Option[Product]`, I've posted some description in the PR description.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
