Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
Non top-level and top-level encoders for `Option[Product]` have a little
difference.
As you said, top-level `Option[Product]` can only be encoded as a single
struct column.
For non top-level one, we can't apply the same change because it is already
a struct column. We don't want to change current behavior of it from a struct
column to a struct column of a struct column.
This means that we can remove the limitation of top-level
`Option[Product]`, but it doesn't help too much Aggregator issue here. We still
need to use non top-level `Option[Product]` encoders for Aggregator case.
@cloud-fan Do you want to incorporate top-level `Option[Product]` encoders
into this PR too? Or let's create another Jira & PR for it? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
