cloud-fan commented on issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders URL: https://github.com/apache/spark/pull/21732#issuecomment-450271381 @HyukjinKwon If we can go back, I'd say we should not have this optimization which flattens top-level `Product` when encoding. This brings the assumption that top level `Product` can't be null, because in Spark the top level `Row` can't be null. Ideally `Option[T]` should be the same as `T` if `T` is nullable. It's just 2 different ways to represent null in Scala and Java. But because of the optimization I mentioned before, top-level `Product` is the only exception. It's too late to revert that optimization, I think we should accept this special case.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
