Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/15918
@srowen and @rxin what is the default behavior that is changed here? i see
a current situation where an implicit encoder is provided that simply cannot
handle the task at hand and this leads to failure.
either the implicits for ExpressionEncoder need to be more narrow so that
they do not claim types they cannot handle (and then other implicit encoders
can be used), or they need to be able to handle these types, for example by
falling back to kryo as is suggested in this JIRA.
currrently ```implicitly[Encoder[Option[Set[Int]]]]``` gives you an
ExpressionEncoder that cannot handle it. that is undesired and makes it
difficult to provide an alternative implicit by the user.
i proposed making the ExpressionEncoders more narrow (that seemed the
easier fix to me at first) but @marmbrus preferred the approach of falling back
to kryo and broadening it. see:
http://apache-spark-developers-list.1001551.n3.nabble.com/getting-encoder-implicits-to-be-more-accurate-td19561.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
