thesamet commented on a change in pull request #35378:
URL: https://github.com/apache/spark/pull/35378#discussion_r798816932
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
##########
@@ -254,6 +254,20 @@ case class StaticInvoke(
returnNullable: Boolean = true,
isDeterministic: Boolean = true) extends InvokeLike {
+ // This additional constructor is added to keep binary compatibility after
the addition of the
Review comment:
I'll leave this up to the maintainers to decide whether to revert, keep
this change, or break binary compatibility. I'll add the library maintainer
context here though (I maintain scalapb and sparksql-scalapb). We currently
don't have a way to provide users the ability to use custom types with Datasets
(such as sealed trait hierarchies). To remedy that, Spark provides `Encoder`
and `Decoder` which I believe are public (?), however implementing them
requires `ExpressionEncoder` which quickly takes you to use catalysts
expressions to do anything useful (instantiating objects, querying them, etc).
Spark currently doesn't provide a general solution in this space and apparently
library maintainers (myself included) dipped in the internals, and end users
depend on us for this.
Maintaining compatibility in the Spark/Scala ecosystem is really time
consuming for maintainers - see
[this](https://github.com/typelevel/frameless#versions-and-dependencies) and
[this](https://scalapb.github.io/docs/sparksql/#setting-up-your-project). The
need for those versions came from users noticing problems, resulting in
debugging by maintainers and so on. I'd like to ask to avoid/minimize binary
breakages between maintenance releases. Breaking binary compatibility on
feature releases makes it hard enough. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
