[
https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484941#comment-17484941
]
Nadav Samet edited comment on SPARK-38077 at 1/31/22, 9:14 PM:
---------------------------------------------------------------
I wasn't aware that the policy allows for it, but I'd like to express the
impact on breaking binary-compatibility on _maintenance_ releases for library
maintainers and ultimately end-users, in hope that it will be reconsidered.
A breakage like this one, doesn't allow a library author to ship a single
artifact that works with both Spark 3.2.0 and Spark 3.2.1. Maintainers have to
create a build for each version. Since sparksql-scalapb (which I maintain)
calls {{Invoke}} directly (it does it through frameless), frameless will need
to produce two versions of their library (for 3.2.0 and 3.2.1)
Today, we already have to cut a release for feature releases (3.0.x, 3.1.x and
3.2.x), see the [table here|#setting-up-your-project],] and the consequence of
this policy is that we also have to cut a release for maintenance versions too,
and that release would be incompatible with 3.2.0.
was (Author: thesamet):
I wasn't aware that the policy allows for it, but I'd like to express the
impact on breaking binary-compatibility on _maintenance_ releases for library
maintainers and ultimately end-users, in hope that it will be reconsidered.
A breakage like this one, doesn't allow a library author to ship a single
artifact that works with both Spark 3.2.0 and Spark 3.2.1. Maintainers have to
create a build for each version. Since my library does call Invoke directly (it
does it through frameless), frameless will need to produce two versions of
their library (for 3.2.0 and 3.2.1)
Today, we already have to cut a release for feature releases (3.0.x, 3.1.x and
3.2.x), see the [table
here|[https://scalapb.github.io/docs/sparksql#setting-up-your-project],] and
the consequence of this policy is that we also have to cut a release for
maintenance versions too, and that release would be incompatible with 3.2.0.
> Spark 3.2.1 breaks binary compatibility with Spark 3.2.0
> --------------------------------------------------------
>
> Key: SPARK-38077
> URL: https://issues.apache.org/jira/browse/SPARK-38077
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.2.1
> Reporter: Nadav Samet
> Priority: Major
>
> [PR 35243|https://github.com/apache/spark/pull/35243] introduced a new
> parameter to class `Invoke` with a default value (`isDeterministic: Boolean =
> true`). Existing Spark libraries (such as
> [frameless|https://github.com/typelevel/frameless]) that invoke
> [Invoke|https://github.com/typelevel/frameless/blob/29961d549e332dddf5cd711ef699dde7460cc48a/dataset/src/main/scala/frameless/RecordEncoder.scala#L154]
> directly expect a method with 7 parameters, and the new version expects 8.
> If Frameless would recompile with Spark 3.2.1, the updated library will NOT
> be binary compatible with Spark 3.2.0. Adding default parameters to existing
> methods [should be
> avoided|https://github.com/jatcwang/binary-compatibility-guide#dont-adding-parameters-with-default-values-to-methods].
> One way forward would be to revert the change in the constructor and
> introduce a second constructor or a companion method that takes all the 8
> parameters.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
