[
https://issues.apache.org/jira/browse/HUDI-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3679:
----------------------------------
Description:
Currently we have inconsistent models of aliasing Spark version in the bundle
name ("spark3", "spark3.1.x", "spark3.2.1", etc).
To avoid confusion in the community, and make things consistent following
compatibility pledge is proposed:
+Hudi bundles will stay compatible with _all patch versions_ of Spark w/in
single minor version branch (ie Spark 3.1.x, 3.2.x, etc)+
Such compatibility model will be correspondingly reflected in the bundle names:
corresponding spark minor version branch will be aliased in the bundle name to
signal its compatibility with Spark versions w/in such branch ("spark3.1" for
Spark 3.1.x, etc)
To transition to the new model following steps are proposed:
# Add new artifacts carrying Spark minor versions in the bundle names (for ex,
"spark3.2", "spark3.1", "spark2.4") in 0.11
# Deprecate single and triple digit bundles ("spark3", "spark2", "spark3.2.1")
in favor of double-digit in 0.11
# Discontinue deprecated bundles (spark3, spark2, etc) from 0.12
As such following bundles will be released in 0.11
* "spark2.4" (Scala 2.11/2.12)
* "spark3.1"
* "spark3.2"
* "spark3" (this is the same as "spark3.2")
* "spark2" (Scala 2.11/2.12, this is the same as "spark2.4")
was:
Currently we have inconsistent models of aliasing Spark version in the bundle
name ("spark3", "spark3.1.x", "spark3.2.1", etc).
To avoid confusion in the community, and make things consistent following
compatibility pledge is proposed:
+Hudi bundles will stay compatible with _all patch versions_ of Spark w/in
single minor version branch (ie Spark 3.1.x, 3.2.x, etc)+
Such compatibility model will be correspondingly reflected in the bundle names:
corresponding spark minor version branch will be aliased in the bundle name to
signal its compatibility with Spark versions w/in such branch ("spark3.1" for
Spark 3.1.x, etc)
To transition to the new model following steps are proposed:
# Add new artifacts carrying Spark minor versions in the bundle names (for ex,
"spark3.2", "spark3.1", "spark2.4") in 0.11
# Deprecate single and triple digit bundles ("spark3", "spark2", "spark3.2.1")
in favor of double-digit in 0.11
# Discontinue deprecated bundles (spark3, spark2, etc) from 0.12
> Spark Compatibility Model
> -------------------------
>
> Key: HUDI-3679
> URL: https://issues.apache.org/jira/browse/HUDI-3679
> Project: Apache Hudi
> Issue Type: Epic
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Major
>
> Currently we have inconsistent models of aliasing Spark version in the bundle
> name ("spark3", "spark3.1.x", "spark3.2.1", etc).
> To avoid confusion in the community, and make things consistent following
> compatibility pledge is proposed:
> +Hudi bundles will stay compatible with _all patch versions_ of Spark w/in
> single minor version branch (ie Spark 3.1.x, 3.2.x, etc)+
>
> Such compatibility model will be correspondingly reflected in the bundle
> names: corresponding spark minor version branch will be aliased in the bundle
> name to signal its compatibility with Spark versions w/in such branch
> ("spark3.1" for Spark 3.1.x, etc)
> To transition to the new model following steps are proposed:
> # Add new artifacts carrying Spark minor versions in the bundle names (for
> ex, "spark3.2", "spark3.1", "spark2.4") in 0.11
> # Deprecate single and triple digit bundles ("spark3", "spark2",
> "spark3.2.1") in favor of double-digit in 0.11
> # Discontinue deprecated bundles (spark3, spark2, etc) from 0.12
>
> As such following bundles will be released in 0.11
> * "spark2.4" (Scala 2.11/2.12)
> * "spark3.1"
> * "spark3.2"
> * "spark3" (this is the same as "spark3.2")
> * "spark2" (Scala 2.11/2.12, this is the same as "spark2.4")
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
