[
https://issues.apache.org/jira/browse/SEDONA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713342#comment-17713342
]
Kristin Cowalcijk commented on SEDONA-276:
------------------------------------------
We can workaround the breaking changes of some internal APIs just like what
we've done for Spark 3.0~3.3, but it is a nightmare to maintain binary
compatibility with all Spark 3.x versions. I've come along with lots of issues
with the binary compatibility of the GeoParquet module.
I suggest that we maintain separated code for submodules that are subjective to
internal API changes for each Spark minor version, and release artifacts for
each minor version individually, instead of maintaining one single copy of code
and releasing one single artifact (sedona-spark-shaded-3.0) for all Spark minor
versions, just as [what Apache Iceberg is
doing|https://github.com/apache/iceberg/tree/master/spark].
> Add support for Spark 3.4
> -------------------------
>
> Key: SEDONA-276
> URL: https://issues.apache.org/jira/browse/SEDONA-276
> Project: Apache Sedona
> Issue Type: Improvement
> Reporter: Martin Andersson
> Priority: Major
>
> Spark 3.4.0 was announced on April 14.
> [https://lists.apache.org/thread/o1o9n1t036rmfp45q1jrjs0zzzx0j156]
> Findings from a quick test build using -Dspark.version=3.4.0:
> # Incompatible logging api versions causes test failures.
> # GeoParquet and Geotiff datasources fails to compile due to api changes in
> Spark.
> Setting log4j and slf4j to the same versions as Spark 3.4.0 uses solves 1. I
> don't know how that will affect other Spark versions.
> To solve 2 we might need to use reflection to handle the api changes in Spark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
