[
https://issues.apache.org/jira/browse/SEDONA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17714052#comment-17714052
]
Kristin Cowalcijk commented on SEDONA-276:
------------------------------------------
[~umartin] It is unnecessary to separate everything in sedona-sql, we can
separate part of it.
# The UDTs and ST_ expressions defined in sedona-sql are not likely to be
subject to API changes so we can maintain a single copy of them.
# The spatial join optimization code may be subject to API changes. If it
still works for Spark 3.4 then we'll also maintain a single copy.
# The file format code (GeoParquet and GeoTiff) uses the DataSource V1
interface, which has breaking changes in Spark 3.4. The code for GeoParquet
already has lots of dirty hacks to maintain binary compatibility with Spark 3.0
~ 3.3, it would make it easier to maintain if we have separate implementations
for different Spark minor versions.
To sum up, I suggest maintaining separate implementations for file format
modules only. API-agnostic logic in file format implementations can also be
moved to a common module to minimize code duplication.
> Add support for Spark 3.4
> -------------------------
>
> Key: SEDONA-276
> URL: https://issues.apache.org/jira/browse/SEDONA-276
> Project: Apache Sedona
> Issue Type: Improvement
> Reporter: Martin Andersson
> Priority: Major
>
> Spark 3.4.0 was announced on April 14.
> [https://lists.apache.org/thread/o1o9n1t036rmfp45q1jrjs0zzzx0j156]
> Findings from a quick test build using -Dspark.version=3.4.0:
> # Incompatible logging api versions causes test failures.
> # GeoParquet and Geotiff datasources fails to compile due to api changes in
> Spark.
> Setting log4j and slf4j to the same versions as Spark 3.4.0 uses solves 1. I
> don't know how that will affect other Spark versions.
> To solve 2 we might need to use reflection to handle the api changes in Spark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
