Re: [DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-07-15 Thread Jia Yu
I agree. Let's start the release process of Sedona 1.2.1. On Fri, Jul 15, 2022 at 11:22 AM Martin Andersson < u.martin.anders...@gmail.com> wrote: > Hi, > > Spark 3.3 support is now merged into Sedona master. > > Would this be a good time to release Sedona 1.2.1 and then drop support for > old

Re: [DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-07-15 Thread Martin Andersson
Hi, Spark 3.3 support is now merged into Sedona master. Would this be a good time to release Sedona 1.2.1 and then drop support for old versions of Spark and Scala? Br, Martin fre 24 juni 2022 kl. 09:52 skrev Jia Yu : > Hi Martin, > > I agree. > > 1. Currently, geom serializers, spatial

Re: [DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-06-24 Thread Jia Yu
Hi Martin, I agree. 1. Currently, geom serializers, spatial partitioning code and some format reader code in Sedona-core (all in Java) is independent from Spark dependency. So Sedona-Flink actually re-uses those. But a refactor of Sedona ST / RS functions are needed as some of them depend on

RE: [DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-06-23 Thread Martin Andersson
Hi, I guess that the pending Spark 3.3 support is a big enough feature to warrant a new Sedona release. It makes no sense to remove Spark 2.4 and Scala 2.11 before the release. After Sedona-next is released I think that Spark 2.4 can safely be removed. Long term, but i think that's another

Re: [DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-06-19 Thread Jia Yu
My opinion is to keep -3.0 in the artifact ID just in case it will be needed in the future. For Flink, Flink is working to be Scala-free: https://flink.apache.org/2022/02/22/scala-free.html And Sedona Flink is purely in Java. So I think it may be OK to stop compiling Sedona against Flink Scala

Re: [DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-06-19 Thread Adam Binford
I'll start with my support. I think it's fair to upgrade Spark versions to get new features at this point. Questions: Since all the supported Spark versions are supported by a single artifact, do you drop the -3.0 in the artifact ID? Or leave it in case it's needed in the future? Does Flink

[DISCUSS] Drop Spark 2.4 and Scala 2.11 support

2022-06-19 Thread Jia Yu
Dear all, I am proposing to drop the support of Spark 2.4 and Scala 2.11 in the next Sedona release. The version number will be 1.3.0 if we drop this support, otherwise it will be 1.2.1. Here is the status of Spark 2.4 and Sedona for Spark 2.4 1. Spark community has announced Spark 2.4 EOL on