Yes, I believe we still need to support Spark 2.4 since many Sedona users are still using it
On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka <[email protected]> wrote: > What do you mean by compile target? > > ? > > Iv'e found Apache Zeppelin handle multiple Spark versions here using > profile for each Spark version: > > https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 > > > Do you think this method is better? > > > > Netanel Malka, > Big Data Consultant > [Description: Description: Description: Description: > cid:[email protected]] > ________________________________ > From: Felix Cheung <[email protected]> > Sent: Thursday, November 12, 2020 04:05 > To: [email protected] > Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi > Zhang; [email protected] > Subject: Re: Support Spark 2.4 in Sedona 1.0 > > I am not sure it should be a branch? It is common to deal with this as a > compile target, not as a separate branch. A separate branch might have > difficulty to release? > > There are a few example in projects where they handle multiple Spark > target version like this. > > > On Wed, Nov 11, 2020 at 12:56 PM Jia Yu <[email protected]<mailto: > [email protected]>> wrote: > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the > compiler used in each branch, > > For Sedona on Spark 3.0, I will compile it using Scala 2.12 > For Sedona on Spark 2.4, I will compile it using Scala 2.11. > > For the Java code in both branches, I will compile them using Java 1.8 > > Am I missing anything here? > > > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka <[email protected]<mailto: > [email protected]>> wrote: > > > Hi, > > > > I also think that we need to support 2.4. > > > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > > 12, 2020) > > > > I also asked about it on *[email protected]<mailto: > [email protected]> <[email protected]<mailto: > [email protected]>>* > > : > > > > > > *Sean Owen (answered the question): * > > > > "I don't think there's an official EOL for Spark 2.4.x but would expect > > another maintenance release in the first half of 2021 at least. I'd also > > guess it wouldn't be maintained by 2022." > > > > > > ?BR, > > > > > > > > Netanel Malka, > > Big Data Consultant > > [image: Description: Description: Description: Description: > > cid:[email protected]] > > ------------------------------ > > *From:* Paweł Kociński <[email protected]<mailto: > [email protected]>> > > *Sent:* Wednesday, November 11, 2020 00:29 > > *To:* Jia Yu > > *Cc:* [email protected]<mailto:[email protected]>; > [email protected]<mailto:[email protected]>; > Jinxuan Wu; > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > > > Hi Jia, > > I think we should support spark 2.4, a lot of users still use it. More > > than that I think more users still have jobs written in spark 2.4 than > > 3.0. We will use an additional branch for that use case ? I mean Spark > 2.4 > > with scala 2.12 is important one. > > Regards, > > Paweł > > > > pon., 9 lis 2020 o 20:44 Jia Yu <[email protected]<mailto: > [email protected]>> napisał(a): > > > >> Dear all, > >> > >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder > whether > >> we should support Spark 2.4. > >> > >> In order to support Spark 2.4, we need to do the following > >> > >> 1. Compile the source using Scala 2.11. Sedona master branch currently > is > >> compiled by Scala 2.12 and Java 1.8 > >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the > (1) > >> UDF registration hook (2) the SQL aggregation function format > >> 3. In the future releases of Sedona, use git cherry-pick to pick > >> important features back to the Spark 2.4 branch. This is what I did in > >> GeoSpark to support Spark 2.1, 2.2, 2.3 > >> > >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it > >> that way and just support Spark 3.0. > >> > >> Do you think we should support Spark 2.4 in the future release? > >> > >> Thanks, > >> Jia Yu > >> > > >
