What do you mean by compile target? ?
Iv'e found Apache Zeppelin handle multiple Spark versions here using profile for each Spark version: https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 Do you think this method is better? Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:[email protected]] ________________________________ From: Felix Cheung <[email protected]> Sent: Thursday, November 12, 2020 04:05 To: [email protected] Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi Zhang; [email protected] Subject: Re: Support Spark 2.4 in Sedona 1.0 I am not sure it should be a branch? It is common to deal with this as a compile target, not as a separate branch. A separate branch might have difficulty to release? There are a few example in projects where they handle multiple Spark target version like this. On Wed, Nov 11, 2020 at 12:56 PM Jia Yu <[email protected]<mailto:[email protected]>> wrote: OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the compiler used in each branch, For Sedona on Spark 3.0, I will compile it using Scala 2.12 For Sedona on Spark 2.4, I will compile it using Scala 2.11. For the Java code in both branches, I will compile them using Java 1.8 Am I missing anything here? On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka <[email protected]<mailto:[email protected]>> wrote: > Hi, > > I also think that we need to support 2.4. > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > 12, 2020) > > I also asked about it on > *[email protected]<mailto:[email protected]> > <[email protected]<mailto:[email protected]>>* > : > > > *Sean Owen (answered the question): * > > "I don't think there's an official EOL for Spark 2.4.x but would expect > another maintenance release in the first half of 2021 at least. I'd also > guess it wouldn't be maintained by 2022." > > > ?BR, > > > > Netanel Malka, > Big Data Consultant > [image: Description: Description: Description: Description: > cid:[email protected]] > ------------------------------ > *From:* Paweł Kociński > <[email protected]<mailto:[email protected]>> > *Sent:* Wednesday, November 11, 2020 00:29 > *To:* Jia Yu > *Cc:* [email protected]<mailto:[email protected]>; > [email protected]<mailto:[email protected]>; > Jinxuan Wu; > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > Hi Jia, > I think we should support spark 2.4, a lot of users still use it. More > than that I think more users still have jobs written in spark 2.4 than > 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 > with scala 2.12 is important one. > Regards, > Paweł > > pon., 9 lis 2020 o 20:44 Jia Yu <[email protected]<mailto:[email protected]>> > napisał(a): > >> Dear all, >> >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether >> we should support Spark 2.4. >> >> In order to support Spark 2.4, we need to do the following >> >> 1. Compile the source using Scala 2.11. Sedona master branch currently is >> compiled by Scala 2.12 and Java 1.8 >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) >> UDF registration hook (2) the SQL aggregation function format >> 3. In the future releases of Sedona, use git cherry-pick to pick >> important features back to the Spark 2.4 branch. This is what I did in >> GeoSpark to support Spark 2.1, 2.2, 2.3 >> >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it >> that way and just support Spark 3.0. >> >> Do you think we should support Spark 2.4 in the future release? >> >> Thanks, >> Jia Yu >> >
