Re: Support Spark 2.4 in Sedona 1.0
You might be able to create shim and build in profile Like https://github.com/graphframes/graphframes/tree/branch-0.5/src/main On Thu, Nov 12, 2020 at 3:23 PM Jia Yu wrote: > In order to support Spark 2.4, Sedona needs to use different logic for SQL > aggregation functions. I am not sure if this could be achieved by using > different profiles. > > Sedona for Spark 3.0: > https://github.com/apache/incubator-sedona/blob/master/sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/AggregateFunctions.scala > Sedona for Spark 2.4: > https://github.com/apache/incubator-sedona/blob/spark-2.3-2.4/sql/src/main/scala/org/apache/spark/sql/geosparksql/expressions/AggregateFunctions.scala > > On Thu, Nov 12, 2020 at 9:42 AM Mo Sarwat wrote: > >> Yes, I believe we still need to support Spark 2.4 since many Sedona users >> are still using it >> >> On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka >> wrote: >> >> > What do you mean by compile target? >> > >> > ? >> > >> > Iv'e found Apache Zeppelin handle multiple Spark versions here using >> > profile for each Spark version: >> > >> > https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 >> > >> > >> > Do you think this method is better? >> > >> > >> > >> > Netanel Malka, >> > Big Data Consultant >> > [Description: Description: Description: Description: >> > cid:image001.jpg@01C85203.36A2AF30] >> > ____ >> > From: Felix Cheung >> > Sent: Thursday, November 12, 2020 04:05 >> > To: dev@sedona.apache.org >> > Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi >> > Zhang; d...@sedona.incubator.apache.org >> > Subject: Re: Support Spark 2.4 in Sedona 1.0 >> > >> > I am not sure it should be a branch? It is common to deal with this as a >> > compile target, not as a separate branch. A separate branch might have >> > difficulty to release? >> > >> > There are a few example in projects where they handle multiple Spark >> > target version like this. >> > >> > >> > On Wed, Nov 11, 2020 at 12:56 PM Jia Yu > > ji...@apache.org>> wrote: >> > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the >> > compiler used in each branch, >> > >> > For Sedona on Spark 3.0, I will compile it using Scala 2.12 >> > For Sedona on Spark 2.4, I will compile it using Scala 2.11. >> > >> > For the Java code in both branches, I will compile them using Java 1.8 >> > >> > Am I missing anything here? >> > >> > >> > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka > > > netan...@sela.co.il>> wrote: >> > >> > > Hi, >> > > >> > > I also think that we need to support 2.4. >> > > >> > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 >> Sep >> > > 12, 2020) >> > > >> > > I also asked about it on *us...@spark.apache.org> > us...@spark.apache.org> > > us...@spark.apache.org>>* >> > > : >> > > >> > > >> > > *Sean Owen (answered the question): * >> > > >> > > "I don't think there's an official EOL for Spark 2.4.x but would >> expect >> > > another maintenance release in the first half of 2021 at least. I'd >> also >> > > guess it wouldn't be maintained by 2022." >> > > >> > > >> > > ?BR, >> > > >> > > >> > > >> > > Netanel Malka, >> > > Big Data Consultant >> > > [image: Description: Description: Description: Description: >> > > cid:image001.jpg@01C85203.36A2AF30] >> > > -- >> > > *From:* Paweł Kociński > > pawel93kocin...@gmail.com>> >> > > *Sent:* Wednesday, November 11, 2020 00:29 >> > > *To:* Jia Yu >> > > *Cc:* dev@sedona.apache.org<mailto:dev@sedona.apache.org>; >> > d...@sedona.incubator.apache.org<mailto:d...@sedona.incubator.apache.org >> >; >> > Jinxuan Wu; >> > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang >> > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 >> > > >> > > Hi Jia, >> > > I think we should support spark 2.4, a lot of users still use it. More >> > > than that I think more users still have jobs writt
Re: Support Spark 2.4 in Sedona 1.0
In order to support Spark 2.4, Sedona needs to use different logic for SQL aggregation functions. I am not sure if this could be achieved by using different profiles. Sedona for Spark 3.0: https://github.com/apache/incubator-sedona/blob/master/sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/AggregateFunctions.scala Sedona for Spark 2.4: https://github.com/apache/incubator-sedona/blob/spark-2.3-2.4/sql/src/main/scala/org/apache/spark/sql/geosparksql/expressions/AggregateFunctions.scala On Thu, Nov 12, 2020 at 9:42 AM Mo Sarwat wrote: > Yes, I believe we still need to support Spark 2.4 since many Sedona users > are still using it > > On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka > wrote: > > > What do you mean by compile target? > > > > ? > > > > Iv'e found Apache Zeppelin handle multiple Spark versions here using > > profile for each Spark version: > > > > https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 > > > > > > Do you think this method is better? > > > > > > > > Netanel Malka, > > Big Data Consultant > > [Description: Description: Description: Description: > > cid:image001.jpg@01C85203.36A2AF30] > > > > From: Felix Cheung > > Sent: Thursday, November 12, 2020 04:05 > > To: dev@sedona.apache.org > > Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi > > Zhang; d...@sedona.incubator.apache.org > > Subject: Re: Support Spark 2.4 in Sedona 1.0 > > > > I am not sure it should be a branch? It is common to deal with this as a > > compile target, not as a separate branch. A separate branch might have > > difficulty to release? > > > > There are a few example in projects where they handle multiple Spark > > target version like this. > > > > > > On Wed, Nov 11, 2020 at 12:56 PM Jia Yu > ji...@apache.org>> wrote: > > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the > > compiler used in each branch, > > > > For Sedona on Spark 3.0, I will compile it using Scala 2.12 > > For Sedona on Spark 2.4, I will compile it using Scala 2.11. > > > > For the Java code in both branches, I will compile them using Java 1.8 > > > > Am I missing anything here? > > > > > > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka > netan...@sela.co.il>> wrote: > > > > > Hi, > > > > > > I also think that we need to support 2.4. > > > > > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > > > 12, 2020) > > > > > > I also asked about it on *us...@spark.apache.org > us...@spark.apache.org> > us...@spark.apache.org>>* > > > : > > > > > > > > > *Sean Owen (answered the question): * > > > > > > "I don't think there's an official EOL for Spark 2.4.x but would expect > > > another maintenance release in the first half of 2021 at least. I'd > also > > > guess it wouldn't be maintained by 2022." > > > > > > > > > ?BR, > > > > > > > > > > > > Netanel Malka, > > > Big Data Consultant > > > [image: Description: Description: Description: Description: > > > cid:image001.jpg@01C85203.36A2AF30] > > > -- > > > *From:* Paweł Kociński > pawel93kocin...@gmail.com>> > > > *Sent:* Wednesday, November 11, 2020 00:29 > > > *To:* Jia Yu > > > *Cc:* dev@sedona.apache.org<mailto:dev@sedona.apache.org>; > > d...@sedona.incubator.apache.org<mailto:d...@sedona.incubator.apache.org>; > > Jinxuan Wu; > > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > > > > > Hi Jia, > > > I think we should support spark 2.4, a lot of users still use it. More > > > than that I think more users still have jobs written in spark 2.4 than > > > 3.0. We will use an additional branch for that use case ? I mean Spark > > 2.4 > > > with scala 2.12 is important one. > > > Regards, > > > Paweł > > > > > > pon., 9 lis 2020 o 20:44 Jia Yu > ji...@apache.org>> napisał(a): > > > > > >> Dear all, > > >> > > >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder > > whether > > >> we should support Spark 2.4. > > >> > > >> In order to support Spark 2.4, we need to do the following > > >> > > >> 1. Compile the source using Scala 2.11. Sedona master branch currently > > is > > >> compiled by Scala 2.12 and Java 1.8 > > >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the > > (1) > > >> UDF registration hook (2) the SQL aggregation function format > > >> 3. In the future releases of Sedona, use git cherry-pick to pick > > >> important features back to the Spark 2.4 branch. This is what I did in > > >> GeoSpark to support Spark 2.1, 2.2, 2.3 > > >> > > >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave > it > > >> that way and just support Spark 3.0. > > >> > > >> Do you think we should support Spark 2.4 in the future release? > > >> > > >> Thanks, > > >> Jia Yu > > >> > > > > > >
Re: Support Spark 2.4 in Sedona 1.0
Yes, I believe we still need to support Spark 2.4 since many Sedona users are still using it On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka wrote: > What do you mean by compile target? > > ? > > Iv'e found Apache Zeppelin handle multiple Spark versions here using > profile for each Spark version: > > https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 > > > Do you think this method is better? > > > > Netanel Malka, > Big Data Consultant > [Description: Description: Description: Description: > cid:image001.jpg@01C85203.36A2AF30] > > From: Felix Cheung > Sent: Thursday, November 12, 2020 04:05 > To: dev@sedona.apache.org > Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi > Zhang; d...@sedona.incubator.apache.org > Subject: Re: Support Spark 2.4 in Sedona 1.0 > > I am not sure it should be a branch? It is common to deal with this as a > compile target, not as a separate branch. A separate branch might have > difficulty to release? > > There are a few example in projects where they handle multiple Spark > target version like this. > > > On Wed, Nov 11, 2020 at 12:56 PM Jia Yu ji...@apache.org>> wrote: > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the > compiler used in each branch, > > For Sedona on Spark 3.0, I will compile it using Scala 2.12 > For Sedona on Spark 2.4, I will compile it using Scala 2.11. > > For the Java code in both branches, I will compile them using Java 1.8 > > Am I missing anything here? > > > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka netan...@sela.co.il>> wrote: > > > Hi, > > > > I also think that we need to support 2.4. > > > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > > 12, 2020) > > > > I also asked about it on *us...@spark.apache.org us...@spark.apache.org> us...@spark.apache.org>>* > > : > > > > > > *Sean Owen (answered the question): * > > > > "I don't think there's an official EOL for Spark 2.4.x but would expect > > another maintenance release in the first half of 2021 at least. I'd also > > guess it wouldn't be maintained by 2022." > > > > > > ?BR, > > > > > > > > Netanel Malka, > > Big Data Consultant > > [image: Description: Description: Description: Description: > > cid:image001.jpg@01C85203.36A2AF30] > > -- > > *From:* Paweł Kociński pawel93kocin...@gmail.com>> > > *Sent:* Wednesday, November 11, 2020 00:29 > > *To:* Jia Yu > > *Cc:* dev@sedona.apache.org<mailto:dev@sedona.apache.org>; > d...@sedona.incubator.apache.org<mailto:d...@sedona.incubator.apache.org>; > Jinxuan Wu; > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > > > Hi Jia, > > I think we should support spark 2.4, a lot of users still use it. More > > than that I think more users still have jobs written in spark 2.4 than > > 3.0. We will use an additional branch for that use case ? I mean Spark > 2.4 > > with scala 2.12 is important one. > > Regards, > > Paweł > > > > pon., 9 lis 2020 o 20:44 Jia Yu ji...@apache.org>> napisał(a): > > > >> Dear all, > >> > >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder > whether > >> we should support Spark 2.4. > >> > >> In order to support Spark 2.4, we need to do the following > >> > >> 1. Compile the source using Scala 2.11. Sedona master branch currently > is > >> compiled by Scala 2.12 and Java 1.8 > >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the > (1) > >> UDF registration hook (2) the SQL aggregation function format > >> 3. In the future releases of Sedona, use git cherry-pick to pick > >> important features back to the Spark 2.4 branch. This is what I did in > >> GeoSpark to support Spark 2.1, 2.2, 2.3 > >> > >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it > >> that way and just support Spark 3.0. > >> > >> Do you think we should support Spark 2.4 in the future release? > >> > >> Thanks, > >> Jia Yu > >> > > >
Re: Support Spark 2.4 in Sedona 1.0
What do you mean by compile target? ? Iv'e found Apache Zeppelin handle multiple Spark versions here using profile for each Spark version: https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 Do you think this method is better? Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:image001.jpg@01C85203.36A2AF30] From: Felix Cheung Sent: Thursday, November 12, 2020 04:05 To: dev@sedona.apache.org Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi Zhang; d...@sedona.incubator.apache.org Subject: Re: Support Spark 2.4 in Sedona 1.0 I am not sure it should be a branch? It is common to deal with this as a compile target, not as a separate branch. A separate branch might have difficulty to release? There are a few example in projects where they handle multiple Spark target version like this. On Wed, Nov 11, 2020 at 12:56 PM Jia Yu mailto:ji...@apache.org>> wrote: OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the compiler used in each branch, For Sedona on Spark 3.0, I will compile it using Scala 2.12 For Sedona on Spark 2.4, I will compile it using Scala 2.11. For the Java code in both branches, I will compile them using Java 1.8 Am I missing anything here? On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka mailto:netan...@sela.co.il>> wrote: > Hi, > > I also think that we need to support 2.4. > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > 12, 2020) > > I also asked about it on > *us...@spark.apache.org<mailto:us...@spark.apache.org> > mailto:us...@spark.apache.org>>* > : > > > *Sean Owen (answered the question): * > > "I don't think there's an official EOL for Spark 2.4.x but would expect > another maintenance release in the first half of 2021 at least. I'd also > guess it wouldn't be maintained by 2022." > > > ?BR, > > > > Netanel Malka, > Big Data Consultant > [image: Description: Description: Description: Description: > cid:image001.jpg@01C85203.36A2AF30] > -- > *From:* Paweł Kociński > mailto:pawel93kocin...@gmail.com>> > *Sent:* Wednesday, November 11, 2020 00:29 > *To:* Jia Yu > *Cc:* dev@sedona.apache.org<mailto:dev@sedona.apache.org>; > d...@sedona.incubator.apache.org<mailto:d...@sedona.incubator.apache.org>; > Jinxuan Wu; > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > Hi Jia, > I think we should support spark 2.4, a lot of users still use it. More > than that I think more users still have jobs written in spark 2.4 than > 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 > with scala 2.12 is important one. > Regards, > Paweł > > pon., 9 lis 2020 o 20:44 Jia Yu mailto:ji...@apache.org>> > napisał(a): > >> Dear all, >> >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether >> we should support Spark 2.4. >> >> In order to support Spark 2.4, we need to do the following >> >> 1. Compile the source using Scala 2.11. Sedona master branch currently is >> compiled by Scala 2.12 and Java 1.8 >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) >> UDF registration hook (2) the SQL aggregation function format >> 3. In the future releases of Sedona, use git cherry-pick to pick >> important features back to the Spark 2.4 branch. This is what I did in >> GeoSpark to support Spark 2.1, 2.2, 2.3 >> >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it >> that way and just support Spark 3.0. >> >> Do you think we should support Spark 2.4 in the future release? >> >> Thanks, >> Jia Yu >> >
Re: Support Spark 2.4 in Sedona 1.0
I am not sure it should be a branch? It is common to deal with this as a compile target, not as a separate branch. A separate branch might have difficulty to release? There are a few example in projects where they handle multiple Spark target version like this. On Wed, Nov 11, 2020 at 12:56 PM Jia Yu wrote: > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the > compiler used in each branch, > > For Sedona on Spark 3.0, I will compile it using Scala 2.12 > For Sedona on Spark 2.4, I will compile it using Scala 2.11. > > For the Java code in both branches, I will compile them using Java 1.8 > > Am I missing anything here? > > > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka wrote: > > > Hi, > > > > I also think that we need to support 2.4. > > > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > > 12, 2020) > > > > I also asked about it on *us...@spark.apache.org >* > > : > > > > > > *Sean Owen (answered the question): * > > > > "I don't think there's an official EOL for Spark 2.4.x but would expect > > another maintenance release in the first half of 2021 at least. I'd also > > guess it wouldn't be maintained by 2022." > > > > > > BR, > > > > > > > > Netanel Malka, > > Big Data Consultant > > [image: Description: Description: Description: Description: > > cid:image001.jpg@01C85203.36A2AF30] > > -- > > *From:* Paweł Kociński > > *Sent:* Wednesday, November 11, 2020 00:29 > > *To:* Jia Yu > > *Cc:* dev@sedona.apache.org; d...@sedona.incubator.apache.org; Jinxuan > Wu; > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > > > Hi Jia, > > I think we should support spark 2.4, a lot of users still use it. More > > than that I think more users still have jobs written in spark 2.4 than > > 3.0. We will use an additional branch for that use case ? I mean Spark > 2.4 > > with scala 2.12 is important one. > > Regards, > > Paweł > > > > pon., 9 lis 2020 o 20:44 Jia Yu napisał(a): > > > >> Dear all, > >> > >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder > whether > >> we should support Spark 2.4. > >> > >> In order to support Spark 2.4, we need to do the following > >> > >> 1. Compile the source using Scala 2.11. Sedona master branch currently > is > >> compiled by Scala 2.12 and Java 1.8 > >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the > (1) > >> UDF registration hook (2) the SQL aggregation function format > >> 3. In the future releases of Sedona, use git cherry-pick to pick > >> important features back to the Spark 2.4 branch. This is what I did in > >> GeoSpark to support Spark 2.1, 2.2, 2.3 > >> > >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it > >> that way and just support Spark 3.0. > >> > >> Do you think we should support Spark 2.4 in the future release? > >> > >> Thanks, > >> Jia Yu > >> > > >
Re: Support Spark 2.4 in Sedona 1.0
OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the compiler used in each branch, For Sedona on Spark 3.0, I will compile it using Scala 2.12 For Sedona on Spark 2.4, I will compile it using Scala 2.11. For the Java code in both branches, I will compile them using Java 1.8 Am I missing anything here? On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka wrote: > Hi, > > I also think that we need to support 2.4. > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > 12, 2020) > > I also asked about it on *us...@spark.apache.org * > : > > > *Sean Owen (answered the question): * > > "I don't think there's an official EOL for Spark 2.4.x but would expect > another maintenance release in the first half of 2021 at least. I'd also > guess it wouldn't be maintained by 2022." > > > BR, > > > > Netanel Malka, > Big Data Consultant > [image: Description: Description: Description: Description: > cid:image001.jpg@01C85203.36A2AF30] > -- > *From:* Paweł Kociński > *Sent:* Wednesday, November 11, 2020 00:29 > *To:* Jia Yu > *Cc:* dev@sedona.apache.org; d...@sedona.incubator.apache.org; Jinxuan Wu; > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > Hi Jia, > I think we should support spark 2.4, a lot of users still use it. More > than that I think more users still have jobs written in spark 2.4 than > 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 > with scala 2.12 is important one. > Regards, > Paweł > > pon., 9 lis 2020 o 20:44 Jia Yu napisał(a): > >> Dear all, >> >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether >> we should support Spark 2.4. >> >> In order to support Spark 2.4, we need to do the following >> >> 1. Compile the source using Scala 2.11. Sedona master branch currently is >> compiled by Scala 2.12 and Java 1.8 >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) >> UDF registration hook (2) the SQL aggregation function format >> 3. In the future releases of Sedona, use git cherry-pick to pick >> important features back to the Spark 2.4 branch. This is what I did in >> GeoSpark to support Spark 2.1, 2.2, 2.3 >> >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it >> that way and just support Spark 3.0. >> >> Do you think we should support Spark 2.4 in the future release? >> >> Thanks, >> Jia Yu >> >
Re: Support Spark 2.4 in Sedona 1.0
Hi, I also think that we need to support 2.4. I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep 12, 2020) I also asked about it on us...@spark.apache.org : Sean Owen (answered the question): "I don't think there's an official EOL for Spark 2.4.x but would expect another maintenance release in the first half of 2021 at least. I'd also guess it wouldn't be maintained by 2022." ?BR, Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:image001.jpg@01C85203.36A2AF30] From: Paweł Kociński Sent: Wednesday, November 11, 2020 00:29 To: Jia Yu Cc: dev@sedona.apache.org; d...@sedona.incubator.apache.org; Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Zongsi Zhang Subject: Re: Support Spark 2.4 in Sedona 1.0 Hi Jia, I think we should support spark 2.4, a lot of users still use it. More than that I think more users still have jobs written in spark 2.4 than 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 with scala 2.12 is important one. Regards, Paweł pon., 9 lis 2020 o 20:44 Jia Yu mailto:ji...@apache.org>> napisał(a): Dear all, In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether we should support Spark 2.4. In order to support Spark 2.4, we need to do the following 1. Compile the source using Scala 2.11. Sedona master branch currently is compiled by Scala 2.12 and Java 1.8 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) UDF registration hook (2) the SQL aggregation function format 3. In the future releases of Sedona, use git cherry-pick to pick important features back to the Spark 2.4 branch. This is what I did in GeoSpark to support Spark 2.1, 2.2, 2.3 GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it that way and just support Spark 3.0. Do you think we should support Spark 2.4 in the future release? Thanks, Jia Yu
Re: Support Spark 2.4 in Sedona 1.0
Hi Jia, I think we should support spark 2.4, a lot of users still use it. More than that I think more users still have jobs written in spark 2.4 than 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 with scala 2.12 is important one. Regards, Paweł pon., 9 lis 2020 o 20:44 Jia Yu napisał(a): > Dear all, > > In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether > we should support Spark 2.4. > > In order to support Spark 2.4, we need to do the following > > 1. Compile the source using Scala 2.11. Sedona master branch currently is > compiled by Scala 2.12 and Java 1.8 > 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) > UDF registration hook (2) the SQL aggregation function format > 3. In the future releases of Sedona, use git cherry-pick to pick important > features back to the Spark 2.4 branch. This is what I did in GeoSpark to > support Spark 2.1, 2.2, 2.3 > > GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it > that way and just support Spark 3.0. > > Do you think we should support Spark 2.4 in the future release? > > Thanks, > Jia Yu >
