Hi all, This is a discussion thread about deprecating Spark3.2 in the following version of Gluten release.
We are planning to deprecate Spark3.2 maintenance for several reasons. 1. To reduce CI/CD resource consumption in github action: currently Gluten needs to run every version of Spark(current support 3.2/3.3/3.4/3.5) during submitting each PR. We would like to reduce resource usage during CI/CD testing. 2. Most customers are moving to Spark 3.4/3.5/4.0: Many Gluten customers are moving to Spark3.4/3.5 and in the upcoming Gluten release, we will support Spark 4.0 as well. 3. For Spark3.2 users, we will recommend to use an older version of Gluten such as Gluten 1.2 or 1.3 and a minor version upgrade(such as 1.2.2/1.3.1) is on demand to give the best support. It's better to support up to four versions of Spark in a Gluten release cycle. The proposed plan will be in below: Gluten 1.4.0: target for Q1'25 and plan to add Spark 4.0 API support only Gluten 1.5.0: target for Q2'25 and plan to add Spark 4.0 UT support & deprecate Spark 3.2 support Please feel free to discuss this if you have any comments on this plan. Thanks, Weiting --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
