> > To Liang-Chi, I'm -1 for postponing the branch cut because this is a soft > cut and the committers still are able to commit to `branch-3.3` according > to their decisions.
First, I think you are saying "branch-3.2"; Second, the "so cut" means no "code freeze", although we cut the branch. To avoid releasing half-baked and unready features, the release manager needs to be very careful when cutting the RC. Based on what is proposed here, the RC date is the actual code freeze date. This way, we can backport the other performance/operability enhancements > tickets under SPARK-33235 into branch-3.2 to be released in future Spark > 3.2.x patch releases. This is not allowed based on the policy. Only bug fixes can be merged to the patch releases. Thus, if we know it will introduce major performance regression, we have to turn the feature off by default. Xiao Min Shen <victor....@gmail.com> 于2021年6月16日周三 下午3:22写道： > Hi Gengliang, > > Thanks for volunteering as the release manager for Spark 3.2.0. > Regarding the ongoing work of push-based shuffle in SPARK-30602, we are > close to having all the patches merged to master to enable push-based > shuffle. > Currently, there are 2 PRs under SPARK-30602 that are under active review > (SPARK-32922 and SPARK-35671), and hopefully can be merged soon. > We should be able to post the PRs for the other 2 remaining tickets > (SPARK-32923 and SPARK-35546) early next week. > > The tickets under SPARK-30602 are the minimum set of patches to enable > push-based shuffle. > We do have other performance/operability enhancements tickets under > SPARK-33235 that are needed to fully contribute what we have internally for > push-based shuffle. > However, these are optional for enabling push-based shuffle. > We do strongly prefer to cut the release for Spark 3.2.0 including all the > patches under SPARK-30602. > This way, we can backport the other performance/operability enhancements > tickets under SPARK-33235 into branch-3.2 to be released in future Spark > 3.2.x patch releases. > I understand the preference of not postponing the branch cut date. > We will check with Dongjoon regarding the soft cut date and the > flexibility for including the remaining tickets under SPARK-30602 into > branch-3.2. > > Best, > Min > > On Wed, Jun 16, 2021 at 1:20 PM Liang-Chi Hsieh <vii...@gmail.com> wrote: > >> >> Thanks Dongjoon. I've talked with Dongjoon offline to know more this. >> As it is soft cut date, there is no reason to postpone it. >> >> It sounds good then to keep original branch cut date. >> >> Thank you. >> >> >> >> Dongjoon Hyun-2 wrote >> > Thank you for volunteering, Gengliang. >> > >> > Apache Spark 3.2.0 is the first version enabling AQE by default. I'm >> also >> > watching some on-going improvements on that. >> > >> > https://issues.apache.org/jira/browse/SPARK-33828 (SQL Adaptive >> Query >> > Execution QA) >> > >> > To Liang-Chi, I'm -1 for postponing the branch cut because this is a >> soft >> > cut and the committers still are able to commit to `branch-3.3` >> according >> > to their decisions. >> > >> > Given that Apache Spark had 115 commits in a week in various areas >> > concurrently, we should start QA for Apache Spark 3.2 by creating >> > branch-3.3 and allowing only limited backporting. >> > >> > https://github.com/apache/spark/graphs/commit-activity >> > >> > Bests, >> > Dongjoon. >> > >> > >> > On Wed, Jun 16, 2021 at 9:19 AM Liang-Chi Hsieh < >> >> > viirya@ >> >> > > wrote: >> > >> >> First, thanks for being volunteer as the release manager of Spark >> 3.2.0, >> >> Gengliang! >> >> >> >> And yes, for the two important Structured Streaming features, RocksDB >> >> StateStore and session window, we're working on them and expect to have >> >> them >> >> in the new release. >> >> >> >> So I propose to postpone the branch cut date. >> >> >> >> Thank you! >> >> >> >> Liang-Chi >> >> >> >> >> >> Gengliang Wang-2 wrote >> >> > Thanks, Hyukjin. >> >> > >> >> > The expected target branch cut date of Spark 3.2 is *July 1st* on >> >> > https://spark.apache.org/versioning-policy.html. However, I notice >> that >> >> > there are still multiple important projects in progress now: >> >> > >> >> > [Core] >> >> > >> >> > - SPIP: Support push-based shuffle to improve shuffle efficiency >> >> > <https://issues.apache.org/jira/browse/SPARK-30602> >> >> > >> >> > [SQL] >> >> > >> >> > - Support ANSI SQL INTERVAL types >> >> > <https://issues.apache.org/jira/browse/SPARK-27790> >> >> > - Support Timestamp without time zone data type >> >> > <https://issues.apache.org/jira/browse/SPARK-35662> >> >> > - Aggregate (Min/Max/Count) push down for Parquet >> >> > <https://issues.apache.org/jira/browse/SPARK-34952> >> >> > >> >> > [Streaming] >> >> > >> >> > - EventTime based sessionization (session window) >> >> > <https://issues.apache.org/jira/browse/SPARK-10816> >> >> > - Add RocksDB StateStore as external module >> >> > <https://issues.apache.org/jira/browse/SPARK-34198> >> >> > >> >> > >> >> > I wonder whether we should postpone the branch cut date. >> >> > cc Min Shen, Yi Wu, Max Gekk, Huaxin Gao, Jungtaek Lim, Yuanjian >> >> > Li, Liang-Chi Hsieh, who work on the projects above. >> >> > >> >> > On Tue, Jun 15, 2021 at 4:34 PM Hyukjin Kwon < >> >> >> >> > gurwls223@ >> >> >> >> > > wrote: >> >> > >> >> >> +1, thanks. >> >> >> >> >> >> On Tue, 15 Jun 2021, 16:17 Gengliang Wang, < >> >> >> >> > ltnwgl@ >> >> >> >> > > wrote: >> >> >> >> >> >>> Hi, >> >> >>> >> >> >>> As the expected release date is close, I would like to volunteer >> as >> >> the >> >> >>> release manager for Apache Spark 3.2.0. >> >> >>> >> >> >>> Thanks, >> >> >>> Gengliang >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe e-mail: >> >> > dev-unsubscribe@.apache >> >> >> >> >> >> >> >> >> >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
