Re: Apache Spark 3.2 Expectation

*GA -> QA On Thu, 17 Jun 2021, 15:16 Hyukjin Kwon, wrote: > I think we would make sure treating these items in the list as exceptions > from the code freeze, and discourage to push new APIs and features though. > > GA period ideally we should focus on bug fixes and polishing. > > It would be gre

Re: Apache Spark 3.2 Expectation

I think we would make sure treating these items in the list as exceptions from the code freeze, and discourage to push new APIs and features though. GA period ideally we should focus on bug fixes and polishing. It would be great if we can speed up on these items in the list too. On Thu, 17 Jun

Re: Apache Spark 3.2 Expectation

Thanks for the suggestions from Dongjoon, Liangchi, Min, and Xiao! Now we make it clear that it's a soft cut and we can still merge important code changes to branch-3.2 before RC. Let's keep the branch cut date as July 1st. On Thu, Jun 17, 2021 at 1:41 PM Dongjoon Hyun wrote: > > First, I think

Re: Apache Spark 3.2 Expectation

> First, I think you are saying "branch-3.2"; To Xiao. Yes, it's was a typo of "branch-3.2". > We do strongly prefer to cut the release for Spark 3.2.0 including all the patches under SPARK-30602. > This way, we can backport the other performance/operability enhancements tickets under SPARK-33235

Re: Apache Spark 3.2 Expectation

> > To Liang-Chi, I'm -1 for postponing the branch cut because this is a soft > cut and the committers still are able to commit to `branch-3.3` according > to their decisions. First, I think you are saying "branch-3.2"; Second, the "so cut" means no "code freeze", although we cut the branch. To

Re: Apache Spark 3.2 Expectation

Hi Gengliang, Thanks for volunteering as the release manager for Spark 3.2.0. Regarding the ongoing work of push-based shuffle in SPARK-30602, we are close to having all the patches merged to master to enable push-based shuffle. Currently, there are 2 PRs under SPARK-30602 that are under active re

UPDATE: Apache Spark 3.2 Release

This is a continuation of the previous thread, `Apache Spark 3.2 Expectation`, in order to give you updates. - https://lists.apache.org/thread.html/r61897da071729913bf586ddd769311ce8b5b068e7156c352b51f7a33%40%3Cdev.spark.apache.org%3E First of all, the AS-IS schedule is here - https://spark.ap

Re: Apache Spark 3.2 Expectation

Thanks Dongjoon. I've talked with Dongjoon offline to know more this. As it is soft cut date, there is no reason to postpone it. It sounds good then to keep original branch cut date. Thank you. Dongjoon Hyun-2 wrote > Thank you for volunteering, Gengliang. > > Apache Spark 3.2.0 is the firs

Re: Apache Spark 3.2 Expectation

Thank you for volunteering, Gengliang. Apache Spark 3.2.0 is the first version enabling AQE by default. I'm also watching some on-going improvements on that. https://issues.apache.org/jira/browse/SPARK-33828 (SQL Adaptive Query Execution QA) To Liang-Chi, I'm -1 for postponing the branch cut

Re: Apache Spark 3.2 Expectation

First, thanks for being volunteer as the release manager of Spark 3.2.0, Gengliang! And yes, for the two important Structured Streaming features, RocksDB StateStore and session window, we're working on them and expect to have them in the new release. So I propose to postpone the branch cut date.

Re: Apache Spark 3.2 Expectation

Thanks, Hyukjin. The expected target branch cut date of Spark 3.2 is *July 1st* on https://spark.apache.org/versioning-policy.html. However, I notice that there are still multiple important projects in progress now: [Core] - SPIP: Support push-based shuffle to improve shuffle efficiency