I like this idea, but we should call out the impact to Spark development. IIUC, we need the following changes:
- If a feature is known to take a long time to complete, we should add a feature flag at the beginning, to avoid releasing a half-baked feature with this faster relase cadence. - We have defined behavior change in https://spark.apache.org/contributing.html. According to it, new APIs, new features, bug fixes, etc. are all behavior changes because they are user-visible. We should be more clear about what can go to minor releases. I think we can follow the same standard for writing migration guide: if the behavior change needs user action. - We will merge most PRs to both master and a new .x branch, and we should avoid merge conflicts as possible as we can. For example, behavior changes should also be merged into the .x branch, with flag off. An alternative idea is to still cut next release from the master branch, but we only allow a 3-month window per year (right before the next major release) to make breaking changes in the master branch, such as dependency upgrade, Java/Scala version upgrade, etc. Of cource we can still have exceptions with voting. The drawback is that sometimes 3 months is not sufficient to make major upgrades, and then the .x branch will be useful. On Tue, Dec 9, 2025 at 9:02 AM L. C. Hsieh <[email protected]> wrote: > I see. > > In the "Code Merging Principle" section, > > For non-behavior changes, always merge to master and the latest .x > branch. This change will be released with the next minor release > > Is this .x branch meaning a branch of a major branch like branch-4.x? > > Also, looks like master and the latest .x branch basically have the > same codebase? > > On Mon, Dec 8, 2025 at 4:37 PM Hyukjin Kwon <[email protected]> wrote: > > > > I actually intentionally disabled the commenter access so the discussion > can happen here :-). Otherwise, we would end up with multiple places to > discuss this. > > > > On Tue, 9 Dec 2025 at 09:33, L. C. Hsieh <[email protected]> wrote: > >> > >> Can you open comment access to the google doc? > >> So it will be easier to ask questions directly on the SPIP doc. > >> > >> On Sun, Dec 7, 2025 at 1:53 PM Hyukjin Kwon <[email protected]> > wrote: > >> > > >> > Hi all, > >> > > >> > I would like to start a discussion on accelerating the Apache Spark > release cadence. Over the past four months, we have been running preview > releases, and the process has been smooth and effective. As mentioned in > the preview release discussion thread, I’d now like to extend this approach > to official releases. > >> > > >> > During this period, I also looked into how other large projects, such > as Kubernetes and Python, manage their release timelines. Based on that > research and our own recent experience, I’ve drafted a proposal for an > updated Apache Spark release plan. > >> > > >> > TL;DR: > >> > > >> > Introduce a predictable release schedule: annual major releases and > quarterly minor releases, so users can benefit from new features earlier. > >> > With a faster cadence for minor releases, we should take a more > conservative approach toward behavior changes in minor versions, while > still allowing new features and improvements. > >> > > >> > I’d love to hear your thoughts and feedback. > >> > > >> > More details can be found in SPIP: Accelerating Apache Spark Release > Cadence > >> > > >> > Thanks! > >> > >> --------------------------------------------------------------------- > >> To unsubscribe e-mail: [email protected] > >> > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
