Re: A new release process?

[William A Rowe Jr]

On Tue, Jan 3, 2017 at 1:32 PM, Jacob Champion  wrote:
> On 12/29/2016 08:16 PM, David Zuelke wrote:
>>
>> The tl;dr of this approach is that
>>
>> - any x.y.z release only introduces bugfixes. These releases are done
>> every four weeks, like clockwork. If a fix doesn't make the cut for a
>> release, it'll end up in the next one; - x.y.0 releases, on the other
>> hand, may introduce new features, fixes, and deprecations, but no
>> breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0
>> in late 2015). where backward compatibility may be changed, etc.
>
> My favorite pieces here are
> - the introduction of a release cadence
> - the separation of bugfix releases from feature releases
>
> Both make life easier -- for us, for users, and for intermediate maintainers
> -- for the reasons you've mentioned.

Consider also our consumers, module authors who rely on us to provide an
API they can design to. All of our subsidiary components, be it apr, or expat,
or pcre, or openssl, or libxml2, or lua, or nghttp2... and most of the
expansions
to httpd, be it python or perl or whatnot, all follow essentially this model.

That isn't how httpd evolved; if you look far far back into the pre-1.0 history
and even through the evolution up to 1.3.12, there wasn't a particular rhyme
or reason other than what the major patch submitters wanted to have happen.
So when pools are introduced, minor version bump - a subversion bump was
obviously not going to cut it. But there was not necessarily any compatibility
at all with previously compiled modules.

This all changed radically with 1.3.14 and 2.0 later on. But the aversion from
back in the 1.2 and 1.3 days to declaring the version minors are precious
resources has not changed.

In the meantime, nghttp2 just published the 18th version minor by year end
last year. And remains solidly compatible due to good design from the get go,
it is still major version 1. Will be interesting to watch and see when enough
API issues have accumulated to ditch major v1 for v2.

We are certainly the enigma, and if there is one reason for developers to
disabused themselves from your API, that is likely making shit up as you
go along. I know a lot of folks frustrated back in the PHP 3-4 days, and it
seems like this solution has worked out very well for the PHP crew. And
there was an equal amount of chaos around Perl's versioning and lifecycle
evolution, but that also seems to be working out pretty well.


>> [...]
>>
>> There are a bunch of technicalities that would need adjusting to fit
>> HTTPD, such as release intervals, release management (for PHP, every
>> x.y.* series has two managers who jointly coordinate releases), etc,
>> but overall the idea is, IMO, worth considering.
>
> Our current process is also fairly labor-intensive, so moving to a quick
> release cadence without simultaneously fixing that is not likely to end up
> well, IMO. There are other discussions on the list for improving our
> automated QA, which I think is probably step 1.

Releases themselves are not terribly challenging. The investment, outside
of the reviews and testing we hope many committers are engaged in,
boils down to less than an hour of actual mechanics, and some time to
prepare specific announcement details or merge back in some sequestered
security patches if those have to be applied.


>> As a, more or less, "outside observer", I happen to think that the
>> current method of voting on finals, instead of a practice of rolling
>> out RCs (that are then left up for testing for at least a week), is
>> fundamentally broken. The 2.4 changelog in particular is littered
>> with releases that were never officially published. For users, that's
>> really confusing.
>
> To a certain extent I think this is something that people get used to
> easily... but for security patches in *particular*, I agree. It's
> disconcerting to hear that CVE 2099-BLAH was fixed in 2.4.Z and then find
> out that 2.4.Z was never released.

We say this, that subversion numbers are cheap, and the lack of 2.4.24
really isn't confusing. But I have to answer questions from at least 15
individuals over the course of the following month when that happens :)
So apparently it might not confuse us, but it does confuse the rest of
the world more than we might expect.


>> For maintainers, it's painful to start over the
>> process each time, and it sometimes leads to months and months
>> without a release that contains certain fixes.
>
> The pain of discarding and restarting your manual tests because someone else
> found a regression on another platform two hours into the release vote comes
> to mind, yes.

On that, adding features to a security release 4 days before tag and roll is
a sure fire path to a bad tag. But features-features-features, gotta collect
them all :)

Looking at a long term solution to the specific quirk that occurred in 2.4.24
and 2.4.25, but the general truth of throwing in one more late last feature

Re: A new release process?

[Jacob Champion]

On 12/29/2016 08:16 PM, David Zuelke wrote:

The tl;dr of this approach is that

- any x.y.z release only introduces bugfixes. These releases are done
every four weeks, like clockwork. If a fix doesn't make the cut for a
release, it'll end up in the next one; - x.y.0 releases, on the other
hand, may introduce new features, fixes, and deprecations, but no
breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0
in late 2015). where backward compatibility may be changed, etc.


My favorite pieces here are
- the introduction of a release cadence
- the separation of bugfix releases from feature releases

Both make life easier -- for us, for users, and for intermediate 
maintainers -- for the reasons you've mentioned.



[...]

There are a bunch of technicalities that would need adjusting to fit
HTTPD, such as release intervals, release management (for PHP, every
x.y.* series has two managers who jointly coordinate releases), etc,
but overall the idea is, IMO, worth considering.


Our current process is also fairly labor-intensive, so moving to a quick 
release cadence without simultaneously fixing that is not likely to end 
up well, IMO. There are other discussions on the list for improving our 
automated QA, which I think is probably step 1.



As a, more or less, "outside observer", I happen to think that the
current method of voting on finals, instead of a practice of rolling
out RCs (that are then left up for testing for at least a week), is
fundamentally broken. The 2.4 changelog in particular is littered
with releases that were never officially published. For users, that's
really confusing.


To a certain extent I think this is something that people get used to 
easily... but for security patches in *particular*, I agree. It's 
disconcerting to hear that CVE 2099-BLAH was fixed in 2.4.Z and then 
find out that 2.4.Z was never released.



For maintainers, it's painful to start over the
process each time, and it sometimes leads to months and months
without a release that contains certain fixes.


The pain of discarding and restarting your manual tests because someone 
else found a regression on another platform two hours into the release 
vote comes to mind, yes.


For the record, though, I think having strong automated QA processes 
will help both of these issues more than the release cadence will. It's 
just that the cadence reinforces the QA and the QA reinforces the 
cadence; they go hand-in-hand in a very nice positive feedback loop.



Then a backport goes
wrong (still using SVN, in my opinion, does not help there, but
that's a whole different discussion :)),


I've started some discussions on how we might introduce feature- and 
fix-branches, and hopefully better backport workflows, into our current 
use of SVN. (I'm a git user myself, and the current backport process is 
more error-prone than I'm used to as well. But I have no desire to start 
a holy war, and I'm confident we can find an SVN workflow that works well.)



and a regression is in the
latest release until someone eventually picks up a fix.

Much of this, and many of the "what do we backport from trunk" and
"I'd like to squeeze in a change I've had sitting around locally,
please wait with the new release, because who knows when the next one
after that will be" are, from what I can tell, a significant source
of discord on this mailing list. All these unnecessary distractions
that deteriorate personal relationships, while at the same time
slowing down the pace of the project (several people have already
pointed out Nginx's rate of innovation in comparison) and raising the
threshold for contributions, can be fixed. PHP is the perfect
example, and I think HTTPD would be wise to at least consider
following this example.

Happy New Year!


Thanks for your input!

--Jacob

Re: A new release process?

[Eric Covener]

On Thu, Dec 29, 2016 at 11:16 PM, David Zuelke  wrote:
> As a, more or less, "outside observer", I happen to think that the current 
> method of voting on finals, instead of a practice of rolling out RCs (that 
> are then left up for testing for at least a week), is fundamentally broken. 
> The 2.4 changelog in particular is littered with releases that were never 
> officially published. For users, that's really confusing. For maintainers, 
> it's painful to start over the process each time, and it sometimes leads to 
> months and months without a release that contains certain fixes. Then a 
> backport goes wrong (still using SVN, in my opinion, does not help there, but 
> that's a whole different discussion :)), and a regression is in the latest 
> release until someone eventually picks up a fix.

It seems to me that most "not released" versions are immediately
superseded.  The practice of throwing away the number instead of
calling the initial contents an RC shouldn't contribute meaningfully
to a fix being delayed.

New blood for release managers would be nice, and smaller/more
frequent /more predictable releases would probably help that.

-- 
Eric Covener
cove...@gmail.com

A new release process?

[David Zuelke]

Hi everyone,

Given the several current threads where there's arguing about what and how and 
when to release features, backported or not, I'd like to offer a tale of a 
project that was, more or less, in the same dire spot, and pulled itself out of 
that misery with great success and universal acclaim.

That project is PHP.

Back in 2010, it was stuck at version 5.3.something, there was no standards 
around what feature goes into a minor release, how a feature is even accepted 
for inclusion, how security updates were treated, and so forth.

Enter: https://wiki.php.net/rfc/releaseprocess

The tl;dr of this approach is that

- any x.y.z release only introduces bugfixes. These releases are done every 
four weeks, like clockwork. If a fix doesn't make the cut for a release, it'll 
end up in the next one;
- x.y.0 releases, on the other hand, may introduce new features, fixes, and 
deprecations, but no breaking changes;
- x.0.0 releases are the big ones (think PHP 7.0.0 in late 2015). where 
backward compatibility may be changed, etc.

This makes for very predictable update cycles, and for pretty easy updating, as 
it's very very unusual that a x.y.z release breaks existing behavior. At the 
same time, since x.y.0 releases are made roughly once a year, bigger changes do 
not exist in a limbo branch forever, but will see the light of day eventually 
(and, again, after a predictable amount of time). There is a defined roadmap 
with active/security maintenance status and EOL dates for all versions; see 
http://php.net/supported-versions.php

From a user perspective, this change has been fantastic. Once virtually 
stagnant, development of the language itself has been much more active; users 
have a much better idea up front around when fixes or features will land; 
upgrades have become easy as there are clear definitions of what kind of 
changes a version may contain.

But most importantly, the language itself has seen a steady flow of new 
features as a result from this discipline. For features and bigger changes, an 
RFC is written with rationale, analysis, suggestion of implementation, and then 
it can be voted on. Once accepted, it can be committed, reviewed, and merged. 
Even for "outside" contributors, this process is transparent, and the mental 
threshold towards contribution has been lowered greatly, as acceptance of an 
RFC basically means that the feature will go into a new release anywhere 
between within a few months and a year and a bit (depending on when in the 
release cycle it happens).

There are a bunch of technicalities that would need adjusting to fit HTTPD, 
such as release intervals, release management (for PHP, every x.y.* series has 
two managers who jointly coordinate releases), etc, but overall the idea is, 
IMO, worth considering.

As a, more or less, "outside observer", I happen to think that the current 
method of voting on finals, instead of a practice of rolling out RCs (that are 
then left up for testing for at least a week), is fundamentally broken. The 2.4 
changelog in particular is littered with releases that were never officially 
published. For users, that's really confusing. For maintainers, it's painful to 
start over the process each time, and it sometimes leads to months and months 
without a release that contains certain fixes. Then a backport goes wrong 
(still using SVN, in my opinion, does not help there, but that's a whole 
different discussion :)), and a regression is in the latest release until 
someone eventually picks up a fix. 

Much of this, and many of the "what do we backport from trunk" and "I'd like to 
squeeze in a change I've had sitting around locally, please wait with the new 
release, because who knows when the next one after that will be" are, from what 
I can tell, a significant source of discord on this mailing list. All these 
unnecessary distractions that deteriorate personal relationships, while at the 
same time slowing down the pace of the project (several people have already 
pointed out Nginx's rate of innovation in comparison) and raising the threshold 
for contributions, can be fixed. PHP is the perfect example, and I think HTTPD 
would be wise to at least consider following this example.

Happy New Year!

David