Re: A proposal...

[Marion et Christophe JAILLET]

Le 23/04/2018 à 23:09, Mark Blackman a écrit :




On 23 Apr 2018, at 19:17, Christophe Jaillet  
wrote:

Le 23/04/2018 à 16:00, Jim Jagielski a écrit :

It seems that, IMO, if there was not so much concern about "regressions" in 
releases, this whole revisit-versioning debate would not have come up. This implies, to 
me at least, that the root cause (as I've said before) appears to be one related to QA 
and testing more than anything. Unless we address this, then nothing else really matters.
We have a test framework. The questions are:
  1. Are we using it?
  2. Are we using it sufficiently well?
  3. If not, what can we do to improve that?
  4. Can we supplement/replace it w/ other frameworks?
It does seem to me that each time we patch something, there should be a test 
added or extended which covers that bug. We have gotten lax in that. Same for 
features. And the more substantial the change (ie, the more core code it 
touches, or the more it refactors something), the more we should envision what 
tests can be in place which ensure nothing breaks.
In other words: nothing backported unless it also involves some changes to the 
Perl test framework or some pretty convincing reasons why it's not required.


Hi,
+1000 on my side for more tests.

But, IMHO, the perl framework is complex to understand for most of us.


Do you believe the Perl element is contributing to the complexity? I’d say Perl 
is perfect for this case in general, although I would have to look at it first 
to confirm.


For my personal case, Yes, I consider that the Perl syntax itself is 
complex and/or tricky. That is certainly because I've never worked that 
much with it.
I think that this can limit the number of people who can increase our 
test coverage.




I certainly believe adequate testing is a bigger and more important problem to 
solve than versioning policies, although some versioning policies might make it 
simpler to allow enough time for decent testing to happen. I personally have a 
stronger incentive to help with testing, than I do with versioning policies.

- Mark

Re: Expanding httpd adoption internationally - POC

[Marion et Christophe JAILLET]

Le 24/04/2018 à 02:58, William A Rowe Jr a écrit :

On Thu, Apr 19, 2018 at 12:20 AM, Marion et Christophe JAILLET
 wrote:

Le 18/04/2018 à 22:12, William A Rowe Jr a écrit :

On Wed, Apr 18, 2018 at 2:31 PM, Nick Kew  wrote:

I suspect the straightforward way to do this, in 2.6/3.0, will be to add
an
i18n table of the error log strings extracted from and indexed by those
APLOGNO() entries. No match? Default English message.

Please, not without an overhaul of APLOGNO to automate it a lot better!
The time to devise your error message is when coding an original
ap_log_*error, not retrospectively once a number has been generated!

++1

This could be achieved easily by just adding in 'log_error_core()':
(+ a few lines to initialize gettext in main() + a few lines to get a pool
in 'log_error_core()' (i.e. p below))

 /* Check if we have a leading AHn: header */
 if (p &&
 fmt[0] == 'A' &&
 fmt[1] == 'H' &&
 apr_isdigit(fmt[2]) &&
 apr_isdigit(fmt[3]) &&
 apr_isdigit(fmt[4]) &&
 apr_isdigit(fmt[5]) &&
 apr_isdigit(fmt[6]) &&
 fmt[7] == ':' &&
 fmt[8] == ' ') {
 const char *tmp;

 /* Get the translation of the message itself */
 tmp = gettext([9]);

 /* Rewrite the format string with the translated message */
 fmt = apr_pstrcat(p, apr_pstrmemdup(p, fmt, 9), tmp, NULL);
 printf("tmp: %s\n", fmt);
 }

That's all.
If interested by this approach, I can give a full working patch.

While we are unconcerned whether build tooling is GPL, we are very concerned
that we retain license and copyright of that output, and that linking system GPL
code does not alter our license.

As a GPL toolchain and call, this seems to require additional research.

If the tooling is clean and the gettext() call is not, then I could
fully support
following GPL gettext conversion file conventions in any custom code we create.

I'm not really a license man, but as far as I understand, the gettext 
library itself is LGPL, not GPL 
(https://www.gnu.org/software/gettext/manual/html_node/Licenses.html). I 
suppose that this is fine with the Apache license (but once again, I'm 
not a license man...)


CJ

Re: Expanding httpd adoption internationally - POC

[William A Rowe Jr]

On Thu, Apr 19, 2018 at 12:20 AM, Marion et Christophe JAILLET
 wrote:
> Le 18/04/2018 à 22:12, William A Rowe Jr a écrit :
>>
>> On Wed, Apr 18, 2018 at 2:31 PM, Nick Kew  wrote:

 I suspect the straightforward way to do this, in 2.6/3.0, will be to add
 an
 i18n table of the error log strings extracted from and indexed by those
 APLOGNO() entries. No match? Default English message.
>>>
>>> Please, not without an overhaul of APLOGNO to automate it a lot better!
>>> The time to devise your error message is when coding an original
>>> ap_log_*error, not retrospectively once a number has been generated!
>>
>> ++1
>
> This could be achieved easily by just adding in 'log_error_core()':
> (+ a few lines to initialize gettext in main() + a few lines to get a pool
> in 'log_error_core()' (i.e. p below))
>
> /* Check if we have a leading AHn: header */
> if (p &&
> fmt[0] == 'A' &&
> fmt[1] == 'H' &&
> apr_isdigit(fmt[2]) &&
> apr_isdigit(fmt[3]) &&
> apr_isdigit(fmt[4]) &&
> apr_isdigit(fmt[5]) &&
> apr_isdigit(fmt[6]) &&
> fmt[7] == ':' &&
> fmt[8] == ' ') {
> const char *tmp;
>
> /* Get the translation of the message itself */
> tmp = gettext([9]);
>
> /* Rewrite the format string with the translated message */
> fmt = apr_pstrcat(p, apr_pstrmemdup(p, fmt, 9), tmp, NULL);
> printf("tmp: %s\n", fmt);
> }
>
> That's all.
> If interested by this approach, I can give a full working patch.

While we are unconcerned whether build tooling is GPL, we are very concerned
that we retain license and copyright of that output, and that linking system GPL
code does not alter our license.

As a GPL toolchain and call, this seems to require additional research.

If the tooling is clean and the gettext() call is not, then I could
fully support
following GPL gettext conversion file conventions in any custom code we create.

Re: A proposal...

[Mark Blackman]

> On 23 Apr 2018, at 19:17, Christophe Jaillet  
> wrote:
> 
> Le 23/04/2018 à 16:00, Jim Jagielski a écrit :
>> It seems that, IMO, if there was not so much concern about "regressions" in 
>> releases, this whole revisit-versioning debate would not have come up. This 
>> implies, to me at least, that the root cause (as I've said before) appears 
>> to be one related to QA and testing more than anything. Unless we address 
>> this, then nothing else really matters.
>> We have a test framework. The questions are:
>>  1. Are we using it?
>>  2. Are we using it sufficiently well?
>>  3. If not, what can we do to improve that?
>>  4. Can we supplement/replace it w/ other frameworks?
>> It does seem to me that each time we patch something, there should be a test 
>> added or extended which covers that bug. We have gotten lax in that. Same 
>> for features. And the more substantial the change (ie, the more core code it 
>> touches, or the more it refactors something), the more we should envision 
>> what tests can be in place which ensure nothing breaks.
>> In other words: nothing backported unless it also involves some changes to 
>> the Perl test framework or some pretty convincing reasons why it's not 
>> required.
> 
> Hi,
> +1000 on my side for more tests.
> 
> But, IMHO, the perl framework is complex to understand for most of us.

Do you believe the Perl element is contributing to the complexity? I’d say Perl 
is perfect for this case in general, although I would have to look at it first 
to confirm.

I certainly believe adequate testing is a bigger and more important problem to 
solve than versioning policies, although some versioning policies might make it 
simpler to allow enough time for decent testing to happen. I personally have a 
stronger incentive to help with testing, than I do with versioning policies.

- Mark

Re: A proposal...

[Alain Toussaint]

> Hi,
> +1000 on my side for more tests.
> 
> But, IMHO, the perl framework is complex to understand for most of us.

>From what I saw, the preferred scripting language having access to httpd's 
>internal seem to be lua
at the present time. I also think that redoing (recoding) the testing framework 
to use Lua and get
to the point where sufficient testing (for regression and anything else) is 
possible is a huge
endeavor. I am also not aware of the history of httpd or its mailing list 
archive but I am willing
to invest some time to make that happen (I also have 5 days of course per week 
and a part-time job 1
day a week plus editorship on BLFS to work on but BLFS and test case of httpd 
will be done jointly).

Still, I will ask you all if reimplementation of the testing framework is 
possible / feasible in
Lua?

my 0.02$

Alain

AW: Revisit Versioning? (Was: 2.4.3x regression w/SSL vhost configs)

[Plüm , Rüdiger , Vodafone Group]

> -Ursprüngliche Nachricht-
> Von: David Zuelke 
> Gesendet: Montag, 23. April 2018 18:09
> An: dev@httpd.apache.org
> Betreff: Re: Revisit Versioning? (Was: 2.4.3x regression w/SSL vhost
> configs)
> 
> On Sat, Apr 21, 2018 at 12:45 PM, Graham Leggett 
> wrote:
> > On 19 Apr 2018, at 5:55 PM, David Zuelke 
> wrote:
> >
> >> I hate to break this to you, and I do not want to discredit the
> >> amazing work all the contributors here are doing, but httpd 2.4 is of
> >> miserable, miserable quality when it comes to breaks and regressions.
> >>
> >> I maintain the PHP/Apache/Nginx infrastructure at Heroku, and I was
> >> able to use the following httpd releases only in the last ~2.5 years:
> >>
> >> - 2.4.16
> >> - 2.4.18
> >> - 2.4.20
> >> - 2.4.29
> >> -2.4.33
> >>
> >> Mostly because of regressions around mod_proxy(_fcgi), REDIRECT_URL,
> whatever.
> >
> > Did you bring these regressions to our attention? Regressions get fix
> very quickly - there was an 18 month period between 2.4.20 and 2.4.29,
> what stopped it being possible to upgrade in that time?
> 
> I double checked. It was actually 2.4.27, not 2.4.29; 15 months, not 18.
> My bad.
> 
> The issue was the PHP-FPM + mod_proxy_fcgi regression introduced in
> 2.4.21; it got reported pretty quickly but took a while to address.
> 
> It finally got fixed in 2.4.26:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60576
> 
> But that fix broke SCRIPT_NAME:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=61202
> 
> So 2.4.27 was functional again.
> 
> That means between April 11, 2016, and July 11, 2017, httpd with
> mod_proxy_fcgi and PHP-FPM was broken.
> 
> The original change was made against trunk
> (https://bz.apache.org/bugzilla/show_bug.cgi?id=59618) and then
> backported to 2.4.next.

Which was an unfortunate regression that took long to be fixed correctly,
but as far as I remember it did not involve any new features nor refactoring.
"Just" a bugfix that caused a regression that was hard to fix.

> 
> >
> > (As other people have said, there was no release between 2.4.16 and
> 2.4.18, 2.4.19 was replaced two weeks later, and there were no releases
> for you to have used between v2.4.29 and 2.4.33)
> >
> >> This is not any person's fault. This is the fault of the process. The
> >> process can be repaired: bugfixes only in 2.4.x, do RC cycles for
> >> bugfix releases as well (that alone makes the changelog look a lot
> >> less confusing, which is useful for the project's image, see also the
> >> Nginx marketing/FUD discussion in the other thread), and start
> testing
> >> new features in modules first.
> >
> > Unfortunately this misses a fundamental reality of what the httpd
> project is - we are the foundation under many many other things, and
> when we jump from v2.4.x to v2.6.x, our complete ecosystem above us
> needs to be recompiled.
> 
> Going from 2.4.x to 2.6.0 doesn't mean that 2.4.x would no longer be
> maintained. There could easily be some predictable, defined support
> policy for keeping older versions alive.

Which requires enough manpower to maintain all these branches. Looking at the 
past
15 years we never maintained more than two stable branches at the same time.
This is no argument against 2.6.0 as we currently only maintain 2.4.x, but 
against
having 2.6+n.0 each time we want to add new features and keep maintaining 
2.6+n-x.0
I think experience shows that if we would release 2.6 the activity to backport 
new
features to 2.4 would drop over time.

> 
> The other thing is ABI versus API stability; you could say 2.x.
> versions retain API compatibility, but not ABI compatibility, so while
> modules would have to be rebuilt against newer version series, that
> could in virtually all cases happen without having to touch the
> module's code.

This might work with open source modules, but even here you would lose the 
chance
e.g. on LTS distributions to compile your own Apache with later features being 
on
the same ABI level as the OS delivered Apache and install a OS delivered package
of a module to use it with this instead of needing to compile it on your own.
It does not work at all with closed source modules, because as soon as these
vendors need to recompile you will either have to wait for a longer period of
time or need to upgrade their product as well to a newer version.

> 
> >> It makes such little sense to land h2 support in 2.4.something, as
> >> opposed to having it as an official "brand new, try it out"
> subproject
> >> first, and then bundle it with 2.6.
> >
> > Not only does it make sense, but it is vital we do so.
> >
> > We needed to get h2 support into the hands of end users - end users
> who were not going to recompile their entire web stack, who install
> software from distros who are not going to upgrade, who were deploying
> modules from vendors that were not going to recompile.
> 
> But that's what I'm saying. Why was h2 not kept as a 

Re: A proposal...

[William A Rowe Jr]

On Mon, Apr 23, 2018 at 1:05 PM, Jim Jagielski  wrote:
>
>> On Apr 23, 2018, at 12:54 PM, William A Rowe Jr  wrote:
>>
>> +1; I see any "patch" releases (semver definition) as adopting well-tested 
>> bug
>> fixes. In some cases, complex patches could arrive first on a new minor 
>> branch
>> for longer alpha/beta scrutiny, before being accepted as-a-patch. This
>> could have
>> helped our php-fpm users with that crazy 2.4.2# cycle of tweak-and-break.
>
> What really helped was having test cases, which are now part
> of the test framework.

More to the point this would have always been iterative. Fix one to break
another. You aren't going to anticipate every side effect writing the
initial test.

It would be great to understand how our PR system failed us in engaging
with PHP users to identify *all* the side effects of 'whatever' change we were
making to the location transcription. And tests were added as things were
broken, more tests added and those broke other things.

To suggest tests would have solved this is silly. The tests were necessary,
and derived from user reports of testing out our code. That it took so many
releases over a year was sort of inexplicable, and if we can sort that out,
we will end up with a better process no matter how we change test rules
or release versioning.

Re: A proposal...

[Paul Querna]

On Mon, Apr 23, 2018 at 11:17 AM, Christophe Jaillet
 wrote:
> Le 23/04/2018 à 16:00, Jim Jagielski a écrit :
>>
>> It seems that, IMO, if there was not so much concern about "regressions"
>> in releases, this whole revisit-versioning debate would not have come up.
>> This implies, to me at least, that the root cause (as I've said before)
>> appears to be one related to QA and testing more than anything. Unless we
>> address this, then nothing else really matters.
>>
>> We have a test framework. The questions are:
>>
>>   1. Are we using it?
>>   2. Are we using it sufficiently well?
>>   3. If not, what can we do to improve that?
>>   4. Can we supplement/replace it w/ other frameworks?
>>
>> It does seem to me that each time we patch something, there should be a
>> test added or extended which covers that bug. We have gotten lax in that.
>> Same for features. And the more substantial the change (ie, the more core
>> code it touches, or the more it refactors something), the more we should
>> envision what tests can be in place which ensure nothing breaks.
>>
>> In other words: nothing backported unless it also involves some changes to
>> the Perl test framework or some pretty convincing reasons why it's not
>> required.
>>
>
> Hi,
> +1000 on my side for more tests.
>
> But, IMHO, the perl framework is complex to understand for most of us.
>
> Last week I tried to play with it. I tried to update proxy_balancer.t
> because only lbmethod=byrequests is tested.
> The current test itself is really simple. It just checks if the module
> didn't crashed (i.e we receive 200).
> I tried to extend it for the other lbmethod available. This looked as an
> easy task. But figuring the relation between:
>
> and
>BalancerMember http://@SERVERNAME@:@PROXY_HTTP_BAL1_PORT@
> still remains a mystery to me.
>
>
> The ./test framework could be useful as well.
> At least it is written in C, so the entry ticket should be cheaper for most
> of us.
> But every thing can't be done with it, I guess.
> Maybe, we should at least have some unit testing for each ap_ function? The
> behavior of these function should not change as it can be used by 3rd party
> modules.

I agree that having a quick way to make function level tests would be
very helpful.  It's something largely missing from httpd. (APR has
more)

Even in making mod_log_json, testing it via the test framework is
complicated, as its not a module that changes the output of an HTTP
Request, vs I could make a few quick C-based tests that make sure
things are being serialized correctly very easily.

> The more tests, the better, but I believe that most regressions come from
> interaction between all what is possible with httpd.
> A test-suite is only a test-suite. Everything can't be tested.
>
>
> IMHO, as a minimum, all CVE should have their dedicated test which
> explicitly fails with version n, and succeeds with version n+1.
> It would help to make sure than known security issues don't come back.
>
>
>
> Another question with the perl framework.
> Is there a way to send "invalid" data/request with it?
> All, I see is some GET(...). I guess that it sends well formed date.
> Checking the behavior when invalid queries are received would be great.
> Some kind of RFC compliancy check.
>
>
> just my 2c,
> CJ

Re: A proposal...

[Christophe Jaillet]

Le 23/04/2018 à 16:00, Jim Jagielski a écrit :

It seems that, IMO, if there was not so much concern about "regressions" in 
releases, this whole revisit-versioning debate would not have come up. This implies, to 
me at least, that the root cause (as I've said before) appears to be one related to QA 
and testing more than anything. Unless we address this, then nothing else really matters.

We have a test framework. The questions are:

  1. Are we using it?
  2. Are we using it sufficiently well?
  3. If not, what can we do to improve that?
  4. Can we supplement/replace it w/ other frameworks?

It does seem to me that each time we patch something, there should be a test 
added or extended which covers that bug. We have gotten lax in that. Same for 
features. And the more substantial the change (ie, the more core code it 
touches, or the more it refactors something), the more we should envision what 
tests can be in place which ensure nothing breaks.

In other words: nothing backported unless it also involves some changes to the 
Perl test framework or some pretty convincing reasons why it's not required.



Hi,
+1000 on my side for more tests.

But, IMHO, the perl framework is complex to understand for most of us.

Last week I tried to play with it. I tried to update proxy_balancer.t 
because only lbmethod=byrequests is tested.
The current test itself is really simple. It just checks if the module 
didn't crashed (i.e we receive 200).
I tried to extend it for the other lbmethod available. This looked as an 
easy task. But figuring the relation between:

   
and
   BalancerMember http://@SERVERNAME@:@PROXY_HTTP_BAL1_PORT@
still remains a mystery to me.


The ./test framework could be useful as well.
At least it is written in C, so the entry ticket should be cheaper for 
most of us.

But every thing can't be done with it, I guess.
Maybe, we should at least have some unit testing for each ap_ function? 
The behavior of these function should not change as it can be used by 
3rd party modules.



The more tests, the better, but I believe that most regressions come 
from interaction between all what is possible with httpd.

A test-suite is only a test-suite. Everything can't be tested.


IMHO, as a minimum, all CVE should have their dedicated test which 
explicitly fails with version n, and succeeds with version n+1.

It would help to make sure than known security issues don't come back.



Another question with the perl framework.
Is there a way to send "invalid" data/request with it?
All, I see is some GET(...). I guess that it sends well formed date. 
Checking the behavior when invalid queries are received would be great.

Some kind of RFC compliancy check.


just my 2c,
CJ

Re: A proposal...

[Jim Jagielski]

> On Apr 23, 2018, at 12:54 PM, William A Rowe Jr  wrote:
> 
> 
> +1; I see any "patch" releases (semver definition) as adopting well-tested bug
> fixes. In some cases, complex patches could arrive first on a new minor branch
> for longer alpha/beta scrutiny, before being accepted as-a-patch. This
> could have
> helped our php-fpm users with that crazy 2.4.2# cycle of tweak-and-break.
> 

What really helped was having test cases, which are now part
of the test framework.

Re: Start using RCs

[Paul Querna]

On Mon, Apr 23, 2018 at 10:10 AM, Micha Lenk  wrote:
> On 04/23/2018 06:33 PM, William A Rowe Jr wrote:
>>
>> On Mon, Apr 23, 2018 at 11:12 AM, Micha Lenk  wrote:
>>>
>>> On Fri, Apr 20, 2018 at 08:54:09AM -0400, Jim Jagielski wrote:

 We have a history, as well as a published "agreement" on what minor
 version numbering means.
>>>
>>>
>>> Just to make sure I am on the same page, would you mind to make that
>>> explicit? Where is that published?
>>
>>
>> http://httpd.apache.org/dev/release.html
>
>
> The only paragraph on that page that contains the word "minor" is the second
> one in the Introduction section. It isn't very verbose either. Am I missing
> something obvious?
>
> Regards,
> Micha

https://svn.apache.org/repos/asf/httpd/httpd/trunk/VERSIONING

Contains a much more verbose... non-semver versioning scheme.

Re: Start using RCs

[Micha Lenk]

On 04/23/2018 06:33 PM, William A Rowe Jr wrote:

On Mon, Apr 23, 2018 at 11:12 AM, Micha Lenk  wrote:

On Fri, Apr 20, 2018 at 08:54:09AM -0400, Jim Jagielski wrote:

We have a history, as well as a published "agreement" on what minor
version numbering means.


Just to make sure I am on the same page, would you mind to make that
explicit? Where is that published?


http://httpd.apache.org/dev/release.html


The only paragraph on that page that contains the word "minor" is the 
second one in the Introduction section. It isn't very verbose either. Am 
I missing something obvious?


Regards,
Micha

Re: A proposal...

[William A Rowe Jr]

On Mon, Apr 23, 2018 at 9:47 AM, Rainer Jung  wrote:
> Am 23.04.2018 um 16:00 schrieb Jim Jagielski:
>>
>> It seems that, IMO, if there was not so much concern about "regressions"
>> in releases, this whole revisit-versioning debate would not have come up.

Additional concerns that amplify the regressions; last minute code dumps
with minimal review upon each point release. A three day review window
for the success of the combined result. Insufficient community review of
new features (w/wo new directives) with no alpha or beta releases in over
half a decade (h2/md excepted.)

>> It does seem to me that each time we patch something, there should be a
>> test added or extended which covers that bug. We have gotten lax in that.
>> Same for features. And the more substantial the change (ie, the more core
>> code it touches, or the more it refactors something), the more we should
>> envision what tests can be in place which ensure nothing breaks.

+1!

>> In other words: nothing backported unless it also involves some changes to
>> the Perl test framework or some pretty convincing reasons why it's not
>> required.

Or horse-before-the-cart, put in the test for a spec violation/problem behavior
in the code, and add it to TODO. The suite doesn't fail, but serves as a flag
for a defect to be corrected.

Even better (and we have been good about this)... make corresponding docs
changes a prereq, in addition to test.

> I agree with the importance of the test framework, but would also like to
> mention that getting additional test feedback from the community seems also
> important. That's why IMHO the RC style of releasing could be helpful by
> attracting more test effort before a release.
>
> And for the more complex modules like mod_proxy, mod_ssl and the event MPM,
> some of the hickups might have been hard to detect with the test framework.
> That's why I think having a more stable branch 2.4 with less feature
> backports and another branch that evolves faster would give downstreams a
> choice.

+1; I see any "patch" releases (semver definition) as adopting well-tested bug
fixes. In some cases, complex patches could arrive first on a new minor branch
for longer alpha/beta scrutiny, before being accepted as-a-patch. This
could have
helped our php-fpm users with that crazy 2.4.2# cycle of tweak-and-break.

I'd hope we would reintroduce alpha/beta review of new features coinciding
with release m.n.0 with a much longer tail for feature review. Maybe it requires
two or three patch releases before GA, maybe it is accepted as GA on the
very first candidate.

A patch release can be reviewed in a week, but needs to be reviewed in days
to move a security defect fix into users' hands after it is revealed
to our svn/git.
On very rare occasions (once a decade or so), we accelerate this to 24 hours.

A feature release/significant behavior change needs a community, and this is
not a review that happens in a week. I'd expect better adoption of new features
by drawing in our users@ and extended communities to help review additions.

Re: A proposal...

[Micha Lenk]

Just a side node, some days ago I just realized that the source package 
of the apache2 package in Debian seems to include the test suite for the 
purpose of running it as part of the continuous integration test 
'run-test-suite': https://ci.debian.net/packages/a/apache2/


In my recently provided bugfix (#62186) I included a change of the test 
suite, but so far it looks like it isn't integrated yet (do I really 
need to file a separate bugzilla in the other project for that?).


From the experience with doing so, I agree with others that in the long 
run maintaining some Perl-based test framework will probably make 
contributions pretty unpopular, especially for contributors that didn't 
work with Perl before.


For the addition of new regression tests (as others suggested) it would 
be pretty cool if they can be added in a somewhat more popular 
(scripting) language (Python and pytest were already mentioned). Yet the 
number of test frameworks to execute should stay at a manageable low number.


That being said, I am all for extending the use of any test framework.

Regards,
Micha

Re: Start using RCs

[William A Rowe Jr]

On Mon, Apr 23, 2018 at 11:12 AM, Micha Lenk  wrote:
> Hi Jim,
>
> On Fri, Apr 20, 2018 at 08:54:09AM -0400, Jim Jagielski wrote:
>> We have a history, as well as a published "agreement" on what minor
>> version numbering means.
>
> Just to make sure I am on the same page, would you mind to make that
> explicit? Where is that published?

http://httpd.apache.org/dev/release.html

Re: Start using RCs (Was: Re: So... when should we do 2.4.34? [WAS: Re: Revisit Versioning? (Was: 2.4.3x regression w/SSL vhost configs)])

[William A Rowe Jr]

On Sun, Apr 22, 2018 at 11:34 AM, Daniel Ruggeri  wrote:
>
> The more I think about it, the more I *really* like a semver-ish
> approach where major represents the ABI that will not be broken, minor
> represents the feature set (for backwards compatibility) and patch
> represents forward-compatible changes/fixes. This may be, in spirit,
> what you are suggesting. I'd further add that no directives are added in
> patch releases, but minor releases would be fair game. How we choose to
> maintain that behind the scenes is a thought exercise. Given that most
> of our users get httpd from distros, I'd be in favor of having only two
> long-lived branches: trunk and release. The distros are already taking
> the point-release they want and curating fixes/changes for it, so they
> will be doing what they do regardless of our direction.

Thought exercise... if it is decided that version 3.5.x added an all around
nice set of features, and we had platform maintainers as committers who
wanted to keep backporting to that tree at this project, well... they could
elect to maintain that version as a long lived branch and collaborate on
it while we went on our merry way through 3.15.0, even 4.1.x as a group.

In other words, wherever we have three committers who will cultivate a
long lived release, they can do that. This doesn't differ from how a few
individuals kept 2.0, and later, 2.2 alive for a significant period of time
after the next major release. When interest cooled, each was retired.

Re: Start using RCs

[Micha Lenk]

Hi Jim,

On Fri, Apr 20, 2018 at 08:54:09AM -0400, Jim Jagielski wrote:
> We have a history, as well as a published "agreement" on what minor
> version numbering means. 

Just to make sure I am on the same page, would you mind to make that
explicit? Where is that published?

> Our module eco-system is huge and we need to factor/consider not only
> the big players, but also the solitary developers who have modules. It
> is a long, long history and if/when we need to reconsider versioning,
> the impact will not be insignificant.
>
> [...]
> Again, it is all in balance, and likely we've just not achieved that
> lately due to the extreme complexities of the eco-system around,
> internal, external and dependent upon us.

But isn't the very purpose of the major.minor construct (that renders
all modules incompatible on a minor version bump) to get a handle on the
"extreme complexity" of the eco-system? If so, is the major.minor
construct used correctly to get the most out of it?

Regards,
Micha

Re: Revisit Versioning? (Was: 2.4.3x regression w/SSL vhost configs)

[David Zuelke]

On Sat, Apr 21, 2018 at 12:45 PM, Graham Leggett  wrote:
> On 19 Apr 2018, at 5:55 PM, David Zuelke  wrote:
>
>> I hate to break this to you, and I do not want to discredit the
>> amazing work all the contributors here are doing, but httpd 2.4 is of
>> miserable, miserable quality when it comes to breaks and regressions.
>>
>> I maintain the PHP/Apache/Nginx infrastructure at Heroku, and I was
>> able to use the following httpd releases only in the last ~2.5 years:
>>
>> - 2.4.16
>> - 2.4.18
>> - 2.4.20
>> - 2.4.29
>> -2.4.33
>>
>> Mostly because of regressions around mod_proxy(_fcgi), REDIRECT_URL, 
>> whatever.
>
> Did you bring these regressions to our attention? Regressions get fix very 
> quickly - there was an 18 month period between 2.4.20 and 2.4.29, what 
> stopped it being possible to upgrade in that time?

I double checked. It was actually 2.4.27, not 2.4.29; 15 months, not 18. My bad.

The issue was the PHP-FPM + mod_proxy_fcgi regression introduced in
2.4.21; it got reported pretty quickly but took a while to address.

It finally got fixed in 2.4.26:
https://bz.apache.org/bugzilla/show_bug.cgi?id=60576

But that fix broke SCRIPT_NAME:
https://bz.apache.org/bugzilla/show_bug.cgi?id=61202

So 2.4.27 was functional again.

That means between April 11, 2016, and July 11, 2017, httpd with
mod_proxy_fcgi and PHP-FPM was broken.

The original change was made against trunk
(https://bz.apache.org/bugzilla/show_bug.cgi?id=59618) and then
backported to 2.4.next.

>
> (As other people have said, there was no release between 2.4.16 and 2.4.18, 
> 2.4.19 was replaced two weeks later, and there were no releases for you to 
> have used between v2.4.29 and 2.4.33)
>
>> This is not any person's fault. This is the fault of the process. The
>> process can be repaired: bugfixes only in 2.4.x, do RC cycles for
>> bugfix releases as well (that alone makes the changelog look a lot
>> less confusing, which is useful for the project's image, see also the
>> Nginx marketing/FUD discussion in the other thread), and start testing
>> new features in modules first.
>
> Unfortunately this misses a fundamental reality of what the httpd project is 
> - we are the foundation under many many other things, and when we jump from 
> v2.4.x to v2.6.x, our complete ecosystem above us needs to be recompiled.

Going from 2.4.x to 2.6.0 doesn't mean that 2.4.x would no longer be
maintained. There could easily be some predictable, defined support
policy for keeping older versions alive.

The other thing is ABI versus API stability; you could say 2.x.
versions retain API compatibility, but not ABI compatibility, so while
modules would have to be rebuilt against newer version series, that
could in virtually all cases happen without having to touch the
module's code.

>> It makes such little sense to land h2 support in 2.4.something, as
>> opposed to having it as an official "brand new, try it out" subproject
>> first, and then bundle it with 2.6.
>
> Not only does it make sense, but it is vital we do so.
>
> We needed to get h2 support into the hands of end users - end users who were 
> not going to recompile their entire web stack, who install software from 
> distros who are not going to upgrade, who were deploying modules from vendors 
> that were not going to recompile.

But that's what I'm saying. Why was h2 not kept as a module (for the
few people that are already deploying HTTP/2 stacks), let it mature
this way, and then put it into everyone's hands as part of 2.6.0,
which could be the big shiny new feature, to give everyone an
incentive to move to that new major version?

> Our average user will deploy whatever comes by default on their operating 
> system, they’re not going to have a dedicated team that deploys a custom 
> stack for their application. It is vital we respect the needs of these groups 
> of users.

That is even more of an argument to move to a more predictable cycle
and have patch releases only fix issues, because it means new features
see the light of day more quickly, so more people who just use what
comes with their OS would benefit from them.

Nobody who uses Apache as part of Debian, Ubuntu, RHEL, whatever, gets
new 2.4.next features. Those distros freeze Apache at whatever is the
latest version when their cutoff date is due, and then only backport
security fixes.

>> Really, I'd suggest taking a close look at the PHP release cycle, with
>> their schedules, their RFC policies, everything. As I said in that
>> other thread, the PHP project was in exactly the same spot a few years
>> ago and they have pulled themselves out of that mess with amazing
>> results.
>
> Specifically what about the php release cycle are you referring to? I was 
> burned badly a number of years ago by php config file formats being changed 
> in point releases, have they improved their stability?

Yes. Nothing like that has happened since PHP 5.4, when the new
release process started and all of 

new module: mod_log_json

[Paul Querna]

Morning dev@,

I just committed mod_log_json to trunk in r1829898.

Right now, to use it you need something like this:

LogFormat "%^JS" json
CustomLog "logs/access_log.json" json

Currently it has a static format of the JSON, and example message is like this:

{
  "log_id": null,
  "vhost": "127.0.0.1",
  "status": "404",
  "proto": "HTTP\/1.1",
  "method": "GET",
  "uri": "\/x",
  "srcip": "127.0.0.1",
  "bytes_sent": 199,
  "hdrs": {
"user-agent": "curl\/7.54.0"
  }
}


Some changes I'd like to figure out:

- Right now mod_log_json is a format string to mod_log_config -- this
is mostly to be able to leverage mod_log_config's support to write to
many outputs (files, buffered logs, pipes, programs, etc).

- Leveraging the existing mod_log_config format strings.  Exposing the
internal hash of callbacks isn't that hard, I experimented with adding
a hook of that here
,
but a bigger refactoring is needed to expose mod_log_config's parsing
of the format strings, rather than just their key-names.

- mod_log_json is not configurable right now, the format is static.
Obviously, being able to configure what is logged, what headers, etc
is valuable. I played with some options here, but wasn't happy with
it.  Any ideas?

- (small) I'd like to be able to log in RFC-3339 timestamps, in
JSON-world this seems like the most common format by far.  Just need
to do the work to export an apr_time_t that way, I don't think there
is existing code for that?

Cheers,

Paul

Re: Revisit Versioning? (Was: 2.4.3x regression w/SSL vhost configs)

[Micha Lenk]

On Fri, Apr 20, 2018 at 08:14:16AM -0400, Jim Jagielski wrote:
> On Apr 20, 2018, at 8:04 AM, Micha Lenk  wrote:
> > [...], I value the ability to distinguish between bugfix-only
> > releases and feature addition releases.
> 
> I understand that, thx. I also understand how a minor bump makes that
> easier. It would make, however, other people's lives and jobs *more*
> difficult, I think, so it's all about balance. I can see how
> distinguishing the difference is also nice, but what "value" do you
> derive from it? I am genuinely curious. Thx!

To be honest, our commercial interest in bugfixes is simply higher than
getting new features. So, I expect integrating a bugfix-only release to
be much less effort (in terms of porting our own modules, patches,
additional testing scrutiny) than a release that re-works internal core
functionality like the request handling for the sake of adding a new
feature like the entirely new support for h2.

But I am equally genuinely curious what would make "other people's lives
and jobs *more* difficult". What exactly do you refer to here?

> This is a "hack", of course, but what if CHANGES specifically called
> out new features like we do w/ SECURITY items?

Not being a native speaker I am not sure I understand your question
correctly. Can you please elaborate that question a bit?

Regards,
Micha

Re: A proposal...

[Stefan Eissing]

> Am 23.04.2018 um 17:07 schrieb Stefan Eissing :
> 
> I do that for stuff I wrote myself. Not because I care only about that, but 
> because the coverage and documentation of other server parts does give me an i
> dea of what should work and what should not. So, I am the 

*the coverage and documentation of other server parts does *NOT* give me

Re: A proposal...

[Stefan Eissing]

> Am 23.04.2018 um 16:00 schrieb Jim Jagielski :
> 
> It seems that, IMO, if there was not so much concern about "regressions" in 
> releases, this whole revisit-versioning debate would not have come up. This 
> implies, to me at least, that the root cause (as I've said before) appears to 
> be one related to QA and testing more than anything. Unless we address this, 
> then nothing else really matters.
> 
> We have a test framework. The questions are:

Personal view/usage answers:

> 1. Are we using it?

On release candidates only.

> 2. Are we using it sufficiently well?

 * I only added very basic tests for h2, since Perl's capabilities here are 
rather limited.
 * the whole framework was hard to figure out. It took me a while to get vhost 
setups working.

> 3. If not, what can we do to improve that?

 * A CI setup would help.

> 4. Can we supplement/replace it w/ other frameworks?

 * For mod_h2 I started with just shell scripts. Those still make my h2 test 
suite,
   using nghttp and curl client as well as go (if available).
 * For mod_md I used pytest which I found an excellent framework. The test suite
   is available in the github repository of mod_md
 * Based on Robert Swiecki's hongfuzz, there is a h2fuzz project for fuzzing
   our server at https://github.com/icing/h2fuzz. This works very well on a 
Linux
   style system.

So, I do run a collection of things. All are documented, but none is really 
tied into
the httpd testing framework.

> It does seem to me that each time we patch something, there should be a test 
> added or extended which covers that bug. We have gotten lax in that. Same for 
> features. And the more substantial the change (ie, the more core code it 
> touches, or the more it refactors something), the more we should envision 
> what tests can be in place which ensure nothing breaks.

I do that for stuff I wrote myself. Not because I care only about that, but 
because the coverage and documentation of other server parts does give me an 
idea of what should work and what should not. So, I am the wrong guy to place 
assertions into test cases for those code parts.

Example: the current mod_ssl enabled quirkyness discovered by Joe would ideally 
be documented now in a new test case. But neither me nor Yann would have found 
that before release via testing (the tests worked) nor did we anticipate such 
breakage.

Such undocumented and untested behaviour, which nevertheless is considered a 
regression, cannot be avoided, since it cannot be anticipated by people 
currently working on those code parts. This is a legacy of the past, it seems, 
which we can only overcome by breakage and resulting, added test cases.

> In other words: nothing backported unless it also involves some changes to 
> the Perl test framework or some pretty convincing reasons why it's not 
> required.

See above, this will not fix the unforeseeable breakage that results from use 
cases unknown and untested.

-Stefan

Re: A proposal...

[Jim Jagielski]

> On Apr 23, 2018, at 10:22 AM, Graham Leggett  wrote:
> 
> My perl knowledge is very rusty, making perl tests is going to be harder for 
> some than others.
> 

Yeah, that IS an issue. It is also not as well documented as desired[1].

Should we look at using something external as a well to complement/supplement 
it? Or even start adding some specific tests under the ./test subdirectory in 
the repo. Maybe say that the requirement is some sort of test "bundled" w/ the 
feature; it doesn't need to be under the perl test framework. Or maybe some way 
the perl test framework can call other test scripts written in whatever 
language someone wants; it simply sets things up, lets the script run and 
checks the return status.


1. https://perl.apache.org/docs/general/testing/testing.html

Re: A proposal...

[Daniel Ruggeri]

On 2018-04-23 09:00, Jim Jagielski wrote:

It seems that, IMO, if there was not so much concern about
"regressions" in releases, this whole revisit-versioning debate would
not have come up. This implies, to me at least, that the root cause
(as I've said before) appears to be one related to QA and testing more
than anything. Unless we address this, then nothing else really
matters.

We have a test framework. The questions are:

 1. Are we using it?
 2. Are we using it sufficiently well?
 3. If not, what can we do to improve that?
 4. Can we supplement/replace it w/ other frameworks?


My opinion (I think mentioned here on-list before, too) is that the 
framework is too... mystical. A lot of us do not understand how it works 
and it's a significant cognitive exercise to get started. Getting it 
installed and up and running is also non-trivial.


I am willing to invest time working with anyone who would like to 
generate more documentation to demystify the framework. Pair 
programming, maybe, to go with this newfangled test driven design 
thought??? :-). I do not understand the ins and outs of the framework 
very well, but am willing to learn more to ferret out the things that 
should be better documented. Things like, "How do I add a vhost for a 
specific test?", "Are there any convenient test wrappers for HTTP(s) 
requests?", "How do I write a test case from scratch?" would be a great 
first start.



Also, FWIW, at $dayjob we use serverspec (https://serverspec.org/) as a 
testing framework for infrastructure like httpd. After some initial 
thrashing and avoidance, I've come to like it quite well. If we prefer 
to keep with a scripting language for tests (I do), Ruby is a decent 
choice since it has all the niceties that we'd expect (HTTP(s), 
XML/JSON/YML, threading, native testing framework, crypto) built in. I'm 
happy to provide an example or two if anyone is interested in exploring 
the topic in more depth.





It does seem to me that each time we patch something, there should be
a test added or extended which covers that bug. We have gotten lax in
that. Same for features. And the more substantial the change (ie, the
more core code it touches, or the more it refactors something), the
more we should envision what tests can be in place which ensure
nothing breaks.

In other words: nothing backported unless it also involves some
changes to the Perl test framework or some pretty convincing reasons
why it's not required.


I completely support creating this as a procedure, provided we tackle 
the "how do I test stuff" doco challenges, too.


--
Daniel Ruggeri

Re: A proposal...

[Rainer Jung]

Am 23.04.2018 um 16:00 schrieb Jim Jagielski:

It seems that, IMO, if there was not so much concern about "regressions" in 
releases, this whole revisit-versioning debate would not have come up. This implies, to 
me at least, that the root cause (as I've said before) appears to be one related to QA 
and testing more than anything. Unless we address this, then nothing else really matters.

We have a test framework. The questions are:

  1. Are we using it?
  2. Are we using it sufficiently well?
  3. If not, what can we do to improve that?
  4. Can we supplement/replace it w/ other frameworks?

It does seem to me that each time we patch something, there should be a test 
added or extended which covers that bug. We have gotten lax in that. Same for 
features. And the more substantial the change (ie, the more core code it 
touches, or the more it refactors something), the more we should envision what 
tests can be in place which ensure nothing breaks.

In other words: nothing backported unless it also involves some changes to the 
Perl test framework or some pretty convincing reasons why it's not required.


I agree with the importance of the test framework, but would also like 
to mention that getting additional test feedback from the community 
seems also important. That's why IMHO the RC style of releasing could be 
helpful by attracting more test effort before a release.


And for the more complex modules like mod_proxy, mod_ssl and the event 
MPM, some of the hickups might have been hard to detect with the test 
framework. That's why I think having a more stable branch 2.4 with less 
feature backports and another branch that evolves faster would give 
downstreams a choice.


Regards,

Rainer

Re: A proposal...

[Graham Leggett]

On 23 Apr 2018, at 4:00 PM, Jim Jagielski  wrote:

> It seems that, IMO, if there was not so much concern about "regressions" in 
> releases, this whole revisit-versioning debate would not have come up. This 
> implies, to me at least, that the root cause (as I've said before) appears to 
> be one related to QA and testing more than anything. Unless we address this, 
> then nothing else really matters.

+1.

> We have a test framework. The questions are:
> 
> 1. Are we using it?

Is there a CI set up for building httpd?

Is there a CI available we could use to trigger the test suite on a regular 
basis?

(I believe the answer is yes for APR).

> 2. Are we using it sufficiently well?
> 3. If not, what can we do to improve that?
> 4. Can we supplement/replace it w/ other frameworks?
> 
> It does seem to me that each time we patch something, there should be a test 
> added or extended which covers that bug. We have gotten lax in that. Same for 
> features. And the more substantial the change (ie, the more core code it 
> touches, or the more it refactors something), the more we should envision 
> what tests can be in place which ensure nothing breaks.
> 
> In other words: nothing backported unless it also involves some changes to 
> the Perl test framework or some pretty convincing reasons why it's not 
> required.

My perl knowledge is very rusty, making perl tests is going to be harder for 
some than others.

Regards,
Graham
—



smime.p7s
Description: S/MIME cryptographic signature

A proposal...

[Jim Jagielski]

It seems that, IMO, if there was not so much concern about "regressions" in 
releases, this whole revisit-versioning debate would not have come up. This 
implies, to me at least, that the root cause (as I've said before) appears to 
be one related to QA and testing more than anything. Unless we address this, 
then nothing else really matters.

We have a test framework. The questions are:

 1. Are we using it?
 2. Are we using it sufficiently well?
 3. If not, what can we do to improve that?
 4. Can we supplement/replace it w/ other frameworks?

It does seem to me that each time we patch something, there should be a test 
added or extended which covers that bug. We have gotten lax in that. Same for 
features. And the more substantial the change (ie, the more core code it 
touches, or the more it refactors something), the more we should envision what 
tests can be in place which ensure nothing breaks.

In other words: nothing backported unless it also involves some changes to the 
Perl test framework or some pretty convincing reasons why it's not required.