Re: arm64 support for Travis CI testing
On Thu, Jan 09, 2020 at 03:01:42PM +0100, Luca Toscano wrote: > Il giorno gio 9 gen 2020 alle ore 14:19 Joe Orton > ha scritto: > > > > The caching does work on arm64 so the build & test only takes 11 minutes > > which is reasonable. > > > > https://travis-ci.org/apache/httpd/jobs/634661216 > > > > Looks like this one is the next unreliable test to attack - > > > > 2425# Failed test 17 in t/apache/expr_string.t at line 87 > > 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2 > > 2427t/apache/expr_string.t .. > > 2428Failed 2/44 subtests > > 2429 > > > > Can anybody reliably reproduce that failure? > > Does it happen only for ARM or also for other architectures? Never > seen this before.. Definitely all architectures, it's one Rainer reports as failing regularly in release testing. Here's another one and I'm pretty sure it has shown up in more of the recent failures too - https://travis-ci.org/apache/httpd/jobs/633442262
Re: arm64 support for Travis CI testing
Il giorno gio 9 gen 2020 alle ore 14:19 Joe Orton ha scritto: > > The caching does work on arm64 so the build & test only takes 11 minutes > which is reasonable. > > https://travis-ci.org/apache/httpd/jobs/634661216 > > Looks like this one is the next unreliable test to attack - > > 2425# Failed test 17 in t/apache/expr_string.t at line 87 > 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2 > 2427t/apache/expr_string.t .. > 2428Failed 2/44 subtests > 2429 > > Can anybody reliably reproduce that failure? Does it happen only for ARM or also for other architectures? Never seen this before..
Re: arm64 support for Travis CI testing
The caching does work on arm64 so the build & test only takes 11 minutes which is reasonable. https://travis-ci.org/apache/httpd/jobs/634661216 Looks like this one is the next unreliable test to attack - 2425# Failed test 17 in t/apache/expr_string.t at line 87 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2 2427t/apache/expr_string.t .. 2428Failed 2/44 subtests 2429 Can anybody reliably reproduce that failure?
Re: arm64 support for Travis CI testing
On Thu, Jan 09, 2020 at 09:10:31AM +, Pluem, Ruediger, Vodafone Group wrote: > Would > > lscpu -p | grep -c -E ^[0-9] > > work on Ubuntu to deliver the number of cores? > > So > > make -j `lscpu -p | grep -c -E ^[0-9]` I've got no idea if this DTRT for the mix of VMs & containers that travis uses, but it's probably worth trying! Regards, Joe
Re: worker MPM test failures (was Re: Still Failing: apache/httpd#190 (trunk - 894b6a1))
> Am 09.01.2020 um 12:39 schrieb Pluem, Ruediger, Vodafone Group > : > > > >> -Ursprüngliche Nachricht- >> Von: Stefan Eissing >> Gesendet: Donnerstag, 9. Januar 2020 11:51 >> An: dev@httpd.apache.org >> Betreff: Re: worker MPM test failures (was Re: Still Failing: >> apache/httpd#190 (trunk - 894b6a1)) >> >> >> >>> Am 09.01.2020 um 11:32 schrieb Pluem, Ruediger, Vodafone Group >> : >>> >>> >>> BTW: Can we really have morphing buckets in ap_request_core_filter? >> ap_request_core_filter runs >>> after a possible HTTP chunking filter that I guess needs to be in >> place when we have a morphing >>> bucket which makes it impossible to determine the content length >> upfront. Hence I guess the >>> HTTP chunking filter will transform all these morphing buckets >> already. >>> Or is this just a safety measure to support other protocols but HTTP? >> >> What exactly do you mean by a "morphing" bucket? If that include file >> buckets, then I guess "yes" is the answer. >> > > A file bucket is not a the morphing bucket I am talking here about > although it morphs as well when read, because it has a known length. > A morphing bucket has a length of -1 and when it is read it "morphs" > into a bucket of known length (mostly in memory) with a subset of the > content of the morphing bucket and the morphing bucket is added back > in the brigade after the morphed bucket sans the content left in the morphed > bucket. > Examples are > > CGI buckets > PIPE buckets > Socket buckets Thanks for the clarification. I agree that all those should have been resolved to a known length in the core filters before a setaside. > > Regards > > Rüdiger
AW: worker MPM test failures (was Re: Still Failing: apache/httpd#190 (trunk - 894b6a1))
> -Ursprüngliche Nachricht- > Von: Stefan Eissing > Gesendet: Donnerstag, 9. Januar 2020 11:51 > An: dev@httpd.apache.org > Betreff: Re: worker MPM test failures (was Re: Still Failing: > apache/httpd#190 (trunk - 894b6a1)) > > > > > Am 09.01.2020 um 11:32 schrieb Pluem, Ruediger, Vodafone Group > : > > > > > > BTW: Can we really have morphing buckets in ap_request_core_filter? > ap_request_core_filter runs > > after a possible HTTP chunking filter that I guess needs to be in > place when we have a morphing > > bucket which makes it impossible to determine the content length > upfront. Hence I guess the > > HTTP chunking filter will transform all these morphing buckets > already. > > Or is this just a safety measure to support other protocols but HTTP? > > What exactly do you mean by a "morphing" bucket? If that include file > buckets, then I guess "yes" is the answer. > A file bucket is not a the morphing bucket I am talking here about although it morphs as well when read, because it has a known length. A morphing bucket has a length of -1 and when it is read it "morphs" into a bucket of known length (mostly in memory) with a subset of the content of the morphing bucket and the morphing bucket is added back in the brigade after the morphed bucket sans the content left in the morphed bucket. Examples are CGI buckets PIPE buckets Socket buckets Regards Rüdiger
Re: worker MPM test failures (was Re: Still Failing: apache/httpd#190 (trunk - 894b6a1))
> Am 09.01.2020 um 11:32 schrieb Pluem, Ruediger, Vodafone Group > : > > > > > C2 General > >> -Ursprüngliche Nachricht- >> Von: Yann Ylavic >> Gesendet: Mittwoch, 8. Januar 2020 16:48 >> An: httpd-dev >> Betreff: Re: worker MPM test failures (was Re: Still Failing: >> apache/httpd#190 (trunk - 894b6a1)) >> >> On Wed, Jan 8, 2020 at 8:36 AM Ruediger Pluem wrote: >>> >>> Just for my further analysis let me try to get the purpose of the >>> ap_request_core_filter: >>> >>> It should leverage at least parts of the benefits of the filter >> setaside / yielding / brigade reinstate framework for >>> filters that do not use this, because e.g. >>> >>> - they are not aware of this framework >> >> Yes, the goal (I think, Graham could confirm) is to let request >> filters work as before but have a catch all (last) filter to not let >> morphing buckets go through connection filters (and consume c->pool). >> Same for setasides. >> >>> - it seems to complex for them to use it and adjust their filter >>> >>> Of course this is not perfect as some resource and content filters >> need to handle this on their own e.g. deflate when >>> compressing contents of file buckets in order to avoid blocking >> writes. >> >> Correct. >> >>> >>> Thinking out loudly a bit here: Except for ap_request_core_filter all >> other resource and content filters should process >>> all the data they might have setaside when they see an EOS in a >> brigade, correct? >> >> I think so, because otherwise they will never be called again since >> the core/mpm only cares about connection filters, so the data must be >> there after EOS. In any case, request filters shall not be called when >> EOR is in the place. >> >> >> Regarding my patch, I'm afraid it's not the right fix, we can't front >> push several req_core output filters like this. >> If an old/outer req_core filter has to setaside buckets (network >> EAGAIN) while the passed brigade contains buckets from the new >> request, then it will setaside the buckets from a newer request pool >> to an oldest's. >> Even though it can (possibly) work, it does not look quite optimal, >> we'd need some kind of Start Of Request (SOR) bucket to delimit >> buckets from/for successive requests. SOR would be passed through >> c->output_filters when a new request is created (e.g. in >> http_create_request()), since anything sent through the request >> filters from that time should be about this last request. >> Then a single req_core connection filter could be responsible for >> setting aside buckets on the relevant request pool (should some >> congestion occur). >> >> Dunno if we want to go that route, though... > > As this looks like to be a larger amount of work I come back to my simple > patch idea > that would possibly cause more blocking writes, but would ensure that the > data is sent > completely at least. Basically it pushes all down the chain once it notices > an EOR bucket. > See > > https://github.com/apache/httpd/pull/88 > > BTW: Can we really have morphing buckets in ap_request_core_filter? > ap_request_core_filter runs > after a possible HTTP chunking filter that I guess needs to be in place when > we have a morphing > bucket which makes it impossible to determine the content length upfront. > Hence I guess the > HTTP chunking filter will transform all these morphing buckets already. > Or is this just a safety measure to support other protocols but HTTP? What exactly do you mean by a "morphing" bucket? If that include file buckets, then I guess "yes" is the answer.
AW: worker MPM test failures (was Re: Still Failing: apache/httpd#190 (trunk - 894b6a1))
C2 General > -Ursprüngliche Nachricht- > Von: Yann Ylavic > Gesendet: Mittwoch, 8. Januar 2020 16:48 > An: httpd-dev > Betreff: Re: worker MPM test failures (was Re: Still Failing: > apache/httpd#190 (trunk - 894b6a1)) > > On Wed, Jan 8, 2020 at 8:36 AM Ruediger Pluem wrote: > > > > Just for my further analysis let me try to get the purpose of the > > ap_request_core_filter: > > > > It should leverage at least parts of the benefits of the filter > setaside / yielding / brigade reinstate framework for > > filters that do not use this, because e.g. > > > > - they are not aware of this framework > > Yes, the goal (I think, Graham could confirm) is to let request > filters work as before but have a catch all (last) filter to not let > morphing buckets go through connection filters (and consume c->pool). > Same for setasides. > > > - it seems to complex for them to use it and adjust their filter > > > > Of course this is not perfect as some resource and content filters > need to handle this on their own e.g. deflate when > > compressing contents of file buckets in order to avoid blocking > writes. > > Correct. > > > > > Thinking out loudly a bit here: Except for ap_request_core_filter all > other resource and content filters should process > > all the data they might have setaside when they see an EOS in a > brigade, correct? > > I think so, because otherwise they will never be called again since > the core/mpm only cares about connection filters, so the data must be > there after EOS. In any case, request filters shall not be called when > EOR is in the place. > > > Regarding my patch, I'm afraid it's not the right fix, we can't front > push several req_core output filters like this. > If an old/outer req_core filter has to setaside buckets (network > EAGAIN) while the passed brigade contains buckets from the new > request, then it will setaside the buckets from a newer request pool > to an oldest's. > Even though it can (possibly) work, it does not look quite optimal, > we'd need some kind of Start Of Request (SOR) bucket to delimit > buckets from/for successive requests. SOR would be passed through > c->output_filters when a new request is created (e.g. in > http_create_request()), since anything sent through the request > filters from that time should be about this last request. > Then a single req_core connection filter could be responsible for > setting aside buckets on the relevant request pool (should some > congestion occur). > > Dunno if we want to go that route, though... As this looks like to be a larger amount of work I come back to my simple patch idea that would possibly cause more blocking writes, but would ensure that the data is sent completely at least. Basically it pushes all down the chain once it notices an EOR bucket. See https://github.com/apache/httpd/pull/88 BTW: Can we really have morphing buckets in ap_request_core_filter? ap_request_core_filter runs after a possible HTTP chunking filter that I guess needs to be in place when we have a morphing bucket which makes it impossible to determine the content length upfront. Hence I guess the HTTP chunking filter will transform all these morphing buckets already. Or is this just a safety measure to support other protocols but HTTP? Regards Rüdiger
AW: arm64 support for Travis CI testing
C2 General > -Ursprüngliche Nachricht- > Von: Joe Orton > Gesendet: Donnerstag, 9. Januar 2020 09:15 > An: dev@httpd.apache.org > Betreff: Re: arm64 support for Travis CI testing > > On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote: > > Okay the first Travis run with arm64 (#201) passed. > > The arm64 job (#201.4) took 33 minutes, 5 seconds. > > Compared tp ppc64le at 4 minutes, 30 seconds. > > This is possibly due to need for extra caching on the first run. > > Otherwise, the jobs seem to have the same results. > > Nice, thanks Mike! From the timings in the log a large chunk of the > time (21m) is building and installing the Perl modules, so if the > caching works it should indeed be much faster second time round. > > I wonder if the non-x86 builds have more CPUs available, we have > hardcoded "make -j2" at the moment but possibly it makes sense to adjust > this. Would lscpu -p | grep -c -E ^[0-9] work on Ubuntu to deliver the number of cores? So make -j `lscpu -p | grep -c -E ^[0-9]` Regards Rüdiger
Re: arm64 support for Travis CI testing
I would like to express my thanks to Luca and Joe and all other people involved in the travis integration! Very nice indeed! > Am 09.01.2020 um 09:15 schrieb Joe Orton : > > On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote: >> Okay the first Travis run with arm64 (#201) passed. >> The arm64 job (#201.4) took 33 minutes, 5 seconds. >> Compared tp ppc64le at 4 minutes, 30 seconds. >> This is possibly due to need for extra caching on the first run. >> Otherwise, the jobs seem to have the same results. > > Nice, thanks Mike! From the timings in the log a large chunk of the > time (21m) is building and installing the Perl modules, so if the > caching works it should indeed be much faster second time round. > > I wonder if the non-x86 builds have more CPUs available, we have > hardcoded "make -j2" at the moment but possibly it makes sense to adjust > this. > > Regards, Joe >
Re: arm64 support for Travis CI testing
Hi, On Thu, Jan 9, 2020 at 11:01 AM Joe Orton wrote: > On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote: > > Okay the first Travis run with arm64 (#201) passed. > > The arm64 job (#201.4) took 33 minutes, 5 seconds. > > Compared tp ppc64le at 4 minutes, 30 seconds. > > This is possibly due to need for extra caching on the first run. > > Otherwise, the jobs seem to have the same results. > > Nice, thanks Mike! From the timings in the log a large chunk of the > time (21m) is building and installing the Perl modules, so if the > caching works it should indeed be much faster second time round. > > I wonder if the non-x86 builds have more CPUs available, we have > hardcoded "make -j2" at the moment but possibly it makes sense to adjust > this. > https://docs.travis-ci.com/user/multi-cpu-architectures/ says: While available to all Open Source repositories, the concurrency available for multiple CPU arch-based jobs is limited during the alpha period. Not sure what exactly they meant with *limited* here. > Regards, Joe > >
Re: arm64 support for Travis CI testing
On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote: > Okay the first Travis run with arm64 (#201) passed. > The arm64 job (#201.4) took 33 minutes, 5 seconds. > Compared tp ppc64le at 4 minutes, 30 seconds. > This is possibly due to need for extra caching on the first run. > Otherwise, the jobs seem to have the same results. Nice, thanks Mike! From the timings in the log a large chunk of the time (21m) is building and installing the Perl modules, so if the caching works it should indeed be much faster second time round. I wonder if the non-x86 builds have more CPUs available, we have hardcoded "make -j2" at the moment but possibly it makes sense to adjust this. Regards, Joe
