Re: Integration tests running on Docker
Il giorno gio 7 nov 2019 alle ore 16:49 Yann Ylavic ha scritto: > > On Thu, Nov 7, 2019 at 4:47 PM Eric Covener wrote: > > > > > @@ -125,6 +125,7 @@ > > > . non-Unix, single-platform code > > > . routine APLOGNO() backports > > > . .gdbinit > > > +. Travis integration: .travis.yml and test/travis*.sh > > > > +1 > > +1 +1 \o/
Re: Integration tests running on Docker
On Thu, Nov 7, 2019 at 4:47 PM Eric Covener wrote: > > > @@ -125,6 +125,7 @@ > > . non-Unix, single-platform code > > . routine APLOGNO() backports > > . .gdbinit > > +. Travis integration: .travis.yml and test/travis*.sh > > +1 +1
Re: Integration tests running on Docker
> @@ -125,6 +125,7 @@ > . non-Unix, single-platform code > . routine APLOGNO() backports > . .gdbinit > +. Travis integration: .travis.yml and test/travis*.sh +1
Re: Integration tests running on Docker
On Thu, Nov 07, 2019 at 10:37:50AM +, Pluem, Ruediger, Vodafone Group wrote: > r1868314 Ah, thank you! Obviously I tested this locally first too but hadn't updated my test/framework directory... duh. Should we use RTC or can I get an exception for 2.4.x travis integration? I have a few more things to merge to trunk then can backport: https://github.com/apache/httpd/compare/2.4.x...notroj:travisify-2.4.x Index: STATUS === --- STATUS (revision 1869499) +++ STATUS (working copy) @@ -125,6 +125,7 @@ . non-Unix, single-platform code . routine APLOGNO() backports . .gdbinit +. Travis integration: .travis.yml and test/travis*.sh RELEASE SHOWSTOPPERS:
Re: Integration tests running on Docker
On Thu, Nov 7, 2019 at 11:31 AM Yann Ylavic wrote: > > Since this test passes on trunk, a backport is missing possibly? Yep, r1868313, proposed in STATUS already.
Re: Integration tests running on Docker
On Thu, Nov 7, 2019 at 11:31 AM Pluem, Ruediger, Vodafone Group wrote: > > I suspect that $deflated_str_q0 is the result of the request with > Accept-Encoding: gzip;q=0 > where mod_deflate actually should not gzip the response because of q=0. Yeah, brain fart :)
Re: Integration tests running on Docker
On Thu, Nov 7, 2019 at 11:19 AM Yann Ylavic wrote: > > Am I missing something? A brain or something like that :) Nevermind, I was just confused by the name of $deflated_str_q0, this string isn't actually deflated because of q=0. Since this test passes on trunk, a backport is missing possibly?
Re: Integration tests running on Docker
On Wed, Nov 6, 2019 at 6:54 PM Joe Orton wrote:
>
> Has anybody seen t/modules/deflate.t and t/modules/brotli.t fail on
> Debian-ish distro? I don't remember seeing this before but maybe it's
> something
> trivial? t/TEST output -
It seems to originate from r1868312.
I don't really understand this new test in mod_deflate (the one which fails):
ok $original_str eq $deflated_str_q0;
Why would we expect a plain string to be equal to a deflated one, Christophe?
I don't see how the failure depends on debian vs any other system either.
Running t/modules/deflate.t on my debian, with this debugging patch:
```
Index: t/modules/deflate.t
===
--- t/modules/deflate.t (revision 1869498)
+++ t/modules/deflate.t (working copy)
@@ -57,6 +57,10 @@ for my $server_deflate_uri (@server_deflate_uris)
ok $original_str eq $inflated_str;
ok $original_str eq $deflated_str_q0;
+if (not $original_str eq $deflated_str_q0) {
+print "original: '$original_str'\n";
+print "deflated: '$deflated_str_q0'\n";
+}
my $resp = POST($server_inflate_uri, @inflate_headers,
content => "foo123456789012346");
if (have_min_apache_version("2.5")) {
```
I get (e.g.):
```
not ok 42
original: '
Some dummy content. Some dummy content. Some dummy content. Some dummy content.
EOF
'
deflated: ''
# Failed test 42 in t/modules/deflate.t at line 59 fail #11
```
Am I missing something?
Regards,
Yann.
Re: Integration tests running on Docker
Has anybody seen t/modules/deflate.t and t/modules/brotli.t fail on Debian-ish distro? I don't remember seeing this before but maybe it's something trivial? t/TEST output - https://travis-ci.org/notroj/httpd/jobs/608306769#L3244 (I'm trying to get Travis working in 2.4.x as well.) Regards, Joe
Re: Integration tests running on Docker
Il giorno mer 6 nov 2019 alle ore 12:15 Joe Orton ha scritto: > > On Wed, Nov 06, 2019 at 10:54:56AM +0100, Luca Toscano wrote: > > I just noticed > > https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems, > > so the 'addons' will be used by travis only if the os can use them (I > > was afraid of having windows and apt combined). The only thing that I > > would add to travis.yml are the ifs to separate > > before_install/script/etc.. for every os, I think that we should add > > windows as soon as possible.. Do we have a list of build commands to > > test in Travis? > > I have no idea how to build on anything but Linux :) Someone could do > osx too, travis supports that... Me too, I am not a big Windows user, but I am pretty sure that somebody else in the dev community have some info! Will wait :) > So it works for PRs and successfully failed for a bogus commit - > > https://travis-ci.org/apache/httpd/pull_requests > > This one is sensible? > > https://github.com/apache/httpd/pull/71/commits/4a46327813e87f3d662e65361e6d9240ce855db5 +1 looks good, we can either keep the ifs in travis config or go for specific files, I like both approaches! > I don't know if anything other than Linux can actually be tested like > this tbh. (Also trying to context switch between svn & git doing this > stuff is going to melt my brain!) We could start with a simple build on Windows, I think it would be sufficient for the moment.. Osx should be easier, will try to add something as well. The current format of travis.yaml is simple enough that people with experience can add their tests, but surely some documentation is needed! Luca
Re: Integration tests running on Docker
On Wed, Nov 06, 2019 at 10:54:56AM +0100, Luca Toscano wrote: > I just noticed > https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems, > so the 'addons' will be used by travis only if the os can use them (I > was afraid of having windows and apt combined). The only thing that I > would add to travis.yml are the ifs to separate > before_install/script/etc.. for every os, I think that we should add > windows as soon as possible.. Do we have a list of build commands to > test in Travis? I have no idea how to build on anything but Linux :) Someone could do osx too, travis supports that... So it works for PRs and successfully failed for a bogus commit - https://travis-ci.org/apache/httpd/pull_requests This one is sensible? https://github.com/apache/httpd/pull/71/commits/4a46327813e87f3d662e65361e6d9240ce855db5 I don't know if anything other than Linux can actually be tested like this tbh. (Also trying to context switch between svn & git doing this stuff is going to melt my brain!)
Re: Integration tests running on Docker
On Tue, Nov 05, 2019 at 10:06:23AM +0100, Luca Toscano wrote: > Basically what Joe did, but following what the Apache Arrow Project > did. There is a little bit of repetition, but in theory with this > config people are free to add tests for other OSes (osx, windows) and > I will probably be able to add custom ones with Docker to see the > difference in execution timings etc.. (Apache Arrow's test do use > docker, I didn't see it a first). I haven't tried to follow all of what the Arrow .yml is doing (too much Java ;) but moving some of the logic into the repo itself out of .travis.yml seems like a good idea. e.g. moving all of script: into something like ./test/travis_script_linux.sh or something for the Linux based tests, does that make sense? This is working now - https://travis-ci.org/apache/httpd/builds and I guess it will work for forks and PRs too, haven't tried yet. Anybody feel free to extend, play, hack, whatever with the travis.yml, I am no expert so may be doing something non-standard/wrong, feel free to correct/improve ;) The jobs are quite quick but the queue time for the "apache" repos is long (10+ minutes) before they get started. I guess it is doing notifications to committers directly, but we won't find out until a job fails? Maybe let's keep experimenting with this a bit longer and then start sending mail to dev@ for state changes (exactly as buildbot does) when it seems stable? Regards, Joe
Re: Integration tests running on Docker
Il giorno mar 5 nov 2019 alle ore 10:06 Luca Toscano ha scritto: > > Il giorno lun 4 nov 2019 alle ore 19:33 Luca Toscano > ha scritto: > > > > Hi Joe, > > > > Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton > > ha scritto: > > > > > > Here is a proof of concept - > > > > > > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml > > > > > > This runs using the full test suite using a few different configurations > > > and also does builds with -Werror and maintainer-mode - > > > > > > https://travis-ci.org/notroj/httpd/builds/607213820 > > > > > > ...this should be very easy to extend with more configurations. > > > > > > Can we merge the docker way with this kind of matrix type travis > > > configuration? > > > > > > > The proof of concept looks great, I wanted to test something similar > > especially to get timings. It looks like a single test takes from 2 to > > 5 minutes maximum, that is really impressive, I thought it would have > > been more. With Docker in theory we could use the os matrix, but the > > documentation doesn't really suggest any best practice. My idea was to > > pull different docker images (previously built and uploaded to Docker > > Hub), and depending on the os in the matrix run the correct docker > > command in the "script" section (to get an idea > > https://github.com/apache/arrow/blob/master/.travis.yml, even if they > > don't use docker). > > > > While I research though it might be good to follow your solution and > > just have a simple Travis file for say ubuntu and windows. Even if it > > takes 30 mins to complete, it would be a good feedback for anybody > > committing code to trunk/2.4.x. And it could be possible that this > > solution is good enough for our use cases, only testing will tell us! > > So feel free to commit your Travis config to trunk, maybe following > > the Apache Arrow example? If so others, more experienced with Windows, > > might be able to chime in and add the missing configuration. > > This is my idea: > > https://github.com/elukey/httpd/blob/trunk/.travis.yml > > https://travis-ci.org/elukey/httpd/builds/607536348 > > Basically what Joe did, but following what the Apache Arrow Project > did. There is a little bit of repetition, but in theory with this > config people are free to add tests for other OSes (osx, windows) and > I will probably be able to add custom ones with Docker to see the > difference in execution timings etc.. (Apache Arrow's test do use > docker, I didn't see it a first). > I just noticed https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems, so the 'addons' will be used by travis only if the os can use them (I was afraid of having windows and apt combined). The only thing that I would add to travis.yml are the ifs to separate before_install/script/etc.. for every os, I think that we should add windows as soon as possible.. Do we have a list of build commands to test in Travis? Luca
Re: Integration tests running on Docker
Il giorno lun 4 nov 2019 alle ore 19:33 Luca Toscano ha scritto: > > Hi Joe, > > Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton > ha scritto: > > > > Here is a proof of concept - > > > > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml > > > > This runs using the full test suite using a few different configurations > > and also does builds with -Werror and maintainer-mode - > > > > https://travis-ci.org/notroj/httpd/builds/607213820 > > > > ...this should be very easy to extend with more configurations. > > > > Can we merge the docker way with this kind of matrix type travis > > configuration? > > > > The proof of concept looks great, I wanted to test something similar > especially to get timings. It looks like a single test takes from 2 to > 5 minutes maximum, that is really impressive, I thought it would have > been more. With Docker in theory we could use the os matrix, but the > documentation doesn't really suggest any best practice. My idea was to > pull different docker images (previously built and uploaded to Docker > Hub), and depending on the os in the matrix run the correct docker > command in the "script" section (to get an idea > https://github.com/apache/arrow/blob/master/.travis.yml, even if they > don't use docker). > > While I research though it might be good to follow your solution and > just have a simple Travis file for say ubuntu and windows. Even if it > takes 30 mins to complete, it would be a good feedback for anybody > committing code to trunk/2.4.x. And it could be possible that this > solution is good enough for our use cases, only testing will tell us! > So feel free to commit your Travis config to trunk, maybe following > the Apache Arrow example? If so others, more experienced with Windows, > might be able to chime in and add the missing configuration. This is my idea: https://github.com/elukey/httpd/blob/trunk/.travis.yml https://travis-ci.org/elukey/httpd/builds/607536348 Basically what Joe did, but following what the Apache Arrow Project did. There is a little bit of repetition, but in theory with this config people are free to add tests for other OSes (osx, windows) and I will probably be able to add custom ones with Docker to see the difference in execution timings etc.. (Apache Arrow's test do use docker, I didn't see it a first). Luca
Re: Integration tests running on Docker
Nice work! Some weeks ago, I played around with travis to get the mod_h2 tests running, but all OS platforms offered included a too old Apache httpd version. Which meant I would need to drag in a recent httpd as well. I got distracted before doing so and this got put back on the useful-things-to-do-list. Maybe I should fork your fork and add these tests there... Do you intend to PR this back to httpd one day? Cheers, Stefan > Am 04.11.2019 um 17:16 schrieb Joe Orton : > > Here is a proof of concept - > > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml > > This runs using the full test suite using a few different configurations > and also does builds with -Werror and maintainer-mode - > > https://travis-ci.org/notroj/httpd/builds/607213820 > > ...this should be very easy to extend with more configurations. > > Can we merge the docker way with this kind of matrix type travis > configuration? > > Regards, Joe >
Re: Integration tests running on Docker
Hi Joe, Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton ha scritto: > > Here is a proof of concept - > > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml > > This runs using the full test suite using a few different configurations > and also does builds with -Werror and maintainer-mode - > > https://travis-ci.org/notroj/httpd/builds/607213820 > > ...this should be very easy to extend with more configurations. > > Can we merge the docker way with this kind of matrix type travis > configuration? > The proof of concept looks great, I wanted to test something similar especially to get timings. It looks like a single test takes from 2 to 5 minutes maximum, that is really impressive, I thought it would have been more. With Docker in theory we could use the os matrix, but the documentation doesn't really suggest any best practice. My idea was to pull different docker images (previously built and uploaded to Docker Hub), and depending on the os in the matrix run the correct docker command in the "script" section (to get an idea https://github.com/apache/arrow/blob/master/.travis.yml, even if they don't use docker). While I research though it might be good to follow your solution and just have a simple Travis file for say ubuntu and windows. Even if it takes 30 mins to complete, it would be a good feedback for anybody committing code to trunk/2.4.x. And it could be possible that this solution is good enough for our use cases, only testing will tell us! So feel free to commit your Travis config to trunk, maybe following the Apache Arrow example? If so others, more experienced with Windows, might be able to chime in and add the missing configuration. Thanks! Luca
Re: Integration tests running on Docker
Here is a proof of concept - https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml This runs using the full test suite using a few different configurations and also does builds with -Werror and maintainer-mode - https://travis-ci.org/notroj/httpd/builds/607213820 ...this should be very easy to extend with more configurations. Can we merge the docker way with this kind of matrix type travis configuration? Regards, Joe
Re: Integration tests running on Docker
Hi Daniel, thanks for following up! Il giorno mer 30 ott 2019 alle ore 13:05 Daniel Ruggeri ha scritto: > > > > On October 29, 2019 8:12:25 AM CDT, Luca Toscano > wrote: > >Il giorno mar 29 ott 2019 alle ore 12:23 Joe Orton > >ha scritto: > >> > >> On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote: > >> > Some updates: > >> > > >> > - We have support for httpd in travis - > >https://travis-ci.org/apache/httpd > >> > >> Nice, thanks Luca & infra! > >> > >> > - In order to configure automatic builds, a travis.yaml file is > >needed > >> > in the base path of the repository, in every branch that needs to > >be > >> > tested IIUC. > >> > - My idea is currently to add in trunk a travis.yaml initial > >> > configuration file, that builds a Docker image like [1] and runs > >the > >> > perl test suite. > >> > > >> > - Building the Docker image in [1] takes quite a while now (between > >9 > >> > and 13 mins on my laptop, depending also on network bandwidth > >etc..) > >> > so it will need more time before it could be as fast as we need, > >but > >> > we have to start from somewhere :) > >> > - In the travis.yaml file we'd need to put config options about > >what > >> > Docker image to build and with what parameters. Ideally I'd like to > >> > store the Docker images in the httpd-framework repository, and > >> > >> Do you mean Dockerfiles rather than images here? > > > >Correct yes, my bad! > > > >> > >> > reference them in the Travis config of the httpd branches, but not > >> > sure if it will be possible. Worst case scenario we'll need to add > >the > >> > Docker image in each httpd branch that we want to test (possibly in > >a > >> > dedicated dir, like "docker" or "testing". > >> > >> I expected two goals for testing: > >> > >> 1. configure, build, and test using a variety of different configure > >> options in the standard Ubuntu environment which Travis provides > >(enable > >> different MPMs, different module sets, different > >> --enable-foo/--with-foo, different gcc versions etc etc) > > > >This is also what I was trying to understand in these days, namely if > >travis could build and test directly without Docker. > > > >> 2. configure build and test in a variety of OS environments - this > >would > >> make sense using container images. > >> > >> I don't have much experience testing inside containers from Travis, > >but > >> if we can do both (1) and (2) inside containers that might make > >sense? > >> Or can we do both separately? If you have a .travis.yml which works > >> then I'd say go commit it and we can work from there. > > > >I have currently a Docker file that works for Debian/Ubuntu (and > >another one for CentOs), and IIUC the same could also be done for > >other OSes (so having different Docker files for OSX and Win, since > >Travis seems to support them). The main problem is that it currently > >takes minutes to build the Docker images, and only after that tests > >will be able to run. I guess that this problem will be the same if we > >run directly tests in Travis, but we can give it a try. Another > >downside about Travis, in my opinion, is that testing locally seems to > >be more difficult that building Docker images (but I admit my > >ignorance so it might be different), this is why I preferred Docker in > >the first place. > > This very closely matches my own testing setup. I *really* like this. Thank > you for driving this! > > > > >> Or can we do both separately? If you have a .travis.yml which works > >> then I'd say go commit it and we can work from there. > > > >I currently don't have a working travis yaml, but I'll come up with > >something during the next days. What I am currently trying to > >understand is if the Docker files could be stored elsewhere (like in > >the httpd-framework repo) and downloaded in Travis on demand, to have > >only one the Travis config in the httpd branches (as opposed to also > >having Docker files etc..). > > > >Let me know your suggestions/thoughts :) > > > >Luca > > The most I know about Travis is how to spell it, but... > What if the various Dockerfile-based testing environments were treated as a > separate "CI-able" item? That is, what if we monitored for upstream changes > then proactively rebuilt and pushed the testing images as > "httpdtest-centos-7" and "httpdtest-ubuntu-trusty", etc? That way, the > testing image is built/ready so the $work needed on check-in is just to build > and test httpd. > > A significant downside is that these discreet test "enviroments" will all > have their own lifecycles and maintenance burden. I found on my Debian-based > Docker test environment that subtle upstream changes caused unexpected > breakage. This was pretty disruptive to workflow because I had to get out of > my "httpd tester" hat and don my "Debian administrator" hat in the middle of > testing whatever I was working on to resolve the breakage. With that in mind, > I view the pre-build mechanism as a way to avoid breakages du
Re: Integration tests running on Docker
Il giorno mar 29 ott 2019 alle ore 12:23 Joe Orton ha scritto: > > On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote: > > Some updates: > > > > - We have support for httpd in travis - https://travis-ci.org/apache/httpd > > Nice, thanks Luca & infra! > > > - In order to configure automatic builds, a travis.yaml file is needed > > in the base path of the repository, in every branch that needs to be > > tested IIUC. > > - My idea is currently to add in trunk a travis.yaml initial > > configuration file, that builds a Docker image like [1] and runs the > > perl test suite. > > > > - Building the Docker image in [1] takes quite a while now (between 9 > > and 13 mins on my laptop, depending also on network bandwidth etc..) > > so it will need more time before it could be as fast as we need, but > > we have to start from somewhere :) > > - In the travis.yaml file we'd need to put config options about what > > Docker image to build and with what parameters. Ideally I'd like to > > store the Docker images in the httpd-framework repository, and > > Do you mean Dockerfiles rather than images here? Correct yes, my bad! > > > reference them in the Travis config of the httpd branches, but not > > sure if it will be possible. Worst case scenario we'll need to add the > > Docker image in each httpd branch that we want to test (possibly in a > > dedicated dir, like "docker" or "testing". > > I expected two goals for testing: > > 1. configure, build, and test using a variety of different configure > options in the standard Ubuntu environment which Travis provides (enable > different MPMs, different module sets, different > --enable-foo/--with-foo, different gcc versions etc etc) This is also what I was trying to understand in these days, namely if travis could build and test directly without Docker. > 2. configure build and test in a variety of OS environments - this would > make sense using container images. > > I don't have much experience testing inside containers from Travis, but > if we can do both (1) and (2) inside containers that might make sense? > Or can we do both separately? If you have a .travis.yml which works > then I'd say go commit it and we can work from there. I have currently a Docker file that works for Debian/Ubuntu (and another one for CentOs), and IIUC the same could also be done for other OSes (so having different Docker files for OSX and Win, since Travis seems to support them). The main problem is that it currently takes minutes to build the Docker images, and only after that tests will be able to run. I guess that this problem will be the same if we run directly tests in Travis, but we can give it a try. Another downside about Travis, in my opinion, is that testing locally seems to be more difficult that building Docker images (but I admit my ignorance so it might be different), this is why I preferred Docker in the first place. > Or can we do both separately? If you have a .travis.yml which works > then I'd say go commit it and we can work from there. I currently don't have a working travis yaml, but I'll come up with something during the next days. What I am currently trying to understand is if the Docker files could be stored elsewhere (like in the httpd-framework repo) and downloaded in Travis on demand, to have only one the Travis config in the httpd branches (as opposed to also having Docker files etc..). Let me know your suggestions/thoughts :) Luca
Re: Integration tests running on Docker
On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote: > Some updates: > > - We have support for httpd in travis - https://travis-ci.org/apache/httpd Nice, thanks Luca & infra! > - In order to configure automatic builds, a travis.yaml file is needed > in the base path of the repository, in every branch that needs to be > tested IIUC. > - My idea is currently to add in trunk a travis.yaml initial > configuration file, that builds a Docker image like [1] and runs the > perl test suite. > > - Building the Docker image in [1] takes quite a while now (between 9 > and 13 mins on my laptop, depending also on network bandwidth etc..) > so it will need more time before it could be as fast as we need, but > we have to start from somewhere :) > - In the travis.yaml file we'd need to put config options about what > Docker image to build and with what parameters. Ideally I'd like to > store the Docker images in the httpd-framework repository, and Do you mean Dockerfiles rather than images here? > reference them in the Travis config of the httpd branches, but not > sure if it will be possible. Worst case scenario we'll need to add the > Docker image in each httpd branch that we want to test (possibly in a > dedicated dir, like "docker" or "testing". I expected two goals for testing: 1. configure, build, and test using a variety of different configure options in the standard Ubuntu environment which Travis provides (enable different MPMs, different module sets, different --enable-foo/--with-foo, different gcc versions etc etc) 2. configure build and test in a variety of OS environments - this would make sense using container images. I don't have much experience testing inside containers from Travis, but if we can do both (1) and (2) inside containers that might make sense? Or can we do both separately? If you have a .travis.yml which works then I'd say go commit it and we can work from there. Regards, Joe > If you are strongly against the above or have more suggestions and > knowledge about Travis please let me know!
Re: Integration tests running on Docker
Some updates: - We have support for httpd in travis - https://travis-ci.org/apache/httpd - In order to configure automatic builds, a travis.yaml file is needed in the base path of the repository, in every branch that needs to be tested IIUC. - My idea is currently to add in trunk a travis.yaml initial configuration file, that builds a Docker image like [1] and runs the perl test suite. - Building the Docker image in [1] takes quite a while now (between 9 and 13 mins on my laptop, depending also on network bandwidth etc..) so it will need more time before it could be as fast as we need, but we have to start from somewhere :) - In the travis.yaml file we'd need to put config options about what Docker image to build and with what parameters. Ideally I'd like to store the Docker images in the httpd-framework repository, and reference them in the Travis config of the httpd branches, but not sure if it will be possible. Worst case scenario we'll need to add the Docker image in each httpd branch that we want to test (possibly in a dedicated dir, like "docker" or "testing". If you are strongly against the above or have more suggestions and knowledge about Travis please let me know! Luca [1] https://github.com/elukey/httpd_integration_testing/blob/master/docker/code-testing/debian/Dockerfile Il giorno mar 22 ott 2019 alle ore 22:04 Luca Toscano ha scritto: > > Opened an issue to Infra to enable Travis support in github: > https://issues.apache.org/jira/browse/INFRA-19325 > > Luca > > Il giorno sab 28 set 2019 alle ore 09:39 Luca Toscano > ha scritto: > > > > Follow up after some work in > > https://github.com/elukey/httpd_integration_testing: > > > > * Use only one Docker image for ubuntu/debian > > * Created two use cases: code snapshot testing (latest trunk and > > 2.4.x) vs release candidate testing > > * Still some issues in installing perl deps on older Debian distro > > (like Stretch), but the perl suite seems to run fine in both use cases > > for Debian Buster. > > * Some issues while running the HTTP/2 test suite, will follow up with > > Stefan. > > * I'll try to add a Docker image for CentOS. > > > > Please open pull requests if you have ideas/comments/suggestions/etc.. :) > > > > Thanks! > > > > Luca > > > > Il giorno mer 25 set 2019 alle ore 11:52 Luca Toscano > > ha scritto: > > > > > > Hi everybody, > > > > > > I spent some time reading the previous discussions around the concept > > > of "CI" and from what I gather, it seems that we didn't reach an > > > agreement about how to proceed and who is working on it (but I might > > > be wrong, in case apologies!). From what I can see, there are two > > > possible working fronts: > > > > > > 1) Simplify the usage of the current perl testing suite, adding > > > docs/tests/etc.. Some people expressed the desire for a more friendly > > > framework, especially when adding new tests. > > > 2) Run the testing framework automatically on different environments > > > to spot anomalies/bugs/etc.. in a timely manner. > > > > > > I am a bit ignorant about how to run a proper CI but I created a > > > little prototype of a Dockerfile able to bootstrap a testing > > > environment on Debian 10 (Buster) and run the perl test suite: > > > > > > https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile > > > > > > The above is only an example, it is missing a lot of things and some > > > follow up work is needed. But with the following commands, on my > > > laptop I was able to create a docker image and run the test suite: > > > > > > docker build . > > > docker run $id-of-the-image make check > > > > > > Some thoughts: > > > > > > 1) The above Dockerfile is really handy since I can easily switch > > > between Debian versions (Jessie/Stretch/Buster to name the last three) > > > and run the test suite with different package versions (openssl, > > > nghttp2, etc..). It should be also easy to create Dockerfiles for > > > other OS/environments, and run make check in a similar way. > > > 1-bis) Testing on Windows would still need to be solved, Docker > > > probably it is not the right solution but we could find something else > > > to integrate for this specific use case. > > > 2) Docker also offers a way to open a bash shell in interactive mode, > > > so it could be easy to run tests on a certain platform when somebody > > > reports a problem. Or make sure that a new set of tests runs correctly > > > everywhere. > > > 3) Another use case could be to create a Dockerfile that pulls a > > > specific new release of httpd, installing it and running the test > > > suite on multiple platforms. > > > 4) The same Docker image could also run tests suites like > > > https://github.com/icing/mod_h2/tree/master/test/e2e (that is really > > > nice, I suggest to check it if you haven't done it) to run HTTP/2 > > > tests as well. > > > 5) We could even think about having daily docker image builds that > > > take a snapshot of trunk/2.4.x and run the test suites,
Re: Integration tests running on Docker
Opened an issue to Infra to enable Travis support in github: https://issues.apache.org/jira/browse/INFRA-19325 Luca Il giorno sab 28 set 2019 alle ore 09:39 Luca Toscano ha scritto: > > Follow up after some work in > https://github.com/elukey/httpd_integration_testing: > > * Use only one Docker image for ubuntu/debian > * Created two use cases: code snapshot testing (latest trunk and > 2.4.x) vs release candidate testing > * Still some issues in installing perl deps on older Debian distro > (like Stretch), but the perl suite seems to run fine in both use cases > for Debian Buster. > * Some issues while running the HTTP/2 test suite, will follow up with Stefan. > * I'll try to add a Docker image for CentOS. > > Please open pull requests if you have ideas/comments/suggestions/etc.. :) > > Thanks! > > Luca > > Il giorno mer 25 set 2019 alle ore 11:52 Luca Toscano > ha scritto: > > > > Hi everybody, > > > > I spent some time reading the previous discussions around the concept > > of "CI" and from what I gather, it seems that we didn't reach an > > agreement about how to proceed and who is working on it (but I might > > be wrong, in case apologies!). From what I can see, there are two > > possible working fronts: > > > > 1) Simplify the usage of the current perl testing suite, adding > > docs/tests/etc.. Some people expressed the desire for a more friendly > > framework, especially when adding new tests. > > 2) Run the testing framework automatically on different environments > > to spot anomalies/bugs/etc.. in a timely manner. > > > > I am a bit ignorant about how to run a proper CI but I created a > > little prototype of a Dockerfile able to bootstrap a testing > > environment on Debian 10 (Buster) and run the perl test suite: > > > > https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile > > > > The above is only an example, it is missing a lot of things and some > > follow up work is needed. But with the following commands, on my > > laptop I was able to create a docker image and run the test suite: > > > > docker build . > > docker run $id-of-the-image make check > > > > Some thoughts: > > > > 1) The above Dockerfile is really handy since I can easily switch > > between Debian versions (Jessie/Stretch/Buster to name the last three) > > and run the test suite with different package versions (openssl, > > nghttp2, etc..). It should be also easy to create Dockerfiles for > > other OS/environments, and run make check in a similar way. > > 1-bis) Testing on Windows would still need to be solved, Docker > > probably it is not the right solution but we could find something else > > to integrate for this specific use case. > > 2) Docker also offers a way to open a bash shell in interactive mode, > > so it could be easy to run tests on a certain platform when somebody > > reports a problem. Or make sure that a new set of tests runs correctly > > everywhere. > > 3) Another use case could be to create a Dockerfile that pulls a > > specific new release of httpd, installing it and running the test > > suite on multiple platforms. > > 4) The same Docker image could also run tests suites like > > https://github.com/icing/mod_h2/tree/master/test/e2e (that is really > > nice, I suggest to check it if you haven't done it) to run HTTP/2 > > tests as well. > > 5) We could even think about having daily docker image builds that > > take a snapshot of trunk/2.4.x and run the test suites, reporting back > > any problem. > > > > Does the above make sense or it is completely out of scope for our > > project? In the former case I could spend some time on figuring out > > how to create what I proposed above, otherwise I'll stop and drop the > > idea :) > > > > Last but not the least, this would be a way to help us testing changes > > in a more automated way, not a replacement of community based testing > > and bug reports (stating the obvious for an Apache project but better > > safe than sorry :). > > > > Luca
Re: Integration tests running on Docker
Follow up after some work in https://github.com/elukey/httpd_integration_testing: * Use only one Docker image for ubuntu/debian * Created two use cases: code snapshot testing (latest trunk and 2.4.x) vs release candidate testing * Still some issues in installing perl deps on older Debian distro (like Stretch), but the perl suite seems to run fine in both use cases for Debian Buster. * Some issues while running the HTTP/2 test suite, will follow up with Stefan. * I'll try to add a Docker image for CentOS. Please open pull requests if you have ideas/comments/suggestions/etc.. :) Thanks! Luca Il giorno mer 25 set 2019 alle ore 11:52 Luca Toscano ha scritto: > > Hi everybody, > > I spent some time reading the previous discussions around the concept > of "CI" and from what I gather, it seems that we didn't reach an > agreement about how to proceed and who is working on it (but I might > be wrong, in case apologies!). From what I can see, there are two > possible working fronts: > > 1) Simplify the usage of the current perl testing suite, adding > docs/tests/etc.. Some people expressed the desire for a more friendly > framework, especially when adding new tests. > 2) Run the testing framework automatically on different environments > to spot anomalies/bugs/etc.. in a timely manner. > > I am a bit ignorant about how to run a proper CI but I created a > little prototype of a Dockerfile able to bootstrap a testing > environment on Debian 10 (Buster) and run the perl test suite: > > https://github.com/elukey/httpd_integration_testing/blob/master/docker/Dockerfile > > The above is only an example, it is missing a lot of things and some > follow up work is needed. But with the following commands, on my > laptop I was able to create a docker image and run the test suite: > > docker build . > docker run $id-of-the-image make check > > Some thoughts: > > 1) The above Dockerfile is really handy since I can easily switch > between Debian versions (Jessie/Stretch/Buster to name the last three) > and run the test suite with different package versions (openssl, > nghttp2, etc..). It should be also easy to create Dockerfiles for > other OS/environments, and run make check in a similar way. > 1-bis) Testing on Windows would still need to be solved, Docker > probably it is not the right solution but we could find something else > to integrate for this specific use case. > 2) Docker also offers a way to open a bash shell in interactive mode, > so it could be easy to run tests on a certain platform when somebody > reports a problem. Or make sure that a new set of tests runs correctly > everywhere. > 3) Another use case could be to create a Dockerfile that pulls a > specific new release of httpd, installing it and running the test > suite on multiple platforms. > 4) The same Docker image could also run tests suites like > https://github.com/icing/mod_h2/tree/master/test/e2e (that is really > nice, I suggest to check it if you haven't done it) to run HTTP/2 > tests as well. > 5) We could even think about having daily docker image builds that > take a snapshot of trunk/2.4.x and run the test suites, reporting back > any problem. > > Does the above make sense or it is completely out of scope for our > project? In the former case I could spend some time on figuring out > how to create what I proposed above, otherwise I'll stop and drop the > idea :) > > Last but not the least, this would be a way to help us testing changes > in a more automated way, not a replacement of community based testing > and bug reports (stating the obvious for an Apache project but better > safe than sorry :). > > Luca
Re: Integration tests running on Docker
Hi, Il giorno mer 25 set 2019 alle ore 13:18 Mads Toftum ha scritto: > > On Wed, Sep 25, 2019 at 11:52:52AM +0200, Luca Toscano wrote: > > Does the above make sense or it is completely out of scope for our > > project? In the former case I could spend some time on figuring out > > how to create what I proposed above, otherwise I'll stop and drop the > > idea :) > > > To me it seems useful enough to put in the repo with the rest of the > test code or at least documenting somewhere on http://httpd.apache.org/dev/ > If rolled into httpd test framework, it might make more sense to have a > make docker (or similar) that spins up an image with the source copied > in. Definitely, I started a separate gh project only to show the prototype, once we have something that works and that we like it might be better to move it elsewhere. "make docker" could also be cool to add! Luca
Re: Integration tests running on Docker
Hi Daniel, Il giorno mer 25 set 2019 alle ore 13:24 Daniel Ruggeri ha scritto: > > I'm a big fan of this idea - it's how I do testing :-) > > Unfortunately, I ran into a few issues trying to build and test with Debian > maintained dependencies and only ended up with one working test suite: build > httpd and all dependencies from sources. But the idea as a whole is sound - > we can test against any Linux-based OS distribution which is available in the > Docker hub. It could also serve as a nice hint to those that would like to > deploy httpd from sources (or even package maintainers) by offering a working > recipe that tests clean. > > I also raised an inquiry to Infra (but admittedly never followed up) to > understand what we'd need to do to have Docker deployed to our CI VM. I > understand they use Puppet to manage said VMs so I was hoping to figure out > if this would be permissible and then which scripts need to be updated. Good point about the Debian maintained dependencies, I didn't think about it. As first step we could simply assume that they don't exist and build all the deps from source :) Do you have any special ./configure arg list that you think it would be worth to use? Or maybe a collection of them, I have seen problems over time rising from using static vs dso modules, enabling only a subset of modules, etc.. For the first iteration maybe something that involves --enable-modules=reallyall is enough? Thanks! Luca
Re: Integration tests running on Docker
On Wed, Sep 25, 2019 at 11:52:52AM +0200, Luca Toscano wrote: > Does the above make sense or it is completely out of scope for our > project? In the former case I could spend some time on figuring out > how to create what I proposed above, otherwise I'll stop and drop the > idea :) > To me it seems useful enough to put in the repo with the rest of the test code or at least documenting somewhere on http://httpd.apache.org/dev/ If rolled into httpd test framework, it might make more sense to have a make docker (or similar) that spins up an image with the source copied in. vh Mads Toftum -- http://flickr.com/photos/q42/
Re: Integration tests running on Docker
+1's! Thanks Luca, looking great! El mié., 25 sept. 2019 a las 12:44, Stefan Eissing () escribió: > > +1 > > I like this a lot. Having a maintained docker setup at hand to run tests in > controlled setups is super good. > > If you get this going for the main test suite, I will be more than happy to > contribute with setups for the mod_h2 and mod_md test suites. Those are part > of the github repros only. > > Cheers, Stefan > > > Am 25.09.2019 um 11:52 schrieb Luca Toscano : > > > > Does the above make sense or it is completely out of scope for our > > project? In the former case I could spend some time on figuring out > > how to create what I proposed above, otherwise I'll stop and drop the > > idea :) > > > -- Daniel Ferradal HTTPD Project #httpd help at Freenode
Re: Integration tests running on Docker
+1 I like this a lot. Having a maintained docker setup at hand to run tests in controlled setups is super good. If you get this going for the main test suite, I will be more than happy to contribute with setups for the mod_h2 and mod_md test suites. Those are part of the github repros only. Cheers, Stefan > Am 25.09.2019 um 11:52 schrieb Luca Toscano : > > Does the above make sense or it is completely out of scope for our > project? In the former case I could spend some time on figuring out > how to create what I proposed above, otherwise I'll stop and drop the > idea :) >
