Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Wed, Oct 29, 2014 at 09:21:00PM -0700, Craig Rodrigues wrote: > On Tue, Oct 28, 2014 at 10:44 PM, Mateusz Guzik wrote: > > > On Fri, Oct 24, 2014 at 09:09:48PM -0700, Craig Rodrigues wrote: > > > > Are you aware of e.g. stress2 by pho? > > > > https://people.freebsd.org/~pho/stress/index.html > > > > Plugging this in should not be hard for someone who knows jenkins. > > > > That's not me though. :) > > > > > Since you are making some good suggestions, would you be interested > in learning enough about Jenkins on your own machine first, > and and see if you can get a successful integration of stress2? > Setting up Jenkins is quite easy under FreeBSD. There are a lot of howto's > on the web for > how to do it, and I have given multiple presentations: > https://wiki.freebsd.org/Jenkins#Presentations_and_Working_Groups > > I don't mind when people make good suggestions, but > when the good suggestions are not backed up by concrete offers to > help out, then that doesn't look so good. We need more people contributing > actual scripts > and tests. There are plenty of people contributing suggestions. :) > I'm not really interested in learning jenkins, got enough backlog as it is. I'm just putting stuff out in case it sounds ok and someone wants to do it. -- Mateusz Guzik ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Tue, Oct 28, 2014 at 10:44 PM, Mateusz Guzik wrote: > On Fri, Oct 24, 2014 at 09:09:48PM -0700, Craig Rodrigues wrote: > > Are you aware of e.g. stress2 by pho? > > https://people.freebsd.org/~pho/stress/index.html > > Plugging this in should not be hard for someone who knows jenkins. > > That's not me though. :) > Since you are making some good suggestions, would you be interested in learning enough about Jenkins on your own machine first, and and see if you can get a successful integration of stress2? Setting up Jenkins is quite easy under FreeBSD. There are a lot of howto's on the web for how to do it, and I have given multiple presentations: https://wiki.freebsd.org/Jenkins#Presentations_and_Working_Groups I don't mind when people make good suggestions, but when the good suggestions are not backed up by concrete offers to help out, then that doesn't look so good. We need more people contributing actual scripts and tests. There are plenty of people contributing suggestions. :) -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Tue, Oct 28, 2014 at 10:44 PM, Mateusz Guzik wrote: ... > Are you aware of e.g. stress2 by pho? > > https://people.freebsd.org/~pho/stress/index.html +1 for this, but I don't have cycles to implement it in Jenkins. All I'll say is that using stress2 won't work with your existing preconditions based on how it gathers crash dumps, how long some of the tests may take, etc. But you could effectively script pho's magic to some degree. > Plugging this in should not be hard for someone who knows jenkins. Plugging it in should be relatively easy. Babysitting it will be a lot more entertaining. There's a reason why we don't run it in our default test runs at Isilon ;)... ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Fri, Oct 24, 2014 at 09:09:48PM -0700, Craig Rodrigues wrote: > On Thu, Oct 23, 2014 at 10:36 PM, Mateusz Guzik wrote: > > > > (1) does a buildworld/buildkernel on amd64 when someone checks new > > > code into the stable/10 branch > > > > Is not this excessive? > > > It has not been a problem. For example, when a build occurs on the HEAD in > svn, > if further commits come in on HEAD, we have Jenkins configured so that it > will not trigger another build on HEAD until the build in progress is done. > It has been working fine. > Ok. > If no commits > > > > > (2) Creates a bootable UFS image with makefs > > > > any chance zfs will be used as well? > > > > Sure, we can look at that as well, but as I said earlier, > there need to be more bodies working on setting up builds and > configurations for this > to happen. > I can't help with that, sorry. > > > > > would be nice to run some kind of stress testing. buildworld with a high > > -j is an example of a general purpose test. This could be done with > > different frequency than regular tests. > > > > > Are you volunteering to write the scripts that incorporate any stress > testing > that you think should be done? We would welcome any contributions. > > Are you aware of e.g. stress2 by pho? https://people.freebsd.org/~pho/stress/index.html Plugging this in should not be hard for someone who knows jenkins. That's not me though. :) > > Do you have crashdumps configured in case stuff goes wrong? > > > > > > No. > > You can look at our scripts used to build and boot the various VM's: > > https://wiki.freebsd.org/Jenkins#Repositories > > It's all on github, so if you think you have new scripts to add, or fixes > to existing scripts, > you can feel free to do a github pull request to contribute. > > -- > Craig -- Mateusz Guzik ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
Susan, In addition to the slides which I previously mentioned, you and your colleagues at Xinuos may wish to take a quick read of this document which I wrote, the "Kyua Quickstart Guide": https://github.com/rodrigc/kyua/wiki/Quickstart-Guide kyua is the test harness that is used to run the tests in the /usr/tests directory in FreeBSD. That doc is a very quick guide on how to run tests under kyua, and also how to write tests which can be run under kyua. Getting any feedback on this would be very useful. Does that doc make sense to a newcomer? Are there typos? I want the FreeBSD project to get to the point where: (1) anybody can download the FreeBSD tests (2) anybody can run the tests (3) people can easily figure out how to write new tests and contribute them back (if they wish) to FreeBSD Better documentation will hopefully help with this. Thanks. -- Craig On Fri, Oct 24, 2014 at 1:37 PM, Susan Stanziano wrote: > Hello Craig, > > I have been reading through your slides and your website announcement, as > requested, and have a few questions. (I have also subscribed to the > mailing list as suggested). > > (1) The slides indicate that with a FreeBSD10 and lower OS, I should be > able to build tests already included with FreeBSD. I have created a VM on > FreeBSD10.0 but do not have the /etc/make.conf file necessary to build and > install the tests. This may be that I am missing some set-up steps > required but not covered in the slide presentation. > > (2) Does the test build require the use of the "Build Flow Plugin" or can > the tests be run independent of the "continuous build" model? > > (3) Can the tests only be run under Bhyve as described in the announcement? > > I would be happy to exchange emails with you or skype as I investigate > this work further. > > Regards, > > Susan > > -- > *From: *"Craig Rodrigues" > *To: *e...@xinuos.com > *Cc: *"Allan Jude" , "John Wolfe" , > freebsd-virtualization@freebsd.org, freebsd-test...@freebsd.org, "Susan > Stanziano" , "Cheryl Blain" < > cheryl.bl...@xinuos.com>, "Andy Nagle" > *Sent: *Friday, October 24, 2014 7:09:04 AM > *Subject: *Re: Re: Automatically running /usr/tests on stable/10 branch > under Jenkins > > > > > On Fri, Oct 24, 2014 at 1:11 AM, Eric le Blan wrote: > >> Allan, >> >> >> >> Always happy to discuss testing. I believe that Susan Stanziono of Xinuos >> system test team already spoke to some of you. Please do not hesitate to >> reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of >> operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach >> to Craig. > > > > It would be helpful if someone (or some people) from Xinuous could do the > following: > > (1) Subscribe to freebsd-test...@freebsd.org at > https://lists.freebsd.org/mailman/listinfo/freebsd-testing to start > monitoring the discussions on that list. > > (2) Review this announcement: > http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing > > (3) Review these slides: > http://www.slideshare.net/CraigRodrigues1/kyua-jenkins > > (4) Talk to me at MeetBSD Vendor Summit > > I have not been given a long timeslot to give a presentation, so reviewing > those links before MeetBSD would > be helpful. Long term, any testing of FreeBSD will only succeed if there > is > an ecosystem of people working together and sharing common scripts and > information for testing FreeBSD. > > Any help that Xinuous (or any other company) can provide would be welcome. > > -- > Craig > > ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Sat, Oct 25, 2014 at 2:46 PM, Garrett Cooper wrote: > On Oct 25, 2014, at 13:20, K. Macy wrote: > > Alan also suggested against integrating the test suite as-is, because as > he said, "Remember, don't run these tests on a production system. They > WILL cause panics and deadlocks, and they may cause data loss too.” > > Cheers, > -Garrett Wait, we want to sweep those bugs under the rug? What exactly is wrong with making a test harness that can very easily reproduce a known problem? The chances are that anyone will dive into it once the bug is easily reproducible. >>> >>>Sweeping bugs under the rug is not what I plan on doing; I’m marking >>> these as expected failures, as opposed to having them continually panic a >>> machine. Once a ZFS dev takes a look at the issue and resolves them, then >>> the ZFS dev can remove the “bail” calls I’m adding to the testcases. >> >> Yes, disabling tests that fail leads to an ineffectual test suite. A >> test suite that never has any failures is not very useful. However, >> there are two factors to take in to account in this context: >> a) frequent failures can lead users to stop running a test suite >> leading to further regressions >> b) long-term repeated failures can desensitize users leading them to >> ignore *new* failures facilitating further regressions >> >> Thus it's really a question of what context you're talking about >> running the test suite in. For purposes of Jenkins we want full >> visibility in to what is passing and what is failing and how long this >> has been going on for. > > (seeing as how my other post isn’t in the -testing archives yet..) > Panicking a node (what the tests are doing before last night) and > exiting with a non-zero exit code (what I’m making them do with the bail outs > in tools/regression/zfs) are both considered test failures. The difference > being that I can safely run all of the tests on a production or a test > machine without having to panic/reboot the box and I get greater coverage in > one fell swoop. If a developer wants they can always delete the lines that > bail out of the tests to get the desired panic. I don't think there is a RIGHT answer. I just ask that it be noisy about the choice that's being made. In other words, it should be very explicit that it's running a "safe" subset of the tests because FreeBSD's ZFS can't actually pass all of them, and that there be a flag to enable those of us who want to see the failure to see it without modifying any scripts or config files. A test framework that requires me to muck with settings to run failing tests isn't living up to its full potential. -K ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Oct 25, 2014, at 13:20, K. Macy wrote: Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system. They WILL cause panics and deadlocks, and they may cause data loss too.” Cheers, -Garrett >>> >>> Wait, we want to sweep those bugs under the rug? What exactly is wrong >>> with making a test harness that can very easily reproduce a known problem? >>> The chances are that anyone will dive into it once the bug is easily >>> reproducible. >> >>Sweeping bugs under the rug is not what I plan on doing; I’m marking >> these as expected failures, as opposed to having them continually panic a >> machine. Once a ZFS dev takes a look at the issue and resolves them, then >> the ZFS dev can remove the “bail” calls I’m adding to the testcases. > > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions > > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. (seeing as how my other post isn’t in the -testing archives yet..) Panicking a node (what the tests are doing before last night) and exiting with a non-zero exit code (what I’m making them do with the bail outs in tools/regression/zfs) are both considered test failures. The difference being that I can safely run all of the tests on a production or a test machine without having to panic/reboot the box and I get greater coverage in one fell swoop. If a developer wants they can always delete the lines that bail out of the tests to get the desired panic. Cheers, -Garrett signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Sat, Oct 25, 2014 at 1:20 PM, K. Macy wrote: Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system. They WILL cause panics and deadlocks, and they may cause data loss too.” Cheers, -Garrett >>> >>> Wait, we want to sweep those bugs under the rug? What exactly is wrong >>> with making a test harness that can very easily reproduce a known problem? >>> The chances are that anyone will dive into it once the bug is easily >>> reproducible. >> >> Sweeping bugs under the rug is not what I plan on doing; I’m marking >> these as expected failures, as opposed to having them continually panic a >> machine. Once a ZFS dev takes a look at the issue and resolves them, then >> the ZFS dev can remove the “bail” calls I’m adding to the testcases. >> Cheers, >> -Garrett > > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions > > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. Agreed. We're talking about doing an OpenZFS bug-tracker and maybe we could have tests post there automatically, once the bug is opened. I am going to put this on the board at MeetBSD Ca. next week, so we can have a discussion about it in real-time with some of the people present. Cheers, -matt ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
>>> Alan also suggested against integrating the test suite as-is, because as he >>> said, "Remember, don't run these tests on a production system. They WILL >>> cause panics and deadlocks, and they may cause data loss too.” >>> >>> Cheers, >>> -Garrett >> >> Wait, we want to sweep those bugs under the rug? What exactly is wrong with >> making a test harness that can very easily reproduce a known problem? The >> chances are that anyone will dive into it once the bug is easily >> reproducible. > > Sweeping bugs under the rug is not what I plan on doing; I’m marking > these as expected failures, as opposed to having them continually panic a > machine. Once a ZFS dev takes a look at the issue and resolves them, then the > ZFS dev can remove the “bail” calls I’m adding to the testcases. > Cheers, > -Garrett Yes, disabling tests that fail leads to an ineffectual test suite. A test suite that never has any failures is not very useful. However, there are two factors to take in to account in this context: a) frequent failures can lead users to stop running a test suite leading to further regressions b) long-term repeated failures can desensitize users leading them to ignore *new* failures facilitating further regressions Thus it's really a question of what context you're talking about running the test suite in. For purposes of Jenkins we want full visibility in to what is passing and what is failing and how long this has been going on for. Cheers. -K ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein wrote: > > On 10/24/14 9:45 PM, Garrett Cooper wrote: > >> I think getting tools/regression/zfs working first would be a better idea >> (which means that ZFS developers will need to go debug/fix the issue noted >> in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I'll go >> ahead and commit my fixes to head from my github fork so it runs. >> > If tools/regression/zfs has Kyuafiles that makes it easier to run under the kyua tool, that would greatly facilitate running under Jenkins automation, and would be useful. I notice some of the fixes you are applying to the regression/zfs tests is to make certain tests not run on FreeBSD because they cause known kernel panics such as this: https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html I'm not entirely convinced that this is a good way to "fix" a test. If a test causes a panic, then that's what it does, it it should not be swept under the rug, as Alfred has pointed out. Printing out a warning with a pointer to the PR just before running this type of test is OK, though. > >> Alan also suggested against integrating the test suite as-is, because as >> he said, "Remember, don't run these tests on a production system. They >> WILL cause panics and deadlocks, and they may cause data loss too." >> >> Cheers, >> -Garrett >> > > Wait, we want to sweep those bugs under the rug? What exactly is wrong > with making a test harness that can very easily reproduce a known problem? > The chances are that anyone will dive into it once the bug is easily > reproducible. > I agree with Alfred on this. Even though Alan's test suite may kernel panic or cause problems, there is still value in running it and making the results visible on jenkins.freebsd.org. Running these tests inside a VM which is generated during the build will allow these types of test to run, but still keep the test machine usable, even if the VM gets corrupted while running the tests. If we have test suites for ZFS, but no one runs them, then no one will bother to investigate and fix the bugs. Running the test suites under automation that is visible on jenkins.freebsd.org is going to force developers to see problems much sooner than they do now. Just having the tests in the tree and hoping that people run them and care to look into the problems is not enough. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Oct 24, 2014, at 23:44, Alfred Perlstein wrote: > On 10/24/14 9:45 PM, Garrett Cooper wrote: >> On Oct 24, 2014, at 21:20, Craig Rodrigues wrote: >> >>> On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: >>> >> (2) Creates a bootable UFS image with makefs > any chance zfs will be used as well? > Seconded. There are residual locking issues issues in ZFS. Particularly in the less exercised areas. >>> I think what would be an interesting exercise is to set up a Jenkins job to >>> build >>> and boot a bhyve VM with ZFS, and then run >>> the ZFS Test Suite, ported by Alan Somers to FreeBSD: >>> >>> https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html >>> >>> >>> Are there any volunteers who would like to help set that up and get it >>> running >>> under Jenkins? >> I think getting tools/regression/zfs working first would be a better idea >> (which means that ZFS developers will need to go debug/fix the issue noted >> in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I’ll go ahead >> and commit my fixes to head from my github fork so it runs. >> >> Alan also suggested against integrating the test suite as-is, because as he >> said, "Remember, don't run these tests on a production system. They WILL >> cause panics and deadlocks, and they may cause data loss too.” >> >> Cheers, >> -Garrett > > Wait, we want to sweep those bugs under the rug? What exactly is wrong with > making a test harness that can very easily reproduce a known problem? The > chances are that anyone will dive into it once the bug is easily reproducible. Sweeping bugs under the rug is not what I plan on doing; I’m marking these as expected failures, as opposed to having them continually panic a machine. Once a ZFS dev takes a look at the issue and resolves them, then the ZFS dev can remove the “bail” calls I’m adding to the testcases. Cheers, -Garrett PS FWIW the panics are recent. As I stated in the first bug, it didn’t occur on 10.0-RELEASE. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On 10/24/14 9:45 PM, Garrett Cooper wrote: On Oct 24, 2014, at 21:20, Craig Rodrigues wrote: On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: (2) Creates a bootable UFS image with makefs any chance zfs will be used as well? Seconded. There are residual locking issues issues in ZFS. Particularly in the less exercised areas. I think what would be an interesting exercise is to set up a Jenkins job to build and boot a bhyve VM with ZFS, and then run the ZFS Test Suite, ported by Alan Somers to FreeBSD: https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html Are there any volunteers who would like to help set that up and get it running under Jenkins? I think getting tools/regression/zfs working first would be a better idea (which means that ZFS developers will need to go debug/fix the issue noted in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I’ll go ahead and commit my fixes to head from my github fork so it runs. Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system. They WILL cause panics and deadlocks, and they may cause data loss too.” Cheers, -Garrett Wait, we want to sweep those bugs under the rug? What exactly is wrong with making a test harness that can very easily reproduce a known problem? The chances are that anyone will dive into it once the bug is easily reproducible. -Alfred ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Oct 24, 2014, at 21:45, Garrett Cooper wrote: > On Oct 24, 2014, at 21:20, Craig Rodrigues wrote: > >> On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: >> > (2) Creates a bootable UFS image with makefs any chance zfs will be used as well? >>> >>> Seconded. There are residual locking issues issues in ZFS. >>> Particularly in the less exercised areas. >>> >> >> I think what would be an interesting exercise is to set up a Jenkins job to >> build >> and boot a bhyve VM with ZFS, and then run >> the ZFS Test Suite, ported by Alan Somers to FreeBSD: >> >> https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html >> >> >> Are there any volunteers who would like to help set that up and get it >> running >> under Jenkins? > > I think getting tools/regression/zfs working first would be a better idea > (which means that ZFS developers will need to go debug/fix the issue noted > inhttps://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I’ll go ahead > and commit my fixes to head from my github fork so it runs. Might have helped if I had referenced the appropriate bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191573 (I’m basically fixing https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 right now). Cheers! signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Oct 24, 2014, at 21:20, Craig Rodrigues wrote: > On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: > (2) Creates a bootable UFS image with makefs >>> >>> any chance zfs will be used as well? >>> >> >> Seconded. There are residual locking issues issues in ZFS. >> Particularly in the less exercised areas. >> > > I think what would be an interesting exercise is to set up a Jenkins job to > build > and boot a bhyve VM with ZFS, and then run > the ZFS Test Suite, ported by Alan Somers to FreeBSD: > > https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html > > > Are there any volunteers who would like to help set that up and get it > running > under Jenkins? I think getting tools/regression/zfs working first would be a better idea (which means that ZFS developers will need to go debug/fix the issue noted in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I’ll go ahead and commit my fixes to head from my github fork so it runs. Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system. They WILL cause panics and deadlocks, and they may cause data loss too.” Cheers, -Garrett signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: > >> (2) Creates a bootable UFS image with makefs > > > > any chance zfs will be used as well? > > > > Seconded. There are residual locking issues issues in ZFS. > Particularly in the less exercised areas. > I think what would be an interesting exercise is to set up a Jenkins job to build and boot a bhyve VM with ZFS, and then run the ZFS Test Suite, ported by Alan Somers to FreeBSD: https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html Are there any volunteers who would like to help set that up and get it running under Jenkins? > > Along those same lines, do you have any sort of watchdog in case it > deadlocks? > No. You can look at our scripts used to build and boot the various VM's: https://wiki.freebsd.org/Jenkins#Repositories It's all on github, so if you think you have new scripts to add, or fixes to existing scripts, you can feel free to do a github pull request to contribute. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Thu, Oct 23, 2014 at 10:36 PM, Mateusz Guzik wrote: > > (1) does a buildworld/buildkernel on amd64 when someone checks new > > code into the stable/10 branch > > Is not this excessive? It has not been a problem. For example, when a build occurs on the HEAD in svn, if further commits come in on HEAD, we have Jenkins configured so that it will not trigger another build on HEAD until the build in progress is done. It has been working fine. If no commits > > (2) Creates a bootable UFS image with makefs > > any chance zfs will be used as well? > Sure, we can look at that as well, but as I said earlier, there need to be more bodies working on setting up builds and configurations for this to happen. > > would be nice to run some kind of stress testing. buildworld with a high > -j is an example of a general purpose test. This could be done with > different frequency than regular tests. > > Are you volunteering to write the scripts that incorporate any stress testing that you think should be done? We would welcome any contributions. > Do you have crashdumps configured in case stuff goes wrong? > > No. You can look at our scripts used to build and boot the various VM's: https://wiki.freebsd.org/Jenkins#Repositories It's all on github, so if you think you have new scripts to add, or fixes to existing scripts, you can feel free to do a github pull request to contribute. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On 2014-10-24 00:20, Craig Rodrigues wrote: > On Thu, Oct 23, 2014 at 8:31 PM, Allan Jude wrote: > >> >> At the Cambridge Dev Summit, Xinuous specifically mentioned helping with >> testing and writing tests. Might be good people to reach out to >> > > Are you referring to Eric Le Blan from Xinuous who was interviewed on > BSDNow ( http://www.bsdnow.tv/episodes/2014_08_20-engineering_nginx , > starting at 0:16:38 )? > > -- > Craig > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" > Yes, Eric and the other guy from Xinuous, specifically cited Testing as something they will have to do, much more extensively than we normally do, and would be glad to help out and get the tests upstreamed etc. They plan to support our stable branches for more than 10 years, in very sensitive environments that require stability. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Fri, Oct 24, 2014 at 1:37 PM, Susan Stanziano wrote: > Hello Craig, > > I have been reading through your slides and your website announcement, as > requested, and have a few questions. (I have also subscribed to the > mailing list as suggested). > > (1) The slides indicate that with a FreeBSD10 and lower OS, I should be > able to build tests already included with FreeBSD. I have created a VM on > FreeBSD10.0 but do not have the /etc/make.conf file necessary to build and > install the tests. This may be that I am missing some set-up steps > required but not covered in the slide presentation. > /etc/make.conf is not there by default, but you can create it and add a line: WITH_TESTS="yes" After you do that, you should rebuild everything and install everything as per the steps listed here: https://svnweb.freebsd.org/base/head/Makefile?revision=271168&view=markup#l70 and the tests should be visible under /usr/tests . > > (2) Does the test build require the use of the "Build Flow Plugin" or can > the tests be run independent of the "continuous build" model? > No, "continuous build" and 'Build Flow Plugin" are not required to run the tests. You should focus on building the tests and running them manually before using "continuous integration". > (3) Can the tests only be run under Bhyve as described in the announcement? > > No, you do not need Bhyve to run the tests. Feel free to ask any additional questions on freebsd-test...@freebsd.org, so that we can clarify the steps. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
Hello Craig, I have been reading through your slides and your website announcement, as requested, and have a few questions. (I have also subscribed to the mailing list as suggested). (1) The slides indicate that with a FreeBSD10 and lower OS, I should be able to build tests already included with FreeBSD. I have created a VM on FreeBSD10.0 but do not have the /etc/make.conf file necessary to build and install the tests. This may be that I am missing some set-up steps required but not covered in the slide presentation. (2) Does the test build require the use of the "Build Flow Plugin" or can the tests be run independent of the "continuous build" model? (3) Can the tests only be run under Bhyve as described in the announcement? I would be happy to exchange emails with you or skype as I investigate this work further. Regards, Susan - Original Message - From: "Craig Rodrigues" To: e...@xinuos.com Cc: "Allan Jude" , "John Wolfe" , freebsd-virtualization@freebsd.org, freebsd-test...@freebsd.org, "Susan Stanziano" , "Cheryl Blain" , "Andy Nagle" Sent: Friday, October 24, 2014 7:09:04 AM Subject: Re: Re: Automatically running /usr/tests on stable/10 branch under Jenkins On Fri, Oct 24, 2014 at 1:11 AM, Eric le Blan < e...@xinuos.com > wrote: Allan, Always happy to discuss testing. I believe that Susan Stanziono of Xinuos system test team already spoke to some of you. Please do not hesitate to reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach to Craig. It would be helpful if someone (or some people) from Xinuous could do the following: (1) Subscribe to freebsd-test...@freebsd.org at https://lists.freebsd.org/mailman/listinfo/freebsd-testing to start monitoring the discussions on that list. (2) Review this announcement: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing (3) Review these slides: http://www.slideshare.net/CraigRodrigues1/kyua-jenkins (4) Talk to me at MeetBSD Vendor Summit I have not been given a long timeslot to give a presentation, so reviewing those links before MeetBSD would be helpful. Long term, any testing of FreeBSD will only succeed if there is an ecosystem of people working together and sharing common scripts and information for testing FreeBSD. Any help that Xinuous (or any other company) can provide would be welcome. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Fri, Oct 24, 2014 at 1:11 AM, Eric le Blan wrote: > Allan, > > > > Always happy to discuss testing. I believe that Susan Stanziono of Xinuos > system test team already spoke to some of you. Please do not hesitate to > reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of > operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach > to Craig. It would be helpful if someone (or some people) from Xinuous could do the following: (1) Subscribe to freebsd-test...@freebsd.org at https://lists.freebsd.org/mailman/listinfo/freebsd-testing to start monitoring the discussions on that list. (2) Review this announcement: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing (3) Review these slides: http://www.slideshare.net/CraigRodrigues1/kyua-jenkins (4) Talk to me at MeetBSD Vendor Summit I have not been given a long timeslot to give a presentation, so reviewing those links before MeetBSD would be helpful. Long term, any testing of FreeBSD will only succeed if there is an ecosystem of people working together and sharing common scripts and information for testing FreeBSD. Any help that Xinuous (or any other company) can provide would be welcome. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
RE: Re: Automatically running /usr/tests on stable/10 branch under Jenkins
Allan, Always happy to discuss testing. I believe that Susan Stanziono of Xinuos system test team already spoke to some of you. Please do not hesitate to reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach to Craig. Keep well. Eric Interim CTO * email: <mailto:e...@xinuos.com> e...@xinuos.com Original Message Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Date: Thu, 23 Oct 2014 23:31:39 -0400 From: Allan Jude <mailto:allanj...@freebsd.org> To: freebsd-virtualization@freebsd.org <mailto:freebsd-virtualization@freebsd.org> , freebsd-test...@freebsd.org <mailto:freebsd-test...@freebsd.org> , freebsd-sta...@freebsd.org <mailto:freebsd-sta...@freebsd.org> On 2014-10-23 18:30, Craig Rodrigues wrote: > Garrett Cooper wrote: > >> >> Hi Craig! >> As much as everyone would like to take i386 out to pasture, >> there's a large degree of value in running i386 tests on 11-CURRENT and >> 10-STABLE (I've caught some interesting build bugs and test bugs by running >> on my i386/CURRENT VM). Are there any plans to have i386 executors running >> tests anytime soon (does bhyve support i386?)? >> > > > We (jenkins-ad...@freebsd.org <mailto:jenkins-ad...@freebsd.org> ) have been busy the 5 months since BSDCan > 2014 as > you can see by our status report, which lists what we have done, and what > our future plans are: > > https://www.freebsd.org/news/status/report-2014-07-2014-09.html#Jenkins-Cont inuous-Integration-for-FreeBSD > > Integrating Jenkins with Kyua on amd64 was a major milestone which was > achieved, and > was even mentioned on the Jenkins web site: > > http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing > > Now, where to take this further (such as i386) is an interesting question. > > Personally, I would like to see: >-> integration of automated kyua testing with the FreeBSD release > engineering process >-> more involvement from FreeBSD developers, and even companies such as > EMC/Isilon, >who can: >-> write tests >-> suggest and implement new tests (network, storage, VM, etc.) >-> help with devops maintenance of the existing > jenkins.freebsd.org cluster >-> improve bhyve support in libvirt >-> set up their own Jenkins build environments outside of > FreeBSD, and help test things in their own private environments > > I've had requests for; >-> running tests in bhyve VM's with very small memory footprints > (Adrian Chadd) >-> running tests in MIPS environment (Adrian Chadd) >-> running Java JDK tests (Kip Macy) >-> i386 builds (Garrett Cooper) > > There are a lot of different directions to take this, but without more > people (and companies) pitching in and helping out, > progress is limited by bodies working on the stuff. > > I will be at the MeetBSD Vendor summit (not the conference) Nov. 3-4, so > hopefully > anyone interested in pushing this stuff forward can talk to me there. > > -- > Craig > ___ > freebsd-virtualization@freebsd.org <mailto:freebsd-virtualization@freebsd.org> mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to <mailto:freebsd-virtualization-unsubscr...@freebsd.org> "freebsd-virtualization-unsubscr...@freebsd.org" > At the Cambridge Dev Summit, Xinuous specifically mentioned helping with testing and writing tests. Might be good people to reach out to -- Allan Jude ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
>> (2) Creates a bootable UFS image with makefs > > any chance zfs will be used as well? > Seconded. There are residual locking issues issues in ZFS. Particularly in the less exercised areas. >> (5) Shuts down the bhyve VM > > Do you have crashdumps configured in case stuff goes wrong? Along those same lines, do you have any sort of watchdog in case it deadlocks? -K ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Sun, Oct 12, 2014 at 11:14:45PM -0700, Craig Rodrigues wrote: > Hi, > > I have created this Jenkins job, which you can see a graphical > representation of: > > https://jenkins.freebsd.org/jenkins/view/FreeBSD_src_stable/job/FreeBSD_stable_10/848/BuildGraph/ > It is noted below stuff is being done this for current as well and in that light: > (1) does a buildworld/buildkernel on amd64 when someone checks new > code into the stable/10 branch Is not this excessive? I would suggest every N commits or M hours if there were not enough commits to reach given threshold. Of course no work in there were no commits whatsoever, but then do the work after the first which gets in. > (2) Creates a bootable UFS image with makefs any chance zfs will be used as well? > (3) Boots the image under bhyve > (4) Runs these commands inside the bhyve VM: > > cd /usr/tests > kyua test > kyua report-junit --output=test-output.xml would be nice to run some kind of stress testing. buildworld with a high -j is an example of a general purpose test. This could be done with different frequency than regular tests. > > (5) Shuts down the bhyve VM Do you have crashdumps configured in case stuff goes wrong? > (6) imports test-output.xml into Jenkins. > > You can see a full test report here: > > https://jenkins.freebsd.org/jenkins/job/FreeBSD_stable_10-tests/4/testReport/ > > We already do the same thing for CURRENT. > > Hopefully by running the tests regularly, we can help improve the quality > of FreeBSD. > > > -- > Craig > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" -- Mateusz Guzik ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Thu, Oct 23, 2014 at 8:31 PM, Allan Jude wrote: > > At the Cambridge Dev Summit, Xinuous specifically mentioned helping with > testing and writing tests. Might be good people to reach out to > Are you referring to Eric Le Blan from Xinuous who was interviewed on BSDNow ( http://www.bsdnow.tv/episodes/2014_08_20-engineering_nginx , starting at 0:16:38 )? -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On 2014-10-23 18:30, Craig Rodrigues wrote: > Garrett Cooper wrote: > >> >> Hi Craig! >> As much as everyone would like to take i386 out to pasture, >> there's a large degree of value in running i386 tests on 11-CURRENT and >> 10-STABLE (I've caught some interesting build bugs and test bugs by running >> on my i386/CURRENT VM). Are there any plans to have i386 executors running >> tests anytime soon (does bhyve support i386?)? >> > > > We (jenkins-ad...@freebsd.org) have been busy the 5 months since BSDCan > 2014 as > you can see by our status report, which lists what we have done, and what > our future plans are: > > https://www.freebsd.org/news/status/report-2014-07-2014-09.html#Jenkins-Continuous-Integration-for-FreeBSD > > Integrating Jenkins with Kyua on amd64 was a major milestone which was > achieved, and > was even mentioned on the Jenkins web site: > > http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing > > Now, where to take this further (such as i386) is an interesting question. > > Personally, I would like to see: >-> integration of automated kyua testing with the FreeBSD release > engineering process >-> more involvement from FreeBSD developers, and even companies such as > EMC/Isilon, >who can: >-> write tests >-> suggest and implement new tests (network, storage, VM, etc.) >-> help with devops maintenance of the existing > jenkins.freebsd.org cluster >-> improve bhyve support in libvirt >-> set up their own Jenkins build environments outside of > FreeBSD, and help test things in their own private environments > > I've had requests for; >-> running tests in bhyve VM's with very small memory footprints > (Adrian Chadd) >-> running tests in MIPS environment (Adrian Chadd) >-> running Java JDK tests (Kip Macy) >-> i386 builds (Garrett Cooper) > > There are a lot of different directions to take this, but without more > people (and companies) pitching in and helping out, > progress is limited by bodies working on the stuff. > > I will be at the MeetBSD Vendor summit (not the conference) Nov. 3-4, so > hopefully > anyone interested in pushing this stuff forward can talk to me there. > > -- > Craig > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" > At the Cambridge Dev Summit, Xinuous specifically mentioned helping with testing and writing tests. Might be good people to reach out to -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
Garrett Cooper wrote: > > Hi Craig! > As much as everyone would like to take i386 out to pasture, > there's a large degree of value in running i386 tests on 11-CURRENT and > 10-STABLE (I've caught some interesting build bugs and test bugs by running > on my i386/CURRENT VM). Are there any plans to have i386 executors running > tests anytime soon (does bhyve support i386?)? > We (jenkins-ad...@freebsd.org) have been busy the 5 months since BSDCan 2014 as you can see by our status report, which lists what we have done, and what our future plans are: https://www.freebsd.org/news/status/report-2014-07-2014-09.html#Jenkins-Continuous-Integration-for-FreeBSD Integrating Jenkins with Kyua on amd64 was a major milestone which was achieved, and was even mentioned on the Jenkins web site: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing Now, where to take this further (such as i386) is an interesting question. Personally, I would like to see: -> integration of automated kyua testing with the FreeBSD release engineering process -> more involvement from FreeBSD developers, and even companies such as EMC/Isilon, who can: -> write tests -> suggest and implement new tests (network, storage, VM, etc.) -> help with devops maintenance of the existing jenkins.freebsd.org cluster -> improve bhyve support in libvirt -> set up their own Jenkins build environments outside of FreeBSD, and help test things in their own private environments I've had requests for; -> running tests in bhyve VM's with very small memory footprints (Adrian Chadd) -> running tests in MIPS environment (Adrian Chadd) -> running Java JDK tests (Kip Macy) -> i386 builds (Garrett Cooper) There are a lot of different directions to take this, but without more people (and companies) pitching in and helping out, progress is limited by bodies working on the stuff. I will be at the MeetBSD Vendor summit (not the conference) Nov. 3-4, so hopefully anyone interested in pushing this stuff forward can talk to me there. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
(does bhyve support i386?)? Yes, and also PAE (props to jhb). later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Automatically running /usr/tests on stable/10 branch under Jenkins
On Oct 12, 2014, at 23:14, Craig Rodrigues wrote: > Hi, > > I have created this Jenkins job, which you can see a graphical > representation of: > > https://jenkins.freebsd.org/jenkins/view/FreeBSD_src_stable/job/FreeBSD_stable_10/848/BuildGraph/ … Hi Craig! As much as everyone would like to take i386 out to pasture, there’s a large degree of value in running i386 tests on 11-CURRENT and 10-STABLE (I’ve caught some interesting build bugs and test bugs by running on my i386/CURRENT VM). Are there any plans to have i386 executors running tests anytime soon (does bhyve support i386?)? Thanks! -Garrett signature.asc Description: Message signed with OpenPGP using GPGMail
