Re: [waffle] CI, static analisys, tests and waffle
On Sat 05 Sep 2015, Emil Velikov wrote: > On 3 September 2015 at 21:27, Chad Versace wrote: > > Yes, I think this is worth doing. I prefer to see the cleanups arrive in > > this order, but it's not a strong preference: > > > > *. Fix bugs and leaks as they're discovered. > > 1. Replace waffle_test with cmocka. > > 2. Split `make check-func` into a separate target for each platform. > Considering the churn during the conversion, we might want to do it in > one step, namely: > 1. Update cmocka (fix integration, add missing include) > 2. Update existing users to cmocka 1.x > 3. Transition/refactor each platform to a separate cmocka based test > `gl_basic_test_$(platform)' > > > 3. Add `make check` and the GBM tests to some CI system. > I don't see any GBM tests in waffle. Am I missing something or you > have something locally ? I misspoke. I used to have GBM tests locally, but never pushed them because `make check-func` was not split per platform. So, that gives us a new task, to be done after refactoring "each platform to a separate cmocka based test": add some GBM tests. ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
On 3 September 2015 at 21:27, Chad Versace wrote: > On Tue 01 Sep 2015, Emil Velikov wrote: >> Hello all, >> >> Not too long ago I had the crazy idea of using some of the free >> utilities out there with waffle. >> >> For example Coverity provides a free scan to open-source projects, >> which nicely integrates with Travis-CI, >> The latter of which supporting Linux and MacOS, while an alternative >> is available for Windows (AppVeyor). >> >> I Travis + Linux + Coverity a try and things turned out ok. Small >> catch is the wayland functionality tests fail, as they run in a VM, >> while (the old) mesa/wayland requires hardware device. >> We can split these into separate gl_basic_foo_test (foo being >> glx,wayland etc.) or just silence the whole lot. > > Running Linux + Coverity in CI system is a good idea. > > As for Waffle's functional tests, we should separate them out so it's > possible to run tests for individual platforms. For example, run the GBM > tests but not the Wayland tests. Then we can configure the CI to run > just the tests that will actually work. > > Being able to run the GBM tests in CI should find real bugs. As Mark > pointed out, Mesa commits have broken Waffle + Piglit before, and GBM > tests would have caught that. > >> I have also moved the gl_basic_test to cmocka (after updating the >> latter to 1.0.1) and nuked waffle_test. With this each test has it's >> own setup/teardown and things are handled correctly, fixing memory >> leaks. > > Nice. I look forward to the replacement of waffle_test with cmocka. > >> Overall I'm soliciting feedback on the topic(s) as some of the patches >> are rather rough and/or complete. >> Do you think that these are worthy additions and which one would you >> like to see first :-) > > Yes, I think this is worth doing. I prefer to see the cleanups arrive in > this order, but it's not a strong preference: > > *. Fix bugs and leaks as they're discovered. > 1. Replace waffle_test with cmocka. > 2. Split `make check-func` into a separate target for each platform. Considering the churn during the conversion, we might want to do it in one step, namely: 1. Update cmocka (fix integration, add missing include) 2. Update existing users to cmocka 1.x 3. Transition/refactor each platform to a separate cmocka based test `gl_basic_test_$(platform)' > 3. Add `make check` and the GBM tests to some CI system. I don't see any GBM tests in waffle. Am I missing something or you have something locally ? Thanks Emil ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
On Tue 01 Sep 2015, Emil Velikov wrote: > Hello all, > > Not too long ago I had the crazy idea of using some of the free > utilities out there with waffle. > > For example Coverity provides a free scan to open-source projects, > which nicely integrates with Travis-CI, > The latter of which supporting Linux and MacOS, while an alternative > is available for Windows (AppVeyor). > > I Travis + Linux + Coverity a try and things turned out ok. Small > catch is the wayland functionality tests fail, as they run in a VM, > while (the old) mesa/wayland requires hardware device. > We can split these into separate gl_basic_foo_test (foo being > glx,wayland etc.) or just silence the whole lot. Running Linux + Coverity in CI system is a good idea. As for Waffle's functional tests, we should separate them out so it's possible to run tests for individual platforms. For example, run the GBM tests but not the Wayland tests. Then we can configure the CI to run just the tests that will actually work. Being able to run the GBM tests in CI should find real bugs. As Mark pointed out, Mesa commits have broken Waffle + Piglit before, and GBM tests would have caught that. > I have also moved the gl_basic_test to cmocka (after updating the > latter to 1.0.1) and nuked waffle_test. With this each test has it's > own setup/teardown and things are handled correctly, fixing memory > leaks. Nice. I look forward to the replacement of waffle_test with cmocka. > Overall I'm soliciting feedback on the topic(s) as some of the patches > are rather rough and/or complete. > Do you think that these are worthy additions and which one would you > like to see first :-) Yes, I think this is worth doing. I prefer to see the cleanups arrive in this order, but it's not a strong preference: *. Fix bugs and leaks as they're discovered. 1. Replace waffle_test with cmocka. 2. Split `make check-func` into a separate target for each platform. 3. Add `make check` and the GBM tests to some CI system. ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
Emil Velikov writes: >> We have experienced cases where a mesa commit broke waffle, and would >> like to prevent this. >> > Hopefully you got to reporting it to mesa/waffle ML ? I usually just contact the committer on IRC and get it corrected. I'll send to the list and open a bug if they aren't on IRC. ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
Hello Mark, On 2 September 2015 at 21:38, Mark Janes wrote: > Emil Velikov writes: > >> On 1 September 2015 at 20:01, Dylan Baker wrote: >>> Hi Emil, >>> >>> Mark and I had talked about testing Waffle in our jenkins setup, but >>> it just wasn't feasable for some reason I can't remember, I've CC'd >>> him, hopefully he'll remember. > > I'm happy to run waffle tests in our CI. My recollection is that the > tests won't succeed without a display server. Chad had an action to > produce a way to execute the tests which are not display-dependent. > Iirc all the functionality (check-func) tests require some form of a display. If you/Chad has a bit more information on the topic I won't mind looking into it. > We have experienced cases where a mesa commit broke waffle, and would > like to prevent this. > Hopefully you got to reporting it to mesa/waffle ML ? >> I'm suspecting that the wayland functionality tests were failing, as >> in my case. >> >> Either way using your jenkins setup does not have to be the only >> solution for CI. After all anyone willing to contribute can check the >> issues, without chasing you guys. After all, you do have some downtime >> - sleep and stuff :P. >> >> Which brings the question - are the results (build tests, piglit runs) >> publicly available somewhere ? Last time I've looked one needed to be >> hooked up to the corporate network to access them :'( Is there any >> plans/ideas about changing this ? > > I would like to improve this, for "important" external developers like > yourself. We haven't implemented sufficient security around our Jenkins > instance to allow it to be accessible outside Intel. However, it should > be possible to improve the notification email to the point where > regressions are clearly identifiable. > > We could set up a build to watch your remote, and trigger a build > whenever you push to a specified branch. Later, you get an email > detailing any breakage vs master. > > Would you be interested in this type of setup? > I still cannot shake the habit of build testing in various ways on my Arch, so I'm not sure if it will be worth it. Having/working on half dozen branches doesn't help much either. Big thanks for the offer though. I might come back to it in the not too distant future. -Emil ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
Emil Velikov writes: > On 1 September 2015 at 20:01, Dylan Baker wrote: >> Hi Emil, >> >> Mark and I had talked about testing Waffle in our jenkins setup, but >> it just wasn't feasable for some reason I can't remember, I've CC'd >> him, hopefully he'll remember. I'm happy to run waffle tests in our CI. My recollection is that the tests won't succeed without a display server. Chad had an action to produce a way to execute the tests which are not display-dependent. We have experienced cases where a mesa commit broke waffle, and would like to prevent this. > I'm suspecting that the wayland functionality tests were failing, as > in my case. > > Either way using your jenkins setup does not have to be the only > solution for CI. After all anyone willing to contribute can check the > issues, without chasing you guys. After all, you do have some downtime > - sleep and stuff :P. > > Which brings the question - are the results (build tests, piglit runs) > publicly available somewhere ? Last time I've looked one needed to be > hooked up to the corporate network to access them :'( Is there any > plans/ideas about changing this ? I would like to improve this, for "important" external developers like yourself. We haven't implemented sufficient security around our Jenkins instance to allow it to be accessible outside Intel. However, it should be possible to improve the notification email to the point where regressions are clearly identifiable. We could set up a build to watch your remote, and trigger a build whenever you push to a specified branch. Later, you get an email detailing any breakage vs master. Would you be interested in this type of setup? -Mark ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
On 1 September 2015 at 20:01, Dylan Baker wrote: > Hi Emil, > > Mark and I had talked about testing Waffle in our jenkins setup, but > it just wasn't feasable for some reason I can't remember, I've CC'd > him, hopefully he'll remember. > I'm suspecting that the wayland functionality tests were failing, as in my case. Either way using your jenkins setup does not have to be the only solution for CI. After all anyone willing to contribute can check the issues, without chasing you guys. After all, you do have some downtime - sleep and stuff :P. Which brings the question - are the results (build tests, piglit runs) publicly available somewhere ? Last time I've looked one needed to be hooked up to the corporate network to access them :'( Is there any plans/ideas about changing this ? Thanks Emil ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
Re: [waffle] CI, static analisys, tests and waffle
Hi Emil, Mark and I had talked about testing Waffle in our jenkins setup, but it just wasn't feasable for some reason I can't remember, I've CC'd him, hopefully he'll remember. Dylan On Tue, Sep 01, 2015 at 01:45:32PM +0100, Emil Velikov wrote: > Hello all, > > Not too long ago I had the crazy idea of using some of the free > utilities out there with waffle. > > For example Coverity provides a free scan to open-source projects, > which nicely integrates with Travis-CI, > The latter of which supporting Linux and MacOS, while an alternative > is available for Windows (AppVeyor). > > I Travis + Linux + Coverity a try and things turned out ok. Small > catch is the wayland functionality tests fail, as they run in a VM, > while (the old) mesa/wayland requires hardware device. > We can split these into separate gl_basic_foo_test (foo being > glx,wayland etc.) or just silence the whole lot. > > I have also moved the gl_basic_test to cmocka (after updating the > latter to 1.0.1) and nuked waffle_test. With this each test has it's > own setup/teardown and things are handled correctly, fixing memory > leaks. > > Overall I'm soliciting feedback on the topic(s) as some of the patches > are rather rough and/or complete. > Do you think that these are worthy additions and which one would you > like to see first :-) > > Thanks > Emil > ___ > waffle mailing list > waffle@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/waffle signature.asc Description: Digital signature ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
[waffle] CI, static analisys, tests and waffle
Hello all, Not too long ago I had the crazy idea of using some of the free utilities out there with waffle. For example Coverity provides a free scan to open-source projects, which nicely integrates with Travis-CI, The latter of which supporting Linux and MacOS, while an alternative is available for Windows (AppVeyor). I Travis + Linux + Coverity a try and things turned out ok. Small catch is the wayland functionality tests fail, as they run in a VM, while (the old) mesa/wayland requires hardware device. We can split these into separate gl_basic_foo_test (foo being glx,wayland etc.) or just silence the whole lot. I have also moved the gl_basic_test to cmocka (after updating the latter to 1.0.1) and nuked waffle_test. With this each test has it's own setup/teardown and things are handled correctly, fixing memory leaks. Overall I'm soliciting feedback on the topic(s) as some of the patches are rather rough and/or complete. Do you think that these are worthy additions and which one would you like to see first :-) Thanks Emil ___ waffle mailing list waffle@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/waffle
