It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then, run those test with a command like 'run-tests'
On Wed, Sep 3, 2014 at 6:06 AM, Martin Preisler <mprei...@redhat.com> wrote: > ----- Original Message ----- > > From: "Shawn Wells" <sh...@redhat.com> > > To: "scap-security-guide" <scap-security-guide@lists.fedorahosted.org> > > Sent: Wednesday, September 3, 2014 6:31:00 AM > > Subject: Test run of Jenkins (CI tool) > > > > Also in the vein of automation, went ahead and installed an instance of > > Jenkins and connected it to SSG: > > > > http://jenkins.ssgproject.org:8080/ > > > > Not meant to be permanent right now, but wanted to get something stood > > up for us to play with. Martin stood one up for the OpenSCAP > > interpreter, which was the source of thinking to stand up a sandbox for > > SSG too. There really isn't much to see in the UI, but one thing it does > > is integrate into GitHub. Whenever someone issues a pull request, > > Jenkins will automagically detect that, apply the patch(s), and run > > "make validate." > > I wanted to create a testing Jenkins instance for all things OpenSCAP. > There are fixed costs in maintaining the master and all the slaves. > You have to take care of the machines, keep them updated, etc... You > have to keep updating Jenkins itself and be on top of plugin updates. > > In my opinion it makes no sense to have 2 Jenkins servers for projects > so closely related. I would prefer to pool resources. > > Unfortunately we can't just host our Jenkins on OpenShift because we need > multiple slaves for RHEL 5, 6, 7 and Fedora. And the storage space > available > is not enough for our use-case. > > > Check out https://github.com/OpenSCAP/scap-security-guide/pull/45 for an > > example. Or for the lazy :-), see below: > > The integration is really cool. I planned to make the Jenkins instance > public after a testing period and do exactly this. > > > Does anyone have experience with Jenkins, thoughts on how we could > > begin building out unit tests, or really any thoughts on usefulness > > beyond "make validate" on pull requests? > > I was experimenting with docker for this purpose. The idea was to start > with EL6 image, put ssg and a test script inside. Then you can call > docker run and get the results. The docker instance is deleted after > each run so there is no risk of side effects. > > The idea is to do the following: > 1) pull ssg HEAD > > 2) make content > > 3) generate a profile with just one Rule selected - the tested rule > (a simple python script can easily do this) > > 4) evaluate that profile, compare the results with expected results on a > freshly installed RHEL6 > > 5) anti-remediate - break the configuration (if applicable) > > 6) evaluate again, check that the result is "fail" (if applicable) > > 7) remediate - fix the configuration (if applicable) > > 8) evaluate for the third time, check that the rule passes > > *) as an added bonus we could run whatever we are testing, let it > parse the config files and do another sanity check (sshd for example) > > This should provide repeatable tests for all the rules. > > Unfortunately we can't do this on OpenShift (AFAIK). We need real > slaves with docker-io installed on them. Another problem is that we can > not test every single rule with this approach. "process test" comes to > mind. Or even systemd tests. However we should be able to test most of > the rules. > > -- > Martin Preisler > -- > SCAP Security Guide mailing list > scap-security-guide@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide > https://github.com/OpenSCAP/scap-security-guide/
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide https://github.com/OpenSCAP/scap-security-guide/
