Hi, Ingo On 22:19 Thu 13 Mar , Ingo Schwarze wrote: > Hi Sergey, > > Sergey Bronnikov wrote on Thu, Mar 13, 2014 at 10:08:40PM +0400: > > > We need more and more tests to cover OpenBSD code as much as possible. > Sure, so improve or write some tests.
amount of OpenBSD developers is about 70, and it is really silly to don't reuse existed tests from another projects. It can allow to free time of developers for another work, for example new features :) If you are right then OpenBSD developers should rewrite piglit OpenGL tests to have regression testing for Intel/ATI DRM. I believe no one from developers don't want ot do double work and it is better to reuse piglit. > > OpenBSD developers has unit and regression tests in source tree > > (src/regress) but you cannot use them without having CVS repo > > on you computer. > > That's a non-issue. Without a source tree, you can't do anything > in the first place, in particular not change the code, so the code > is already safe from your interference. You look from point of view when user is equal to developer. But what about case when man can help with running tests on specific hardware but cannot help with fixing potential problems itself? According to you he should download sources and then run it. Too complicated for task with running tests. Decreasing learning curve here can help project to get more feedback from users. IMO it is the same as integration bsd.ports.mk with tests from ports: you run 'make test' and don't dig into guts of tests inside port while tests passed. But it is criteria for workness of port. It should not be commited when tests failed. > > I want to port as much as possible opensource tests used on Linux > > and FreeBSD to OpenBSD and give developers and volunteers ability > > for simple and easy run these tests on OpenBSD. > > When i looked at test suites elsewhere, they were often overengineered > to the point that i wouldn't want to use them at all. Given a > typical framework, it can be terribly difficult to find out what > actually went wrong when the framework reports an error. Ingo, it usually depends on developers implemented that framework. Isn't it? I am totally agree that it is better when simple. But sometimes frameworks are necessary evil. Look at 'smtpscript' framework (https://github.com/poolpOrg/smtpscript) It doesn't look as complex test. But without framework itself SMTP functional test can be more complex and less flexible. >From my experience sometimes tests contains too much linuxisms: - trinity syscall fuzzer (http://codemonkey.org.uk/projects/trinity/) or bloat by design: - stress2 (https://people.freebsd.org/~pho/stress) Need to understand cost of efforts for porting and maintenance of test and profit from that test. > Then again, if somebody finds some tests that are really good *and > simple*, sure, bring them in to src/regress. > > > And I am on that way: > > - kyua and dependencies (atf, kyua-testers, lutok) > > It is a test framework used in NetBSD and FreeBSD maintained > > by Julio Merino. > > I heard about that one during BSDCan 2011: > > http://www.bsdcan.org/2011/schedule/events/223.en.html > > It was incredibly bloated and overengineered already at that point > in time. I didn't look since then, but usually, projects that are > overengineered to begin with usually get worse as time goes by, not > better. > > Would be nice to have kyua ported to OpenBSD. > > Sure, having a port can do little harm, even if only a few people > use a port, it may be useful. Maybe somebody will use it and find > and fix a few bugs. > > However, to get on with OpenBSD unit testing, that is irrelevant. > The test framework we have now is quite simple, and even that one > is used too infrequently, except in very few areas that are actively > maintained. If people aren't even using *that*, a more complicated > framework will get used even less. Anything involving ports has no > chance to get used at all by a relevant number of developers, as far > as i can see. > > Regarding test automation: That's completely pointless in my opinion. > It's yet more layers of complexity and abstraction, the reports are > almost invariably ignored and unintelligible, and if you try to enforce > its usage, you teach developers to provide pseudo-tests providing > formal coverage but not actually testing what's relevant. I don't know how developers run tests on different machines, but suppose it now looks as somehow manual action and it will be routine when you have more than several machines. Don't see something bad to have automation in that place. Test automation is necessary evil here. Elevator is also too complicated than ladder, but in some cases you are using a ladder, in other - elevator :) The same with automation. > To summarize, if you are interested in improving OpenBSD regression > tests, i'd suggest working on *actual tests*, not bloated frameworks. > One could look at the existing tests, clean them up such that they > actually run through, do not generate bogus errors, improve the style > in some places such that they are as simple as possible. One could > also write new tests for areas not yet covered, trying to maintain > a simple, if possible uniform style. > > For a framework, bsd.regress.mk (122 lines of code right now) is about > the right size. Someone working on actual tests will almost certainly > come up with some improvements to bsd.regress.mk, too, removing some > bloat here and there and maybe, sparingly, add a missing feature > now and then. Just like rc.subr(8): Start small, stay small, and > people will understand what's going on. That's the OpenBSD way. Totally agree. It are things for which I love OpenBSD. Ingo, thank you for oppinions. I will correct my actions directed to improving tests in OpenBSD according to your comments. > Yours, > Ingo
