2014-11-11 4:00 GMT+01:00 singh.janmejay <[email protected]>: > On Tue, Nov 11, 2014 at 7:43 AM, David Lang <[email protected]> wrote: > > > On Mon, 10 Nov 2014, Thomas D. wrote: > > > > Hi, > >> > >> On 2014-11-06 08:10, Rainer Gerhards wrote: > >> > >>> Well, you can run the mysql test suite against any mysql server. It is > >>>> independent from source. So that's not the best example at all... > >>>> > >>> > >>> Don't you then need root (or similar) credentials to do some very > >>> delicate > >>> database operations like creating databases and tables? I wouldn't > permit > >>> this to be done to my production server. > >>> > >> > >> That's why I asked for running the test suite in a 'sandbox'. > >> > > > > you still need those credentials for your sandbox, right? > > > > If MySQL isn't installed on the system you will need to install the > > package (requiring root + distro specific knowledge), if it is installed, > > you need to configure users and permissions for the test user (requiring > DB > > root equivalent + distro specific knowlege) > > > > > > > > Well, finally I installed valgrind and had to re-install glibc with > >>>> debugging symbols (another error the test suite logged but did not > >>>> detect). > >>>> > >>> > >>> why did you need the debugging symbols? > >>> > >> > >> In Gentoo imdiag is currently only available when compiling with "debug" > >> USE flag. This will also enable "--enable-valgrind".. > >> > >> The error message logged in the "udp-msgreduc-vg" log file was something > >> like > >> > >> valgrind: Fatal error at startup: a function redirection > >>> valgrind: which is mandatory for this platform-tool combination > >>> valgrind: cannot be set up. Details of the redirection are: > >>> valgrind: > >>> valgrind: [...] > >>> valgrind: > >>> valgrind: Possible fixes: (1, short term): install glibc's debuginfo > >>> valgrind: package on this machine. (2, longer term): ask the > packagers > >>> valgrind: for your Linux distribution to please in future ship a non- > >>> valgrind: stripped ld.so (or whatever the dynamic linker .so is > called) > >>> valgrind: that exports the above-named function using the standard > >>> valgrind: calling conventions for this platform. > >>> valgrind: > >>> valgrind: Cannot continue -- exiting now. Sorry. > >>> > >> > >> > >> > >> But let me be clear. From this thread, it looks like these should be > our > >>> project priorities: > >>> > >>> 1. make the testbench fully automatic and self-contained > >>> 2. create unit tests > >>> 3. create more and better integration tests > >>> 4. document > >>> 5. fix bugs > >>> 6. develop new features > >>> > >>> Given the fact that I am I and not a big team like at MySQL, I don't > >>> believe this is the right order of priorities. > >>> > >>> In reality, it is me alone doing >95% of all work, and funding for this > >>> work is almost solely by Adiscon (and only partially backed by paying > >>> customers). > >>> > >> > >> Asking this question shows me that you understand my point. Great! ;) > >> > >> Yes, I am voting for changing the priority order like you said. > >> > >> Don't get me wrong. I am not saying that you MUST change anything. I am > >> only answering the question "If you could change something in the > >> project, what would you change?" ... and I would change the priority to > >> improve the current project quality using automatic tests. > >> > > > > I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6 > lives > > is subject to a lot of debate) > > > > you could completely halt all documentation, bugfixing, and new features > > for a year while creating tests to cover everything. > > > > Testing is useful, but the end purpose of the software is not to pass > > tests, but rather to provide capabilities to the user. > > > > As an extreme example, I know of a company that lost a month of > > development and missed having the product ready for a national trade show > > because they got so caught up in writing tests for everything that a test > > that didn't work right when the software was being shutdown (after which > it > > would be reset for the next test) HAD to be written before anything more > > could be done, and there was an upstream bug in one component that caused > > this exit to never work cleanly. Don't fall into a similar trap. > > > > +1 tests are just means to an end, we should be disciplined to write them > as we go, but every project has corners(and sometimes large portions) that > are ill-tested or worse. Paralysing development completely to test-cover > them in one go doesn't sound like the best plan. > > FWIW, here is something that has worked for teams I have been on in the > past. We were aware of which areas were not test-covered, bug-prone, > hard-to-read etc. When we were to touch anything in those areas (like > adding new features that touched those parts of code, or fix a bug etc), we > used to test-cover directly and indirectly affected portions of that code, > and then refactor them heavily to bring it up to the mark. >
I try to follow that same paradigm since quite a while. I have to admit it depends on workload, and some test are really *extremely* hard to create. For example, when you want to test out thing with the journal or imuxsock, you need to create a total mock environment and be extremely precise in when, how and where messages are injected and processed. Plus you need to make sure no other component on the running system interferes. In general, this means you must have a very specialised VM in place for that very purpose. I frankly admit that I have not added some tests simply for that reason that creating a decent test would have taken a week or two while the actual bugfix just took half an hour. In many cases, though, creating the tests takes muuuuuuuuch longer than creating the feature (in case of connector modules). So far, I thought it would be be better to provide a working connector and ask those folks that actually know what it does help debug it. That philosophy probably needs to come to an end if we insist on tests, as the system setup alone is often over my head - or what I could do with decent effort. I often have a day to write a connector if someone gives me instructions, but I usually don't have the 1..4 weeks it takes to learn, install, automate a full subsystem use. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
