On Tue, 11 Nov 2014, Rainer Gerhards wrote:
2014-11-11 4:00 GMT+01:00 singh.janmejay <[email protected]>:
On Tue, Nov 11, 2014 at 7:43 AM, David Lang <[email protected]> wrote:
On Mon, 10 Nov 2014, Thomas D. wrote:
Hi,
On 2014-11-06 08:10, Rainer Gerhards wrote:
Well, you can run the mysql test suite against any mysql server. It is
independent from source. So that's not the best example at all...
Don't you then need root (or similar) credentials to do some very
delicate
database operations like creating databases and tables? I wouldn't
permit
this to be done to my production server.
That's why I asked for running the test suite in a 'sandbox'.
you still need those credentials for your sandbox, right?
If MySQL isn't installed on the system you will need to install the
package (requiring root + distro specific knowledge), if it is installed,
you need to configure users and permissions for the test user (requiring
DB
root equivalent + distro specific knowlege)
Well, finally I installed valgrind and had to re-install glibc with
debugging symbols (another error the test suite logged but did not
detect).
why did you need the debugging symbols?
In Gentoo imdiag is currently only available when compiling with "debug"
USE flag. This will also enable "--enable-valgrind"..
The error message logged in the "udp-msgreduc-vg" log file was something
like
valgrind: Fatal error at startup: a function redirection
valgrind: which is mandatory for this platform-tool combination
valgrind: cannot be set up. Details of the redirection are:
valgrind:
valgrind: [...]
valgrind:
valgrind: Possible fixes: (1, short term): install glibc's debuginfo
valgrind: package on this machine. (2, longer term): ask the
packagers
valgrind: for your Linux distribution to please in future ship a non-
valgrind: stripped ld.so (or whatever the dynamic linker .so is
called)
valgrind: that exports the above-named function using the standard
valgrind: calling conventions for this platform.
valgrind:
valgrind: Cannot continue -- exiting now. Sorry.
But let me be clear. From this thread, it looks like these should be
our
project priorities:
1. make the testbench fully automatic and self-contained
2. create unit tests
3. create more and better integration tests
4. document
5. fix bugs
6. develop new features
Given the fact that I am I and not a big team like at MySQL, I don't
believe this is the right order of priorities.
In reality, it is me alone doing >95% of all work, and funding for this
work is almost solely by Adiscon (and only partially backed by paying
customers).
Asking this question shows me that you understand my point. Great! ;)
Yes, I am voting for changing the priority order like you said.
Don't get me wrong. I am not saying that you MUST change anything. I am
only answering the question "If you could change something in the
project, what would you change?" ... and I would change the priority to
improve the current project quality using automatic tests.
I disagree that 1, 2, and 3 are more important than 4 and 5 (where 6
lives
is subject to a lot of debate)
you could completely halt all documentation, bugfixing, and new features
for a year while creating tests to cover everything.
Testing is useful, but the end purpose of the software is not to pass
tests, but rather to provide capabilities to the user.
As an extreme example, I know of a company that lost a month of
development and missed having the product ready for a national trade show
because they got so caught up in writing tests for everything that a test
that didn't work right when the software was being shutdown (after which
it
would be reset for the next test) HAD to be written before anything more
could be done, and there was an upstream bug in one component that caused
this exit to never work cleanly. Don't fall into a similar trap.
+1 tests are just means to an end, we should be disciplined to write them
as we go, but every project has corners(and sometimes large portions) that
are ill-tested or worse. Paralysing development completely to test-cover
them in one go doesn't sound like the best plan.
FWIW, here is something that has worked for teams I have been on in the
past. We were aware of which areas were not test-covered, bug-prone,
hard-to-read etc. When we were to touch anything in those areas (like
adding new features that touched those parts of code, or fix a bug etc), we
used to test-cover directly and indirectly affected portions of that code,
and then refactor them heavily to bring it up to the mark.
I try to follow that same paradigm since quite a while. I have to admit it
depends on workload, and some test are really *extremely* hard to create.
For example, when you want to test out thing with the journal or imuxsock,
you need to create a total mock environment and be extremely precise in
when, how and where messages are injected and processed. Plus you need to
make sure no other component on the running system interferes. In general,
this means you must have a very specialised VM in place for that very
purpose. I frankly admit that I have not added some tests simply for that
reason that creating a decent test would have taken a week or two while the
actual bugfix just took half an hour. In many cases, though, creating the
tests takes muuuuuuuuch longer than creating the feature (in case of
connector modules).
So far, I thought it would be be better to provide a working connector and
ask those folks that actually know what it does help debug it. That
philosophy probably needs to come to an end if we insist on tests, as the
system setup alone is often over my head - or what I could do with decent
effort. I often have a day to write a connector if someone gives me
instructions, but I usually don't have the 1..4 weeks it takes to learn,
install, automate a full subsystem use.
This is a perfect example of the issue.
Tests are a means to an end, not the end themselves.
The purpose of the test is to help get working code in the hands of users, so
when the overhead of creating tests interferes too badly with ongoing
development, the tests can (and should) be skipped.
It's good to have the practice of cleaning up anything you touch. But it's bad
to make it a ridged rule that you _have_ to fully cleanup anything you touch,
because that will discourage someone from touching an area that need cleanup due
to the huge amount of work that it entails.
So it's best to have the practice to always leave the code in better shape than
it started, and do as much cleanup of anything that you touch as time allows.
Without requiring full cleanup in all cases.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
