Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Re: How to implement SIGHUP tests (Stephen Morris)
2. Re: How to implement SIGHUP tests (Marcin Siodelski)
3. Re: How to implement SIGHUP tests (Tomek Mrugalski)
4. Re: How to implement SIGHUP tests (Marcin Siodelski)
----------------------------------------------------------------------
Message: 1
Date: Tue, 27 May 2014 13:11:22 +0100
From: Stephen Morris <[email protected]>
To: [email protected]
Subject: Re: How to implement SIGHUP tests
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1
On 27/05/14 10:19, Marcin Siodelski wrote:
> 1. Shell is there on every OS and doesn't cause any additional
> dependencies for Kea to be tested, whilst this is not true for
> Python. Even if we can detect that there is no Python on the system
> and not run these tests, the down side is that we don't run these
> tests.
That's true.
> 2. One of the tasks for 0.9 is to implement a script that controls
> the startup of Kea. We might not have decided on what language it
> will be written in, but I also don't think anybody wanted Python as
> we are in the process of removing Python-based code from Kea. The
> shell script makes a lot of sense to me and if so, I would like to
> test Kea startup and reconfiguration using the same approach as
> this script would use.
We would want to test that the script starts and stops Kea; we don't
need to test Kea in the same way.
>
> 3. As Tomek pointed out, the shell scripts don't provide any
> framework for testing (nothing similar to gtest) which makes it
> more reasonable to implement our tests in C++ and within gtest
> framework. But, python based tests that we already have don't have
> any framework either and we used to live with it. So, is lack of
> framework really a problem?
I thought Python did have a unit test framework:
https://docs.python.org/release/2.6/library/unittest.html
... and that BIND 10 used it.
>
> 4. Implementing tests in C++ would certainly make tests consistent
> with other tests we have and I am in general in favor of that. But,
> C++ is not the easiest environment to deal with the type of tests
> we are talking about. Dealing with spawning background processes,
> grepping log files, creating configuration files is something that
> is natural in shell because shell is basically for this purpose
> mainly. I don't disagree that it is doable in C++: you can fork,
> execlp, read file in C++ (harder to grep but doable) but it doesn't
> make code implementation and maintenance any easier.
I agree that searching log files is awkward, but there is no need to
do that. The logging framework permits log messages to be sent to
stdout (or stderr) and that can be read by the parent process via a pipe.
I would think a C++ class for handing subprocesses that has methods like:
<constructor>(command)
- start a process in the background and store the PID. Connect stdout
of the output process to a pipe between it and the parent process.
isSubrocessRunning()
- return true or false if the background process is running.
searchOutput(text, timeout)
- wait a maximum of "timeout" for the specified text to be output by
the background process. As each log message has a unique ID, passing
that ID as an argument to this method is a way of checking whether the
message has been output.
sendSignal(signal)
- send the specified signal to the background process.
waitForProcessTermination(timeout)
- wait up to "timeout" for the process to terminate.
... would be sufficient to do the testing
We would only have to write this once and the advantage about C++ is
that it would naturally fit into the existing unit test framework.
There are already some unit testing utilities in Kea, in
src/lib/util/unittests - this would just be another.
> Tomek's major objections to this seem to be:
>
> 1. We already have tests in Python and they should be portable
> with Python 2.x if people hate Python 3 and its incompatibilities
>
> 2. If you run unit tests you're probably a developer and you can
> afford installing Python 2.x.
>
> 3. There is no test framework for shell but there is for C++. This
> is in favor of writing a test in C++.
>
> 4. We are already using Forge that is written in Python so we could
> move some tests between Kea and Forge.
>
> And Tomek, has a list of preferred languages for these tests: 1.
> C++ 2. Python 3. Shell
I agree with Tomek here.
One of the problems with BIND 10 was the use of two languages because
few people were masters of both. Unfortunately, we can't get rid of
Python code altogether: Lettuce is written in Python. However,
Lettuce is all about writing unit tests in a natural form of English.
To my mind, that far outweighs the inconvenience of maintaining the
Python needed to implement Lettuce features.
> I would really like to have input from others why you like or
> dislike a certain approach because it may affect the way we develop
> similar tests in the future. Apart from 3 test cases we are talking
> about, there will be 6 additional test cases for other components:
> 3 for D2 and 3 for v4. It is better to alter the course now, not
> when they are implemented. Having said that, I should have
> initiated this discussion prior to implementing tests for v6, but
> better now than later.
In summary, I think if we were to implement a C++ class for handling
subprocesses that output to stdout, we could do a lot of this type of
testing in the C++ unit tests.
Stephen
------------------------------
Message: 2
Date: Tue, 27 May 2014 14:38:32 +0200
From: Marcin Siodelski <[email protected]>
To: Stephen Morris <[email protected]>, [email protected]
Subject: Re: How to implement SIGHUP tests
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1
On 27/05/14 14:11, Stephen Morris wrote:
> On 27/05/14 10:19, Marcin Siodelski wrote:
>
>> 1. Shell is there on every OS and doesn't cause any additional
>> dependencies for Kea to be tested, whilst this is not true for
>> Python. Even if we can detect that there is no Python on the system
>> and not run these tests, the down side is that we don't run these
>> tests.
>
> That's true.
>
>
>> 2. One of the tasks for 0.9 is to implement a script that controls
>> the startup of Kea. We might not have decided on what language it
>> will be written in, but I also don't think anybody wanted Python as
>> we are in the process of removing Python-based code from Kea. The
>> shell script makes a lot of sense to me and if so, I would like to
>> test Kea startup and reconfiguration using the same approach as
>> this script would use.
>
> We would want to test that the script starts and stops Kea; we don't
> need to test Kea in the same way.
>
Sure, but the more real life test scenario it is, the better. When you
test the script and it fails you don't know whether it is a bug in the
script or in the server. Having a test that validates the server
directly eliminates the influence of the script and makes it easier to
identify the issue.
>>
>> 3. As Tomek pointed out, the shell scripts don't provide any
>> framework for testing (nothing similar to gtest) which makes it
>> more reasonable to implement our tests in C++ and within gtest
>> framework. But, python based tests that we already have don't have
>> any framework either and we used to live with it. So, is lack of
>> framework really a problem?
>
> I thought Python did have a unit test framework:
>
> https://docs.python.org/release/2.6/library/unittest.html
>
> ... and that BIND 10 used it.
>
Ok, point taken.
>>
>> 4. Implementing tests in C++ would certainly make tests consistent
>> with other tests we have and I am in general in favor of that. But,
>> C++ is not the easiest environment to deal with the type of tests
>> we are talking about. Dealing with spawning background processes,
>> grepping log files, creating configuration files is something that
>> is natural in shell because shell is basically for this purpose
>> mainly. I don't disagree that it is doable in C++: you can fork,
>> execlp, read file in C++ (harder to grep but doable) but it doesn't
>> make code implementation and maintenance any easier.
>
> I agree that searching log files is awkward, but there is no need to
> do that. The logging framework permits log messages to be sent to
> stdout (or stderr) and that can be read by the parent process via a pipe.
Whether I get it through a pipe or by reading a file it still seems that
it is easier to parse the output with a script, rather than with C++.
This includes things like counting how many occurrences of some log
message appeared so far - e.g. how many reconfigurations.
>
> I would think a C++ class for handing subprocesses that has methods like:
>
> <constructor>(command)
> - start a process in the background and store the PID. Connect stdout
> of the output process to a pipe between it and the parent process.
>
> isSubrocessRunning()
> - return true or false if the background process is running.
>
> searchOutput(text, timeout)
> - wait a maximum of "timeout" for the specified text to be output by
> the background process. As each log message has a unique ID, passing
> that ID as an argument to this method is a way of checking whether the
> message has been output.
>
> sendSignal(signal)
> - send the specified signal to the background process.
>
> waitForProcessTermination(timeout)
> - wait up to "timeout" for the process to terminate.
>
> ... would be sufficient to do the testing
It is sufficient, unless we will come up with some more sophisticated
test scenarios in the future it will will just make it harder and harder
for unit testing.
>
> We would only have to write this once and the advantage about C++ is
> that it would naturally fit into the existing unit test framework.
> There are already some unit testing utilities in Kea, in
> src/lib/util/unittests - this would just be another.
>
>> Tomek's major objections to this seem to be:
>>
>> 1. We already have tests in Python and they should be portable
>> with Python 2.x if people hate Python 3 and its incompatibilities
>>
>> 2. If you run unit tests you're probably a developer and you can
>> afford installing Python 2.x.
>>
>> 3. There is no test framework for shell but there is for C++. This
>> is in favor of writing a test in C++.
>>
>> 4. We are already using Forge that is written in Python so we could
>> move some tests between Kea and Forge.
>>
>> And Tomek, has a list of preferred languages for these tests: 1.
>> C++ 2. Python 3. Shell
>
> I agree with Tomek here.
>
> One of the problems with BIND 10 was the use of two languages because
> few people were masters of both. Unfortunately, we can't get rid of
> Python code altogether: Lettuce is written in Python. However,
> Lettuce is all about writing unit tests in a natural form of English.
> To my mind, that far outweighs the inconvenience of maintaining the
> Python needed to implement Lettuce features.
What do we use lettuce for? I know we do use lettuce in Forge. But, what
in Kea is tested with lettuce?
If we want to write a script controlling Kea in shell, we will be using
shell anyway. So, it seems that knowledge of shell is sort of important
for Kea team anyway.
>
>> I would really like to have input from others why you like or
>> dislike a certain approach because it may affect the way we develop
>> similar tests in the future. Apart from 3 test cases we are talking
>> about, there will be 6 additional test cases for other components:
>> 3 for D2 and 3 for v4. It is better to alter the course now, not
>> when they are implemented. Having said that, I should have
>> initiated this discussion prior to implementing tests for v6, but
>> better now than later.
>
> In summary, I think if we were to implement a C++ class for handling
> subprocesses that output to stdout, we could do a lot of this type of
> testing in the C++ unit tests.
If people feel this is a general consensus, that's fine. I'll still wait
for additional input from other mail list members.
Marcin
------------------------------
Message: 3
Date: Tue, 27 May 2014 15:59:02 +0200
From: Tomek Mrugalski <[email protected]>
To: [email protected]
Subject: Re: How to implement SIGHUP tests
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1
On 27.05.2014 11:19, Marcin Siodelski wrote:
> So, the test presented above requires spawning a new process, sending
> signals and monitoring whether it reacts in an expected way. A couple of
> reasons why I think it should be written in shell (not in C++, not in
> Python):
>
> 1. Shell is there on every OS and doesn't cause any additional
> dependencies for Kea to be tested, whilst this is not true for Python.
> Even if we can detect that there is no Python on the system and not run
> these tests, the down side is that we don't run these tests.
True, but not shells are equal. Most linuxes use bash, but some BSDs use
tcsh, zsh, ash or one of several other shells. Sure, we can dodge the
problem by declaring that our environment requires bash to run, but this
is similar to declaring that we require specific version of python.
> 2. One of the tasks for 0.9 is to implement a script that controls the
> startup of Kea. We might not have decided on what language it will be
> written in, but I also don't think anybody wanted Python as we are in
> the process of removing Python-based code from Kea. The shell script
> makes a lot of sense to me and if so, I would like to test Kea startup
> and reconfiguration using the same approach as this script would use.
>
> 3. As Tomek pointed out, the shell scripts don't provide any framework
> for testing (nothing similar to gtest) which makes it more reasonable to
> implement our tests in C++ and within gtest framework. But, python based
> tests that we already have don't have any framework either and we used
> to live with it. So, is lack of framework really a problem?
It's not a big problem. I think that over time, our build farm should
start collecting information about number of unit-tests being executed.
Let's assume that a commit broke something in configure.ac and
--with-gtest and --with-gtest-source switches no longer work. Our
current build farm wouldn't pick it. make & make check would complete
just fine. As I said, it's not a big issue, just something we should
keep in mind. The code you wrote in dhcp6_test_func.sh could be a start
of such framework. It doesn't have to be fancy: just uniform printing of
test name and test result will do the trick. The code currently does
that. The (minor) problem is that there's no way to enforce that
behavior in shell tests.
> 4. Implementing tests in C++ would certainly make tests consistent with
> other tests we have and I am in general in favor of that. But, C++ is
> not the easiest environment to deal with the type of tests we are
> talking about. Dealing with spawning background processes, grepping log
> files, creating configuration files is something that is natural in
> shell because shell is basically for this purpose mainly. I don't
> disagree that it is doable in C++: you can fork, execlp, read file in
> C++ (harder to grep but doable) but it doesn't make code implementation
> and maintenance any easier.
>
> Tomek's major objections to this seem to be:
>
> 1. We already have tests in Python and they should be portable with
> Python 2.x if people hate Python 3 and its incompatibilities
As I said before, this is a somewhat weak argument. We already have 7
python tests for v4 and 7 for v6. Regardless if we decide to tweak it to
run on python 2.x or rewrite them in shell, we'll have to spend some
time on it.
> 2. If you run unit tests you're probably a developer and you can afford
> installing Python 2.x.
I think this is quite important point. The fuss about python3 nonsense
was mostly from users perspective. It is obvious that developer-specific
features (like tests) has (and will have) extra dependencies.
> 4. We are already using Forge that is written in Python so we could move
> some tests between Kea and Forge.
I think that over time some of the more advanced "unit" tests will no
longer be unit-tests, they'll just become full featured system tests. If
you don't think this is the case, please check the code that starts the
server with various parameters. The unit here is the DHCPv6 server.
> And Tomek, has a list of preferred languages for these tests:
> 1. C++
> 2. Python
> 3. Shell
That's what I initially wrote, but after giving it more thought I'd like
to amend that list:
1. C++
2. Some scripting language (one of: Python or shell)
I marginally prefer python to shell, as it gives you more flexibility
(e.g. when you want to generate something, like a config with 1000
subnets). Sure, you can do it in shell, but it's just easier in python.
But I won't object is we decide to write some tests in shell. But I will
strongly insist that we should not mix them (some tests in shell and
some in python). If we chose python, I'm 100% that it must not require 3.x.
I think that python has additional benefit. Forge is written in Python.
So if a unit-test grow too large, we can move it to Forge. Also, Wlodek
is quite skilled with python, so we could ask him one day to help us
with unit-tests.
As for Tom's comment that we may do something in 0.9 timeframe and then
possibly revisit our decision later... well, I don't buy it. There will
never be a time when we would say "ok, let's spend 2 weeks rewriting our
unit-tests, because we'd like to have them in a different language".
There will always be tasks that give you better bang for your buck.
But I'm not insisting on it. If we collectively decide that shell is the
way to go, we'll just have a bit more python crap to get rid of :)
Tomek
------------------------------
Message: 4
Date: Tue, 27 May 2014 16:19:06 +0200
From: Marcin Siodelski <[email protected]>
To: Tomek Mrugalski <[email protected]>, [email protected]
Subject: Re: How to implement SIGHUP tests
Message-ID: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1
On 27/05/14 15:59, Tomek Mrugalski wrote:
> On 27.05.2014 11:19, Marcin Siodelski wrote:
>> So, the test presented above requires spawning a new process, sending
>> signals and monitoring whether it reacts in an expected way. A couple of
>> reasons why I think it should be written in shell (not in C++, not in
>> Python):
>>
>> 1. Shell is there on every OS and doesn't cause any additional
>> dependencies for Kea to be tested, whilst this is not true for Python.
>> Even if we can detect that there is no Python on the system and not run
>> these tests, the down side is that we don't run these tests.
> True, but not shells are equal. Most linuxes use bash, but some BSDs use
> tcsh, zsh, ash or one of several other shells. Sure, we can dodge the
> problem by declaring that our environment requires bash to run, but this
> is similar to declaring that we require specific version of python.
Yes, there are differences between shells. But I had this in mind
writing unit tests that you reviewed and it didn't blow up anywhere yet.
>
>> 2. One of the tasks for 0.9 is to implement a script that controls the
>> startup of Kea. We might not have decided on what language it will be
>> written in, but I also don't think anybody wanted Python as we are in
>> the process of removing Python-based code from Kea. The shell script
>> makes a lot of sense to me and if so, I would like to test Kea startup
>> and reconfiguration using the same approach as this script would use.
>>
>> 3. As Tomek pointed out, the shell scripts don't provide any framework
>> for testing (nothing similar to gtest) which makes it more reasonable to
>> implement our tests in C++ and within gtest framework. But, python based
>> tests that we already have don't have any framework either and we used
>> to live with it. So, is lack of framework really a problem?
> It's not a big problem. I think that over time, our build farm should
> start collecting information about number of unit-tests being executed.
> Let's assume that a commit broke something in configure.ac and
> --with-gtest and --with-gtest-source switches no longer work. Our
> current build farm wouldn't pick it. make & make check would complete
> just fine. As I said, it's not a big issue, just something we should
> keep in mind. The code you wrote in dhcp6_test_func.sh could be a start
> of such framework. It doesn't have to be fancy: just uniform printing of
> test name and test result will do the trick. The code currently does
> that. The (minor) problem is that there's no way to enforce that
> behavior in shell tests.
Shell allows for libraries. The proposed tests use a common shell
function to print test name. The function can be tweaked at any point to
make it uniform with whatever format we want.
>
>> 4. Implementing tests in C++ would certainly make tests consistent with
>> other tests we have and I am in general in favor of that. But, C++ is
>> not the easiest environment to deal with the type of tests we are
>> talking about. Dealing with spawning background processes, grepping log
>> files, creating configuration files is something that is natural in
>> shell because shell is basically for this purpose mainly. I don't
>> disagree that it is doable in C++: you can fork, execlp, read file in
>> C++ (harder to grep but doable) but it doesn't make code implementation
>> and maintenance any easier.
>>
>> Tomek's major objections to this seem to be:
>>
>> 1. We already have tests in Python and they should be portable with
>> Python 2.x if people hate Python 3 and its incompatibilities
> As I said before, this is a somewhat weak argument. We already have 7
> python tests for v4 and 7 for v6. Regardless if we decide to tweak it to
> run on python 2.x or rewrite them in shell, we'll have to spend some
> time on it.
I was under the impression that we wanted to remove python from Kea.
This means to me that these tests should be replaced anyway. But, maybe
I was wrong. Don't we really want to remove python dependency from Kea
completely? What are the pieces that we MUST keep?
>
>> 2. If you run unit tests you're probably a developer and you can afford
>> installing Python 2.x.
> I think this is quite important point. The fuss about python3 nonsense
> was mostly from users perspective. It is obvious that developer-specific
> features (like tests) has (and will have) extra dependencies.
>
I disagree. I don't know the developer's who would like to install more
than necessary.
>> 4. We are already using Forge that is written in Python so we could move
>> some tests between Kea and Forge.
> I think that over time some of the more advanced "unit" tests will no
> longer be unit-tests, they'll just become full featured system tests. If
> you don't think this is the case, please check the code that starts the
> server with various parameters. The unit here is the DHCPv6 server.
I know there are such tests. I believe they are important. But, I'd
rather see them written in shell.
>
>> And Tomek, has a list of preferred languages for these tests:
>> 1. C++
>> 2. Python
>> 3. Shell
> That's what I initially wrote, but after giving it more thought I'd like
> to amend that list:
> 1. C++
> 2. Some scripting language (one of: Python or shell)
>
> I marginally prefer python to shell, as it gives you more flexibility
> (e.g. when you want to generate something, like a config with 1000
> subnets). Sure, you can do it in shell, but it's just easier in python.
> But I won't object is we decide to write some tests in shell. But I will
> strongly insist that we should not mix them (some tests in shell and
> some in python). If we chose python, I'm 100% that it must not require 3.x.
I was rather thinking to migrate out of python completely. But, it seems
that there are bits that we want to keep in Python. I am just not sure
which bits and why.
>
> I think that python has additional benefit. Forge is written in Python.
> So if a unit-test grow too large, we can move it to Forge. Also, Wlodek
> is quite skilled with python, so we could ask him one day to help us
> with unit-tests.
There is a set of unit tests that should always be provided along with
the product (Kea). I don't see any point in moving tests around between
one project and another. Forge as a whole is a great tool that allows
remote execution, crafting packets etc. but all this is not necessary to
test DHCP daemons startup etc. I don't see any reason why such tests
should be moved outside of Kea. They test very basic functionality.
Also, such decisions shouldn't be made on the assumption that someone
feels better in whatever language, because people move around.
>
> As for Tom's comment that we may do something in 0.9 timeframe and then
> possibly revisit our decision later... well, I don't buy it. There will
> never be a time when we would say "ok, let's spend 2 weeks rewriting our
> unit-tests, because we'd like to have them in a different language".
> There will always be tasks that give you better bang for your buck.
I think this is correct. If we want to move to something else, let's
spend some extra time on this now.
>
> But I'm not insisting on it. If we collectively decide that shell is the
> way to go, we'll just have a bit more python crap to get rid of :)
>
I am not sure who is going to have final word on this.
Marcin
------------------------------
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
End of kea-dev Digest, Vol 2, Issue 8
*************************************
