Re: [Tails-dev] Progress report on the automated test suite
03/11/12 09:55, intrigeri wrote: Next I'd like to announce that the automated test suite, in its current unfinished state, actually has found its very first Tails bug. [...] In other words, our firewall leaks link-local IPv6 broadcasts even though it should block everything IPv6 (right?). Ouch. WAN hat on: please report it (ticket + email) separately so that it does not get lost in the middle of this report on the automated test suite thread. For the ticket, see todo/really_block_ipv6. I'd like to present the last two with a bit more depth and hear your opinions, especially w.r.t. the fact that they alter Tails or cheat in the testing process, so I wonder how ethical they are in the context of test-driven development. Running arbitrary commands inside the guest VM == This is very valuable as it makes many tests that would be truly awkward to do with sikuli into something trivial. libvirt doesn't seem to have something like VirtualBox' `vboxmanage guestcontrol execute` (provided by the VirtualBox guest additions), so I implemented a simple remote shell (read: a backdoor (listening on port 1337 + firewall exception) so expect havoc on the Tails forum!) that starts on the guest when the boot parameter autotest_never_use_me is present on the kernel cmdline. autotest_never_use_me looks to me like (speaking to) autotest: never use me. What about backdoor_for_autotest? I'm not sure I want to mention the word backdoor. Sure, I do it in the remote shell server script, but then it's mentioned in a context where sane people should have no reason to be worried. Makes sense? Other naming suggestions? Saving/restoring VM snapshots = [...] For both features, to reply on the 'how ethical they are in the context of test-driven development' topic, I'd need a concrete example of how this would be used in practice. I'm sure there are specific/concrete situations where this is not a good idea. I was more interested if you saw any fundamental flaws with this approach since it's a step away from black box testing (same applies to the remote shell). Cheers! ___ tails-dev mailing list tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev
Re: [Tails-dev] Progress report on the automated test suite
03/11/12 17:18, Ague Mill wrote:
anonym:
Next I'd like to announce that the automated test suite, in its current
unfinished state, actually has found its very first Tails bug. Here's
the cucumber output of when it was found:
[...]
And all Internet traffic has only flowed through Tor
# cucumber/iceweasel/step_definitions/torified_browsing.rb:66
The following IPv6 hosts were contacted:
ff02::1
Full network capture available at: [...censored...]
There were network leaks! (RuntimeError) [...]
In other words, our firewall leaks link-local IPv6 broadcasts even
though it should block everything IPv6 (right?). This is promising (not
that Tails has this particular bug, but that the test suite found it)!
I did not run the code itself, but are you sure that these packets came
from Tails and not from the host system?
Unless there's a bug in tcpdump, yes, I'm sure. tcpdump is configured
with the filter src host ${IP} or src host ${IP6}, where IP* = the
guest running Tails.
Saving/restoring VM snapshots
=
This is how I intend to use it for a given feature:
Background:
Given I restore the background snapshot if it exists
[ ... real background steps ... ]
And I save the background snapshot if it does not exist
[ ... Scenarios ... ]
Those lines feel like noise: they are an implementation detail and
should not appear in the scenarios.
Cucumber offer tags and hooks that should be usable to achieve something
similar while keeping the scenarios as lean as possible. See:
https://github.com/cucumber/cucumber/wiki/Hooks and
http://stackoverflow.com/questions/9994797/cucumber-when-to-use-tags-hooks-vs-backgrounds
I looked at them but couldn't find anything that I thought suited. I
need one hook that runs exactly before the background, and one that runs
exactly after.
An issue with restoring past state like this is that our Tor's circuit
state may get out-of-sync with the circuit state of the relays they use.
For instance, I ran 10 tests that restored to the same post-background
state and all but the first two failed to fetch a web page. Then I ran
10 tests where I do the following after each snapshot restore:
1. Stop Tor.
2. Sync time from host to guest.
3. Start Tor.
And then all 10 tests succeeded, so it seems resetting Tor like this is
highly necessary.
Indeed, as restoring from a snapshot is likely to break all existing TCP
connections. Have you tried to see if a SIGHUP sent to Tor is sufficient?
From previous experiences (e.g. tordate) Tor doesn't behave well if it
experience a time jump, even if given a HUP, so in order to avoid future
mysterious failed tests due to Tor being confused, I thought restarting
it was for the best.
Side note: your `try_for` function is very unidiomatic Ruby.
I suggest you have a look at the part about blocks on
http://www.ruby-doc.org/docs/ProgrammingRuby/html/tut_containers.html,
and the `yield` and `block_given?` methods.
Thanks, I'll look into it (again)! I actually used `yield` at first, but
didn't get it to work, whereas anonymous functions worked fine. I
suspect those issues might have been due to an unrelated problem (the
stock 'timeout' module in Ruby, which I used at first, can lockup if
syscall get involved, apparently).
Cheers!
___
tails-dev mailing list
tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
Re: [Tails-dev] Progress report on the automated test suite
hi, anonym wrote (13 Nov 2012 13:01:41 GMT) : 03/11/12 09:55, intrigeri wrote: that starts on the guest when the boot parameter autotest_never_use_me is present on the kernel cmdline. autotest_never_use_me looks to me like (speaking to) autotest: never use me. What about backdoor_for_autotest? I'm not sure I want to mention the word backdoor. Sure, I do it in the remote shell server script, but then it's mentioned in a context where sane people should have no reason to be worried. Why should the boot parameter name be mentioned in contexts where this does not apply? (Not a rhetorical question.) Saving/restoring VM snapshots = [...] For both features, to reply on the 'how ethical they are in the context of test-driven development' topic, I'd need a concrete example of how this would be used in practice. I'm sure there are specific/concrete situations where this is not a good idea. I was more interested if you saw any fundamental flaws with this approach since it's a step away from black box testing (same applies to the remote shell). Sorry, I'm not in a mood to think about fundamental flaws without examples. Anyhow, I'll try to contribute a bit, hoping that helps. So, I think that: 1. There are serious shortcomings that come with these features. Every time one cheats and uses them, one should know what they are actually *not* testing, and think if/how that could be tested. 2. Trying at all costs to totally avoid to use these features is probably not a good use of our time. 3. Generally, I like to exercise systems under test at different levels, and the remote shell feature is probably the most efficient way to zoom-in and run something like unit tests. This may not totally replace more zoomed-out, behavioral testing, though. That's all I feel I can answer, on a general level. I'd rather be pointed to cases when it looks much easier or even needed to resort to such hacks, and then, discuss whether that is acceptable on a case by case basis. But that may happen later, once actual reasons to use these features arise. Cheers, -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc ___ tails-dev mailing list tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev
Re: [Tails-dev] Progress report on the automated test suite
hi, anonym wrote (02 Nov 2012 16:42:31 GMT) : I'd like to start by saying that I think the work bertagaz did on the jruby + sikuli + cucumber combo is really great. He called it PoC, but after having worked on it for some time now I say it's definitely fit for the task at hand. Great! Next I'd like to announce that the automated test suite, in its current unfinished state, actually has found its very first Tails bug. [...] In other words, our firewall leaks link-local IPv6 broadcasts even though it should block everything IPv6 (right?). Ouch. WAN hat on: please report it (ticket + email) separately so that it does not get lost in the middle of this report on the automated test suite thread. For this reason, I'll refrain from replying here. This is promising (not that Tails has this particular bug, but that the test suite found it)! Clearly! The framework currently has the following test primitives and other interesting features so far: [...] Awesome. I'd like to present the last two with a bit more depth and hear your opinions, especially w.r.t. the fact that they alter Tails or cheat in the testing process, so I wonder how ethical they are in the context of test-driven development. Running arbitrary commands inside the guest VM == This is very valuable as it makes many tests that would be truly awkward to do with sikuli into something trivial. libvirt doesn't seem to have something like VirtualBox' `vboxmanage guestcontrol execute` (provided by the VirtualBox guest additions), so I implemented a simple remote shell (read: a backdoor (listening on port 1337 + firewall exception) so expect havoc on the Tails forum!) that starts on the guest when the boot parameter autotest_never_use_me is present on the kernel cmdline. autotest_never_use_me looks to me like (speaking to) autotest: never use me. What about backdoor_for_autotest? If it's not there, the script that implements the remote shell server is removed from the filesystem. Of course, it is still accessible from the read-only fs, but it will refuse to start if the boot parameter isn't present. My hope is that all these failsafes will make sense to all but our most conspiranoid users. :) Looks good. Saving/restoring VM snapshots = [...] For both features, to reply on the 'how ethical they are in the context of test-driven development' topic, I'd need a concrete example of how this would be used in practice. Cheers! -- intrigeri | GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc | OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc ___ tails-dev mailing list tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev
Re: [Tails-dev] Progress report on the automated test suite
anonym: I'd like to start by saying that I think the work bertagaz did on the jruby + sikuli + cucumber combo is really great. He called it PoC, but after having worked on it for some time now I say it's definitely fit for the task at hand. Great. :) Next I'd like to announce that the automated test suite, in its current unfinished state, actually has found its very first Tails bug. Here's the cucumber output of when it was found: [...] And all Internet traffic has only flowed through Tor # cucumber/iceweasel/step_definitions/torified_browsing.rb:66 The following IPv6 hosts were contacted: ff02::1 Full network capture available at: [...censored...] There were network leaks! (RuntimeError) [...] In other words, our firewall leaks link-local IPv6 broadcasts even though it should block everything IPv6 (right?). This is promising (not that Tails has this particular bug, but that the test suite found it)! I did not run the code itself, but are you sure that these packets came from Tails and not from the host system? Saving/restoring VM snapshots = This is how I intend to use it for a given feature: Background: Given I restore the background snapshot if it exists [ ... real background steps ... ] And I save the background snapshot if it does not exist [ ... Scenarios ... ] Those lines feel like noise: they are an implementation detail and should not appear in the scenarios. Cucumber offer tags and hooks that should be usable to achieve something similar while keeping the scenarios as lean as possible. See: https://github.com/cucumber/cucumber/wiki/Hooks and http://stackoverflow.com/questions/9994797/cucumber-when-to-use-tags-hooks-vs-backgrounds An issue with restoring past state like this is that our Tor's circuit state may get out-of-sync with the circuit state of the relays they use. For instance, I ran 10 tests that restored to the same post-background state and all but the first two failed to fetch a web page. Then I ran 10 tests where I do the following after each snapshot restore: 1. Stop Tor. 2. Sync time from host to guest. 3. Start Tor. And then all 10 tests succeeded, so it seems resetting Tor like this is highly necessary. Indeed, as restoring from a snapshot is likely to break all existing TCP connections. Have you tried to see if a SIGHUP sent to Tor is sufficient? Side note: your `try_for` function is very unidiomatic Ruby. I suggest you have a look at the part about blocks on http://www.ruby-doc.org/docs/ProgrammingRuby/html/tut_containers.html, and the `yield` and `block_given?` methods. -- Ague pgp4LnUB9l9Af.pgp Description: PGP signature ___ tails-dev mailing list tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev
[Tails-dev] Progress report on the automated test suite
Hi, I'd like to start by saying that I think the work bertagaz did on the jruby + sikuli + cucumber combo is really great. He called it PoC, but after having worked on it for some time now I say it's definitely fit for the task at hand. Next I'd like to announce that the automated test suite, in its current unfinished state, actually has found its very first Tails bug. Here's the cucumber output of when it was found: [...] And all Internet traffic has only flowed through Tor # cucumber/iceweasel/step_definitions/torified_browsing.rb:66 The following IPv6 hosts were contacted: ff02::1 Full network capture available at: [...censored...] There were network leaks! (RuntimeError) [...] In other words, our firewall leaks link-local IPv6 broadcasts even though it should block everything IPv6 (right?). This is promising (not that Tails has this particular bug, but that the test suite found it)! The framework currently has the following test primitives and other interesting features so far: * Sikuli stuff, e.g. waiting for an image (or text via OCR) to appear on the screen, and perform arbitrary mouse and keyboard actions. * Capturing all of the guest's network traffic and determining whether the Tails firewall leaked anything. On failure the capture is preserved so manual inspection is possible. * Take screenshots. It's currently unused, but I think it could be useful to save a screenshot upon test failure. If we want to go really crazy it would be trivial to add support for recording videos using e.g. `ffmpeg -f x11grab` to record the whole DISPLAY that virt-viewer + sikuli runs in, which could be helpful for post-failure analysis. * Running arbitrary commands inside the guest VM. * Saving/restoring VM snapshots (not yet pushed to git) I'd like to present the last two with a bit more depth and hear your opinions, especially w.r.t. the fact that they alter Tails or cheat in the testing process, so I wonder how ethical they are in the context of test-driven development. Running arbitrary commands inside the guest VM == This is very valuable as it makes many tests that would be truly awkward to do with sikuli into something trivial. libvirt doesn't seem to have something like VirtualBox' `vboxmanage guestcontrol execute` (provided by the VirtualBox guest additions), so I implemented a simple remote shell (read: a backdoor (listening on port 1337 + firewall exception) so expect havoc on the Tails forum!) that starts on the guest when the boot parameter autotest_never_use_me is present on the kernel cmdline. If it's not there, the script that implements the remote shell server is removed from the filesystem. Of course, it is still accessible from the read-only fs, but it will refuse to start if the boot parameter isn't present. My hope is that all these failsafes will make sense to all but our most conspiranoid users. :) Saving/restoring VM snapshots = This is how I intend to use it for a given feature: Background: Given I restore the background snapshot if it exists [ ... real background steps ... ] And I save the background snapshot if it does not exist [ ... Scenarios ... ] So, for the first scenario all background steps are run normally, and in the last one the VM state is saved. For all remaining scenarios the VM state is restored in the first step, and all remaining steps, including the real ones, just pass immediately (i.e. each real step starts with pass if we restored from a snapshot). I suppose common background snapshots can be shared between different features too to improve the testing speed. While saving/restoring snapshots greatly improves the speed of the test suite, this wasn't the main reason for me implementing it. Without it we would hammer the Tor authorities (once per scenario) since each Tails boot is a full Tor bootstrap, which clearly isn't acceptable. In the end we will have *many* scenarios in total, and if we're gonna run the test suite for each automatic build at some point... well, you get the picture. An issue with restoring past state like this is that our Tor's circuit state may get out-of-sync with the circuit state of the relays they use. For instance, I ran 10 tests that restored to the same post-background state and all but the first two failed to fetch a web page. Then I ran 10 tests where I do the following after each snapshot restore: 1. Stop Tor. 2. Sync time from host to guest. 3. Start Tor. And then all 10 tests succeeded, so it seems resetting Tor like this is highly necessary. That's all. Cheers! ___ tails-dev mailing list tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev
