Re: [Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework
On 25.6.2013 14:08, Petr Viktorin wrote: Adding two additional patches for better Beaker integration: Patch 0241 allows e.g. adding ticket numbers for automatic test case management Patch 0242 should bring the BeakerLib logging closer to what traditional Beaker tests output make test seems to run fine with patches 230-242 applied, however ipa-run-tests produces the following output: == ERROR: Test that `ipa help` only writes to stdout -- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py", line 63, in test_ipa_help return_value = api.Backend.cli.run(['help']) File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1067, in run result = self.execute(name, **kw) File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 146, in execute raise error #pylint: disable=E0702 InternalError: an internal error has occurred >> begin captured logging << IPA: DEBUG: [ipa.ipalib.cli.help] raw: help(None, version=u'2.60') IPA: DEBUG: [ipa.ipalib.cli.help] help(None, version=u'2.60') IPA: ERROR: [ipa.ipalib.cli.cli] non-public: AttributeError: 'API' object has no attribute 'parser' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 129, in execute result = self.Command[_name](*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 435, in __call__ ret = self.run(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 765, in run self.api.parser.print_help(outfile) AttributeError: 'API' object has no attribute 'parser' IPA: DEBUG: [ipa.ipalib.rpc.xmlclient] Destroyed connection context.xmlclient - >> end captured logging << - == ERROR: Failure: IOError ([Errno 2] No such file or directory: 'ipatests/test_xmlrpc/service.crt') -- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/lib/python2.7/site-packages/ipatests/test_xmlrpc/test_host_plugin.py", line 59, in fd = open('ipatests/test_xmlrpc/service.crt', 'r') IOError: [Errno 2] No such file or directory: 'ipatests/test_xmlrpc/service.crt' == ERROR: Failure: IOError ([Errno 2] No such file or directory: 'ipatests/test_xmlrpc/service.crt') -- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/lib/python2.7/site-packages/ipatests/test_xmlrpc/test_service_plugin.py", line 42, in fd = open('ipatests/test_xmlrpc/service.crt', 'r') IOError: [Errno 2] No such file or directory: 'ipatests/test_xmlrpc/service.crt' == FAIL: Test that `ipa` errors out, and prints the help to stderr -- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py", line 71, in test_ipa_without_arguments api.Backend.cli.run([]) File "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py", line 55, in __exit__ assert isinstance(exc_value, self.exception), exc_value AssertionError: 'API' object has no attribute 'parser' >> begin captured logging << IPA: DEBUG: [ipa.ipalib.cli.help] raw: help(None, outfile=) IPA: DEBUG: [ipa.ipalib.cli.help] help(None, outfile=instance at 0x486cb48>) - >> end captured logging << - I guess the location of the test certificate should be made configurable in order to
Re: [Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework
On 06/14/2013 05:44 PM, Petr Viktorin wrote: On 06/14/2013 04:41 PM, Jan Cholasta wrote: On 14.6.2013 16:19, Jan Cholasta wrote: Hi, On 13.6.2013 14:16, Petr Viktorin wrote: On 06/10/2013 04:48 PM, Petr Viktorin wrote: On 05/31/2013 01:46 PM, Petr Viktorin wrote: Apply on top of my patches 0227-0234. These patches add an initial integration testing framework. Patch 0230 adds a plugin for ordered test classes. Nose orders methods within a test suite alphabetically, but we generally want to run them in the order defined. This adds the @ordered decorator that causes Nose to do just that, provided the plugin is loaded and enabled, and that the methods are defined in the same file. The ipa-run-tests wrapper is changed to enable the plugin. In the future we may want to use this for unit tests as well. It might also make sense to separate it from the FreeIPA project altogether. Patch 0231 adds configuration for tests. This reads environment variables like: - MASTER (FQDN of initial server) - REPLICA (space-separated FQDNs of replicas) - CLIENT (space-separated FQDNs of clients), - IPATEST_DIR (directory the tests use on the remote machines) etc., and loads them into an easy-to use Python object. A tool called ipa-test-config is provided that generates a full set of environment variables for shell-based tests from these, either global or specific for a given host. If environment variables don't work for us, alternate configuration methods can be added in the future. I think you forgot to add "%dir %{python_sitelib}/ipatests/test_integration" to the spec file. Is the "self = cls()" line at the beginning of Config.from_env() intentional? +self = cls() +env_normalize(env) + +self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests', No. I removed it, thanks for thee catch. Also typo in commit message: "Integration tests are be configured ..." Thanks, fixed. Patch 0232 adds an integration test framework. This extends Host object available from the configuration with methods to run commands and copy files on the remote host, and adds a base class for integration tests which can currently install and uninstall IPA in a "star" topology. (In the future, the install/uninstall code should also be made available as a shell command.) A simple test for user replication between two masters is provided. Log files from the remote hosts can be marked for collection, but the actual collection is left to a Nose plugin. The base class uses the @ordered decorator mentioned above. Patch 0233 improves on how commands are run on remote hosts. In the previous patch, the process's stdin and stdout were combined as a quick hack to avoid the problem that if we first read stdout and then stderr, then stderr's buffer can fill up and we'd deadlock (and the other way around). With this patch the streams are read in parallel. In the future this can be extended to calling whole commands in parallel (e.g. uninstalling IPA on all the hosts at once). Patch 0234 adds log collection to the BeakerLib integration plugin. This tars up the marked logs, downloads then, and calls rlFileSubmit on them. Missing space in commit message: "... log files fromthe remote ..." Thanks, fixed. --- Attaching additional patches: Patch 0237 configures logging in ipa-run-tests to forward messages from the IPA logging machinery to a normal Python logger. This way the logs are captured The logs are also printed to stderr so that there's some activity on the terminal after you run the utility. Patch 0238 makes it possible to use RSA private SSH keys to log in to the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if $IPA_ROOT_SSH_PASSWORD is empty. I've added this to the design page. It seems that the code prefers password authentication over public key authentication if both are configured. IMO it would be better if it did the opposite. Good point, fixed, design page updated. Patch 0239 makes test setup change the hostname, /etc/hosts, and /etc/resolv.conf to match the configured values. These should be equivalent to the fixes in https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733 In test teardown, the changes are undone. I've rebased the patrchset and added small fixes for patches 0232 and 0239. New patch 0240 contains a few fixes/improvements to the Host class that were not trivial to rebase into previous patches. The WIP patch adds a sketch of some of the tests for CA-less (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please comment if you can see where things can be improved for test authors! Just a word of warning, there are still a few test cases I need to add to the CA-less test plan. Sure. I did this mainly to see how things look from the test author's point of view. Adding two additional patches for better Beaker integration: Patch 0241 allows e.g. adding ticket numbers for automatic test case managem
Re: [Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework
On 14.6.2013 16:19, Jan Cholasta wrote: Hi, On 13.6.2013 14:16, Petr Viktorin wrote: On 06/10/2013 04:48 PM, Petr Viktorin wrote: On 05/31/2013 01:46 PM, Petr Viktorin wrote: Apply on top of my patches 0227-0234. These patches add an initial integration testing framework. Patch 0230 adds a plugin for ordered test classes. Nose orders methods within a test suite alphabetically, but we generally want to run them in the order defined. This adds the @ordered decorator that causes Nose to do just that, provided the plugin is loaded and enabled, and that the methods are defined in the same file. The ipa-run-tests wrapper is changed to enable the plugin. In the future we may want to use this for unit tests as well. It might also make sense to separate it from the FreeIPA project altogether. Patch 0231 adds configuration for tests. This reads environment variables like: - MASTER (FQDN of initial server) - REPLICA (space-separated FQDNs of replicas) - CLIENT (space-separated FQDNs of clients), - IPATEST_DIR (directory the tests use on the remote machines) etc., and loads them into an easy-to use Python object. A tool called ipa-test-config is provided that generates a full set of environment variables for shell-based tests from these, either global or specific for a given host. If environment variables don't work for us, alternate configuration methods can be added in the future. I think you forgot to add "%dir %{python_sitelib}/ipatests/test_integration" to the spec file. Is the "self = cls()" line at the beginning of Config.from_env() intentional? +self = cls() +env_normalize(env) + +self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests', Also typo in commit message: "Integration tests are be configured ..." Patch 0232 adds an integration test framework. This extends Host object available from the configuration with methods to run commands and copy files on the remote host, and adds a base class for integration tests which can currently install and uninstall IPA in a "star" topology. (In the future, the install/uninstall code should also be made available as a shell command.) A simple test for user replication between two masters is provided. Log files from the remote hosts can be marked for collection, but the actual collection is left to a Nose plugin. The base class uses the @ordered decorator mentioned above. Patch 0233 improves on how commands are run on remote hosts. In the previous patch, the process's stdin and stdout were combined as a quick hack to avoid the problem that if we first read stdout and then stderr, then stderr's buffer can fill up and we'd deadlock (and the other way around). With this patch the streams are read in parallel. In the future this can be extended to calling whole commands in parallel (e.g. uninstalling IPA on all the hosts at once). Patch 0234 adds log collection to the BeakerLib integration plugin. This tars up the marked logs, downloads then, and calls rlFileSubmit on them. Missing space in commit message: "... log files fromthe remote ..." --- Attaching additional patches: Patch 0237 configures logging in ipa-run-tests to forward messages from the IPA logging machinery to a normal Python logger. This way the logs are captured The logs are also printed to stderr so that there's some activity on the terminal after you run the utility. Patch 0238 makes it possible to use RSA private SSH keys to log in to the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if $IPA_ROOT_SSH_PASSWORD is empty. I've added this to the design page. It seems that the code prefers password authentication over public key authentication if both are configured. IMO it would be better if it did the opposite. Patch 0239 makes test setup change the hostname, /etc/hosts, and /etc/resolv.conf to match the configured values. These should be equivalent to the fixes in https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733 In test teardown, the changes are undone. I've rebased the patrchset and added small fixes for patches 0232 and 0239. New patch 0240 contains a few fixes/improvements to the Host class that were not trivial to rebase into previous patches. The WIP patch adds a sketch of some of the tests for CA-less (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please comment if you can see where things can be improved for test authors! Just a word of warning, there are still a few test cases I need to add to the CA-less test plan. Honza -- Jan Cholasta ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework
Hi, On 13.6.2013 14:16, Petr Viktorin wrote: On 06/10/2013 04:48 PM, Petr Viktorin wrote: On 05/31/2013 01:46 PM, Petr Viktorin wrote: Apply on top of my patches 0227-0234. These patches add an initial integration testing framework. Patch 0230 adds a plugin for ordered test classes. Nose orders methods within a test suite alphabetically, but we generally want to run them in the order defined. This adds the @ordered decorator that causes Nose to do just that, provided the plugin is loaded and enabled, and that the methods are defined in the same file. The ipa-run-tests wrapper is changed to enable the plugin. In the future we may want to use this for unit tests as well. It might also make sense to separate it from the FreeIPA project altogether. Patch 0231 adds configuration for tests. This reads environment variables like: - MASTER (FQDN of initial server) - REPLICA (space-separated FQDNs of replicas) - CLIENT (space-separated FQDNs of clients), - IPATEST_DIR (directory the tests use on the remote machines) etc., and loads them into an easy-to use Python object. A tool called ipa-test-config is provided that generates a full set of environment variables for shell-based tests from these, either global or specific for a given host. If environment variables don't work for us, alternate configuration methods can be added in the future. I think you forgot to add "%dir %{python_sitelib}/ipatests/test_integration" to the spec file. Is the "self = cls()" line at the beginning of Config.from_env() intentional? +self = cls() +env_normalize(env) + +self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests', Patch 0232 adds an integration test framework. This extends Host object available from the configuration with methods to run commands and copy files on the remote host, and adds a base class for integration tests which can currently install and uninstall IPA in a "star" topology. (In the future, the install/uninstall code should also be made available as a shell command.) A simple test for user replication between two masters is provided. Log files from the remote hosts can be marked for collection, but the actual collection is left to a Nose plugin. The base class uses the @ordered decorator mentioned above. Patch 0233 improves on how commands are run on remote hosts. In the previous patch, the process's stdin and stdout were combined as a quick hack to avoid the problem that if we first read stdout and then stderr, then stderr's buffer can fill up and we'd deadlock (and the other way around). With this patch the streams are read in parallel. In the future this can be extended to calling whole commands in parallel (e.g. uninstalling IPA on all the hosts at once). Patch 0234 adds log collection to the BeakerLib integration plugin. This tars up the marked logs, downloads then, and calls rlFileSubmit on them. --- Attaching additional patches: Patch 0237 configures logging in ipa-run-tests to forward messages from the IPA logging machinery to a normal Python logger. This way the logs are captured The logs are also printed to stderr so that there's some activity on the terminal after you run the utility. Patch 0238 makes it possible to use RSA private SSH keys to log in to the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if $IPA_ROOT_SSH_PASSWORD is empty. I've added this to the design page. It seems that the code prefers password authentication over public key authentication if both are configured. IMO it would be better if it did the opposite. Patch 0239 makes test setup change the hostname, /etc/hosts, and /etc/resolv.conf to match the configured values. These should be equivalent to the fixes in https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733 In test teardown, the changes are undone. I've rebased the patrchset and added small fixes for patches 0232 and 0239. New patch 0240 contains a few fixes/improvements to the Host class that were not trivial to rebase into previous patches. The WIP patch adds a sketch of some of the tests for CA-less (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please comment if you can see where things can be improved for test authors! Just a word of warning, there are still a few test cases I need to add to the CA-less test plan. Honza -- Jan Cholasta ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel