Re: [Freeipa-devel] [RFE] Integration testing
On 06/05/2013 12:49 PM, Petr Vobornik wrote: On 06/04/2013 07:19 PM, Petr Viktorin wrote: On 06/04/2013 06:15 PM, Petr Vobornik wrote: Hello, Some Qs: 1) does the configuration only mean: "We have this topology, run all test which fits into it"? Not quite, it means "We have these machines, run tests whose topology can fit on them". IDK how should Web UI or XML-RPC tests fit into it. The existing test suite, including the RPC tests, stays as it is. The single-machine tests generally expect IPA to be set up on the local machine. I wouldn't make that assumption. We should allow to have test runner and FreeIPA server on different machines. That would mean changing our entire existing test suite. For a single machine, the test runner would basically just SSH into the server and run a command or script there. You (or Jenkins) can do that easily. The difference is that configuration for Web UI tests should mean: "This is how FreeIPA and related stuff is installed. Tests all available functionality and don't test the missing." IE. if I install freeipa server without CA, the UI tests needs to get that info (ie. by no_ca flag) and then skip certificate tests or parts of other tests which touches this feature (like navigation tests). The same for no-dns and has-trust... Complete UI testing would be something like following: a) set configuration A b) install server with configuration A c) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome d) uninstall server e) set configuration B f) install server with configuration B g) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome h) uninstall server i) repeat for config C ... Note:: browser change is also done by change of configuration. Is it possible? Can the configuration change or should there be other level of configuration which can change? Do we have time to do it? It may take almost a day (sequentially, with full coverage). This would be configured in Jenkins as several jobs: 1) Run UI tests with configuration A + Firefox 2) Run UI tests with configuration A + IE 3) Run UI tests with configuration A + Chrome 4) Run UI tests with configuration B + Firefox 5) Run UI tests with configuration B + IE 6) Run UI tests with configuration B + Chrome If we wanted exhaustive tests it could be done as a "matrix job" ([A, B]×[FF, IE, Chrome]), but as you say we don't have resources for that, so we'll have to use some sane subset. (Note: Jenkins can aggregate results from multiple jobs, so we'd get a single report from all these) Does python nose support some master tests or can it be somehow configured in Jenkins so we can automate installation of IPA with different configurations (IE by the 'To-be-done command-line tools'). I'm not sure what you mean by master tests. Jenkins can be configured to run different tests using different configurations. But the way I see it, the different topologies would be in different tests, and run different checks. remember that we want a CI, not exhaustive testing of all the possible cases. Thanks for the info. So it would be safe to have only one job for Web UI in CI, the one with most complete feature set and most supported browser(FF). All tests for various config/browser combinations can be executed manually when needed. Yes. Additionally, if there is some subset of tests that we want to always run on another browser, we can mark them with Nose's attrib plugin, and run them in a separate job. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFE] Integration testing
On 06/04/2013 07:19 PM, Petr Viktorin wrote: On 06/04/2013 06:15 PM, Petr Vobornik wrote: Hello, Some Qs: 1) does the configuration only mean: "We have this topology, run all test which fits into it"? Not quite, it means "We have these machines, run tests whose topology can fit on them". IDK how should Web UI or XML-RPC tests fit into it. The existing test suite, including the RPC tests, stays as it is. The single-machine tests generally expect IPA to be set up on the local machine. I wouldn't make that assumption. We should allow to have test runner and FreeIPA server on different machines. The difference is that configuration for Web UI tests should mean: "This is how FreeIPA and related stuff is installed. Tests all available functionality and don't test the missing." IE. if I install freeipa server without CA, the UI tests needs to get that info (ie. by no_ca flag) and then skip certificate tests or parts of other tests which touches this feature (like navigation tests). The same for no-dns and has-trust... Complete UI testing would be something like following: a) set configuration A b) install server with configuration A c) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome d) uninstall server e) set configuration B f) install server with configuration B g) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome h) uninstall server i) repeat for config C ... Note:: browser change is also done by change of configuration. Is it possible? Can the configuration change or should there be other level of configuration which can change? Do we have time to do it? It may take almost a day (sequentially, with full coverage). This would be configured in Jenkins as several jobs: 1) Run UI tests with configuration A + Firefox 2) Run UI tests with configuration A + IE 3) Run UI tests with configuration A + Chrome 4) Run UI tests with configuration B + Firefox 5) Run UI tests with configuration B + IE 6) Run UI tests with configuration B + Chrome If we wanted exhaustive tests it could be done as a "matrix job" ([A, B]×[FF, IE, Chrome]), but as you say we don't have resources for that, so we'll have to use some sane subset. (Note: Jenkins can aggregate results from multiple jobs, so we'd get a single report from all these) Does python nose support some master tests or can it be somehow configured in Jenkins so we can automate installation of IPA with different configurations (IE by the 'To-be-done command-line tools'). I'm not sure what you mean by master tests. Jenkins can be configured to run different tests using different configurations. But the way I see it, the different topologies would be in different tests, and run different checks. remember that we want a CI, not exhaustive testing of all the possible cases. Thanks for the info. So it would be safe to have only one job for Web UI in CI, the one with most complete feature set and most supported browser(FF). All tests for various config/browser combinations can be executed manually when needed. The command-line tools would be for external shell-based tests; I've heard some people might want to write those :) 2) There is no configuration options for trusts. IMO we would like to test that. Options for that can be added when the trust tests are written. 3) Test runner needs to connect to remote machine as root. Should a config option for root password be added or can we safely assume that there will always be other authentication method available? My bad, I forgot to add $IPA_ROOT_SSH_PASSWORD to the wiki page. Fixed. Other authentication methods can be added if needed. -- Petr Vobornik ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFE] Integration testing
On Wed, Jun 05, 2013 at 10:42:30AM +0200, Petr Viktorin wrote: > On 06/04/2013 09:48 PM, Jakub Hrozek wrote: > >On Mon, Jun 03, 2013 at 02:05:09PM +0200, Petr Viktorin wrote: > >>Hello, > >>A design document for integration testing is available at > >>http://www.freeipa.org/page/V3/Integration_testing. I've copied it below for > >>easier quoting. > >> > >> > >>__NOTOC__ > >> > >>= Overview = > >> > >>Make it possible to write and run multi-host integration tests (such as: > >>install master & replica, add user on replica, verify it's added on master). > >> > >>These tests will be run from continuous integration. > >>Any developer can also run them manually. > >> > >>= Use Cases = > >> > >>== Continuous integration == > >> > >>The developer team at Red Hat will run a Jenkins continuous integration > >>server > >>that will run the tests automatically (after each commit if resources are > >>available). > >> > >>The CI results will be posted publicly. > >> > >>== Developer testing == > >> > >>Anyone is be able to run integration tests without advanced infrastructure, > >>only a number of virtual machines to run the tests on is needed. > >> > >>== Beaker integration == > >> > >>The tests will run seamlessly inside [http://beaker-project.org/ > >>Beaker]/[https://fedoraproject.org/wiki/QA/RHTS RHTS]. > >>A special option enables reporting via BeakerLib. > >> > >>= Non-goals = > >> > >>A complete testing/continuous integration setup needs some steps that will > >>not > >>be included in IPA's test suite: > >> > >>* Building the code > >>* VM provisioning > >> > >>* Configuring the basic system, installing the packages > >> > >> > >>= Design= > >> > >>The Python package with the IPA test suite is renamed to ipatests, > >>and > >>packaged for RPM-based systems as freeipa-tests. > >>Eventually the package will be included in Fedora. > >> > >>Integration tests will be controlled from a single machine, and executed > >>on a number of "remote" machines that act as servers, replicas, clients, > >>etc. > >>The controlling machine communicates with the others via the SSH protocol. > >>(The controlling machine may be the same as one of the "remote" ones.) > >> > >>Integration tests are included in the main IPA set suite, and configured > >>using > >>environment variables. If the variables are missing, all integration tests > >>are > >>skipped. > >>If an insufficient number of hosts is configured for a test, the individiual > >>test will be skipped. > >> > >>A tool is provided to run installed tests. > >> > >>The remote machines used for integration testing are required to have > >>relevant > >>IPA packages installed, firewall opened up, any needed workarounds applied > >>(RPM > >>downgrades, SELinux mode,...), and sshd set up to allow root login. > >>The test runner will connect to these machines, install IPA, perform the > >>tests, > >>and then uninstall IPA & return the systems to their previous state. > >> > >>A plugin for integration with BeakerLib is provided. > >> > >>= Test configuration = > >> > >>Tests are configured using these environment variables. > >> > >>== Host configuration == > >> > >>; $MASTER > >>: FQDN of the first IPA server > >>; $REPLICA > >>: FQDNs of other IPA servers (space-separated) > >>; $CLIENT > >>: FQDNs of IPA clients (space-separated) > >>; $MASTER_env2, $REPLICA_env2, $CLIENT_env2, $MASTER_env3, ... > >>: can be used for additional domains when needed > >> > >>DNS needs to be set up so that IP addresses can be obtained for these hosts. > >> > >>== Basic configuration == > >> > >>; $IPATEST_DIR > >>: Directory for test data on the remote hosts > >>: Default: /root/ipatests > >>; $DNSFORWARD > >>: IP of a DNS forwarder > >>: Default: 8.8.8.8 > >> > >>== Test customization == > >> > >>; $DOMAIN > >>: IPA domain name > >>: Default: taken from $MASTER > >>; $NISDOMAIN > >>: NIS domain name > >>: Default: ipatest > >>; $NTPSERVER > >>: NIS domain name > >>: Default: ipatest > >>; $IPv6SETUP > >>: Set to TRUE for IPv6-only connectivity > >>; $IPADEBUG > >>: Set to enable test debugging > >> > >>; $ADMINID > >>: Admin username > >>: Default: admin > >>; $ADMINPW > >>: Admin user password > >>: Default: Secret123 > >>; $ROOTDN > >>: Directory manager DN > >>: Default: cn=Directory Manager > >>; $ROOTDNPWD > >>: Directory manager password > >>: Default: Secret123 > >> > >>= Supporting tools = > >> > >>== ipa-test-config == > >> > >>This tool reads the configuration variables above and outputs a Bash script > >>that sets a much more complete set of variables for easy shell-based testing > >>or test set-up. > >> > >>Without arguments, ipa-test-config outputs information specific > >>to the host it is run on. When given a hostname, it prints config for that > >>host. > >>With the --global flag, it outputs configuration common to all > >>hosts. > >> > >>== ipa-run-tests == > >> > >>This tool is a wrapper arount nosetests and accepts the same > >>arguments > >>as Nose.
Re: [Freeipa-devel] [RFE] Integration testing
On 06/04/2013 09:48 PM, Jakub Hrozek wrote: On Mon, Jun 03, 2013 at 02:05:09PM +0200, Petr Viktorin wrote: Hello, A design document for integration testing is available at http://www.freeipa.org/page/V3/Integration_testing. I've copied it below for easier quoting. __NOTOC__ = Overview = Make it possible to write and run multi-host integration tests (such as: install master & replica, add user on replica, verify it's added on master). These tests will be run from continuous integration. Any developer can also run them manually. = Use Cases = == Continuous integration == The developer team at Red Hat will run a Jenkins continuous integration server that will run the tests automatically (after each commit if resources are available). The CI results will be posted publicly. == Developer testing == Anyone is be able to run integration tests without advanced infrastructure, only a number of virtual machines to run the tests on is needed. == Beaker integration == The tests will run seamlessly inside [http://beaker-project.org/ Beaker]/[https://fedoraproject.org/wiki/QA/RHTS RHTS]. A special option enables reporting via BeakerLib. = Non-goals = A complete testing/continuous integration setup needs some steps that will not be included in IPA's test suite: * Building the code * VM provisioning * Configuring the basic system, installing the packages = Design= The Python package with the IPA test suite is renamed to ipatests, and packaged for RPM-based systems as freeipa-tests. Eventually the package will be included in Fedora. Integration tests will be controlled from a single machine, and executed on a number of "remote" machines that act as servers, replicas, clients, etc. The controlling machine communicates with the others via the SSH protocol. (The controlling machine may be the same as one of the "remote" ones.) Integration tests are included in the main IPA set suite, and configured using environment variables. If the variables are missing, all integration tests are skipped. If an insufficient number of hosts is configured for a test, the individiual test will be skipped. A tool is provided to run installed tests. The remote machines used for integration testing are required to have relevant IPA packages installed, firewall opened up, any needed workarounds applied (RPM downgrades, SELinux mode,...), and sshd set up to allow root login. The test runner will connect to these machines, install IPA, perform the tests, and then uninstall IPA & return the systems to their previous state. A plugin for integration with BeakerLib is provided. = Test configuration = Tests are configured using these environment variables. == Host configuration == ; $MASTER : FQDN of the first IPA server ; $REPLICA : FQDNs of other IPA servers (space-separated) ; $CLIENT : FQDNs of IPA clients (space-separated) ; $MASTER_env2, $REPLICA_env2, $CLIENT_env2, $MASTER_env3, ... : can be used for additional domains when needed DNS needs to be set up so that IP addresses can be obtained for these hosts. == Basic configuration == ; $IPATEST_DIR : Directory for test data on the remote hosts : Default: /root/ipatests ; $DNSFORWARD : IP of a DNS forwarder : Default: 8.8.8.8 == Test customization == ; $DOMAIN : IPA domain name : Default: taken from $MASTER ; $NISDOMAIN : NIS domain name : Default: ipatest ; $NTPSERVER : NIS domain name : Default: ipatest ; $IPv6SETUP : Set to TRUE for IPv6-only connectivity ; $IPADEBUG : Set to enable test debugging ; $ADMINID : Admin username : Default: admin ; $ADMINPW : Admin user password : Default: Secret123 ; $ROOTDN : Directory manager DN : Default: cn=Directory Manager ; $ROOTDNPWD : Directory manager password : Default: Secret123 = Supporting tools = == ipa-test-config == This tool reads the configuration variables above and outputs a Bash script that sets a much more complete set of variables for easy shell-based testing or test set-up. Without arguments, ipa-test-config outputs information specific to the host it is run on. When given a hostname, it prints config for that host. With the --global flag, it outputs configuration common to all hosts. == ipa-run-tests == This tool is a wrapper arount nosetests and accepts the same arguments as Nose. It loads any additional plugins and runs tests from the system-installed IPA test suite. == Other == TBD: Additional command-line tools may be provided for tasks such as installing IPA in a given topology. = Implementation = Test cases are implemented as Nose test classes, with installation/uninstallation as class setup/teardown. A BeakerLib plugin is provided that starts/ends Beaker phases for Nose test contexts and cases, issues a Beaker assertion (rlPass/rlFail) for each test case, and collects and submits relevant logs. A separate plugin will be provided to collect logs outside of a Beaker environment. = Example instructions = To run the test called test_integration/test_simple
Re: [Freeipa-devel] [RFE] Integration testing
On Mon, Jun 03, 2013 at 02:05:09PM +0200, Petr Viktorin wrote: > Hello, > A design document for integration testing is available at > http://www.freeipa.org/page/V3/Integration_testing. I've copied it below for > easier quoting. > > > __NOTOC__ > > = Overview = > > Make it possible to write and run multi-host integration tests (such as: > install master & replica, add user on replica, verify it's added on master). > > These tests will be run from continuous integration. > Any developer can also run them manually. > > = Use Cases = > > == Continuous integration == > > The developer team at Red Hat will run a Jenkins continuous integration > server > that will run the tests automatically (after each commit if resources are > available). > > The CI results will be posted publicly. > > == Developer testing == > > Anyone is be able to run integration tests without advanced infrastructure, > only a number of virtual machines to run the tests on is needed. > > == Beaker integration == > > The tests will run seamlessly inside [http://beaker-project.org/ > Beaker]/[https://fedoraproject.org/wiki/QA/RHTS RHTS]. > A special option enables reporting via BeakerLib. > > = Non-goals = > > A complete testing/continuous integration setup needs some steps that will > not > be included in IPA's test suite: > > * Building the code > * VM provisioning > > * Configuring the basic system, installing the packages > > > = Design= > > The Python package with the IPA test suite is renamed to ipatests, > and > packaged for RPM-based systems as freeipa-tests. > Eventually the package will be included in Fedora. > > Integration tests will be controlled from a single machine, and executed > on a number of "remote" machines that act as servers, replicas, clients, > etc. > The controlling machine communicates with the others via the SSH protocol. > (The controlling machine may be the same as one of the "remote" ones.) > > Integration tests are included in the main IPA set suite, and configured > using > environment variables. If the variables are missing, all integration tests > are > skipped. > If an insufficient number of hosts is configured for a test, the individiual > test will be skipped. > > A tool is provided to run installed tests. > > The remote machines used for integration testing are required to have > relevant > IPA packages installed, firewall opened up, any needed workarounds applied > (RPM > downgrades, SELinux mode,...), and sshd set up to allow root login. > The test runner will connect to these machines, install IPA, perform the > tests, > and then uninstall IPA & return the systems to their previous state. > > A plugin for integration with BeakerLib is provided. > > = Test configuration = > > Tests are configured using these environment variables. > > == Host configuration == > > ; $MASTER > : FQDN of the first IPA server > ; $REPLICA > : FQDNs of other IPA servers (space-separated) > ; $CLIENT > : FQDNs of IPA clients (space-separated) > ; $MASTER_env2, $REPLICA_env2, $CLIENT_env2, $MASTER_env3, ... > : can be used for additional domains when needed > > DNS needs to be set up so that IP addresses can be obtained for these hosts. > > == Basic configuration == > > ; $IPATEST_DIR > : Directory for test data on the remote hosts > : Default: /root/ipatests > ; $DNSFORWARD > : IP of a DNS forwarder > : Default: 8.8.8.8 > > == Test customization == > > ; $DOMAIN > : IPA domain name > : Default: taken from $MASTER > ; $NISDOMAIN > : NIS domain name > : Default: ipatest > ; $NTPSERVER > : NIS domain name > : Default: ipatest > ; $IPv6SETUP > : Set to TRUE for IPv6-only connectivity > ; $IPADEBUG > : Set to enable test debugging > > ; $ADMINID > : Admin username > : Default: admin > ; $ADMINPW > : Admin user password > : Default: Secret123 > ; $ROOTDN > : Directory manager DN > : Default: cn=Directory Manager > ; $ROOTDNPWD > : Directory manager password > : Default: Secret123 > > = Supporting tools = > > == ipa-test-config == > > This tool reads the configuration variables above and outputs a Bash script > that sets a much more complete set of variables for easy shell-based testing > or test set-up. > > Without arguments, ipa-test-config outputs information specific > to the host it is run on. When given a hostname, it prints config for that > host. > With the --global flag, it outputs configuration common to all > hosts. > > == ipa-run-tests == > > This tool is a wrapper arount nosetests and accepts the same > arguments > as Nose. > It loads any additional plugins and runs tests from the system-installed IPA > test suite. > > == Other == > > TBD: Additional command-line tools may be provided for tasks such as > installing > IPA in a given topology. > > = Implementation = > > Test cases are implemented as Nose test classes, with > installation/uninstallation as class setup/teardown. > > A BeakerLib plugin is provided that starts/ends Beaker phases f
Re: [Freeipa-devel] [RFE] Integration testing
On 06/04/2013 06:15 PM, Petr Vobornik wrote: Hello, Some Qs: 1) does the configuration only mean: "We have this topology, run all test which fits into it"? Not quite, it means "We have these machines, run tests whose topology can fit on them". IDK how should Web UI or XML-RPC tests fit into it. The existing test suite, including the RPC tests, stays as it is. The single-machine tests generally expect IPA to be set up on the local machine. The difference is that configuration for Web UI tests should mean: "This is how FreeIPA and related stuff is installed. Tests all available functionality and don't test the missing." IE. if I install freeipa server without CA, the UI tests needs to get that info (ie. by no_ca flag) and then skip certificate tests or parts of other tests which touches this feature (like navigation tests). The same for no-dns and has-trust... Complete UI testing would be something like following: a) set configuration A b) install server with configuration A c) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome d) uninstall server e) set configuration B f) install server with configuration B g) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome h) uninstall server i) repeat for config C ... Note:: browser change is also done by change of configuration. Is it possible? Can the configuration change or should there be other level of configuration which can change? Do we have time to do it? It may take almost a day (sequentially, with full coverage). This would be configured in Jenkins as several jobs: 1) Run UI tests with configuration A + Firefox 2) Run UI tests with configuration A + IE 3) Run UI tests with configuration A + Chrome 4) Run UI tests with configuration B + Firefox 5) Run UI tests with configuration B + IE 6) Run UI tests with configuration B + Chrome If we wanted exhaustive tests it could be done as a "matrix job" ([A, B]×[FF, IE, Chrome]), but as you say we don't have resources for that, so we'll have to use some sane subset. (Note: Jenkins can aggregate results from multiple jobs, so we'd get a single report from all these) Does python nose support some master tests or can it be somehow configured in Jenkins so we can automate installation of IPA with different configurations (IE by the 'To-be-done command-line tools'). I'm not sure what you mean by master tests. Jenkins can be configured to run different tests using different configurations. But the way I see it, the different topologies would be in different tests, and run different checks. remember that we want a CI, not exhaustive testing of all the possible cases. The command-line tools would be for external shell-based tests; I've heard some people might want to write those :) 2) There is no configuration options for trusts. IMO we would like to test that. Options for that can be added when the trust tests are written. 3) Test runner needs to connect to remote machine as root. Should a config option for root password be added or can we safely assume that there will always be other authentication method available? My bad, I forgot to add $IPA_ROOT_SSH_PASSWORD to the wiki page. Fixed. Other authentication methods can be added if needed. -- Petr³ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] [RFE] Integration testing
Hello, Some Qs: 1) does the configuration only mean: "We have this topology, run all test which fits into it"? IDK how should Web UI or XML-RPC tests fit into it. The difference is that configuration for Web UI tests should mean: "This is how FreeIPA and related stuff is installed. Tests all available functionality and don't test the missing." IE. if I install freeipa server without CA, the UI tests needs to get that info (ie. by no_ca flag) and then skip certificate tests or parts of other tests which touches this feature (like navigation tests). The same for no-dns and has-trust... Complete UI testing would be something like following: a) set configuration A b) install server with configuration A c) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome d) uninstall server e) set configuration B f) install server with configuration B g) run all UI tests (some may be skipped) i) in Firefox ii) in IE iii) in Chrome h) uninstall server i) repeat for config C ... Note:: browser change is also done by change of configuration. Is it possible? Can the configuration change or should there be other level of configuration which can change? Do we have time to do it? It may take almost a day (sequentially, with full coverage). Does python nose support some master tests or can it be somehow configured in Jenkins so we can automate installation of IPA with different configurations (IE by the 'To-be-done command-line tools'). 2) There is no configuration options for trusts. IMO we would like to test that. 3) Test runner needs to connect to remote machine as root. Should a config option for root password be added or can we safely assume that there will always be other authentication method available? Thanks On 06/03/2013 02:05 PM, Petr Viktorin wrote: Hello, A design document for integration testing is available at http://www.freeipa.org/page/V3/Integration_testing. I've copied it below for easier quoting. __NOTOC__ = Overview = Make it possible to write and run multi-host integration tests (such as: install master & replica, add user on replica, verify it's added on master). These tests will be run from continuous integration. Any developer can also run them manually. = Use Cases = == Continuous integration == The developer team at Red Hat will run a Jenkins continuous integration server that will run the tests automatically (after each commit if resources are available). The CI results will be posted publicly. == Developer testing == Anyone is be able to run integration tests without advanced infrastructure, only a number of virtual machines to run the tests on is needed. == Beaker integration == The tests will run seamlessly inside [http://beaker-project.org/ Beaker]/[https://fedoraproject.org/wiki/QA/RHTS RHTS]. A special option enables reporting via BeakerLib. = Non-goals = A complete testing/continuous integration setup needs some steps that will not be included in IPA's test suite: * Building the code * VM provisioning * Configuring the basic system, installing the packages = Design= The Python package with the IPA test suite is renamed to ipatests, and packaged for RPM-based systems as freeipa-tests. Eventually the package will be included in Fedora. Integration tests will be controlled from a single machine, and executed on a number of "remote" machines that act as servers, replicas, clients, etc. The controlling machine communicates with the others via the SSH protocol. (The controlling machine may be the same as one of the "remote" ones.) Integration tests are included in the main IPA set suite, and configured using environment variables. If the variables are missing, all integration tests are skipped. If an insufficient number of hosts is configured for a test, the individiual test will be skipped. A tool is provided to run installed tests. The remote machines used for integration testing are required to have relevant IPA packages installed, firewall opened up, any needed workarounds applied (RPM downgrades, SELinux mode,...), and sshd set up to allow root login. The test runner will connect to these machines, install IPA, perform the tests, and then uninstall IPA & return the systems to their previous state. A plugin for integration with BeakerLib is provided. = Test configuration = Tests are configured using these environment variables. == Host configuration == ; $MASTER : FQDN of the first IPA server ; $REPLICA : FQDNs of other IPA servers (space-separated) ; $CLIENT : FQDNs of IPA clients (space-separated) ; $MASTER_env2, $REPLICA_env2, $CLIENT_env2, $MASTER_env3, ... : can be used for additional domains when needed DNS needs to be set up so that IP addresses can be obtained for these hosts. == Basic configuration == ; $IPATEST_DIR : Directory for test data on the remote hosts : Default: /root/ipatests ; $DNSFORWARD : IP of a DN
