CTI - Common Test Infrastructure -----------------------------------------------------------------------
CTI is a "next generation" Testing infrastructure. CTI is currently supported in Linux and HP-UX (IPF and PA). Overview: Leverage of common test sources Allows individual developers to customize how each test is run Use a common machine pool for building and running tests Automatically generate shell scripts to help in triage/re-run Mostly a mix of perl and shell scripts Availability ----------------------------------------------------------------------- The latest code in development can be obtained via SVN at http://svn.open64.net/svnroot/open64/opencti/trunk. Installation ----------------------------------------------------------------------- Following steps give an example of typical installation process: 1. This example assumes there are 4 machines host1, host2, host3, host4. host1 and host2 are used for quality testing, host3 and host4 are used for performance testing. In this case, host1 is also used as the dTM (Distributed Test Manager) server and webserver. Before start, lets make sure the following prerequisites are met: 1.1 Set up NIS server, the 4 machines should connect to the same NIS server 1.2 Set up a NFS mount point which is accessible to all this 4 machines, say /export/cti 1.3 Install perl 5.8 in all these machines 1.4 Make a new directory /tmp/dTM and change mod to 777 like > mkdir /tmp/dTM ; chmod 777 /tmp/dTM 1.5 Install Java JVM in the CTI server machine, host1 1.6 Configure a web server with "php", "cgi" and "perl" support. 2. Logon to host1, download CTI source code to dir /export/cti/open_cti, run ./configure.pl, you can accept most of the default option except "dTM server hostname", "Web server hostname", "Web server user id" and "Web server root path". jack@host1> cd /export/cti/open_cti jack@host1> ./configure.pl CTI home path [/export/cti/open_cti]: CTI admin user id [jack]: dTM server hostname [host1.xxx.com]: dTM port number [6969]: dTM auxiliary port number [6968]: Perl path [/usr/local/bin/perl]: Java home path [/usr/lib64/jvm/java]: Java options [-Xmx768m]: load daemon port number [5010]: Remote shell command [/usr/bin/ssh]: Web server hostname []: webserver.xxx.com Web server user id [www]: Web server root path [~jack/CTI]: CTI Checking ... SUCCESS: Required Perl modules found. SUCCESS: Required Java '1.6.0' found at '/usr/lib64/jvm/java/bin/java'. Updated 'host1.xxx.com' in dtm/conf/dTM_conf.xml ... Updated 'host1.xxx.com' in doc/dtm_faq.html ... ..... 3. Set up the Web server root path. Link open_cti/www directory to web server default document directory as CTI, the command is similar as following. The value /var/www/htdocs depends on where your root web server is. jack@host1> ln -s /export/cti/open_cti/www /var/www/htdocs/CTI Now, use your web browser to access the address http://host1.xxx.com/CTI/index.html. You should be able to see the main CTI page, if everything is setup correctly. 4. Modify file export/cti/open_cti/dtm/conf/dTM_conf.xml, add Pools and machines. For ex: add 2 servers in Default pool and 2 servers in Performance pool ****BEFORE MODIFICATION**** <!-- POOLS --> <pool> <name>Default</name> <host>host_name1</host> </pool> <pool> <name>Performance</name> <host>host_name1</host> </pool> ****AFTER MODIFICATION**** <!-- POOLS --> <pool> <name>Default</name> <host>host1</host> <host>host2</host> </pool> <pool> <name>Performance</name> <host>host3</host> <host>host4</host> </pool> 5. Start the dtm server jack@host1> /export/cti/open_cti/dtm/bin/dtm.pl -start ... dTM server is coming up on host1.xxx.com:6969... dTM server is up on host1.xxx.com:6969 :-) 6. Before enabling the servers, make sure the web server can interpret CGI at directory dtm/cgi-bin and www/dtm/www/cgi-bin , for apache2 server, insert following lines to httpd.conf. The value /var/www/htdocs depends on where your root web server is. ScriptAlias /CTI/cgi-bin/ "/var/www/htdocs/CTI/cgi-bin/" ScriptAlias /CTI/dtm/www/cgi-bin/ "/var/www/htdocs/CTI/dtm/www/cgi-bin/" <Directory "/var/www/htdocs/CTI/cgi-bin"> AllowOverride None Options +ExecCGI -Includes AddHandler cgi-script cgi Order allow,deny Allow from all </Directory> <Directory "/var/www/htdocs/CTI/dtm/www/cgi-bin"> AllowOverride None Options +ExecCGI -Includes AddHandler cgi-script cgi Order allow,deny Allow from all </Directory> Restart the web server. 7. Visit the dtm servers on web, enable servers you just add. Check the dtm server at http://host1.xxx.com/CTI/dtm/www/cgi-bin/dTMState.php. You can see the servers with a yellow background in both Default and Performance Pool. To enable them, click Admin->Pool Summary, there is an column named "Enable", the value is "N", click it to enable each servers. After this, you can see the background of the server turn to blue. Quick Start To Run CTI Tests ----------------------------------------------------------------------- Contents - Prepare - Run tests with command line - Run tests through web-page - Triage test results 1. Prepare Before start the first testing, there are some items to be prepared. 1.1 Make sure CTI has been configured, if not please refer to <Installation> part 1.2 Make sure your test case has been added to {ctihome}/ cti_groups, if not please refer to: Link 1: "http://{webserver}/{webroot}/cgi-bin/addtest.cgi"; to add Regression test case Link 2: "http://{webserver}/{webroot}/CTI/doc/ cti-application-import.txt" to add Application or Performance test case 1.3 Prepare option files to specify the environment variables such as Compilers, OPT_LEVEL... There are already template option files for regression test and Perf test at "{ctihome}/cti_regression/*.opt" and "{ctihome}/cti_perf/*.opt". For more info of option files please refer to: "http://{webserver}/{webroot}/doc/CTI_Users_Guide.html#options"; 1.4 [optional] If run tests through web-page, a schedule file is also needed. Templates can be found at:"{ctihome}/ cti_regression/*.sched" and "{ctihome}/cti_perf/*.sched" 2. Run tests with command line 2.1 Tools to run tests: "TM.pl" is located at {ctihome}/bin/ 2.2 View available options for "TM.pl" ${ctihome}/bin/TM.pl --help 2.3 Take regression test for example: 2.3.1 $ "{ctihome}/bin/TM.pl -d -w {workdir} -l {logfile} \ -f {ctihome}/cti_regression/Regression.opt -nomail \ -x DTM_POOL=Default run" {workdir} and {logfile} can be set by user which are used to specify the test work directory and test log file. 2.3.2 Check result The result log is written to file {logfile}, it will show the number of PASS and FAIL. Failed cases will be listed with different error type groups[COMPILATION FAILURES , EXECUTION FAILURES, OUTPUT DIFFERENCE FAILURES] Here is a sample piece of log.[All fail because did not set compiler path] # TOTAL TESTS=112, PASS=0, FAIL=112, RUNNING=0 #_________________________________________________________ # COMPILATION FAILURES #_________________________________________________________ # Total Number of COMPILATION FAILURES = 112 Regression/apo/bug15235.c Regression/cg/bug15191_4.c Regression/cg/bug15191_7.c Regression/cg/bug15199.c Regression/cg/bug285.C Regression/fe/bug15035.cxx ...... 3. Run tests through web-page Take perf test for example[option and schedule locate at {ctihome}/cti_perf]: 3.1 Modify {ctihome}/cti_perf/*.opt Change line: export REAL_HARDWARE_MACHINE="{PERF_MACHINE_LISTS}" to the real performance machines with blank as separator ex: export REAL_HARDWARE_MACHINE="Perf_machine1 Perf_machine2 .... Perf_machineN" 3.2 Open link "http://{webserver}/{webroot}/cgi-bin/show-schedule.cgi? sched={ctihome}/cti_perf/perf.sched" There are link at the top of the page: CTI Home: Document page dTM server: DTM server page to show the current machine status and current tasks Show failures: More details of daily failure Schedule file:current page Start tests: Run test Show time taken: show how much time taken for current testing 3.3 Click "Start tests", choose the list to run, input "username", Click "Do it!" 3.4 Check the on-going tests at "http://{webserver}/{webroot} /CTI/dtm/www/cgi-bin/dTMState.php?dumpUser="username" ", the username is set at step3.3. 3.5 Go back to page "http://{webserver}/{webroot}/cgi-bin/show-schedule.cgi ?sched={ctihome}/cti_perf/perf.sched" It will show the result in the table, like below: Options file, view/dates {dates} fail/total Wed fail/total ... perf.rate.opt {dates} 0 / 1 ... perf.speed.opt {dates} 0 / 1 ... 3.6 Check log: Click link "{dates}" [http://{webserver}/{webroot}/cgi-bin/ get-log-file.cgi?log={ctihome}/cti_perf/log.{dates}.perf.rate.opt] will show the log file 4. Triage test rest results After tests finishes, there will be a workdir saved which maintains auto-generated shell scripts for triage. Take above perf test for example, it's workdir is at {ctihome}/cti_perf/work.{dates}.perf.rate.opt 4.1 cd {workdir} 4.2 There are generally following files: *.env - Environment variables + options that you passed in your option file *.compile.sh - Script to recompile *.run.sh - Script to rerun test *.result - Success/failure message compare.results - should be empty if successful, else has the diff of your output with the master output Soft links to actual sources (*.c, cpp) under {cti_groups} directory 4.3 If the error type is COMPILATION FAILURE run script *.compile.sh, it will generates *.comp.err with the actual invocation and output of the compile 4.4 If the error type is EXECUTION FAILURE run script *.run.sh, it will generate *.run.out with the output The best way to learn is to read the *.sh and try to understand the calling relations. For more info of CTI document please refer to "http://{webserver}/{webroot}"; Have Fun with CTI testing! -- Regards, Lai Jian-Xin ------------------------------------------------------------------------------ Virtualization & Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ _______________________________________________ Open64-devel mailing list Open64-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/open64-devel
