Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
On Nov 9, 2010, at 1:00 AM, DongInn Kim wrote: > No, I did not know that it should be added in the MPI Get phase. You have to call all the phases, even if they don't "do" anything. That's why we have no-op / alreadyinstalled versions of plugins, for example. Each phase sets up data structures that are used by subsequent phases. > OK, I tried to add it the MPI Get phase and mpi_details are recognized but I > could not have "Test Run" phase run the scripts in before_any_exec and > after_all_exec. What exactly do you have in your ini file again for these fields? I have this in an old ini file -- it *used* to work (when launching MPICH2 and Intel MPI jobs through MTT): before_any_exec = < $file # Add localhost if it's not in there (e.g., srun -A) local=`grep $h $file` touch /tmp/mtt-mpiexec-options.$SLURM_JOBID if test "$local" = ""; then echo $h >> $file echo -nolocal > /tmp/mpiexec-options.$SLURM_JOBID fi num=`wc -l $file | awk '{ print $1 }'` mpdboot -n $num -r ssh --verbose --file=$file mpdtrace rm -f $file EOF -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
No, I did not know that it should be added in the MPI Get phase. OK, I tried to add it the MPI Get phase and mpi_details are recognized but I could not have "Test Run" phase run the scripts in before_any_exec and after_all_exec. DB<25> p Dumper($self->{v}{$sect}) $VAR1 = { 'program_dir' => '/u/ftbteam/projects/mtt/client', 'after_all_exec' => [ 'ftb_db_pid=`pgrep ftb_database_server`', 'kill $ftb_db_pid', 'ftb_agent_pid=`pgrep ftb_agent`', 'kill $ftb_agent_pid' ], 'before_any_exec' => [ 'install_dir=&test_prefix_pretty()', 'ftb_server_daemon="$install_dir/sbin/ftb_database_server"', 'ftb_agent_daemon="$install_dir/sbin/ftb_agent"', '$ftb_server_daemon &', '$ftb_agent_daemon' ], 'program_name' => '/u/ftbteam/projects/mtt/client/ftt', 'ini_section_name' => 'platform', 'ini_name' => '/u/ftbteam/ftt-runs/odin/20101108-Nightly/ini/g-br-trunk_i-32gccFT_ftb-gen' }; It seems that entries of before_** and after_** are filled in to the steps to run but it could not run because of the malformed syntax. So, I removed the entries and then just "touch /tmp/foo" in before_** and after_** but it did not create the empty file /tmp/foo and I got the following errors in the perl debug mode. *** ERROR: Module aborted: MTT::Test::Specify::Simple:Specify: Bizarre copy of ARRAY in leave at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Values.pm line 118, line 63. at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Messages.pm line 122 MTT::Messages::Abort('Module aborted: MTT::Test::Specify::Simple:Specify: Bizarre c...') called at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Messages.pm line 85 MTT::Messages::Error('Module aborted: MTT::Test::Specify::Simple:Specify: Bizarre c...') called at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Module.pm line 52 MTT::Module::Run('MTT::Test::Specify::Simple', 'Specify', 'Config::IniFiles=HASH(0x19dfea0)', 'test run: ftb', 'HASH(0x19ebba0)', 'HASH(0x19e4940)', 'HASH(0x19df600)') called at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Test/Specify.pm line 24 MTT::Test::Specify::Specify('Simple', 'Config::IniFiles=HASH(0x19dfea0)', 'test run: ftb', 'HASH(0x19ebba0)', 'HASH(0x19e4940)', 'HASH(0x19df600)') called at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Test/Run.pm line 552 MTT::Test::Run::_do_run('Config::IniFiles=HASH(0x19dfea0)', 'test run: ftb', 'HASH(0x19ebba0)', 'HASH(0x19dfdf0)', 'HASH(0x19e4940)', '/u/ftbteam/ftt-runs/odin/20101109-Nightly/pb_0/installs', '/u/ftbteam/ftt-runs/odin/20101109-Nightly/pb_0/test_runs', 'undef') called at /nfs/rinfs/san/homedirs/ftbteam/projects/mtt/lib/MTT/Test/Run.pm line 200 MTT::Test::Run::Run('Config::IniFiles=HASH(0x19dfea0)', 'Config::IniFiles=HASH(0x19ba940)', '/u/ftbteam/ftt-runs/odin/20101109-Nightly/pb_0/installs', '/u/ftbteam/ftt-runs/odin/20101109-Nightly/pb_0/test_runs', 'undef') called at /u/ftbteam/projects/mtt/client/ftt line 557 Can you please help this out? Regards, On 11/8/10 5:27 PM, Jeff Squyres wrote: > Are you executing an MPI Get section? > > I believe that the mpi_details section is filled in via the MPI Get phase and > then propagated down through all the other phases (i.e., each of the other > phases looks a the way back to their corresponding mpi get phase to find > the mpi_details value). > > > On Nov 8, 2010, at 5:16 PM, DongInn Kim wrote: > >> Jeff, thank you. >> >> BTW, I have looked at the ompi-core-perf-testing.ini file which seems to >> have used the mpi detail sections and I tried to use it in our ftb.ini file >> but I still get the same warning message. >> >> *** Test Run phase starting Test Run [ftb] Running with [ftb-nightly-trunk] / [0.6.2] / [platform] >> *** WARNING: Unable to find MPI details section for [MPI Install: platform; >>skipping >> *** Run test phase complete Phase: Test Run >> Started: Mon Nov 8 17:10:30 2010 >> Stopped: Mon Nov 8 17:10:31 2010 >> Elapsed: 00:00:01 0.02u 0.06s >> Total elapsed: 00:00:01 0.02u 0.06s Phase: Trim >> Started: Mon Nov 8 17:10:31 2010 >> Stopped: Mon Nov 8 17:10:31 2010 >> Elapsed: 00:00:00 0.00u 0.00s >> Total elapsed: 00:00:01 0.02u 0.06s >> *** Reporter finalizing >> *** Reporter finalized >> >> >> Here is the entry in the new ftb.ini file. >> #-- >> >> >> [MPI Details: platform] >> >> # Need a before
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Are you executing an MPI Get section? I believe that the mpi_details section is filled in via the MPI Get phase and then propagated down through all the other phases (i.e., each of the other phases looks a the way back to their corresponding mpi get phase to find the mpi_details value). On Nov 8, 2010, at 5:16 PM, DongInn Kim wrote: > Jeff, thank you. > > BTW, I have looked at the ompi-core-perf-testing.ini file which seems to have > used the mpi detail sections and I tried to use it in our ftb.ini file but I > still get the same warning message. > > *** Test Run phase starting >>> Test Run [ftb] >>> Running with [ftb-nightly-trunk] / [0.6.2] / [platform] > *** WARNING: Unable to find MPI details section for [MPI Install: platform; >skipping > *** Run test phase complete >>> Phase: Test Run > Started: Mon Nov 8 17:10:30 2010 > Stopped: Mon Nov 8 17:10:31 2010 > Elapsed: 00:00:01 0.02u 0.06s > Total elapsed: 00:00:01 0.02u 0.06s >>> Phase: Trim > Started: Mon Nov 8 17:10:31 2010 > Stopped: Mon Nov 8 17:10:31 2010 > Elapsed: 00:00:00 0.00u 0.00s > Total elapsed: 00:00:01 0.02u 0.06s > *** Reporter finalizing > *** Reporter finalized > > > Here is the entry in the new ftb.ini file. > #-- > > > [MPI Details: platform] > > # Need a before_any_exec step to test all the FTB example tests > > before_any_exec = < install_dir=&test_prefix_pretty() > > ftb_server_daemon="$install_dir/sbin/ftb_database_server" > > ftb_agent_daemon="$install_dir/sbin/ftb_agent" > > $ftb_server_daemon & > $ftb_agent_daemon > EOF > > after_all_exec = < > ftb_db_pid=`pgrep ftb_database_server` > kill $ftb_db_pid > ftb_agent_pid=`pgrep ftb_agent` > > kill $ftb_agent_pid > EOT > > #-- > > I have tried to replace "platform" with "FTB" in "[MPI Details: platform]" > but it still did not work. > > Any helps on this? > > Regards, > > > On 11/8/10 3:42 PM, Jeff Squyres wrote: >> Sorry for jumping in late -- been swamped recently... >> >> In the MPI details section, there's 4 fields that should let you do what you >> want. >> >> before_any_exec -- run once before all the tests in a given Test Run >> before_each_exec -- run once before every single exec (including all >> variants) >> after_each_exec -- run after after every single exec (include all variants) >> after_all_exec -- run after all tests in a given Test Run section have >> completed >> >> So you can use the before_any_exec / after_all_exec to launch the daemons >> once at the beginning and then take them down, or you can use >> before_each_exec / after_each_exec to launch the daemons before each test >> and then take them down at the end of that test. >> >> I'm assuming that the *each* variants will cause your tests to run much >> slower. >> >> I see that we don't have an MPI Details section on the wiki describing these >> parameters. Sorry! :-( >> >> > > > -- > - DongInn > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Jeff, thank you. BTW, I have looked at the ompi-core-perf-testing.ini file which seems to have used the mpi detail sections and I tried to use it in our ftb.ini file but I still get the same warning message. *** Test Run phase starting >> Test Run [ftb] >> Running with [ftb-nightly-trunk] / [0.6.2] / [platform] *** WARNING: Unable to find MPI details section for [MPI Install: platform; skipping *** Run test phase complete >> Phase: Test Run Started: Mon Nov 8 17:10:30 2010 Stopped: Mon Nov 8 17:10:31 2010 Elapsed: 00:00:01 0.02u 0.06s Total elapsed: 00:00:01 0.02u 0.06s >> Phase: Trim Started: Mon Nov 8 17:10:31 2010 Stopped: Mon Nov 8 17:10:31 2010 Elapsed: 00:00:00 0.00u 0.00s Total elapsed: 00:00:01 0.02u 0.06s *** Reporter finalizing *** Reporter finalized Here is the entry in the new ftb.ini file. #-- [MPI Details: platform] # Need a before_any_exec step to test all the FTB example tests before_any_exec = < Sorry for jumping in late -- been swamped recently... > > In the MPI details section, there's 4 fields that should let you do what you > want. > > before_any_exec -- run once before all the tests in a given Test Run > before_each_exec -- run once before every single exec (including all variants) > after_each_exec -- run after after every single exec (include all variants) > after_all_exec -- run after all tests in a given Test Run section have > completed > > So you can use the before_any_exec / after_all_exec to launch the daemons > once at the beginning and then take them down, or you can use > before_each_exec / after_each_exec to launch the daemons before each test and > then take them down at the end of that test. > > I'm assuming that the *each* variants will cause your tests to run much > slower. > > I see that we don't have an MPI Details section on the wiki describing these > parameters. Sorry! :-( > > -- - DongInn
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Sorry for jumping in late -- been swamped recently... In the MPI details section, there's 4 fields that should let you do what you want. before_any_exec -- run once before all the tests in a given Test Run before_each_exec -- run once before every single exec (including all variants) after_each_exec -- run after after every single exec (include all variants) after_all_exec -- run after all tests in a given Test Run section have completed So you can use the before_any_exec / after_all_exec to launch the daemons once at the beginning and then take them down, or you can use before_each_exec / after_each_exec to launch the daemons before each test and then take them down at the end of that test. I'm assuming that the *each* variants will cause your tests to run much slower. I see that we don't have an MPI Details section on the wiki describing these parameters. Sorry! :-( On Nov 7, 2010, at 10:05 AM, DongInn Kim wrote: > Hi Josh, > > Hmm, I still can not define "mpi_details" in the ini file. Is there something > wrong with my ini file? > DB<32> p $mpi_install->{mpi_details} > > DB<33> > > DB<33> p Dumper($mpi_install) > $VAR1 = { > 'ftb_get_full_section_name' => 'ftb get: ftb-nightly-trunk', > 'prepend_path' => undef, > 'mpi_name' => 'ftb-nightly-trunk', > 'compiler_version' => 'unknown', > 'pid' => 23511, > 'elapsed_iters' => '0.02', > 'bindir' => > '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD/install/bin', > 'merge_stdout_stderr' => 0, > 'elapsed_user' => '0', > 'unsetenv' => undef, > 'cxx_bindings' => '0', > 'description' => undef, > 'f90_bindings' => '0', > 'exit_status' => undef, > 'libdir' => > '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD/install/lib', > 'mpi_get_simple_section_name' => 'ftb-nightly-trunk', > 'elapsed_real' => 0, > 'f77_bindings' => '0', > 'source_dir' => 'ftb-0.6.2', > 'mpi_version' => '0.6.2', > 'sym_link_name' => 'ftb-nightly-trunk--platform--0.6.2', > 'c_bindings' => 1, > 'full_section_name' => 'ftb install: platform', > 'vpath_mode' => 0, > 'elapsed_children_system' => '0.02', > 'timed_out' => 0, > 'refcount' => 0, > 'env_modules' => undef, > 'elapsed_children_user' => '0', > 'start_benchmark' => bless( [ >1289110689, >'0.4', >'0.12', >'2.48', >'1.8', >0 > ], 'Benchmark' ), > 'version_dir' => > '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD', > 'build_version_dir' => > '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/ftb-install/ohiy', > 'ftb_get_simple_section_name' => 'ftb-nightly-trunk', > 'ftb_version' => '0.6.2', > 'result_message' => 'Success', > 'append_path' => undef, > 'configure_arguments' => '"CFLAGS=-g -pipe" --enable-debug ', > 'installdir' => > '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD/install', > 'test_result' => 1, > 'compiler_name' => 'unknown', > 'build_dir' => > '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/ftb-install/ohiy/src/ftb-0.6.2', > 'simple_section_name' => 'platform', > 'setenv' => undef, > 'start_timestamp' => 1289110690, > 'stop_benchmark' => bless( [ > 1289110689, > '0.4', > '0.12', > '2.5', > '1.82', > 0 > ], 'Benchmark' ) >}; > > DB<34> > > > The ini files that I used are > general_ini_base = $HOME/iu-ftt/odin/etc/general > > general_ini = generic-subs2.config > > specific_ini_base = $HOME/iu-ftt/etc/ftt2 > specific_ini = BEGIN > g-br-trunk,i-32gccFT,ftb > END > > It seems that mtt does not pickup the entries that I put in > generic-sub2.config. > #-- > #-- > > [FTB Details: ftb] > exec = ./run-ftb-tests.pl -test &test_executable() > > > #-- > #
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Hi Josh, Hmm, I still can not define "mpi_details" in the ini file. Is there something wrong with my ini file? DB<32> p $mpi_install->{mpi_details} DB<33> DB<33> p Dumper($mpi_install) $VAR1 = { 'ftb_get_full_section_name' => 'ftb get: ftb-nightly-trunk', 'prepend_path' => undef, 'mpi_name' => 'ftb-nightly-trunk', 'compiler_version' => 'unknown', 'pid' => 23511, 'elapsed_iters' => '0.02', 'bindir' => '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD/install/bin', 'merge_stdout_stderr' => 0, 'elapsed_user' => '0', 'unsetenv' => undef, 'cxx_bindings' => '0', 'description' => undef, 'f90_bindings' => '0', 'exit_status' => undef, 'libdir' => '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD/install/lib', 'mpi_get_simple_section_name' => 'ftb-nightly-trunk', 'elapsed_real' => 0, 'f77_bindings' => '0', 'source_dir' => 'ftb-0.6.2', 'mpi_version' => '0.6.2', 'sym_link_name' => 'ftb-nightly-trunk--platform--0.6.2', 'c_bindings' => 1, 'full_section_name' => 'ftb install: platform', 'vpath_mode' => 0, 'elapsed_children_system' => '0.02', 'timed_out' => 0, 'refcount' => 0, 'env_modules' => undef, 'elapsed_children_user' => '0', 'start_benchmark' => bless( [ 1289110689, '0.4', '0.12', '2.48', '1.8', 0 ], 'Benchmark' ), 'version_dir' => '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD', 'build_version_dir' => '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/ftb-install/ohiy', 'ftb_get_simple_section_name' => 'ftb-nightly-trunk', 'ftb_version' => '0.6.2', 'result_message' => 'Success', 'append_path' => undef, 'configure_arguments' => '"CFLAGS=-g -pipe" --enable-debug ', 'installdir' => '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/installs/c1aD/install', 'test_result' => 1, 'compiler_name' => 'unknown', 'build_dir' => '/nfs/rinfs/san/homedirs/ftbteam/ftt-runs/odin/20101107-Nightly/pb_0/ftb-install/ohiy/src/ftb-0.6.2', 'simple_section_name' => 'platform', 'setenv' => undef, 'start_timestamp' => 1289110690, 'stop_benchmark' => bless( [ 1289110689, '0.4', '0.12', '2.5', '1.82', 0 ], 'Benchmark' ) }; DB<34> The ini files that I used are general_ini_base = $HOME/iu-ftt/odin/etc/general general_ini = generic-subs2.config specific_ini_base = $HOME/iu-ftt/etc/ftt2 specific_ini = BEGIN g-br-trunk,i-32gccFT,ftb END It seems that mtt does not pickup the entries that I put in generic-sub2.config. #-- #-- [FTB Details: ftb] exec = ./run-ftb-tests.pl -test &test_executable() #-- #-- So, I manually added the line "exec = ./run-ftb-tests.pl -test &test_executable()" into the auto-generated ini file "20101107-Nightly/ini/g-br-trunk_i-32gccFT_ftb-gen" and I did not put "mpi_details" entry to it because it does not seem to be added there and it should be defined in generic-sub2.config. Anyway, this tweak did not work either. :-( Any helps would be really appreciated. I attached the current ftb.ini file. This is the "test run" hash entry in Dumper($ini). 'test run: ftb' => { 'simple_pass:tests' => '&enumerate( "ftb_eventhandle_example", "ftb_example_callback_unsubscribe", "ftb_multiplecomp", "ftb_notify_logger", "ftb_pingpong", "ftb_polling_logger", "ftb_simple_publisher", "ftb_simple_subscriber", "ftb_watchdog")', 'np' => '&env_max
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Josh, thanks but actually I don't know how to set the base directory of the test suite in the Test Get/Build phase. Is it setup in the ini file or somewhere? and how? Regards, On 11/5/10 11:32 AM, Joshua Hursey wrote: > > On Nov 5, 2010, at 10:11 AM, DongInn Kim wrote: > >> Josh, I have another question. >> How can mtt find the script to run? exec = ./run-cr-correctness.pl -test ... >> >> I can write a similar script like run-correctness.pl but if I put my >> script(e.g., run-ftb-tests.pl) to ftb-tests/iu/ftt/ftb/, how can I make mtt >> recognize this path and file? > > The file is in the ompi-tests repo, under: >ompi-tests/iu/ft/correctness/run-correctness.pl > > Remember that the Test Run phase will 'cd' into the base directory of the > test suite that you specify in the Test Get/Build phase. So you can reference > a relative path with respect to the test suite. So just put the script in > with the test suite, and the Test Get phase will grab it for you. > > -- Josh > >> >> Regards, >> >> On 11/5/10 9:01 AM, DongInn Kim wrote: >>> Hi Josh, Do you need to restart the FTB daemons before every test, or just start them once before running an entire test suite? >>> I believe that the daemon needs to run once before running an entire test >>> set. Abhishek, can you confirm? >>> >>> I would follow your suggestion(i.e., add the script to run the daemons to >>> the "Test Run" phase) and I will let you know how it goes. Thanks. >>> >>> BTW, can I take a look at the script "run-cr-correctness.pl"? It seems that >>> it is not under the mtt repository. >>> >>> Regards, >>> >>> On 11/5/10 8:40 AM, Joshua Hursey wrote: DongInn, Do you need to restart the FTB daemons before every test, or just start them once before running an entire test suite? Adding another phase is pretty involved (will likely require changes to every piece of MTT - client, database, reporter), so I would suggest doing something like this in 'Test Run'. Something like a pre-exec hook. As an alternative you could write a script that would encapsulate: - Launch FTB Daemons - Run a specific test - Shutdown FTB Daemons This would occur for each test in the 'Test Run' phase. This is similar to what I did for the C/R support. So in the 'MPI Details' section I have something like: [MPI Details: OMPI] exec = ./run-cr-correctness.pl -test &test_executable() \ -storage-dir /storage/tmp/snapshot/ \ -np &test_np() \ -mca "@fixes@ -am ft-enable-cr @pml@" \ --prefix &test_prefix() -v 2 fixes = -mca orte_base_help_aggregate 0 Then for each of the Test Runs the '&test_executable()' funclet is filled in with the test name, and the perl script handles the rest of the launch from there. If it is helpful, the C/R 'Test Get/Build/Run' phases are in ompi-tests/iu/mtt/etc/mtt2/ftcr.ini. And the 'MPI Details' is under "OMPI DETAILS COMMON FTCR TEST" in ompi-tests/iu/mtt/odin/etc/general/generic-subs2.config Let me know if that helps. -- Josh On Nov 5, 2010, at 2:06 AM, DongInn Kim wrote: > Hi, > > Is there any way to put some lines to the mtt ini file to run the FTB > daemons(ftb_database_server and ftb_agent)? > > Should we make another "FTB Run" phase after "FTB Install" or run the two > daemons inside "Test run" but run the daemons before running the test > scripts? > I think either way would be fine if I have a working syntax for the both > cases. > > Any help would be really appreciated. > > Regards, > > -- > - DongInn > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey ___ mtt-devel mailing list mtt-de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >>> >> >> -- >> - DongInn >> ___ >> mtt-devel mailing list >> mtt-de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >> > > > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel -- - DongInn
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
On Nov 5, 2010, at 10:11 AM, DongInn Kim wrote: > Josh, I have another question. > How can mtt find the script to run? >>> exec = ./run-cr-correctness.pl -test ... > > I can write a similar script like run-correctness.pl but if I put my > script(e.g., run-ftb-tests.pl) to ftb-tests/iu/ftt/ftb/, how can I make mtt > recognize this path and file? The file is in the ompi-tests repo, under: ompi-tests/iu/ft/correctness/run-correctness.pl Remember that the Test Run phase will 'cd' into the base directory of the test suite that you specify in the Test Get/Build phase. So you can reference a relative path with respect to the test suite. So just put the script in with the test suite, and the Test Get phase will grab it for you. -- Josh > > Regards, > > On 11/5/10 9:01 AM, DongInn Kim wrote: >> Hi Josh, >>> Do you need to restart the FTB daemons before every test, or just start >>> them once before running an entire test suite? >> I believe that the daemon needs to run once before running an entire test >> set. Abhishek, can you confirm? >> >> I would follow your suggestion(i.e., add the script to run the daemons to >> the "Test Run" phase) and I will let you know how it goes. Thanks. >> >> BTW, can I take a look at the script "run-cr-correctness.pl"? It seems that >> it is not under the mtt repository. >> >> Regards, >> >> On 11/5/10 8:40 AM, Joshua Hursey wrote: >>> DongInn, >>> >>> Do you need to restart the FTB daemons before every test, or just start >>> them once before running an entire test suite? >>> >>> >>> Adding another phase is pretty involved (will likely require changes to >>> every piece of MTT - client, database, reporter), so I would suggest doing >>> something like this in 'Test Run'. Something like a pre-exec hook. >>> >>> As an alternative you could write a script that would encapsulate: >>> - Launch FTB Daemons >>> - Run a specific test >>> - Shutdown FTB Daemons >>> >>> This would occur for each test in the 'Test Run' phase. This is similar to >>> what I did for the C/R support. So in the 'MPI Details' section I have >>> something like: >>> >>> [MPI Details: OMPI] >>> exec = ./run-cr-correctness.pl -test &test_executable() \ >>> -storage-dir /storage/tmp/snapshot/ \ >>> -np &test_np() \ >>> -mca "@fixes@ -am ft-enable-cr @pml@" \ >>> --prefix &test_prefix() -v 2 >>> >>> fixes = -mca orte_base_help_aggregate 0 >>> >>> >>> Then for each of the Test Runs the '&test_executable()' funclet is filled >>> in with the test name, and the perl script handles the rest of the launch >>> from there. >>> >>> If it is helpful, the C/R 'Test Get/Build/Run' phases are in >>> ompi-tests/iu/mtt/etc/mtt2/ftcr.ini. >>> And the 'MPI Details' is under "OMPI DETAILS COMMON FTCR TEST" in >>> ompi-tests/iu/mtt/odin/etc/general/generic-subs2.config >>> >>> Let me know if that helps. >>> >>> -- Josh >>> >>> On Nov 5, 2010, at 2:06 AM, DongInn Kim wrote: >>> Hi, Is there any way to put some lines to the mtt ini file to run the FTB daemons(ftb_database_server and ftb_agent)? Should we make another "FTB Run" phase after "FTB Install" or run the two daemons inside "Test run" but run the daemons before running the test scripts? I think either way would be fine if I have a working syntax for the both cases. Any help would be really appreciated. Regards, -- - DongInn ___ mtt-devel mailing list mtt-de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >>> >>> >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> >>> ___ >>> mtt-devel mailing list >>> mtt-de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >> > > -- > - DongInn > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
On Fri, Nov 5, 2010 at 9:01 AM, DongInn Kim wrote: > Hi Josh, > > Do you need to restart the FTB daemons before every test, or just start > them once before running an entire test suite? > I believe that the daemon needs to run once before running an entire test > set. Abhishek, can you confirm? > > I believe you would want to run the testsuite with a variable number of nodes participating in the FTB. Ergo, it would be good to respawn the daemons before every testsuite. Some tests might also involve killing a FTB daemon (ftb_agent) while a test is being run to test the fault-tolerant routing in the network. The ft/correctness examples in the ompi-tests/ are really a good way to start. You'd also want a wrapper similar to run-cr-correctness.pl to maybe check if the tests have passed successfully or not. Abhishek > I would follow your suggestion(i.e., add the script to run the daemons to > the "Test Run" phase) and I will let you know how it goes. Thanks. > > BTW, can I take a look at the script "run-cr-correctness.pl"? It seems > that it is not under the mtt repository. > Regards, > > On 11/5/10 8:40 AM, Joshua Hursey wrote: > > DongInn, > > > > Do you need to restart the FTB daemons before every test, or just start > them once before running an entire test suite? > > > > > > Adding another phase is pretty involved (will likely require changes to > every piece of MTT - client, database, reporter), so I would suggest doing > something like this in 'Test Run'. Something like a pre-exec hook. > > > > As an alternative you could write a script that would encapsulate: > > - Launch FTB Daemons > > - Run a specific test > > - Shutdown FTB Daemons > > > > This would occur for each test in the 'Test Run' phase. This is similar > to what I did for the C/R support. So in the 'MPI Details' section I have > something like: > > > > [MPI Details: OMPI] > > exec = ./run-cr-correctness.pl -test &test_executable() \ > >-storage-dir /storage/tmp/snapshot/ \ > >-np &test_np() \ > >-mca "@fixes@ -am ft-enable-cr @pml@" \ > >--prefix &test_prefix() -v 2 > > > > fixes = -mca orte_base_help_aggregate 0 > > > > > > Then for each of the Test Runs the '&test_executable()' funclet is filled > in with the test name, and the perl script handles the rest of the launch > from there. > > > > If it is helpful, the C/R 'Test Get/Build/Run' phases are in > ompi-tests/iu/mtt/etc/mtt2/ftcr.ini. > > And the 'MPI Details' is under "OMPI DETAILS COMMON FTCR TEST" in > ompi-tests/iu/mtt/odin/etc/general/generic-subs2.config > > > > Let me know if that helps. > > > > -- Josh > > > > On Nov 5, 2010, at 2:06 AM, DongInn Kim wrote: > > > >> Hi, > >> > >> Is there any way to put some lines to the mtt ini file to run the FTB > daemons(ftb_database_server and ftb_agent)? > >> > >> Should we make another "FTB Run" phase after "FTB Install" or run the > two daemons inside "Test run" but run the daemons before running the test > scripts? > >> I think either way would be fine if I have a working syntax for the both > cases. > >> > >> Any help would be really appreciated. > >> > >> Regards, > >> > >> -- > >> - DongInn > >> ___ > >> mtt-devel mailing list > >> mtt-de...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > >> > > > > > > Joshua Hursey > > Postdoctoral Research Associate > > Oak Ridge National Laboratory > > http://users.nccs.gov/~jjhursey > > > > > > ___ > > mtt-devel mailing list > > mtt-de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > > -- > - DongInn > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Josh, I have another question. How can mtt find the script to run? >> exec = ./run-cr-correctness.pl -test ... I can write a similar script like run-correctness.pl but if I put my script(e.g., run-ftb-tests.pl) to ftb-tests/iu/ftt/ftb/, how can I make mtt recognize this path and file? Regards, On 11/5/10 9:01 AM, DongInn Kim wrote: > Hi Josh, >> Do you need to restart the FTB daemons before every test, or just start them >> once before running an entire test suite? > I believe that the daemon needs to run once before running an entire test > set. Abhishek, can you confirm? > > I would follow your suggestion(i.e., add the script to run the daemons to the > "Test Run" phase) and I will let you know how it goes. Thanks. > > BTW, can I take a look at the script "run-cr-correctness.pl"? It seems that > it is not under the mtt repository. > > Regards, > > On 11/5/10 8:40 AM, Joshua Hursey wrote: >> DongInn, >> >> Do you need to restart the FTB daemons before every test, or just start them >> once before running an entire test suite? >> >> >> Adding another phase is pretty involved (will likely require changes to >> every piece of MTT - client, database, reporter), so I would suggest doing >> something like this in 'Test Run'. Something like a pre-exec hook. >> >> As an alternative you could write a script that would encapsulate: >> - Launch FTB Daemons >> - Run a specific test >> - Shutdown FTB Daemons >> >> This would occur for each test in the 'Test Run' phase. This is similar to >> what I did for the C/R support. So in the 'MPI Details' section I have >> something like: >> >> [MPI Details: OMPI] >> exec = ./run-cr-correctness.pl -test &test_executable() \ >>-storage-dir /storage/tmp/snapshot/ \ >>-np &test_np() \ >>-mca "@fixes@ -am ft-enable-cr @pml@" \ >>--prefix &test_prefix() -v 2 >> >> fixes = -mca orte_base_help_aggregate 0 >> >> >> Then for each of the Test Runs the '&test_executable()' funclet is filled in >> with the test name, and the perl script handles the rest of the launch from >> there. >> >> If it is helpful, the C/R 'Test Get/Build/Run' phases are in >> ompi-tests/iu/mtt/etc/mtt2/ftcr.ini. >> And the 'MPI Details' is under "OMPI DETAILS COMMON FTCR TEST" in >> ompi-tests/iu/mtt/odin/etc/general/generic-subs2.config >> >> Let me know if that helps. >> >> -- Josh >> >> On Nov 5, 2010, at 2:06 AM, DongInn Kim wrote: >> >>> Hi, >>> >>> Is there any way to put some lines to the mtt ini file to run the FTB >>> daemons(ftb_database_server and ftb_agent)? >>> >>> Should we make another "FTB Run" phase after "FTB Install" or run the two >>> daemons inside "Test run" but run the daemons before running the test >>> scripts? >>> I think either way would be fine if I have a working syntax for the both >>> cases. >>> >>> Any help would be really appreciated. >>> >>> Regards, >>> >>> -- >>> - DongInn >>> ___ >>> mtt-devel mailing list >>> mtt-de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >>> >> >> >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> ___ >> mtt-devel mailing list >> mtt-de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > -- - DongInn
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
Hi Josh, > Do you need to restart the FTB daemons before every test, or just start them > once before running an entire test suite? I believe that the daemon needs to run once before running an entire test set. Abhishek, can you confirm? I would follow your suggestion(i.e., add the script to run the daemons to the "Test Run" phase) and I will let you know how it goes. Thanks. BTW, can I take a look at the script "run-cr-correctness.pl"? It seems that it is not under the mtt repository. Regards, On 11/5/10 8:40 AM, Joshua Hursey wrote: > DongInn, > > Do you need to restart the FTB daemons before every test, or just start them > once before running an entire test suite? > > > Adding another phase is pretty involved (will likely require changes to every > piece of MTT - client, database, reporter), so I would suggest doing > something like this in 'Test Run'. Something like a pre-exec hook. > > As an alternative you could write a script that would encapsulate: > - Launch FTB Daemons > - Run a specific test > - Shutdown FTB Daemons > > This would occur for each test in the 'Test Run' phase. This is similar to > what I did for the C/R support. So in the 'MPI Details' section I have > something like: > > [MPI Details: OMPI] > exec = ./run-cr-correctness.pl -test &test_executable() \ >-storage-dir /storage/tmp/snapshot/ \ >-np &test_np() \ >-mca "@fixes@ -am ft-enable-cr @pml@" \ >--prefix &test_prefix() -v 2 > > fixes = -mca orte_base_help_aggregate 0 > > > Then for each of the Test Runs the '&test_executable()' funclet is filled in > with the test name, and the perl script handles the rest of the launch from > there. > > If it is helpful, the C/R 'Test Get/Build/Run' phases are in > ompi-tests/iu/mtt/etc/mtt2/ftcr.ini. > And the 'MPI Details' is under "OMPI DETAILS COMMON FTCR TEST" in > ompi-tests/iu/mtt/odin/etc/general/generic-subs2.config > > Let me know if that helps. > > -- Josh > > On Nov 5, 2010, at 2:06 AM, DongInn Kim wrote: > >> Hi, >> >> Is there any way to put some lines to the mtt ini file to run the FTB >> daemons(ftb_database_server and ftb_agent)? >> >> Should we make another "FTB Run" phase after "FTB Install" or run the two >> daemons inside "Test run" but run the daemons before running the test >> scripts? >> I think either way would be fine if I have a working syntax for the both >> cases. >> >> Any help would be really appreciated. >> >> Regards, >> >> -- >> - DongInn >> ___ >> mtt-devel mailing list >> mtt-de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel >> > > > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel -- - DongInn
Re: [MTT devel] Best way to run ftb_database_server and ftb_agent
DongInn, Do you need to restart the FTB daemons before every test, or just start them once before running an entire test suite? Adding another phase is pretty involved (will likely require changes to every piece of MTT - client, database, reporter), so I would suggest doing something like this in 'Test Run'. Something like a pre-exec hook. As an alternative you could write a script that would encapsulate: - Launch FTB Daemons - Run a specific test - Shutdown FTB Daemons This would occur for each test in the 'Test Run' phase. This is similar to what I did for the C/R support. So in the 'MPI Details' section I have something like: [MPI Details: OMPI] exec = ./run-cr-correctness.pl -test &test_executable() \ -storage-dir /storage/tmp/snapshot/ \ -np &test_np() \ -mca "@fixes@ -am ft-enable-cr @pml@" \ --prefix &test_prefix() -v 2 fixes = -mca orte_base_help_aggregate 0 Then for each of the Test Runs the '&test_executable()' funclet is filled in with the test name, and the perl script handles the rest of the launch from there. If it is helpful, the C/R 'Test Get/Build/Run' phases are in ompi-tests/iu/mtt/etc/mtt2/ftcr.ini. And the 'MPI Details' is under "OMPI DETAILS COMMON FTCR TEST" in ompi-tests/iu/mtt/odin/etc/general/generic-subs2.config Let me know if that helps. -- Josh On Nov 5, 2010, at 2:06 AM, DongInn Kim wrote: > Hi, > > Is there any way to put some lines to the mtt ini file to run the FTB > daemons(ftb_database_server and ftb_agent)? > > Should we make another "FTB Run" phase after "FTB Install" or run the two > daemons inside "Test run" but run the daemons before running the test scripts? > I think either way would be fine if I have a working syntax for the both > cases. > > Any help would be really appreciated. > > Regards, > > -- > - DongInn > ___ > mtt-devel mailing list > mtt-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel > Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey
[MTT devel] Best way to run ftb_database_server and ftb_agent
Hi, Is there any way to put some lines to the mtt ini file to run the FTB daemons(ftb_database_server and ftb_agent)? Should we make another "FTB Run" phase after "FTB Install" or run the two daemons inside "Test run" but run the daemons before running the test scripts? I think either way would be fine if I have a working syntax for the both cases. Any help would be really appreciated. Regards, -- - DongInn