Re: [MTT devel] MTToGDS

[Jeff Squyres]

Mike --

Many thanks!  This rocks.

I'm embarrassed to say that I broke Cisco's MTT a little while ago and  
haven't found the cycles yet to fix it.  This is excellent motivation  
for me to a) fix my MTT runs, and b) start trying to submit to Google.


Woo hoo!


On Sep 29, 2009, at 3:21 PM, Mike Dubman wrote:


Hello guys and gals,

We have completed development and testing of Google DataStore  
support in MTT and are glad to submit it for community tests.




New Files:



The following new files were added to support GDS inside MTT:

1. client/bquery.pl

   Perl-based GDS client, provides basic DB querying/fetching  
capabilities. It creates resultset (files in YAML format) from user- 
provided sql-like query


2. client/breport.pl

   Perl-based report tool, creates excel reports from yaml files,  
generated by bquery.pl tool.


3. client/custom_launchers/

   For brave only: custom launchers for non-standard HPC, mpi-based  
applications


4. lib/MTT/Reporter/MTTGDS.pm

   GDS Reporter, saves mtt results to GDS (see samples/gds-demo.ini  
for configuration examples)


5. lib/MTT/Utils/ClusterInfo.pm

   Helper library to gather node hw/sw configuration information  
which is saved in GDS together with tests results.


6. New TestResults analyzers for HPC applications:

   lib/MTT/test/Analyze/Performance/Fluent.pm

   lib/MTT/test/Analyze/Performance/HPCC.pm

   lib/MTT/test/Analyze/Performance/HPLGDS.pm

   lib/MTT/test/Analyze/Performance/OpenFoam.pm

   lib/MTT/test/Analyze/Performance/PamCrash.pm

7. samples/gds-demo.ini

   Example of howto configure GDS in MTT and run bquery/breport  
tools at the end of MTT session


8. server/gds/

   GDS backend part, which is running at Google and providing Object  
to YAML, YAML to Object translation service as well as helper code  
for bquery.pl DB client.


9. docs/gds/

   Various documentation



Known Issues and Limitations:

==

* lib/MTT/Utils/ClusterInfo.pm uses "sudo" command to gather node`s  
hardware information.


* When using client/custom_launchers/ to run tests, it is impossible  
to kill the test application when timeout reached.




How to start using MTToGDS:

==

* Contact Jeff to provide you with GDS login/password which is  
needed for querying/saving to DB (http://open-mpi-mtt.appspot.com)


* See samples/gds-demo.ini for configuration examples as well as for  
DB querying and reports generation.


* Read Google GQL syntax documentation and use it with bquery.pl in  
order to query objects from GDB.


* The following perl modules are required for all MTToGDS components:
 libYAML
YAML::Syck
YAML::XS

for breport:
GD::Graph
Spreadsheet::WriteExcel

You can install it on linux systems with yum as following:
yum install perl-libyaml perl-YAML-Syck perl-YAML-XS perl-GD-Graph  
perl-Spreadsheet-WriteExcel


Special Thanks to:

==

Igor Ivanov, Andrew Senin, Alexander Alekhin from Argus-Cv.com for  
they contribution in developing and testing of this feature!


Regards

Mike

___
mtt-devel mailing list
mtt-de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel



--
Jeff Squyres
jsquy...@cisco.com

Re: [MTT devel] [MTT svn] svn:mtt-svn r1320

[Mike Dubman]

it seems it can be retired. executable() covers more cases.
shell() can be the alias of executable() for backwards compatibility.

Also, DoCommand::CmdScript should be changed to DoCommand::Cmd inside
executable() to really cover more cases.
regards

Mike

On Tue, Sep 29, 2009 at 8:35 PM, Ethan Mallove wrote:

> Should () be deprecated? It looks awfully similar to
> ().
>
> -Ethan
>
> On Tue, Sep/29/2009 08:34:44AM, mi...@osl.iu.edu wrote:
> > Author: miked
> > Date: 2009-09-29 08:34:44 EDT (Tue, 29 Sep 2009)
> > New Revision: 1320
> > URL: https://svn.open-mpi.org/trac/mtt/changeset/1320
> >
> > Log:
> > applied Jeff,Ethan comments:
> > 1. rename on_stop,on_start to after_mtt_start_exec, before_mtt_start_exec
> > 2. treat *_mtt_start_exec params in the same way like others
> before/after_* params
> > 3. rename shell_script to executable
> > 4. fix DoCommand:CmdScript() to recognize shebang chars and do not add
> ":\n" if #! is present
> >
> >
> > Text files modified:
> >trunk/client/mtt  |15 +--
> >trunk/lib/MTT/DoCommand.pm| 5 +++--
> >trunk/lib/MTT/Values/Functions.pm | 2 +-
> >3 files changed, 17 insertions(+), 5 deletions(-)
> >
> > Modified: trunk/client/mtt
> >
> ==
> > --- trunk/client/mtt  (original)
> > +++ trunk/client/mtt  2009-09-29 08:34:44 EDT (Tue, 29 Sep 2009)
> > @@ -496,7 +496,8 @@
> >  MTT::Lock::Init($ini);
> >
> >  # execute on_start callback if exists
> > -_process_get_value_option("mtt,on_start", $ini);
> > + _do_step($ini, "mtt", "before_mtt_start_exec");
> > +
> >
> >  # Set the logfile, if specified
> >
> > @@ -565,7 +566,7 @@
> >  }
> >
> >  # execute on_stop callback if exists
> > -_process_get_value_option("mtt,on_stop", $ini);
> > + _do_step($ini, "mtt", "after_mtt_start_exec");
> >
> >  # Shut down locks
> >
> > @@ -737,3 +738,13 @@
> >  print "$value\n";
> >  }
> >  }
> > +
> > +# Run cmd, specified in the non Test* sections
> > +sub _do_step {
> > + my ($ini, $section,$param) = @_;
> > + my $cmd = $ini->val($section, $param);
> > + if ( defined $cmd ) {
> > + my $x = MTT::DoCommand::RunStep(1, $cmd, -1, $ini,
> $section, $param);
> > + Verbose("  Output: $x->{result_stdout}\n")
> > + }
> > +}
> >
> > Modified: trunk/lib/MTT/DoCommand.pm
> >
> ==
> > --- trunk/lib/MTT/DoCommand.pm(original)
> > +++ trunk/lib/MTT/DoCommand.pm2009-09-29 08:34:44 EDT (Tue, 29
> Sep 2009)
> > @@ -794,9 +794,10 @@
> >  # protects against a common funclet syntax error.
> >  # We can safely do this since "foo" (literally, with
> >  # quotes included) would never be a valid shell command.
> > -$cmds =~ s/\"$//
> > -if ($cmds =~ s/^\"//);
> > +$cmds =~ s/\"$// if ($cmds =~ s/^\"//);
> >
> > +
> > + print $fh ":\n" if ($cmds !~ /^\s*\#\!/); # no shell specified -
> use default
> >  print $fh "$cmds\n";
> >  close($fh);
> >  chmod(0700, $filename);
> >
> > Modified: trunk/lib/MTT/Values/Functions.pm
> >
> ==
> > --- trunk/lib/MTT/Values/Functions.pm (original)
> > +++ trunk/lib/MTT/Values/Functions.pm 2009-09-29 08:34:44 EDT (Tue, 29
> Sep 2009)
> > @@ -3038,7 +3038,7 @@
> >  #
> >  #
> >
> > -sub shell_script {
> > +sub executable {
> >   my ($cmd_section, $cmd_param) = @_;
> >   my $cmd = _ini_val($cmd_section, $cmd_param);
> >   my $x = MTT::DoCommand::CmdScript(1, $cmd);
> > ___
> > mtt-svn mailing list
> > mtt-...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-svn
> ___
> mtt-devel mailing list
> mtt-de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
>