Re: [MTT users] Discussion on teleconf yesterday?
On Fri, Oct/27/2006 10:31:44AM, Josh Hursey wrote: > > On Oct 27, 2006, at 7:39 AM, Jeff Squyres wrote: > > > On Oct 25, 2006, at 10:37 AM, Josh Hursey wrote: > > > >> The discussion started with the bug characteristics of v1.2 versus > >> the trunk. > > > > Gotcha. > > > >> It seemed from the call that IU was the only institution that can > >> asses this via MTT as noone else spoke up. Since people were > >> interested in seeing things that were breaking I suggested that I > >> start forwarding the IU internal MTT reports (run nightly and > >> weekly) to the test...@open-mpi.org. This was meet by Brain > >> insisting that it would result in "thousands" of emails to the > >> development list. I clarified that it is only 3 - 4 messages a day > >> from IU. However if all other institutions do this then it would be > >> a bunch of email (where 'a bunch' would still be less than > >> 'thousands'). That's how we got to a 'we need a single summary > >> presented to the group' comment. It should be noted that we brought > >> up IU sending to the 'test...@open-mpi.org' list as a bandaid until > >> MTT could do it better. > > > > How about sending them to me and Ethan? > > Sure I can add you both to the list if you like. > > > > >> This single summary can be email or a webpage that people can > >> check. Rich said that he would prefer a webpage, and noone else > >> really had a comment. That got us talking about the current summary > >> page that MTT generates. Tim M mentioned that the current website > >> is difficult to figure out how to get the answers you need. I > >> agree, it is hard [usability] for someone to go to the summary page > >> and answer the question "So what failed from IU last night, and how > >> does that differ from Yesterday -- e.g., what regressed and > >> progressed yesterday at IU?". The website is flexible enough to due > >> it, but having a couple of basic summary pages would be nice for > >> basic users. What that should look like we can discuss further. > > > > Agreed; we aren't super-fond of the current web page, either. Do you > > guys want to have a teleconf to go over the current status of MTT, > > where you want it to go, etc.? I consider IU's input here quite > > important, since you're the ones pushing the boundaries, flexing > > MTT's muscles, etc. > > In my previous email I suggested a couple of questions that I would > like a webpage to answer. A teleconf might be good to talk about some > of the various items that IU is trying to do around MTT. > > > > >> The IU group really likes the emails that we currently generate. A > >> plain-text summary of the previous run. I posted copies on the MTT > >> bug tracker here: > >> http://svn.open-mpi.org/trac/mtt/ticket/61 > >> Currently we have not put the work in to aggregate the runs, so for > >> each INI file that we run we get 1 email to the IU group. This is > >> fine for the moment, but as we add the rest of the clusters and > >> dimensions in the testing matrix we will need MTT to aggregate the > >> results for us and generate such an email. > > > > Ok. > > > > We created another ticket yesterday to make a new MTT Reporter (our > > internal plugins) that duplicates this output format. It actually > > shouldn't be that hard -- we don't have to do parsing to get the > > numbers that you're reporting; we have access to the actual data. So > > it's mostly caching the data, calculating the totals that you're > > calculating, and printing in your output format. > > > > Ethan has some other short tasks to do before he gets to this, but > > its near the top of the priority list. You can see the current > > workflow on the wiki (this is a living document; it keeps changing as > > requirements, etc. change): > > > > http://svn.open-mpi.org/trac/mtt/wiki/TaskPlan > > > > Awesome Thanks! :) > > >> So I think the general feel of the discussion is that we need the > >> following from MTT: > >> - A 'basic' summary page providing answers to some general > >> frequently asked queries. The current interface is too advanced for > >> the current users. > > > > We have the summary.php page, but I personally have never found it > > too useful. :-) > > > > We're getting towards a full revamp of reporter.php (got some other > > tasks to complete first, but we're definitely starting to think about > > it) -- got any ideas / input? Our "haven't thought about it much > > yet" idea is to be more menu/Q-A driven with a few common queries > > easily available (rather than a huge, complicated single screen). > > See previous email for some general ideas. Tim M might have a few > more that he would like to see since he is the one at IU that is > watching the nightly results the closest. > > > > >> - A summary email [in plain-text preferably] similar to the one > >> that IU generated showing an aggregation of the previous nights > >> results for (a) all reporters (b) my institution [so I can track > >> them down and file bugs]. >
Re: [MTT users] Discussion on teleconf yesterday?
On Oct 25, 2006, at 10:37 AM, Josh Hursey wrote: The discussion started with the bug characteristics of v1.2 versus the trunk. Gotcha. It seemed from the call that IU was the only institution that can asses this via MTT as noone else spoke up. Since people were interested in seeing things that were breaking I suggested that I start forwarding the IU internal MTT reports (run nightly and weekly) to the test...@open-mpi.org. This was meet by Brain insisting that it would result in "thousands" of emails to the development list. I clarified that it is only 3 - 4 messages a day from IU. However if all other institutions do this then it would be a bunch of email (where 'a bunch' would still be less than 'thousands'). That's how we got to a 'we need a single summary presented to the group' comment. It should be noted that we brought up IU sending to the 'test...@open-mpi.org' list as a bandaid until MTT could do it better. How about sending them to me and Ethan? This single summary can be email or a webpage that people can check. Rich said that he would prefer a webpage, and noone else really had a comment. That got us talking about the current summary page that MTT generates. Tim M mentioned that the current website is difficult to figure out how to get the answers you need. I agree, it is hard [usability] for someone to go to the summary page and answer the question "So what failed from IU last night, and how does that differ from Yesterday -- e.g., what regressed and progressed yesterday at IU?". The website is flexible enough to due it, but having a couple of basic summary pages would be nice for basic users. What that should look like we can discuss further. Agreed; we aren't super-fond of the current web page, either. Do you guys want to have a teleconf to go over the current status of MTT, where you want it to go, etc.? I consider IU's input here quite important, since you're the ones pushing the boundaries, flexing MTT's muscles, etc. The IU group really likes the emails that we currently generate. A plain-text summary of the previous run. I posted copies on the MTT bug tracker here: http://svn.open-mpi.org/trac/mtt/ticket/61 Currently we have not put the work in to aggregate the runs, so for each ini file that we run we get 1 email to the IU group. This is fine for the moment, but as we add the rest of the clusters and dimensions in the testing matrix we will need MTT to aggregate the results for us and generate such an email. Ok. We created another ticket yesterday to make a new MTT Reporter (our internal plugins) that duplicates this output format. It actually shouldn't be that hard -- we don't have to do parsing to get the numbers that you're reporting; we have access to the actual data. So it's mostly caching the data, calculating the totals that you're calculating, and printing in your output format. Ethan has some other short tasks to do before he gets to this, but its near the top of the priority list. You can see the current workflow on the wiki (this is a living document; it keeps changing as requirements, etc. change): http://svn.open-mpi.org/trac/mtt/wiki/TaskPlan So I think the general feel of the discussion is that we need the following from MTT: - A 'basic' summary page providing answers to some general frequently asked queries. The current interface is too advanced for the current users. We have the summary.php page, but I personally have never found it too useful. :-) We're getting towards a full revamp of reporter.php (got some other tasks to complete first, but we're definitely starting to think about it) -- got any ideas / input? Our "haven't thought about it much yet" idea is to be more menu/Q-A driven with a few common queries easily available (rather than a huge, complicated single screen). - A summary email [in plain-text preferably] similar to the one that IU generated showing an aggregation of the previous nights results for (a) all reporters (b) my institution [so I can track them down and file bugs]. For the moment, we don't have the dynamic capability for you to login to the web page, create a report, and say "mail this to me nightly". However, Ethan can make up custom reports on the server quite easily -- if you want some IU-specific reports, just file a ticket and Ethan can Make It So. - 1 email a day on the previous nights testing results. That's what we intended for the mails that are coming today, but it seemed to not be sufficient -- we ended up with 4 nightly mails, one for each relevant phase failures and a 4th for showing stderr of mpi installs. Some relevant bugs currently in existence: http://svn.open-mpi.org/trac/mtt/ticket/92 http://svn.open-mpi.org/trac/mtt/ticket/61 http://svn.open-mpi.org/trac/mtt/ticket/94 The other concern is that given the frequency of testing as bugs appear from the
