Sure I will as soon as I resume dealing with regression. I am completing the feature first, as I see now that regression is not small task :-). This is great to see all this stuff being checked automatically, but it comes at a price. Eric
-----Original Message----- From: Sandhya Sundaresan [mailto:[email protected]] Sent: Wednesday, August 5, 2015 8:56 PM To: [email protected] Subject: RE: Where are the logs for the first set of regression tests? I have added the details to the wiki at : https://wiki.trafodion.org/wiki/index.php/Test_Suites#Core Eric, please take a look and see if this is helpful and if anything else should be added. Thanks Sandhya -----Original Message----- From: Anoop Sharma [mailto:[email protected]] Sent: Tuesday, August 4, 2015 3:42 PM To: [email protected] Subject: RE: Where are the logs for the first set of regression tests? if a known diff file already exists for a test, then you cannot copy the log file as the expected file. Doing that will mask the current known diffs. You will need to manually update the expected file and regenerate a new known diff file as diff line numbers may have changed. anoop -----Original Message----- From: Eric Owhadi [mailto:[email protected]] Sent: Tuesday, August 4, 2015 3:37 PM To: [email protected] Subject: Re: Where are the logs for the first set of regression tests? OK clear, Thanks again Eric On Tue, Aug 4, 2015 at 5:34 PM, Suresh Subbiah <[email protected]> wrote: > Hi Eric, > > Standard procedure is to modify the EXPECTEDaaa file. If you are > satisfied with your new results you could simply copy the LOGaaa file > as the new EXPECTEDaaa file. There are filters in the test suite that > will hide things like timestamp from the final comparison. > DIFFaaa.KNOWN files are meant to be temporary. To allow a test to pass > while a fix is being worked on for a particular failure. > > Thanks > Suresh > > On Tue, Aug 4, 2015 at 4:52 PM, Eric Owhadi <[email protected]> wrote: > > > Thanks Sandhya, very useful! > > BTW, if I want to change a test to take into account valid changes > > that translate into false failure, should I modify the comparison > > file, or the file that logs known legitimate differences? > > Thanks again for the help :-), > > Eric > > > > -----Original Message----- > > From: Sandhya Sundaresan [mailto:[email protected]] > > Sent: Tuesday, August 4, 2015 1:19 PM > > To: [email protected] > > Subject: RE: Where are the logs for the first set of regression tests? > > > > RE: Where are the logs for the first set of regression tests? > > > > There are logs and diffs. I notice I added the instructions on the > > wiki only for the individual "pull request" automated jobs. I will > > add them for the official test results as well. > > > > Here they are : > > > > Click on the link in the daily build email : > > > > *https://jenkins02.trafodion.org/job/Check-Daily/48/* > > <https://jenkins02.trafodion.org/job/Check-Daily/48/> > > > > ==> Click on "Console Output" for any test suite you want to check > > > > ==> Around the 5th or 6th line you will see a link to detailed logs . > Click > > that. > > > > > > > > 2015-08-04 08:30:13 Started by upstream project "Check-Daily" build > > number 48 > > > > 2015-08-04 08:30:13 originally caused by: > > > > 2015-08-04 08:30:13 Started by timer > > > > 2015-08-04 08:30:13 [EnvInject] - Loading node environment > variables. > > > > 2015-08-04 08:30:13 Building remotely on ahw22-172.16.0.75 > > (redhat_ahw2.2) in workspace > > /home/jenkins/workspace/core-regress-fullstack2-ahw2.2 > > > > 2015-08-04 08:30:14 [core-regress-fullstack2-ahw2.2] $ /bin/sh > > /tmp/hudson2056470402644332566.sh > > > > 2015-08-04 08:30:14 Detailed logs: > > * > > > http://logs.trafodion.org/Daily/2015-08-04_08-30-00/regress-fullstack2 > -ahw2.2/* > > < > > > http://logs.trafodion.org/Daily/2015-08-04_08-30-00/regress-fullstack2 > -ahw2.2/ > > > > > > > == > Navigate to sql-regress-log > > > > You will see a directory for the component test > > suite – > eg > > “executor” and a full log – “Regress.log”. > > > > If you navigate to the component directory eg > > executor, > you > > will see all the DIFFs and LOG files . > > > > Sandhya > > > > -----Original Message----- > > From: Eric Owhadi [mailto:[email protected] > > <[email protected]>] > > Sent: Tuesday, August 4, 2015 11:06 AM > > To: [email protected] > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Hi Sandhya, > > > > Thanks for the offer, I'll first do a re-run with the feature off > > and get familiar with the official results, that'll give me some > > learning opportunity :-). > > > > There is no "diff" file on the official results automatic daily run? > > So that it is easy to compare with the failure one get running the > > test manually? > > > > Eric > > > > -----Original Message----- > > > > From: Sandhya Sundaresan [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 12:50 PM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Hi Eric, > > > > They are run on a nightly basis. Attached are the results form last > night > > and previous night. As you can see 2 failures last night and both > > are > being > > investigated. Otherwise we have been having good clean runs. So any > > other failures you see need to be investigated. If you can give me > > VNC access > to > > your machine, I can take a quick look. > > > > I have documented how to read the official results on the wiki : > > > > > > > https://wiki.trafodion.org/wiki/index.php/Contributor_Workflow#Checkin > g_Test_Results > > > > Sandhya > > > > -----Original Message----- > > > > From: Eric Owhadi [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 10:40 AM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Out of curiosity, anyone have recently run the regression test all > > on master branch? I am getting some failure, and I wonder if they > > are linked with > my > > changes or if the current regression suit is having known temporary > issues? > > > > Eric > > > > -----Original Message----- > > > > From: Dave Birdsall [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 11:26 AM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Right. Start with the simple stuff I think. Eric's e-mail trail is > valuable > > as it is fresh newbie experience; if we could answer Eric's > > questions on the wiki I think that would be a great start. > > > > -----Original Message----- > > > > From: Sandhya Sundaresan [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 9:24 AM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Hi Eric, Dave, > > > > I can add a page to the wiki for this. We did have the steps > documented > > in our old internal wiki for those starting on Trafodion for the > > first time. > > > > Also Roberta had a comprehensive document describing the whole > > regress environment back in MX days , I think. Not sure if anyone > > has that saved. > > > > Either way, for the wiki, perhaps a smaller page would be sufficient. > > > > Thanks > > > > Sandhya > > > > -----Original Message----- > > > > From: Eric Owhadi [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 9:03 AM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > I didn't want to ask, but indeed that would be fantastic and time > > saving. > > > > Also how to write and/or modify when tests are complaining, but we > > know that this is just because regression is based on delta, and > > cannot recognize legitimate expected delta (due to know feature we > > just implemented), and real regression. > > > > Eric > > > > -----Original Message----- > > > > From: Dave Birdsall [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 10:21 AM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Do we have a page on our Apache Trafodion wiki that describes how to > > run the regression tests and how to interpret and deal with the > > results? If > someone > > would like to volunteer to write such a page, it would be very > > useful to newbies, I think. > > > > -----Original Message----- > > > > From: Eric Owhadi [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 7:45 AM > > > > To: [email protected] > > > > Subject: RE: Where are the logs for the first set of regression tests? > > > > Thanks Suresh, that's it indeed :-). Was driving me crazy. > > > > Eric > > > > -----Original Message----- > > > > From: Suresh Subbiah [mailto:[email protected] > > <[email protected]>] > > > > Sent: Tuesday, August 4, 2015 9:41 AM > > > > To: [email protected] > > > > Subject: Re: Where are the logs for the first set of regression tests? > > > > Hi Eric, > > > > I think the "first" set of tests is the same as the last set. This > > is an artifact of how runallsb prints out results. At the end of > > every suite a summary of that suite is printed out (with no header > > what suite it refers to since we have been running that suite for > > the last X minutes). Then at > the > > end of a run it concatenates the summary of every suite, this time > > with a header for each suite. So if you kick off regressions now and > > come back several hours later there will be a screen with a summary > > for all suites and then before the summary for core there will be > > list of results with no suite name. This is the same as the last > > suite in the summary (the hive test suite). > > > > Thanks > > > > Suresh > > > > On Tue, Aug 4, 2015 at 9:20 AM, Qifan Chen <[email protected]> wrote: > > > > > Hi Eric, > > > > > > > > > > The results should be in one of the directories under $rundir. > > > For > > > > > example, under $rundir/core, the result for TEST001 (under > > > > > $scriptsdir/core) is LOG001. > > > > > > > > > > In my ~/.bashrc file, I have these two variables defined. > > > > > > > > > > export rundir=/mnt/qfc/rundir > > > > > > > > > > export scriptsdir=$trafodionRoot/sql/regress > > > > > > > > > > > > > > > After mkdir $rundir, make sure $rundir contains directories, one > > > for > > > > > each category in $scriptsdir. Once it is setup, you can run > > > > > regressions as many times as you want :-) > > > > > > > > > > On Tue, Aug 4, 2015 at 9:01 AM, Eric Owhadi > > > <[email protected]> > > wrote: > > > > > > > > > > > That may be an obvious question, but I can’t find the logs and > > > > diffs > > > > > > for the first set of test showing up in the result of the runallsb: > > > > > > > > > > > > I can see all the logs and diffs for categories like core, > > > > > > compGeneral > > > > > etc, > > > > > > but the first set of test labelled: > > > > > > > > > > > > TEST001 > > > > > > > > > > > > TEST002 > > > > > > > > > > > > TEST003 > > > > > > > > > > > > TEST004 > > > > > > > > > > > > TEST005 > > > > > > > > > > > > TEST006 > > > > > > > > > > > > TEST015 > > > > > > > > > > > > TEST017 > > > > > > > > > > > > TEST018 > > > > > > > > > > > > TEST020 > > > > > > > > > > > > > > > > > > > > > > > > are not associated with a category, and they don’t have a > > > > > > subdirectory > > > > > with > > > > > > categoryname where I can find the other test results. > > > > > > > > > > > > Thanks in advance for the help. > > > > > > > > > > > > > > > > > > > > > > > > As a side note, if a test fails because of the compare, but the > > > > > > delta is legit, what should one do? Update known diffs? Or set a > > > > new > > > > > > compare file? > > > > > > > > > > > > > > > > > > > > > > > > Eric > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, --Qifan > > > > > > > >
