Thanks Wyatt. Yeah I am aware how to do it in code, was just making sure that I wasn't missing some TC option that would enable me to switch the builds / verbosity without code pushes. No big deal, I can check in code and push and get feedback that way.
Looking at the output now..... On Thu, May 21, 2015 at 11:44 AM, Wyatt Barnett <[email protected]> wrote: > I found a few minutes to play with things. > > First, the general technique to switching things to running under RELEASE > vs DEBUG while keeping VERBOSE output on is to modify line 299 in > LuceneTestCase.cs to be "true" -- the preprocessor directives there are > basically tying DEBUG and VERBOSE together. > > Output for TestTieBreaker can be found at http://pastebin.com/VXQYF32T > Output for TestDMQ8 can be found at http://pastebin.com/WYS07KqB > Output for TestSort_2 can be downloaded from > > https://drive.google.com/file/d/0By01KKFIe0LEX1NubW1mRlVMVWM/view?usp=sharing > > I hope this helps get to the bottom of this one, let me know if you need > further assistance. > > On Tue, May 19, 2015 at 2:02 PM Wyatt Barnett <[email protected]> > wrote: > > > The screenshot didn't come through so I'm kind of guessing here but I > > switched to debug mode which also triggers VERBOSE to get more output. > > > > This confirmed one of the things I saw locally earlier -- that some of > the > > semantics switching from debug to release (or non-verbose to verbose come > > to think of it) makes those tests fail for some reason. The other > challenge > > is that the output is so verbose that having verbose on makes the 4th > test > > run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute > > test run slow ( > > > http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests > ) > > compared to ~1 minute test runs. > > > > Let me see if I can get output without triggering debug mode when I get > > home, need to get to the right PC to make that happen. > > > > On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <[email protected]> > > wrote: > > > >> Wyatt, > >> > >> Would it be possible to pass that in through the configuration? I tried > >> passing in verbose flag through but did not appear to work. The same > with > >> configuration. Here is the properties section I am referring to: > >> > >> [image: Inline image 1] > >> > >> Do you know if it something that is possible to have for the TC builds? > >> > >> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett < > [email protected]> > >> wrote: > >> > >>> Great. One thing just hit me -- would it be better for this to run as > >>> DEBUG > >>> rather than RELEASE? I can flip that switch too . . . > >>> > >>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <[email protected]> > >>> wrote: > >>> > >>> > Thank you! Just kicked off the build. Let's see what it tells us :) > >>> > > >>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett < > >>> [email protected]> > >>> > wrote: > >>> > > >>> > > Liamonas -- you should be all set, I just added > >>> > > > >>> > > > >>> > > >>> > http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests > >>> > > which runs the core build with focused tests. Please ignore the > build > >>> > > number. Test category is more of a general setting for a build so > >>> there > >>> > > isn't an easy checkbox to hit. > >>> > > > >>> > > If it makes more sense to re-point that at your github repo just > say > >>> the > >>> > > word and I'll make it so. > >>> > > > >>> > > > >>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <[email protected] > > > >>> > wrote: > >>> > > > >>> > > > Wyatt, > >>> > > > > >>> > > > I have a branch pushed for this named "failingtests", it is now > >>> > running a > >>> > > > build on TC. Where does one specify which category of tests to > >>> run? I > >>> > see > >>> > > > in the settings tab "NUnit categories include/exclude" but don't > >>> see > >>> > > where > >>> > > > to actually modify these values. The tests I would like to run > >>> belong > >>> > to > >>> > > > category "Focus" :) Do you know where to change this? > >>> > > > > >>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett < > >>> > [email protected] > >>> > > > > >>> > > > wrote: > >>> > > > > >>> > > > > Good to hear I checked the right box. > >>> > > > > > >>> > > > > I'll see what I can pull together when I get home in terms of > >>> debug > >>> > > > output. > >>> > > > > In terms of testing procedure what I was thinking is we make a > >>> new > >>> > > > category > >>> > > > > -- call it "Focus" and then configure a build looking at your > >>> fork > >>> > > > > filtering for just those tests. You can then push away, fire > off > >>> > remote > >>> > > > > builds and check the output yourself. > >>> > > > > > >>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis < > >>> [email protected]> > >>> > > > wrote: > >>> > > > > > >>> > > > > > Wyatt, > >>> > > > > > > >>> > > > > > I see the new options on TC, thanks for that. I still haven't > >>> > thought > >>> > > > > about > >>> > > > > > how I will go about capturing the failures exactly, but will > >>> give > >>> > > you a > >>> > > > > > shout if I need some help with TC configuration just for > those > >>> > runs. > >>> > > > > > > >>> > > > > > If you can reproduce any of those test failures locally, do > you > >>> > mind > >>> > > > > > running them in VERBOSE mode (debug build without any other > >>> changes > >>> > > > will > >>> > > > > > do), and emailing the console output that you get? I might be > >>> too > >>> > > > > > optimistic, but perhaps something there will stand out. > >>> > > > > > > >>> > > > > > > >>> > > > > > Thanks again! > >>> > > > > > > >>> > > > > > Laimis > >>> > > > > > > >>> > > > > > > >>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett < > >>> > > > [email protected]> > >>> > > > > > wrote: > >>> > > > > > > >>> > > > > > > For TestSort_2 -- It appears to be passing based on data at > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails > >>> > > > > > > ; I am having locally reproducable problems on the others > >>> though. > >>> > > > > > > > >>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett < > >>> > > > > [email protected]> > >>> > > > > > > wrote: > >>> > > > > > > > >>> > > > > > > > Done -- you should now see a run button when you visit > >>> > > > > > > > > >>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet > >>> > > > > > > > > >>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis < > >>> > > > [email protected]> > >>> > > > > > > > wrote: > >>> > > > > > > > > >>> > > > > > > >> Wyatt, > >>> > > > > > > >> > >>> > > > > > > >> Could you add me to the lucene.net group on TC? I have > a > >>> > login > >>> > > > > there, > >>> > > > > > > >> username: laimis. > >>> > > > > > > >> > >>> > > > > > > >> > >>> > > > > > > >> Thanks! > >>> > > > > > > >> > >>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett < > >>> > > > > > [email protected] > >>> > > > > > > > > >>> > > > > > > >> wrote: > >>> > > > > > > >> > >>> > > > > > > >> > Sounds good Laimis. You will need to setup a login to > >>> the > >>> > > > > CodeBetter > >>> > > > > > > >> > teamcity server and get added to the lucene.net group > >>> if > >>> > you > >>> > > > > > haven't > >>> > > > > > > >> > already. Let me know if you need help there too. > >>> > > > > > > >> > > >>> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis < > >>> > > > > [email protected]> > >>> > > > > > > >> wrote: > >>> > > > > > > >> > > >>> > > > > > > >> > > Wyatt, > >>> > > > > > > >> > > > >>> > > > > > > >> > > Sweet, I will let you know once I have a branch out > >>> with > >>> > > > > > additional > >>> > > > > > > >> > logging > >>> > > > > > > >> > > and separate category for tests that you can > >>> configure to > >>> > > run. > >>> > > > > > > >> > > > >>> > > > > > > >> > > Re: release mode, tried that and was able to fix a > few > >>> > bugs > >>> > > > > after > >>> > > > > > > >> > switching > >>> > > > > > > >> > > to it. They were in that PR with debug.assert > >>> changes. Who > >>> > > > > knows, > >>> > > > > > > the > >>> > > > > > > >> > > remaining failures might still be related to that, > but > >>> > can't > >>> > > > > > > >> reproduce it > >>> > > > > > > >> > > locally. > >>> > > > > > > >> > > > >>> > > > > > > >> > > Laimis > >>> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" < > >>> > > > > [email protected] > >>> > > > > > > > >>> > > > > > > >> > wrote: > >>> > > > > > > >> > > > >>> > > > > > > >> > > > Sorry about the blank one -- getting used to > google > >>> > inbox > >>> > > > here > >>> > > > > > > and I > >>> > > > > > > >> > > > misclicked. > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and > >>> reason -- > >>> > > > > > TeamCity > >>> > > > > > > is > >>> > > > > > > >> > > > running in release mode and I think we have > >>> difffering > >>> > > > > behavior > >>> > > > > > > >> there. > >>> > > > > > > >> > If > >>> > > > > > > >> > > > you switch your copy of visual studio to release > >>> mode > >>> > you > >>> > > > will > >>> > > > > > get > >>> > > > > > > >> the > >>> > > > > > > >> > > same > >>> > > > > > > >> > > > failures we are seeing in TeamCity. Does that help > >>> > narrow > >>> > > it > >>> > > > > > down > >>> > > > > > > a > >>> > > > > > > >> > bit? > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett < > >>> > > > > > > >> [email protected] > >>> > > > > > > >> > > > >>> > > > > > > >> > > > wrote: > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > > > >>> > > > > > > >> > > > > > >>> > > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett < > >>> > > > > > > >> > [email protected] > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > > wrote: > >>> > > > > > > >> > > > > > >>> > > > > > > >> > > > >> I agree with Itamar -- it feels environmental. > >>> I'll > >>> > do > >>> > > > some > >>> > > > > > > >> digging > >>> > > > > > > >> > > into > >>> > > > > > > >> > > > >> the teamcity output but I think the plan of > >>> setting > >>> > up > >>> > > > some > >>> > > > > > > extra > >>> > > > > > > >> > > > verbose > >>> > > > > > > >> > > > >> logging here would make sense. I can set you up > >>> with > >>> > a > >>> > > > > > separate > >>> > > > > > > >> > build > >>> > > > > > > >> > > > >> pointed at your fork if that helps -- it will > >>> keep > >>> > the > >>> > > > > > feedback > >>> > > > > > > >> > cycle > >>> > > > > > > >> > > > >> tighter. The other thing we could do is > >>> categorize > >>> > the > >>> > > > > tests > >>> > > > > > > and > >>> > > > > > > >> > focus > >>> > > > > > > >> > > > that > >>> > > > > > > >> > > > >> build at running only that category so you > don't > >>> need > >>> > > to > >>> > > > > wait > >>> > > > > > > on > >>> > > > > > > >> the > >>> > > > > > > >> > > > whole > >>> > > > > > > >> > > > >> suite to get responses. Let me know if you want > >>> me to > >>> > > > > proceed > >>> > > > > > > >> there. > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar > >>> Syn-Hershko < > >>> > > > > > > >> > > [email protected] > >>> > > > > > > >> > > > > > >>> > > > > > > >> > > > >> wrote: > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> Yes, that would be the best way to do this. On > >>> Java > >>> > > > > Lucene, > >>> > > > > > > the > >>> > > > > > > >> > > > >>> randomized > >>> > > > > > > >> > > > >>> tests framework allows you to re-use the > random > >>> seed > >>> > > > > > > associated > >>> > > > > > > >> > with > >>> > > > > > > >> > > > the > >>> > > > > > > >> > > > >>> failure, but we are not there yet. Either > way, I > >>> > > suspect > >>> > > > > > this > >>> > > > > > > >> to be > >>> > > > > > > >> > > an > >>> > > > > > > >> > > > >>> environment issue rather than a code path one. > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > >>> -- > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > >>> Itamar Syn-Hershko > >>> > > > > > > >> > > > >>> http://code972.com | @synhershko < > >>> > > > > > > >> https://twitter.com/synhershko> > >>> > > > > > > >> > > > >>> Freelance Developer & Consultant > >>> > > > > > > >> > > > >>> Lucene.NET committer and PMC member > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas > >>> Simutis < > >>> > > > > > > >> > [email protected] > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > >>> wrote: > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > >>> > There are three tests that consistently fail > >>> on TC > >>> > > but > >>> > > > > no > >>> > > > > > > >> matter > >>> > > > > > > >> > > how > >>> > > > > > > >> > > > >>> many > >>> > > > > > > >> > > > >>> > times I try, I can't reproduce it locally. > >>> These > >>> > > tests > >>> > > > > > are: > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > >>> > > > > > > >> > > >>> > > > > > > >> > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798 > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8 > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > >>> > > > > > > >> > > >>> > > > > > > >> > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127 > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2 > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > >>> > > > > > > >> > > >>> > > > > > > >> > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892 > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > I would fix them if I could reproduce it -- > >>> and I > >>> > am > >>> > > > > > running > >>> > > > > > > >> out > >>> > > > > > > >> > of > >>> > > > > > > >> > > > >>> ideas > >>> > > > > > > >> > > > >>> > how to do it. Even if I put them in a loop > >>> running > >>> > > > > > hundreds > >>> > > > > > > of > >>> > > > > > > >> > > > times, I > >>> > > > > > > >> > > > >>> > can't trigger the failure. > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > Anyone have any ideas how to go about > >>> reproducing > >>> > > it? > >>> > > > I > >>> > > > > am > >>> > > > > > > >> > thinking > >>> > > > > > > >> > > > to > >>> > > > > > > >> > > > >>> push > >>> > > > > > > >> > > > >>> > very verbose code in a separate branch that > >>> logs > >>> > the > >>> > > > > input > >>> > > > > > > >> > values / > >>> > > > > > > >> > > > >>> random > >>> > > > > > > >> > > > >>> > values that are used and see what happens. > >>> > Checking > >>> > > if > >>> > > > > > > anyone > >>> > > > > > > >> has > >>> > > > > > > >> > > any > >>> > > > > > > >> > > > >>> other > >>> > > > > > > >> > > > >>> > suggestions. > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > Thanks, > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > Laimis > >>> > > > > > > >> > > > >>> > > >>> > > > > > > >> > > > >>> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > >>> > > > > > > >> > > >>> > > > > > > >> > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >> > >> >
