The screenshot didn't come through so I'm kind of guessing here but I switched to debug mode which also triggers VERBOSE to get more output.
This confirmed one of the things I saw locally earlier -- that some of the semantics switching from debug to release (or non-verbose to verbose come to think of it) makes those tests fail for some reason. The other challenge is that the output is so verbose that having verbose on makes the 4th test run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute test run slow ( http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests) compared to ~1 minute test runs. Let me see if I can get output without triggering debug mode when I get home, need to get to the right PC to make that happen. On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <[email protected]> wrote: > Wyatt, > > Would it be possible to pass that in through the configuration? I tried > passing in verbose flag through but did not appear to work. The same with > configuration. Here is the properties section I am referring to: > > [image: Inline image 1] > > Do you know if it something that is possible to have for the TC builds? > > On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <[email protected]> > wrote: > >> Great. One thing just hit me -- would it be better for this to run as >> DEBUG >> rather than RELEASE? I can flip that switch too . . . >> >> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <[email protected]> >> wrote: >> >> > Thank you! Just kicked off the build. Let's see what it tells us :) >> > >> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <[email protected] >> > >> > wrote: >> > >> > > Liamonas -- you should be all set, I just added >> > > >> > > >> > >> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests >> > > which runs the core build with focused tests. Please ignore the build >> > > number. Test category is more of a general setting for a build so >> there >> > > isn't an easy checkbox to hit. >> > > >> > > If it makes more sense to re-point that at your github repo just say >> the >> > > word and I'll make it so. >> > > >> > > >> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <[email protected]> >> > wrote: >> > > >> > > > Wyatt, >> > > > >> > > > I have a branch pushed for this named "failingtests", it is now >> > running a >> > > > build on TC. Where does one specify which category of tests to run? >> I >> > see >> > > > in the settings tab "NUnit categories include/exclude" but don't see >> > > where >> > > > to actually modify these values. The tests I would like to run >> belong >> > to >> > > > category "Focus" :) Do you know where to change this? >> > > > >> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett < >> > [email protected] >> > > > >> > > > wrote: >> > > > >> > > > > Good to hear I checked the right box. >> > > > > >> > > > > I'll see what I can pull together when I get home in terms of >> debug >> > > > output. >> > > > > In terms of testing procedure what I was thinking is we make a new >> > > > category >> > > > > -- call it "Focus" and then configure a build looking at your fork >> > > > > filtering for just those tests. You can then push away, fire off >> > remote >> > > > > builds and check the output yourself. >> > > > > >> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis < >> [email protected]> >> > > > wrote: >> > > > > >> > > > > > Wyatt, >> > > > > > >> > > > > > I see the new options on TC, thanks for that. I still haven't >> > thought >> > > > > about >> > > > > > how I will go about capturing the failures exactly, but will >> give >> > > you a >> > > > > > shout if I need some help with TC configuration just for those >> > runs. >> > > > > > >> > > > > > If you can reproduce any of those test failures locally, do you >> > mind >> > > > > > running them in VERBOSE mode (debug build without any other >> changes >> > > > will >> > > > > > do), and emailing the console output that you get? I might be >> too >> > > > > > optimistic, but perhaps something there will stand out. >> > > > > > >> > > > > > >> > > > > > Thanks again! >> > > > > > >> > > > > > Laimis >> > > > > > >> > > > > > >> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett < >> > > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > For TestSort_2 -- It appears to be passing based on data at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails >> > > > > > > ; I am having locally reproducable problems on the others >> though. >> > > > > > > >> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett < >> > > > > [email protected]> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Done -- you should now see a run button when you visit >> > > > > > > > >> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet >> > > > > > > > >> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis < >> > > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > >> Wyatt, >> > > > > > > >> >> > > > > > > >> Could you add me to the lucene.net group on TC? I have a >> > login >> > > > > there, >> > > > > > > >> username: laimis. >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> Thanks! >> > > > > > > >> >> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > >> wrote: >> > > > > > > >> >> > > > > > > >> > Sounds good Laimis. You will need to setup a login to the >> > > > > CodeBetter >> > > > > > > >> > teamcity server and get added to the lucene.net group if >> > you >> > > > > > haven't >> > > > > > > >> > already. Let me know if you need help there too. >> > > > > > > >> > >> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis < >> > > > > [email protected]> >> > > > > > > >> wrote: >> > > > > > > >> > >> > > > > > > >> > > Wyatt, >> > > > > > > >> > > >> > > > > > > >> > > Sweet, I will let you know once I have a branch out >> with >> > > > > > additional >> > > > > > > >> > logging >> > > > > > > >> > > and separate category for tests that you can configure >> to >> > > run. >> > > > > > > >> > > >> > > > > > > >> > > Re: release mode, tried that and was able to fix a few >> > bugs >> > > > > after >> > > > > > > >> > switching >> > > > > > > >> > > to it. They were in that PR with debug.assert changes. >> Who >> > > > > knows, >> > > > > > > the >> > > > > > > >> > > remaining failures might still be related to that, but >> > can't >> > > > > > > >> reproduce it >> > > > > > > >> > > locally. >> > > > > > > >> > > >> > > > > > > >> > > Laimis >> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" < >> > > > > [email protected] >> > > > > > > >> > > > > > > >> > wrote: >> > > > > > > >> > > >> > > > > > > >> > > > Sorry about the blank one -- getting used to google >> > inbox >> > > > here >> > > > > > > and I >> > > > > > > >> > > > misclicked. >> > > > > > > >> > > > >> > > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and >> reason -- >> > > > > > TeamCity >> > > > > > > is >> > > > > > > >> > > > running in release mode and I think we have >> difffering >> > > > > behavior >> > > > > > > >> there. >> > > > > > > >> > If >> > > > > > > >> > > > you switch your copy of visual studio to release mode >> > you >> > > > will >> > > > > > get >> > > > > > > >> the >> > > > > > > >> > > same >> > > > > > > >> > > > failures we are seeing in TeamCity. Does that help >> > narrow >> > > it >> > > > > > down >> > > > > > > a >> > > > > > > >> > bit? >> > > > > > > >> > > > >> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett < >> > > > > > > >> [email protected] >> > > > > > > >> > > >> > > > > > > >> > > > wrote: >> > > > > > > >> > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett < >> > > > > > > >> > [email protected] >> > > > > > > >> > > > >> > > > > > > >> > > > > wrote: >> > > > > > > >> > > > > >> > > > > > > >> > > > >> I agree with Itamar -- it feels environmental. >> I'll >> > do >> > > > some >> > > > > > > >> digging >> > > > > > > >> > > into >> > > > > > > >> > > > >> the teamcity output but I think the plan of >> setting >> > up >> > > > some >> > > > > > > extra >> > > > > > > >> > > > verbose >> > > > > > > >> > > > >> logging here would make sense. I can set you up >> with >> > a >> > > > > > separate >> > > > > > > >> > build >> > > > > > > >> > > > >> pointed at your fork if that helps -- it will keep >> > the >> > > > > > feedback >> > > > > > > >> > cycle >> > > > > > > >> > > > >> tighter. The other thing we could do is categorize >> > the >> > > > > tests >> > > > > > > and >> > > > > > > >> > focus >> > > > > > > >> > > > that >> > > > > > > >> > > > >> build at running only that category so you don't >> need >> > > to >> > > > > wait >> > > > > > > on >> > > > > > > >> the >> > > > > > > >> > > > whole >> > > > > > > >> > > > >> suite to get responses. Let me know if you want >> me to >> > > > > proceed >> > > > > > > >> there. >> > > > > > > >> > > > >> >> > > > > > > >> > > > >> >> > > > > > > >> > > > >> >> > > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar >> Syn-Hershko < >> > > > > > > >> > > [email protected] >> > > > > > > >> > > > > >> > > > > > > >> > > > >> wrote: >> > > > > > > >> > > > >> >> > > > > > > >> > > > >>> Yes, that would be the best way to do this. On >> Java >> > > > > Lucene, >> > > > > > > the >> > > > > > > >> > > > >>> randomized >> > > > > > > >> > > > >>> tests framework allows you to re-use the random >> seed >> > > > > > > associated >> > > > > > > >> > with >> > > > > > > >> > > > the >> > > > > > > >> > > > >>> failure, but we are not there yet. Either way, I >> > > suspect >> > > > > > this >> > > > > > > >> to be >> > > > > > > >> > > an >> > > > > > > >> > > > >>> environment issue rather than a code path one. >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >>> -- >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >>> Itamar Syn-Hershko >> > > > > > > >> > > > >>> http://code972.com | @synhershko < >> > > > > > > >> https://twitter.com/synhershko> >> > > > > > > >> > > > >>> Freelance Developer & Consultant >> > > > > > > >> > > > >>> Lucene.NET committer and PMC member >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas >> Simutis < >> > > > > > > >> > [email protected] >> > > > > > > >> > > > >> > > > > > > >> > > > >>> wrote: >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >>> > There are three tests that consistently fail >> on TC >> > > but >> > > > > no >> > > > > > > >> matter >> > > > > > > >> > > how >> > > > > > > >> > > > >>> many >> > > > > > > >> > > > >>> > times I try, I can't reproduce it locally. >> These >> > > tests >> > > > > > are: >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > >> >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798 >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8 >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > >> >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127 >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2 >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > >> >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892 >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > I would fix them if I could reproduce it -- >> and I >> > am >> > > > > > running >> > > > > > > >> out >> > > > > > > >> > of >> > > > > > > >> > > > >>> ideas >> > > > > > > >> > > > >>> > how to do it. Even if I put them in a loop >> running >> > > > > > hundreds >> > > > > > > of >> > > > > > > >> > > > times, I >> > > > > > > >> > > > >>> > can't trigger the failure. >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > Anyone have any ideas how to go about >> reproducing >> > > it? >> > > > I >> > > > > am >> > > > > > > >> > thinking >> > > > > > > >> > > > to >> > > > > > > >> > > > >>> push >> > > > > > > >> > > > >>> > very verbose code in a separate branch that >> logs >> > the >> > > > > input >> > > > > > > >> > values / >> > > > > > > >> > > > >>> random >> > > > > > > >> > > > >>> > values that are used and see what happens. >> > Checking >> > > if >> > > > > > > anyone >> > > > > > > >> has >> > > > > > > >> > > any >> > > > > > > >> > > > >>> other >> > > > > > > >> > > > >>> > suggestions. >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > Thanks, >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> > Laimis >> > > > > > > >> > > > >>> > >> > > > > > > >> > > > >>> >> > > > > > > >> > > > >> >> > > > > > > >> > > > >> >> > > > > > > >> > > > >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > >> >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
