Hi everyone, sorry for the radio silence but we are getting very close. The basic CI setup where we grab the latest and then run an automated build and test was fairly easy to get worked out. But getting it running right has been a bit of a challenge. The main stumbling block the last few weeks was that, if I ran the test suite locally, I was getting through it in 15-20 minutes. Once I went to the build servers this stretched to 2+ hours. After playing a game of "what am I doing differently" I realized I was causing this delay. TeamCity has a feature for running test coverage on your tests. I had enabled this on our tests because test coverage is great and part of the point of doing this. It turns out test coverage also very slow which was causing this time issue -- not something horribly fubar in the code or the tests.
The other problem I am seeing is that there seems to be some variabiltiy in what tests fail here are the last 3 test runs, all with zero code changes: * #45: 1178 tests failed, 53 new failures, 1476 passed, 119 ignored * #44: 1170 tests failed, 49 new failures, 1483 passed, 119 ignored * #42: 1178 tests failed, 50 new failures, 1476 passed, 119 ignored * #41: 1178 tests failed, 58 new failures, 1475 passed, 119 ignored In this context a "new" failure means that the test had passed in a previous iteration and then failed anew, potentially indicating some broken code was introduced. Unfortunately there were zero code changes during any or all of those tests -- I was just trying to get a consistent run given consistent input. I also have seen a similar pattern on my private setup where I have done the lion's share of this development work. In terms of sources, I am successfully running against the apache github mirror. The official apache git repo is still hanging about 90 tests into things and I'm not sure why -- it appears to be the same commit hash as the github branch (2d7533d4e5a3278f242c2915c6f8dfd10ea77847) so it is presumably the same code involved. Will investigate this further but for now we are pointed at https://github.com/apache/lucenenet.git. In terms of proceeding here, here are the questions: * How important is constantly running / reported code coverage to us? Worth having a very lengthly build process? Is it something where we setup a single weekly code-coverage reporting job? * Does anyone have any insight into the inconsistent behavior by some of the tests? There are CSV exports of all the test runs so we could probably do some anaysis to focus the targets. * Is there any difference I am *not* seeing with the ASF repo? Why is the github one working and the presumably identical ASF one not? * What kind of schedule do we want this running on? Right now it has been manual, mainly because I didn't want to loose 3 hour test cycles on the world. But we are past that a bit so I think we can switch it to "run after master branch has an update" at this point and not overstay our welcome over at codebetter and get to the kind of feedback you all wanted. Still on the todo list [with questions]: * Get a better build system in place in general to make room for . . . . * Setup nuget generation and other packaging -- really kind of moot until we get tests passing in general but still notable. * Take a dive into integrating with the git flow system Itamar mentioned someplace but I can't seem to find at the moment. Hope this helps explain a bit, please let me know if you've got questions. On Sat, Dec 6, 2014 at 11:51 PM, Itamar Syn-Hershko <[email protected]> wrote: > > Hi Wyatt, I wonder if you have any good news for us on this? > > Thanks > > -- > > Itamar Syn-Hershko > http://code972.com | @synhershko <https://twitter.com/synhershko> > Freelance Developer & Consultant > Author of RavenDB in Action <http://manning.com/synhershko/> > > On Mon, Nov 24, 2014 at 4:48 PM, Wyatt Barnett <[email protected]> > wrote: > > > So, after fighting a number of teething problems -- many of them self > > inflicted -- I've got a solid, repeatable test run that finishes in 38 > > minutes or so on our hardware. > > > > I can certainly reprise this setup over on teamcity.codebetter.com > without > > much effort. We will need to merge some changes into things before > > proceeding -- the current state of tests in the trunk will just hang if > we > > tried to run them there. I'm guessing I'll need to sign a contributing > > agreement here as I don't believe I filed one. > > > > On Tue, Nov 18, 2014 at 5:19 PM, Wyatt Barnett <[email protected]> > > wrote: > > > > > Sounds good, we'll be [Explicit()] with reason. > > > > > > On Tue, Nov 18, 2014 at 5:06 PM, Itamar Syn-Hershko < > [email protected]> > > > wrote: > > > > > >> Yes, but Explicit() will help us isolate those issues from other real > > bugs > > >> we can concentrate on solving in parallel. > > >> > > >> -- > > >> > > >> Itamar Syn-Hershko > > >> http://code972.com | @synhershko <https://twitter.com/synhershko> > > >> Freelance Developer & Consultant > > >> Author of RavenDB in Action <http://manning.com/synhershko/> > > >> > > >> On Wed, Nov 19, 2014 at 12:01 AM, Wyatt Barnett < > > [email protected]> > > >> wrote: > > >> > > >> > [Ignore()] is good too, I actually prefer a slightly nuanced version > > >> called > > >> > [Explicit()] as that lets you still fire off the test from resharper > > or > > >> > nunit gui. > > >> > > > >> > That said I proposed Assert.Fail() here because we can put the > failure > > >> > point at the *exact* point where the folks should start debugging > this > > >> from > > >> > versus having them start a at a whole test or test fixture. > > >> > > > >> > On Tue, Nov 18, 2014 at 4:56 PM, Itamar Syn-Hershko < > > [email protected] > > >> > > > >> > wrote: > > >> > > > >> > > Don't forget to stick a reason to the Ignore property tho! > > >> > > > > >> > > -- > > >> > > > > >> > > Itamar Syn-Hershko > > >> > > http://code972.com | @synhershko <https://twitter.com/synhershko> > > >> > > Freelance Developer & Consultant > > >> > > Author of RavenDB in Action <http://manning.com/synhershko/> > > >> > > > > >> > > On Tue, Nov 18, 2014 at 11:55 PM, Itamar Syn-Hershko < > > >> [email protected] > > >> > > > > >> > > wrote: > > >> > > > > >> > > > Or just Skip (or Ignore, however that's called in NUnit) :) > > >> > > > > > >> > > > -- > > >> > > > > > >> > > > Itamar Syn-Hershko > > >> > > > http://code972.com | @synhershko < > https://twitter.com/synhershko> > > >> > > > Freelance Developer & Consultant > > >> > > > Author of RavenDB in Action <http://manning.com/synhershko/> > > >> > > > > > >> > > > On Tue, Nov 18, 2014 at 11:47 PM, Wyatt Barnett < > > >> > [email protected] > > >> > > > > > >> > > > wrote: > > >> > > > > > >> > > >> 5 minutes sounds like a reasonable going in proposition. That > > said > > >> > > >> depending on how many of these there are there might well be a > > >> *lot* > > >> > of > > >> > > 5 > > >> > > >> minute waits. > > >> > > >> > > >> > > >> Would it help if I just stuck Assert.Fails() where I run into > > these > > >> > > loops > > >> > > >> for the folks smarter than I to run down? > > >> > > >> > > >> > > >> On Tue, Nov 18, 2014 at 4:37 PM, Itamar Syn-Hershko < > > >> > [email protected] > > >> > > > > > >> > > >> wrote: > > >> > > >> > > >> > > >> > 5 mins max for one test maybe? > > >> > > >> > > > >> > > >> > To fix that infinite loop we need to revisit the original > Java > > >> code, > > >> > > >> most > > >> > > >> > likely its porting of an iterator-style code that went > wrong. I > > >> hope > > >> > > to > > >> > > >> > have time to look at it next week, please anyone else who > feels > > >> like > > >> > > it > > >> > > >> > beat me to it.. > > >> > > >> > > > >> > > >> > -- > > >> > > >> > > > >> > > >> > Itamar Syn-Hershko > > >> > > >> > http://code972.com | @synhershko < > > https://twitter.com/synhershko > > >> > > > >> > > >> > Freelance Developer & Consultant > > >> > > >> > Author of RavenDB in Action <http://manning.com/synhershko/> > > >> > > >> > > > >> > > >> > On Tue, Nov 18, 2014 at 11:13 PM, Wyatt Barnett < > > >> > > >> [email protected]> > > >> > > >> > wrote: > > >> > > >> > > > >> > > >> > > So, after the 2nd day of watching the tests just spin I > > decided > > >> > > that a > > >> > > >> > > little visibility might make sense as something just was > not > > >> > adding > > >> > > up > > >> > > >> > -- I > > >> > > >> > > expected some failing tests, and some long-running tests > but > > >> there > > >> > > >> just > > >> > > >> > > ain't enough data here to keep something running for 8+ > > hours. > > >> > > >> > > > > >> > > >> > > Anyhow, I stood up VS and the debugger and started looking > > into > > >> > > things > > >> > > >> > and > > >> > > >> > > I found that I was hitting at least one infiinte loop > dealing > > >> with > > >> > > >> > > randomized values -- specifically at > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > https://github.com/apache/lucene.net/blob/master/src/Lucene.Net.TestFramework/Index/BasePostingsFormatTestCase.cs#L394 > > >> > > >> > > ; > > >> > > >> > > > > >> > > >> > > Anyhow, I'm not sure how to proceed here as if we want test > > >> > > automation > > >> > > >> > the > > >> > > >> > > tests need to run through in a reasonable amount of time > but > > I > > >> > don't > > >> > > >> know > > >> > > >> > > enough about the project to know what should run or not. > One > > >> > thought > > >> > > >> > would > > >> > > >> > > be to use nunit timeout attributes ( > > >> > > >> > > http://www.nunit.org/index.php?p=timeout&r=2.5) to > constrain > > >> > > things. > > >> > > >> If > > >> > > >> > so > > >> > > >> > > what is a reasonable timeout? > > >> > > >> > > > > >> > > >> > > I'm quite open to other, non kludgy thoughts too . . . > > >> > > >> > > > > >> > > >> > > On Tue, Nov 18, 2014 at 12:00 PM, Wyatt Barnett < > > >> > > >> [email protected] > > >> > > >> > > > > >> > > >> > > wrote: > > >> > > >> > > > > >> > > >> > > > I looked over GitVersion -- looks like a great fit for > this > > >> > > project > > >> > > >> > > though > > >> > > >> > > > it will require a bit of forethought about branching > > >> strategies. > > >> > > >> > > > > > >> > > >> > > > I'll take a run at getting it integrated once I get > through > > >> the > > >> > > test > > >> > > >> > > suite > > >> > > >> > > > running successfully. > > >> > > >> > > > > > >> > > >> > > > On Sat, Nov 15, 2014 at 11:59 AM, Itamar Syn-Hershko < > > >> > > >> > [email protected] > > >> > > >> > > > > > >> > > >> > > > wrote: > > >> > > >> > > > > > >> > > >> > > >> Inline > > >> > > >> > > >> > > >> > > >> > > >> -- > > >> > > >> > > >> > > >> > > >> > > >> Itamar Syn-Hershko > > >> > > >> > > >> http://code972.com | @synhershko < > > >> > https://twitter.com/synhershko > > >> > > > > > >> > > >> > > >> Freelance Developer & Consultant > > >> > > >> > > >> Author of RavenDB in Action < > > http://manning.com/synhershko/ > > >> > > > >> > > >> > > >> > > >> > > >> > > >> On Sat, Nov 15, 2014 at 6:53 PM, Wyatt Barnett < > > >> > > >> > [email protected] > > >> > > >> > > > > > >> > > >> > > >> wrote: > > >> > > >> > > >> > > >> > > >> > > >> > Thanks. I registered at CodeBetter.com under wwb. Is > > there > > >> > any > > >> > > >> > > >> > documentation on their TeamCity setup or someone who I > > can > > >> > > reach > > >> > > >> out > > >> > > >> > > to > > >> > > >> > > >> > regarding questions about the build environment? > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> I'm asking around, will let you know > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > >> > > >> > > >> > Thanks for the rundown -- things sound pretty > > >> straightforward > > >> > > and > > >> > > >> > > >> doable. > > >> > > >> > > >> > One thing we'll need to think a bit a bout is how do > we > > >> want > > >> > to > > >> > > >> > manage > > >> > > >> > > >> the > > >> > > >> > > >> > git branching strategy to best integrate with TeamCity > > and > > >> > best > > >> > > >> > > automate > > >> > > >> > > >> > the release cycle. Doing things like constantly > > building a > > >> > > >> "trunk" > > >> > > >> > and > > >> > > >> > > >> > releasing based on tags are very doable. > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> Are you familiar with > > >> > > https://github.com/ParticularLabs/GitVersion > > >> > > >> ? > > >> > > >> > > I'll > > >> > > >> > > >> be interested in adopting this to our process. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > >> > > >> > > >> > I've grabbed the code, looks like things are pretty > > clean > > >> in > > >> > > >> terms > > >> > > >> > of > > >> > > >> > > >> being > > >> > > >> > > >> > able to build and run cleanly. One question -- I > > started > > >> > > running > > >> > > >> > the > > >> > > >> > > >> test > > >> > > >> > > >> > suite, it appears to execute about 80% successfully. > I'm > > >> > > >> presuming > > >> > > >> > > this > > >> > > >> > > >> is > > >> > > >> > > >> > because we are still porting 4.8.0 here and is > expected > > >> > > behavior. > > >> > > >> > > Please > > >> > > >> > > >> > confirm. > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> Yes, we still have some failing tests. The hope is to > also > > >> > > utilize > > >> > > >> > > >> TeamCity's reports to measure the affects of internal > > >> changes > > >> > we > > >> > > >> make > > >> > > >> > > >> faster. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > >> > > >> > > >> > I'll take a few stabs at a build cycle over the next > few > > >> days > > >> > > and > > >> > > >> > see > > >> > > >> > > >> what > > >> > > >> > > >> > I can shake out. > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> Thanks! > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > >> > > >> > > >> > On Sat, Nov 15, 2014 at 11:06 AM, Itamar Syn-Hershko < > > >> > > >> > > >> [email protected]> > > >> > > >> > > >> > wrote: > > >> > > >> > > >> > > > >> > > >> > > >> > > Thanks Wyatt > > >> > > >> > > >> > > > > >> > > >> > > >> > > Can you please register at > > >> http://teamcity.codebetter.com/ > > >> > > and > > >> > > >> > send > > >> > > >> > > >> me > > >> > > >> > > >> > > your > > >> > > >> > > >> > > user name (privately if you prefer)? > > >> > > >> > > >> > > > > >> > > >> > > >> > > Please use the master branch of our Apache git > git:// > > >> > > >> > > >> > > git.apache.org/lucene.net.git - or the mirror at > > >> > > >> > > >> > > https://github.com/apache/lucene.net (would rather > > you > > >> use > > >> > > the > > >> > > >> > > >> original > > >> > > >> > > >> > > one > > >> > > >> > > >> > > to avoid delays) > > >> > > >> > > >> > > > > >> > > >> > > >> > > The way I see it is this: > > >> > > >> > > >> > > * Lucene.Net.TestFramework compiles and generates a > > >> nuget > > >> > > >> package > > >> > > >> > > >> > > * Lucene.Net.Core and Lucene.Net.Tests compiles, > > taking > > >> > > >> dependency > > >> > > >> > > on > > >> > > >> > > >> > > Lucene.Net.TestFramework > > >> > > >> > > >> > > * Lucene.Net.Tests is run and if successful versions > > the > > >> > > build > > >> > > >> and > > >> > > >> > > >> > > generates nuget on the local feed (we also have a > > MyGet > > >> > > >> account to > > >> > > >> > > >> work > > >> > > >> > > >> > > with) > > >> > > >> > > >> > > * Same process to all sub-projects: > Lucene.Net.Queries > > >> for > > >> > > >> > example: > > >> > > >> > > >> > > * Compile > > >> > > >> > > >> > > * Compile Lucene.Net.Tests.Queries > > >> > > >> > > >> > > * Run tests from Lucene.Net.Tests.Queries (take > > >> > dependency > > >> > > on > > >> > > >> > > >> > > Lucene.Net.TestFramework) > > >> > > >> > > >> > > * Create nuget for Lucene.Net.Queries > > >> > > >> > > >> > > * Every sub-project should be re-compiled and tests > > >> re-run > > >> > if > > >> > > >> the > > >> > > >> > > >> > projects > > >> > > >> > > >> > > it depends on have changed (you can probably > separate > > >> the > > >> > > >> cases by > > >> > > >> > > >> > defining > > >> > > >> > > >> > > watch folders under src\) > > >> > > >> > > >> > > > > >> > > >> > > >> > > We can then iterate from there. > > >> > > >> > > >> > > > > >> > > >> > > >> > > I updated the README to explain the new structure > > >> > > >> > > >> > > > > >> > > >> > https://github.com/apache/lucene.net/blob/master/README.md#files > > >> > > >> > - > > >> > > >> > > >> let > > >> > > >> > > >> > me > > >> > > >> > > >> > > know if you have any questions. > > >> > > >> > > >> > > > > >> > > >> > > >> > > -- > > >> > > >> > > >> > > > > >> > > >> > > >> > > Itamar Syn-Hershko > > >> > > >> > > >> > > http://code972.com | @synhershko < > > >> > > >> https://twitter.com/synhershko> > > >> > > >> > > >> > > Freelance Developer & Consultant > > >> > > >> > > >> > > Author of RavenDB in Action < > > >> > http://manning.com/synhershko/> > > >> > > >> > > >> > > > > >> > > >> > > >> > > On Sat, Nov 15, 2014 at 5:51 AM, Wyatt Barnett < > > >> > > >> > > >> [email protected]> > > >> > > >> > > >> > > wrote: > > >> > > >> > > >> > > > > >> > > >> > > >> > > > That would be great -- let me know what I need to > do > > >> to > > >> > > help > > >> > > >> > make > > >> > > >> > > >> that > > >> > > >> > > >> > > > happen. > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > In the meantime I've got a teamcity server to work > > >> with > > >> > > here, > > >> > > >> > > >> should I > > >> > > >> > > >> > be > > >> > > >> > > >> > > > looking at > > >> > > >> > > >> > > > > >> > > https://github.com/synhershko/lucene.net/tree/Lucene.Net_4.8.0 > > >> > > >> > > >> > > > as the project layout. > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > Also, what is envisioned for the output of the > build > > >> > > >> pipeline? > > >> > > >> > > Many > > >> > > >> > > >> > > things > > >> > > >> > > >> > > > are within the art of the possible. > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > On Fri, Nov 14, 2014 at 9:08 PM, Prescott Nasser < > > >> > > >> > > >> > [email protected]> > > >> > > >> > > >> > > > wrote: > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > Would love the help setting this up - Itamar do > > you > > >> > know > > >> > > >> if we > > >> > > >> > > can > > >> > > >> > > >> > > > provide > > >> > > >> > > >> > > > > Wyatt access for this? > > >> > > >> > > >> > > > > ________________________________ > > >> > > >> > > >> > > > > From: Wyatt Barnett<mailto: > > [email protected]> > > >> > > >> > > >> > > > > Sent: 11/14/2014 4:26 PM > > >> > > >> > > >> > > > > To: [email protected]<mailto: > > >> > > >> [email protected]> > > >> > > >> > > >> > > > > Subject: Re: Setting up the CI pipeline > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > I do a lot of CI and CD and I've got loads of > seat > > >> time > > >> > > >> with > > >> > > >> > > >> > TeamCity, > > >> > > >> > > >> > > I > > >> > > >> > > >> > > > > would be happy to help the cause. > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > On Fri, Nov 14, 2014 at 5:49 AM, Itamar > > Syn-Hershko > > >> < > > >> > > >> > > >> > > [email protected]> > > >> > > >> > > >> > > > > wrote: > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > > Heya, > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > So JetBrains and CodeBetter have setup a > > TeamCity > > >> > > account > > >> > > >> > for > > >> > > >> > > >> us - > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > http://teamcity.codebetter.com/project.html?projectId=LuceneNet&tab=projectOverview > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > I have asked them to add Prescott and Troy as > > >> > > >> collaborators, > > >> > > >> > > so > > >> > > >> > > >> we > > >> > > >> > > >> > 3 > > >> > > >> > > >> > > > have > > >> > > >> > > >> > > > > > access to change stuff there. > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > The idea is to have every sub-project (Core, > > >> Codecs, > > >> > > >> > Queries, > > >> > > >> > > >> > Facets, > > >> > > >> > > >> > > > > etc) > > >> > > >> > > >> > > > > > compile and available as a nuget package, and > > also > > >> > > >> > thoroughly > > >> > > >> > > >> > tested > > >> > > >> > > >> > > > via > > >> > > >> > > >> > > > > > the test agents on TeamCity. > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > Prescott, Troy - you said you can work on > > setting > > >> > this > > >> > > >> up, > > >> > > >> > > will > > >> > > >> > > >> be > > >> > > >> > > >> > > > happy > > >> > > >> > > >> > > > > > for you to go ahead and do this now. > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > If there's anyone else on this list who has > > >> > experience > > >> > > >> with > > >> > > >> > > >> > TeamCity > > >> > > >> > > >> > > > and > > >> > > >> > > >> > > > > > setting up CI pipelines, please drop us a line > > if > > >> you > > >> > > are > > >> > > >> > > >> willing > > >> > > >> > > >> > to > > >> > > >> > > >> > > > help > > >> > > >> > > >> > > > > > with this effort. > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > Cheers, > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > -- > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > Itamar Syn-Hershko > > >> > > >> > > >> > > > > > http://code972.com | @synhershko < > > >> > > >> > > >> https://twitter.com/synhershko> > > >> > > >> > > >> > > > > > Freelance Developer & Consultant > > >> > > >> > > >> > > > > > Author of RavenDB in Action < > > >> > > >> http://manning.com/synhershko/ > > >> > > >> > > > > >> > > >> > > >> > > > > > > > >> > > >> > > >> > > > > > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
