I don't know the history on why the timeouts were added but I suspect they were self-defensive in nature. I think in most cases the timeouts are really back-stopping things so that if there is a race condition the test suite bails and fails rather than runs forever. Is there a way to backstop stuff like that with xunit?
On Tue, Jun 21, 2016 at 8:24 PM Connie Yau <[email protected]> wrote: > Hi, > > I've been looking into migrating existing tests from NUnit to xUnit and I > think it's doable with the suggestions that were proposed. The only > question I need answered from the community is: > > "How important are timeouts in your test cases?" > - xUnit runs tests asynchronously and parallel, so they removed the > ability to have timeout because the ability to time these tests are not > reliable. > - There are 4 timeouts that I see in the cases. > - int.MaxValue > - 5 minutes > - 2.5 minutes > - 40000 milliseconds > My current approach to dealing with timeouts is removing them because the > tests run concurrently, so we aren’t waiting for a single test to finish. > > Thanks! > Connie > > -----Original Message----- > From: Elizabeth Maher (NEWMAN) [mailto:[email protected]] > Sent: Monday, June 20, 2016 12:26 PM > To: [email protected] > Subject: RE: Lucene.NET to .NET Core > > Our goal with the tests was to keep it working with the existing system > with a few changes as possible. We thought there would be more changes > than the community was comfortable with. However, it sounds like the > interest in xUnit is more that we realized. > > There are three items that concerned us in the xUnit move. > 1. No categories - Traits can be used instead. Hopefully just some > simple cut and paste. > 2. No timeout for test cases - This is the biggest issue as tc timeout is > frequently used in the test cases I saw. We are not sure the effect of > this or how to overcome any issues related to this. > 3. TeamCity integration - Looks like Wyatt found a good article > explaining how to do this. > > Perhaps Connie can look into the first two issues and Wyatt can look into > the last one? Connie will respond with an update by tomorrow evening and > then we can decide together. > > On a sad note, I've been pulled off onto another project. Connie will > take point on this project and finish up necessary work. > > Thanks, > Elizabeth > > > -----Original Message----- > From: Wyatt Barnett [mailto:[email protected]] > Sent: Monday, June 20, 2016 8:12 AM > To: [email protected] > Subject: Re: Lucene.NET to .NET Core > > Sorry for the delayed response, I was off at the beach last weekend. > > Not a whole lot of seat-time w/ xunit here. In terms of running it, there > is a plugin, we can see if CodeBetter would be willing to install it. The > hand-plumbed option doesn't seem to daunting either. In terms of categories > that is a better question but it seems like Xunit has a [Traits] attribute > that could be used -- see > https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fstackoverflow.com%2fquestions%2f21791739%2fexecute-tests-based-on-xunit-filtered-by-traits-in-teamcity&data=01%7c01%7cElizabeth.Maher%40microsoft.com%7c93aa1f13228a4142f26108d3991d5513%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=SaJ%2b%2b5r2mjXE6s550uotyCpNxB9eTP5EH577DmSU8to%3d > for > an example. > > On Sun, Jun 19, 2016 at 5:49 AM Itamar Syn-Hershko <[email protected]> > wrote: > > > "All these issues together lead us to the conclusion that we would be > > doing a disservice to just blindly port to xUnit." - why do you say > > that? if a move to xUnit benefits us on multiple occasions (better > > API, .NET Core support, faster runs) this one-tine effort would be worth > it - no? > > > > Wyatt, can you chime in perhaps - you have the most experience getting > > the build system to run, and test categorisation > > > > Thanks > > > > -- > > > > Itamar Syn-Hershko > > https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fcode97 > > 2.com&data=01%7c01%7cElizabeth.Maher%40microsoft.com%7c93aa1f13228a414 > > 2f26108d3991d5513%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=UpeiUeT > > A6iiP7fV7dZ9Z9bKiMjgqo990K6xyt4D6LTs%3d | @synhershko > > <https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2ftwit > > ter.com%2fsynhershko&data=01%7c01%7cElizabeth.Maher%40microsoft.com%7c > > 93aa1f13228a4142f26108d3991d5513%7c72f988bf86f141af91ab2d7cd011db47%7c > > 1&sdata=kc%2fH3fEbR68ajKlN3pZ7vISmdPb%2bIRTHWrDTZ4FdplY%3d> > > Freelance Developer & Consultant > > Lucene.NET committer and PMC member > > > > On Fri, Jun 17, 2016 at 7:06 AM, Elizabeth Maher (NEWMAN) < > > [email protected]> wrote: > > > > > Hi all. I have a quick update on our progress. > > > > > > We did a test of what it would be like to port Lucene.NET to use xUnit. > > > As Itamer guessed, it is quite a bit of work. The basic work of > > > renaming attributes and changing out package names and the like is a > > > little > > tedious, > > > but it is doable. The issues comes in because of the use of > > > categories > > and > > > timeout attributes. Creating categories in xUnit is possible, but > > > rather complicated. The bigger issue is that my research indicated > > > that > > TeamCity > > > only has basic integration with xUnit and you have to run all the > tests. > > > Running only categories requires extra scripting. The bigger issue > > > was that there is no TimeOut attribute in xUnit so longer running > > > tests are > > an > > > issue. We would also have to re-arrange some the test fixture setup > > > and teardown code into class constructors and dispose methods, but > > > that work was doable. All these issues together lead us to the > > > conclusion that we would be doing a disservice to just blindly port to > xUnit. > > > > > > nUnit has just release an alpha package that runs on .NET Core RC2. > > > Connie is looking into getting working this week or next to so we > > > can > > keep > > > the use of such features as testcase timeouts and categories. We'll > > > send an update when we've tried out the alpha package. > > > > > > Thanks, > > > Elizabeth > > > > > > -----Original Message----- > > > From: Elizabeth Maher (NEWMAN) > > > [mailto:[email protected]] > > > Sent: Monday, May 23, 2016 4:40 PM > > > To: [email protected] > > > Subject: RE: Lucene.NET to .NET Core > > > > > > Thank you everyone for the feedback. > > > > > > It's sounds everyone is favor of the move to xUnit. I've talked to > > > a couple of my co-workers and they said the conversion can be a > > > little tedious, but often worth the effort. Give me a couple days > > > to see if > > it’s > > > a rabbits hole. I'll respond back if its more effort that I can > expend. > > > > > > Itamar, > > > Regarding your last question. There are no more know tasks other > > > than what I initially outlined. Once I merge Connie's change in > > > from master > > we > > > will have a building/ported assembly. I just need to get the tests > > running > > > to verify everything works as expected. > > > > > > Thanks, > > > Elizabeth > > > > > > -----Original Message----- > > > From: [email protected] > > > [mailto:[email protected]] > > On > > > Behalf Of Itamar Syn-Hershko > > > Sent: Friday, May 20, 2016 6:46 AM > > > To: [email protected] > > > Subject: Re: Lucene.NET to .NET Core > > > > > > Hey there, > > > > > > Thank you so much again for your great work here. Lots of good > > > contributions we could definitely use. > > > > > > Moving to xUnit.Net was actually on my list of things to do - > > > basically also because it's API is much better and more flexible > > > (and less buggy!) and my experience with it was much better than > > > with NUnit (which we use > > an > > > outdated version of). Some people claim xUnit is also faster, so hey > > > - > > why > > > not. > > > > > > Are you sure you can "easily move all tests to xUnit"? From what I > > > can tell this will be a rather rigorous copy-paste job? While the > > > JavaCompatibility bits we have ( > > > > > > > > https://github.com/apache/lucenenet/tree/master/src/Lucene.Net.TestFra > > mework/JavaCompatibility > > > ) > > > do contain centralized function like assertTrue() etc that save > > > loads of time and effort porting tests, so that could be changed > > > easily to use xUnit, there are many places where asserts etc where > > > ported in full to > > the > > > NUnit API. > > > > > > I am definitely pro this change, but let's make sure we don't go > > > down > > into > > > a rabbit hole first. Let me know what you think. > > > > > > With regards to the status update: > > > > > > 1. I made some last minor comments on the merge scheduler PR - we > > > should be able to merge it within a few days now. Great work there > > > > > > 2. What is keeping us from merging your ICU etc changes? please note > > > some subprojects are absolutely necessary for Lucene.NET to run on > > > Core CLR as well - even though they are "sub" projects and not part > > > of the Lucene > > core > > > itself. Queries and Analysis are probably the biggest ones (latter > > > not fully ported yet, and it's the one with the ICU dep). > > > > > > 3. Other than what you posted - are there any other pending items to > > > get this running on .NET Core? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fcode97 > > 2.com&data=01%7c01%7cElizabeth.Maher%40microsoft.com%7c443f03152dfd451 > > ebb1c08d380b51eba%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=HxE52%2 > > bFAC6FJAN%2fifQ5V8v27tZZALzBuDw2yhbw%2bkc8%3d > > > | @synhershko < > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2ftwitt > > er.com%2fsynhershko&data=01%7c01%7cElizabeth.Maher%40microsoft.com%7c4 > > 43f03152dfd451ebb1c08d380b51eba%7c72f988bf86f141af91ab2d7cd011db47%7c1 > > &sdata=7iqRqwQwlcNbmwkckUjcS1MWjUPJi3cxLFfRdLc3BnQ%3d > > > > > > > Freelance Developer & Consultant > > > Lucene.NET committer and PMC member > > > > > > On Fri, May 20, 2016 at 3:17 AM, Elizabeth Maher (NEWMAN) < > > > [email protected]> wrote: > > > > > > > Lucene.NET Community, > > > > > > > > Connie and I continue to make progress on moving Lucene.NET to > > > > .NET > > Core. > > > > The last challenge we face is to get the tests to successfully run > > > > on .NET Core. Currently the unit tests use NUnit. There is a > > > > .NET Core version called NUnitLite. However, the issue is there > > > > is some manual steps to get NUnitLite tests to work on .NET Core. > > > > We would have to create console applications for each of the test > > > > library, call the appropriate APIs to create a test results file > > > > and then work with the CI system to manually upload the results. > > > > Also, there is not Test Runner integration for NUnitLite. While, > > > > this is all possible to do, I would like to propose a different > > > > change. I could easily move all the test to use xUnit. The would > > > > benefit Lucene.NET in the long run as there are both full fx< > > > > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fx > > > > unit > > > > .github.io%2fdocs%2fgetting-started-desktop.html&data=01%7c01%7cEl > > > > izab > > > > eth.Maher%40microsoft.com%7c443f03152dfd451ebb1c08d380b51eba%7c72f > > > > 988b > > > > f86f141af91ab2d7cd011db47%7c1&sdata=2orwADQfb%2fkpCykXkySekBp0Dxan > > > > w0WR > > > > rUupy%2fzRFTY%3d> and core fx< > > > > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fx > > > > unit > > > > .github.io%2fdocs%2fgetting-started-dotnet-core.html&data=01%7c01% > > > > 7cEl > > > > izabeth.Maher%40microsoft.com%7c443f03152dfd451ebb1c08d380b51eba%7 > > > > c72f > > > > 988bf86f141af91ab2d7cd011db47%7c1&sdata=UGKx5nz6%2foaV2vt2LQDTqkpn > > > > 0g%2 bu2XXHzE4pDZ0K%2fhU%3d> versions of xUnit, there is a > > > > TeamCity plugin that automatically works< > > > > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fx > > > > unit > > > > .github.io%2fdocs%2fgetting-test-results-in-teamcity.html&data=01% > > > > 7c01 > > > > %7cElizabeth.Maher%40microsoft.com%7c443f03152dfd451ebb1c08d380b51 > > > > eba% > > > > 7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=an0p3MTPFYe24wQiPq3Ss > > > > F3Eb v3gwJwrr88TjoQ4DnM%3d>, as well as the ability to run tests > > > > in Visual Studio.< > > > > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fx > > > > unit > > > > .github.io%2fdocs%2fgetting-started-dotnet-core.html%23run-tests-v > > > > s&da > > > > ta=01%7c01%7cElizabeth.Maher%40microsoft.com%7c443f03152dfd451ebb1 > > > > c08d > > > > 380b51eba%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=BN%2f3XEmrU > > > > yKQu 8tdZtoghxSskpFBPP1N1YGxT7e45Vc%3d.> > > > > I believe the long-term maintenance cost for both full and core fx > > > > tests would be lower using xUnit. Is this an acceptable direction > > > > for me to take the unit tests? Please let me know. > > > > > > > > Below is the progress of our migration work, for the curious. > > > > > > > > 1. Remove use of SharpZipLib dependency - completed > > > > > > > > 2. Update ICU4NET dependency for .NET Core - completed in dev > > > branch > > > > > > > > a. In https://github.com/conniey/lucenenet/tree/move2dnx. > > > > > > > > 3. Remove Appache.NMS dependency - completed > > > > > > > > 4. Move Scheduler to use Tasks - PR submitted > > > > > > > > a. Pull request at > https://github.com/apache/lucenenet/pull/171 > > . > > > > > > > > 5. Make portable libraries to build .NET Core binaries - > > completed > > > > in dev branch > > > > > > > > a. In https://github.com/conniey/lucenenet/tree/move2dnx. > > > > > > > > 6. Verify tests pass on both full framework and .Net Core. - In > > > > progress > > > > > > > > a. Need to get tests to run on .NET Core. > > > > > > > > > > > > Thanks, > > > > Elizabeth > > > > > > > > > >
