BTW - the contrib NuGet package is now dead - the functionality has been moved into new sub-projects just like in Lucene.
But, that reminds me - I still don't have access to https://www.nuget.org/packages/Spatial4n.Core.NTS/, so I have been unable to update it to the same version that https://www.nuget.org/packages/Spatial4n.Core/ is on. NTS is not required by Lucene.Net, but it is required for anyone who needs to run the tests. -----Original Message----- From: Shad Storhaug [mailto:[email protected]] Sent: Saturday, April 1, 2017 5:31 AM To: [email protected] Subject: RE: API Work/Stabilization Update Wyatt, Great. Yea, actually after I sent my last email I was starting to wonder if Connie ever tested the build on a box without Visual Studio installed. We might need to determine what the prerequisites for the build are so we know what to put on the build server. I haven't yet had a chance to determine what tests are causing the build to crash (although, I know I can reliably reproduce it). I plan on getting that straightened out shortly. Worst case, we can just turn off verbosity on the build, but it might be helpful to leave it on if something goes wrong. I'll also look into the build dependencies and the script itself. I am just going through now and deleting all of the old 3.x source files that are now just causing noise in the project and resurrecting the tests for the support classes. But I am nearly finished. If you are available, I would like to try to get the beta2 release done over the weekend (at least to the point where it is on MyGet). We need to get the README and CONTRIBUTING files updated with the latest status (it would be helpful to sync the wiki page as well: https://cwiki.apache.org/confluence/display/LUCENENET/Current+Status - I don't think I have access for that). Several issues regarding the main readme need to be addressed: 1. There is no link to the wiki 2. There is no link to the issue tracker 3. It is not clear what the status is 4. There are no build instructions 5. There is no mention that we need help to finish 6. There is no link to the license 7. The documentation links are out of date 8. The top part of the repo already lists the files, do we really need to describe them again? Itamar mentioned he would be working on a new web site - I am not sure what the status of that is, but either way these issues are causing friction with anyone who is willing to help, but can't navigate through these hurdles. People who are used to working with GitHub, its issue tracker, and wiki don't find it natural to go looking somewhere else to these tools. For beta testing, we definitely need to make it crystal clear how to report bugs. Also, the "known issues" list is getting short enough so I can start adding them to the issue tracker. I have been using the downtime while running tests to fix up the documentation comments in Lucene.Net.Core, but it's still only about 50% done. I could use some help getting them fixed so they are at least visible in Visual Studio intellisense. See the latest status on #203. And then of course we can use them to generate new documents. Thanks, Shad Storhaug (NightOwl888) -----Original Message----- From: Wyatt Barnett [mailto:[email protected]] Sent: Saturday, April 1, 2017 4:22 AM To: [email protected] Subject: Re: API Work/Stabilization Update Shad -- definitely makes sense. Json files are fine -- functionally this is a bit too fancy to use Teamcity's automagic nuget package generation so as long as we've got a file to edit we are fine. Myget -> nuget works for me but that doesn't solve the key problem. I don't have it, maybe Prescott or Itamar know where it is kept but I can't claim to have ever seen it. I joined this party after the last nuget push. It is a bit foggy but I think I ran into the nunit-console issue with the Build.ps1 script. Remember that with build servers the pre-requisites often need to be embedded in the project for things to work properly. Anyhow, let me know when you are in a good place with your branch to start slogging through getting the new build working. In the interests of full disclosure I'm working an event the last week and a half of April and will be completely out of pocket then. But I'm about otherwise. On Sat, Mar 25, 2017 at 10:04 AM Shad Storhaug <[email protected]> wrote: > Wyatt, > > > We will probably want to build out .nuspec files to get all the > > nuget > stuff right for these projects -- I don't think the generation will > work for us to get things quite right. > > Connie has set us up to use .json files instead of .nuspec files to > generate the NuGet packages (the new way instead of the old way). The > build script Build.ps1 does it all (it even has help documentation), > but it is missing an option to override versioning. Ideally we would > be able to override the version that is in the .json file with an > environment variable (which you can pass from TeamCity), and be able > to override that on the CLI for local "one-off" builds. See the build > instructions on #191: > https://github.com/apache/lucenenet/pull/191 > > > Regarding the nuget key -- that plan works for me, the trick is I > > don't > have the key to add to myget. > > I don't know what order the infrastructure was setup in on your end, > but my thought was that if someone had previously pushed from MyGet to > NuGet the key is probably already configured there. But yea, you would > need access to MyGet to confirm. > > > I would love to start beating on that a bit but the .net core > > version > seems to want NUnit 3.5+ which needs to be added to the project to run. > > I will take a look at your pull request, but I think this is a symptom > of trying to run using the older tooling. The Build.ps1 script already > has the ability to test, and all of the tooling is there to do it (I > think - maybe I should do a fresh clone to be sure). It does have some > prerequisites, though (see #191). It builds both the .NET Framework > and .NET Core versions and packages them into NuGet. > > Per #191: Hopefully Lucene.Net.sln can be removed in the future > because the .NET Core projects compile for .NET 4.5.1 already. > > So I think the aim is to eventually eliminate those .csproj files (and > for that matter .nuspec files) and use strictly .json files for > project configuration going forward. > > > Thanks, > Shad Storhaug (NightOwl888) > > -----Original Message----- > From: Wyatt Barnett [mailto:[email protected]] > Sent: Saturday, March 25, 2017 5:00 AM > To: [email protected] > Subject: Re: API Work/Stabilization Update > > Shad -- the overall plan sounds good. We will probably want to build > out .nuspec files to get all the nuget stuff right for these projects > -- I don't think the generation will work for us to get things quite right. > > Regarding the nuget key -- that plan works for me, the trick is I > don't have the key to add to myget. Come to think of it I don't think > I have the proverbial keys to the myget page either but I think Martin > can help us out there. > > Buffers could be the issue on the tests -- I've long suspected that or > I/O causing the meltdown, I just haven't been able to reproduce. I > would love to start beating on that a bit but the .net core version > seems to want NUnit 3.5+ which needs to be added to the project to > run. If you get that added I can start beating on the test problems a bit > more. > > Thanks for all your hard work putting this together, let me know how I > can help you get it out the proverbial door. > > On Fri, Mar 24, 2017 at 9:34 AM Shad Storhaug <[email protected]> > wrote: > > > Wyatt, > > > > Thanks. Actually, I was thinking this should go in a few steps > > instead of > > one: > > > > 1. Merge #203. > > 2. Change the pre-release label to "beta2" and work out any issues > > to build/push to MyGet (might take a few tries) 3. Update the README > > and CONTRIBUTING pages 4. Push the package to NuGet > > > > I have always just used the control panel at MyGet to push upstream > > to NuGet, and it is capable of storing someone's key so the person > > who pushes it doesn't actually need it. > > > > As far as the tests burning down are concerned, I discovered that > > some of them write so much "verbose" data that they overflow NUnit's > > buffer and cause it to crash (sometimes this even causes Visual > > Studio to crash locally). I think I have found all of the tests in > > the Core that were causing this and hard-coded them to set verbose > > off (with the ability to manually override), but I noticed that > > there are still tests in Analysis.Common that can cause it to crash. > > I haven't investigated if there is a setting in NUnit to increase > > the buffer size, which might be a better fix, but I could probably > > track down the rest of the tests that are causing this before the merge. > > > > Thanks, > > Shad Storhaug (NightOwl888) > > > > -----Original Message----- > > From: Wyatt Barnett [mailto:[email protected]] > > Sent: Friday, March 24, 2017 8:14 PM > > To: [email protected] > > Subject: Re: API Work/Stabilization Update > > > > I'm about and happy to update when you are ready. > > > > I think the build should start working for 203 if you add the > > nunit-console nuget package. At least work in the sense of the build > > will start. I'm still chasing those build time bugbears I haven't > > been able to slay yet. > > > > As for getting to nuget.org -- who has the key? I've never had > > access to it so I'm not sure we can update what is out there. > > > > > > On Fri, Mar 24, 2017 at 09:10 Shad Storhaug <[email protected]> > wrote: > > > > > Wyatt, Prescott, Itamar, anybody? We've almost got all of the > > > tests passing on #203 and would like to merge to master and > > > release it with the new pre-release label "beta2". > > > > > > If there is nobody available to get the build running after > > > merging, could someone give me access to TeamCity to work on it? > > > > > > Thanks, > > > Shad Storhaug (NightOwl888) > > > > > > From: Shad Storhaug > > > Sent: Tuesday, March 21, 2017 5:25 AM > > > To: '[email protected]' > > > Subject: API Work/Stabilization Update > > > > > > I am getting very close to getting #203 merged. I wouldn't go so > > > far as to say that the API is finished, but the most significant > > > of the breaking API changes are now behind us. > > > > > > BUILD/VERSIONING > > > > > > I just wanted to be sure there is someone available to help get > > > the build working after the merge. I think it would be appropriate > > > to change the pre-release label from "beta" to "beta2" (without > > > resetting the build number, since that is actually what NuGet uses). > > > This would be primarily because of a major breaking API change, > > > but also to indicate another advancement toward release. > > > > > > We should probably also get this onto NuGet as soon as possible to > > > (hopefully) make it easier to recruit help to stabilize and create > > > some integration packages for popular Microsoft frameworks. > > > > > > KNOWN ISSUES > > > > > > > > > 1. The QueryParser.Flexible custom localized message > functionality > > > is currently not implemented for .NET core, so those tests are now > > failing. > > > > > > 2. The implementation of Lucene.Net.Expressions currently reads > > data > > > from the configuration file. This is not how modern libraries are > > > supposed to be built - instead we want any configuration to be > > > pushed in from the application that uses Lucene.Net. Reading from > > > the configuration file directly means no opportunity to use > > > dependency injection. There is also a namespace > > > Support/Configuration that can and should be removed after the > > > implementation is refactored to be DI-friendly (see > > > http://blog.ploeh.dk/2014/05/19/di-friendly-framework/). I haven't > > > yet worked out how the implementation was done in .NET - in Java, > > > the defaults were read from an embedded resource file and could be > > > overridden by passing in a ClassLoader (similar to .NET's Assembly > > class) - if anyone has any information on how the "auto generated" > > C# code was generated, please share. > > > > > > 3. The Collation functionality in Analysis.Common doesn't work > with > > > icu-dotnet, and has been excluded from compilation using the > > > constant FEATURE_COLLATION. I am now convinced after reading the > > > docs that it would be better to port the similar functionality > > > from Analysis.ICU because it was designed to work with icu4j and > > > is therefore more likely to work with icu-dotnet. > > > > > > 4. The Highlighter PostingsHighlight and VectorHighlight > > > functionality relies on icu-dotnet, which doesn't have a close > > > match for the BreakIterator in the JRE, so there are likely some > > > big differences in the functionality. Several hacks were put in to > > > make the tests pass, but these are not likely to fix all of the > > > issues in the > > wild. > > > > > > 5. There are several namespaces in Lucene.Net.Core and > > > Lucene.Net.Codecs that have broken documentation comments. > > > > > > 6. There are some concurrency and performance issues (as pointed > > out > > > by Vincent Van Den Burghe): > > > http://git.net/ml/general/2017-02/msg00168.html > > > > > > 7. We have around 2 dozen tests that fail during randomization > > > (averaging about 17 broken per run), and 8 tests that fail > > > all/most of the time. > > > > > > RESOLVED ISSUES (in addition to API refactoring) > > > > > > > > > 1. Finished implementing the randomization of Codecs, Culture, > Time > > > Zone, and InfoStream in the TestFramework. > > > > > > 2. Added factories for Codec, DocValuesFormat, and PostingsFormat > > so > > > custom implementations can be provided via dependency injection > > > instead of using the Java-ish NamedSPILoader class. The name must > > > now be provided by an attribute (or by class naming convention) > > > rather than via constructor, so it can be read without creating an > > > instance of > > the class. > > > > > > 3. Fixed several of the codecs in Lucene.Net.Codecs that were > still > > > not functioning (and not being tested because of the unfinished > > > RandomCodec class and test mocks). > > > > > > 4. Reviewed all catch blocks in Lucene.Net.Core to ensure the > right > > > type of exceptions are being caught and the right type re-thrown. > > > > > > 5. Fixed culture-sensitive comparison and sort order issues when > > > using strings in Lucene.Net.Core and Lucene.Net.Codecs. > > > > > > 6. Merged similar functionality in Support into the same class > and > > > deleted several unused Support classes. > > > > > > 7. Made the API CLS compliant, so it now works with all .NET > > > languages. > > > > > > > > > Shad Storhaug (NightOwl888) > > > > > >
