Good question Shad. I think I de-escaped things properly -- the powershell
command it is running is ./runbuild.ps1 -RunTests -FrameworksToTest
@("net451", "netcoreapp1.0")I wholeheartedly agree that this build is complicated enough that we should get something a little beefier than powershell in place. I was looking at FAKE before but PSake could work too. I can't claim to have significant seat-time with either but that is why one does these open source projects. Anyhow more coming later this week/weekend when I can find some time to write things up . . . On Thu, Apr 6, 2017 at 11:37 AM Shad Storhaug <[email protected]> wrote: > Thanks. > > > > Hmm…Did you play with the escaping of quotes? What I pasted was the raw > command to run from CMD, but that might not be what will run in TeamCity. > > > > I am not entirely certain why it sometimes prompts for that (even when you > don’t specify to run a push), or why there are even required parameters. > These are some of the reasons why I am suggesting that PSake might be > better, but since I don’t know for certain if you can exit the script and > re-enter the same target without having to execute all of the target’s > dependencies all over again, I would like to run a few tests in TeamCity. > Worst case, we don’t specify any dependent targets for “test” so we can run > it multiple times without triggering the restore and build processes over > and over (which is probably harmless, but takes up lots of time). But it > helps if you have separate targets you can call in the script without > having to set lots of different Boolean flags to tell it what to run and > what not to. > > > > *From:* Wyatt Barnett [mailto:[email protected]] > *Sent:* Thursday, April 6, 2017 6:20 PM > > > *To:* Shad Storhaug > *Cc:* Connie Yau; [email protected] > *Subject:* Re: API Work/Stabilization Update > > > > Hi Shad -- you now have teamcity permissions. Please be a bit careful in > editing things -- it is a house of cards without great documentation as to > why things are doing what and there isn't an easy rollback button like with > source controlled code. Let me know if you want a tour of what is what and > we can find a time to do some screen sharing. > > > > That command won't run for me on the server -- the error is: > > > > [14:14:38]W: [Step 1/2] > C:\BuildAgent\work\dc5a565ec74d072b\runbuild.ps1 : Cannot process command > because of one or more missing mandatory > > [14:14:38]W: [Step 1/2] parameters: NuGetApiKey UploadPackages. > > [14:14:38]W: [Step 1/2] + CategoryInfo : InvalidArgument: > (:) [runbuild.ps1], ParentContainsErrorRecordException > > [14:14:38]W: [Step 1/2] + FullyQualifiedErrorId : > MissingMandatoryParameter,runbuild.ps1 > > > > Is there a way to run it without trying to upload the packages -- that is > a fairly mechanical thing we can certainly make work once we get build, > test and packaging the right things going. > > > > I will need a bit more time than I can find on weekdays to respond to your > other notes -- lots of good thoughts in there, they need cogent responses. > > > > On Wed, Apr 5, 2017 at 6:19 PM Shad Storhaug <[email protected]> > wrote: > > Wyatt, > > It turns out that adding the 4.5.1 framework to project.json was all that > was required to get the tests to run, but dotnet.exe seems to crash > whenever an exception is thrown from a test on that framework (saw 3 > different exception types - NullReferenceException, > AccessViolationException, IndexOutOfRangeException and they all brought it > to a halt). So, I gave the NUnit build runner a shot and it seems to be > more stable (and much faster as well). I wasn't able to get it working on > .NET core, but I haven't yet seen it crash when running the .NET core tests. > > Give it a shot to see if this will run all the way through without > crashing in TeamCity. > > powershell -ExecutionPolicy Bypass -Command "& .\runbuild.ps1 > -PackageVersion 4.8.0.1000-beta2 -RunTests -FrameworksToTest @(\"net451\", > \"netcoreapp1.0\")" > > Then we can work on getting this setup so tests can be run as separate > steps. > > Thanks, > Shad Storhaug (NightOwl888) > > > -----Original Message----- > From: Shad Storhaug [mailto:[email protected]] > Sent: Wednesday, April 5, 2017 8:42 PM > To: Wyatt Barnett > Cc: Connie Yau; [email protected] > Subject: RE: API Work/Stabilization Update > > I found one issue that is preventing the tests from running - the runner > isn't running .NET 4.5.1 framework tests at all. That framework is missing > from project.json in the test projects altogether, so it seems this has > never been run. I tried putting that section in, but I am getting an error > that the "dotnet-test-nunit" target is missing on .NET 4.5.1 (and tracking > down the cause of this doesn't seem to be plausible amongst all of the > advice telling to "upgrade"). The build works, so I don't think it would be > wise to upgrade just for the sake of testing. > > So, it seems we are back to square 1 - you are onto something with > installing the NUnit console runner for this. That is also easier said than > done - for some reason both Visual Studio and dotnet restore are ignoring > the fact that I added the NuGet package to the project and are not > downloading the binaries. I guess worst case scenario I can just check the > binaries into the repo, but it would be more difficult to upgrade if we did > that, so I am trying to find a fix to the NuGet download issue first. > > The good news is in .NET core I was able to successfully run all tests in > the Lucene.Net.Tests project. > > As for splitting this up, I am thinking that starting a script with a true > build runner like PSake is probably a good idea for the long term. Perhaps > we could just take the test part out of the current script so we could set > it up in TeamCity like: > > 1. Restore/Build step using runbuild.ps1 2. Test (using a different script > that can be called multiple times) 3. Package/Push using runbuild.ps1 > (side-effect of going through the Restore/Build steps all over again) > > > Also, allow the raw "where" clause to be passed directly to the command > line tool rather than doing all of the building that is done currently. > Then the tests could be setup in different steps however you like. To > ensure we can add projects to the setup without having them ignored by > default, I suggest: > > 1. Test Lucene.Net (50% of the tests) > 2. Test Lucene.Net.Analysis.Common (25% of the tests) 3. Test everything > but Lucene.Net and Lucene.Net.Analysis.Common (25% of the tests) > > And repeat for each framework. > > Or...wait a minute. I just noticed that one of the packages that downloads > with NUnit.Console is a TeamCity event listener. Would it make more sense > just to skip a test script for this (at least for .NET 4.5.1) and set it up > using TeamCity tools? > > Thanks, > Shad Storhaug (NightOwl888) > > -----Original Message----- > From: Shad Storhaug [mailto:[email protected]] > Sent: Wednesday, April 5, 2017 1:10 PM > To: Wyatt Barnett > Cc: Connie Yau; [email protected] > Subject: RE: API Work/Stabilization Update > > Wyatt, > > I have been getting that randomly, too (but I thought it was mainly due to > Visual Studio locking the file). Re-running has always made it recover > here. Basically, I have set it up so it backs up the project.json files > before making edits so it can restore them at the end. This is primarily to > prevent someone running locally from accidentally checking in the changes > the build makes. However, if it cannot backup the files for some reason (a > permission error perhaps) it won’t be able to do this step. > > So, looks like we need to add an option to override the backup/restore > file process on the build server, which will work around any windows > permission issues that prevent that from working. I will also add a check > if the file exists before attempting to restore to make it more robust. > > The modified process does one very important step – it names the main > NuGet package Lucene.Net instead of Lucene.Net.Core. I tried making the > script compensate for the fact that dotnet.exe doesn’t support an option > for changing the package name to be different than the folder name, but > that became very complicated – it was simpler just to rename the folder > from Lucene.Net.Core to Lucene.Net. > > As per your suggestion, we should probably break up the restore, build, > test, and pack steps so they work separately. Unfortunately, this script > wasn’t built with a task runner with separate entry points, so that is > going to be a bit complicated. Do we need the part of the script that > pushes to NuGet/MyGet, or is that something you can setup on TeamCity as a > separate step without a script? > > What specific handling do you need on the version string? I set it up so > you can pass the PackageVersion (of the NuGet package) separate from the > Version (of the assembly). It also sets the PackageVersion as the > AssemblyInformationalVersion of the assembly and also takes care of setting > all of the versions for inter-package dependencies between NuGet packages. > Basically, this makes it simple to specify only 1 parameter for version and > the script will take care of the rest. > > Keep in mind, however we set it up that this process is temporary. We will > likely change it up again when Microsoft finally gets .csproj working > right. So, we should focus on the *interface* of the build process (that > is, how TeamCity will interact with the script) so we can change the script > around later without breaking the build. Perhaps we should use a task > runner like PSake so we have separate entry points: > https://github.com/psake/psake? I have always used it in conjunction with > PowerShell in the past (although I have to admit, I never needed any of the > other entry points – but then, I never had 2 hours of testing to do). That > would eliminate the need to specify a lot of different switches to turn on > and off the right stuff for the current task. i.e (Invoke-psake > .\runbuild.ps1 Test -framework 4.0x64 -properties @{ > “framework”=”netcore1.0” }) would just run the tests for .NET Core and do > nothing else (see the Spatial4n script: > https://github.com/NightOwl888/Spatial4n/blob/master/.build/build.ps1). > > Build.bat is primarily to make the synatax of building simpler than what > is required to call powershell when building locally. (build > –pv:4.8.0.891-beta2 rather than powershell –Command “& .\runbuild.ps1 > –CreatePackages –PackageVersion 4.8.0.891-beta2”). It could be expanded > with more parameters and/or environment variables if necessary – I find it > simpler to maintain and test if there is both a way to manually build and > automatically build that run through the same script (the manual part > should be as simple as possible, the automated part only as complex as it > needs to be for the build server to do its magic). > > I tried running all of the tests last night from the command line and it > crashed during the run. Fortunately, I have a stack trace that might help > track it down. > > Culture: vai-Latn-LR > Time Zone: (UTC-05:00) Bogota, Lima, Quito, Rio Branco Default Codec: > Lucene41 (Lucene.Net.Codecs.Lucene41.Lucene41RWCodec) > Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes): > => Lucene.Net.Search.TestConstantScoreQuery.TestQueryWrapperFilter > Passed > Culture: vai-Latn-LR > Time Zone: (UTC-05:00) Bogota, Lima, Quito, Rio Branco Default Codec: > Lucene41 (Lucene.Net.Codecs.Lucene41.Lucene41RWCodec) > Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes): > [field, DFR G3(800)] => > Lucene.Net.Search.TestConstantScoreQuery.TestWrapped2Times > Passed > Culture: he-IL > Time Zone: (UTC-04:00) Asuncion > Default Codec: Lucene3x (Lucene.Net.Codecs.Lucene3x.PreFlexRWCodec) > Default Similarity: DefaultSimilarity > : hit exc > System.NullReferenceException: Object reference not set to an instance of > an object. > at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in > F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413 > at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState > state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91 > at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState > state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line > 47 > at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84 > at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line > 573 > at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread > flushingDWPT) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638 > at Lucene.Net.Index.DocumentsWriter.PreUpdate() in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461 > at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, > Analyzer analyzer, Term delTerm) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513 > at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, > IEnumerable`1 docs, Analyzer analyzer) in > F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562 > at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, > IEnumerable`1 docs) in > F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103 > at > Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term > id, IEnumerable`1 docs) in > F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line > 116 > at > Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() > in > F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line > 276 > at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in > F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413 > at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState > state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91 > at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState > state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line > 47 > at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84 > index=_0(4.2):C10/1:delGen=1 _1(4.2):C10 _2(4.2):C10 _3(4.2):C5 > _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: merging _1(4.2):C10 _2(4.2):C10 > _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: > seg=_1(4.2):C10 no deletesIW 8 [5/4/2017 01:48:05; ]: seg=_2(4.2):C10 no > deletesIW 8 [5/4/2017 01:48:05; ]: seg=_0(4.2):C10/1:delGen=1 delCount=1IW > 8 [5/4/2017 01:48:05; ]: seg=_3(4.2):C5 no deletesIW 8 [5/4/2017 01:48:05; > ]: seg=_4(4.2):C1 no deletesSM 8 [5/4/2017 01:48:05; ]: merge store > matchedCount=5 vs 5SM 8 [5/4/2017 01:48:05; ]: 0 msec to merge stored > fields [35 docs]SM 8 [5/4/2017 01:48:05; ]: 3 msec to merge postings [35 > docs]SM 8 [5/4/2017 01:48:05; ]: 5 msec to merge doc values [35 docs]SM 8 > [5/4/2017 01:48:05; ]: 0 msec to merge norms [35 docs]SM 8 [5/4/2017 > 01:48:05; ]: 1 msec to merge vectors [35 docs]IW 8 [5/4/2017 01:48:05; ]: > merge codec=Lucene46: [[trieLong, PostingsFormat(name=Memory doPackFST= > False)], [content2, PostingsFormat(name=Memory doPackFST= True)], [id, > PostingsFormat(name=Memory doPackFST= True)], [content5, FST41], [autf8, > PostingsFormat(name=Direct)], [trieInt, PostingsFormat(name=Memory > doPackFST= False)], [utf8, PostingsFormat(name=Memory doPackFST= False)], > [fieΓ▒╖ld, FST41], [content3, PostingsFormat(name=Direct)], [content, > PostingsFormat(name=Memory doPackFST= True)], [content6, > PostingsFormat(name=Direct)]], docValues:[[dvSortedSet, > DocValuesFormat(name=Disk)], [dvPacked, DocValuesFormat(name=Disk)], > [dvBytesStraightVar, DocValuesFormat(name=Asserting)], [dvLong, > DocValuesFormat(name=Disk)], [dvInt, DocValuesFormat(name=Memory)], > [dvBytesDerefFixed, DocValuesFormat(name=Memory)], [dvFloat, > DocValuesFormat(name=Asserting)], [dvBytesSortedFixed, > DocValuesFormat(name=SimpleText)], [dvBytesStraightFixed, > DocValuesFormat(name=Memory)], [dvBytesSortedVar, > DocValuesFormat(name=Disk)], [dvDouble, DocValuesFormat(name=Disk)], > [dvBytesDerefVar, DocValuesFormat(name=Memory)], [dvByte, > DocValuesFormat(name=Disk)], [dvShort, DocValuesFormat(name=Disk)]] > docCount=35; merged segment has vectors; norms; docValues; prox; freqsIW 8 > [5/4/2017 01:48:05; ]: merged segment size=%.3f MB vs estimate=%.3f MBTP 8 > [5/4/2017 01:48:05; ]: startCommitMergeIW 8 [5/4/2017 01:48:05; ]: > commitMerge: _1(4.2):C10 _2(4.2):C10 _0(4.2):C10/1:delGen=1 _3(4.2):C5 > _4(4.2):C1 index=_0(4.2):C10/1:delGen=1 _1(4.2):C10 _2(4.2):C10 _3(4.2):C5 > _4(4.2):C1TP 8 [5/4/2017 01:48:05; ]: startCommitMergeDeletesIW 8 [5/4/2017 > 01:48:05; ]: commitMergeDeletes _1(4.2):C10 _2(4.2):C10 > _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: no > new deletes or field updates since merge startedIFD 8 [5/4/2017 01:48:05; > ]: now checkpoint "_5(4.8):C35" [1 segments ; isCommit = False]IFD 8 > [5/4/2017 01:48:05; ]: 0 msec to checkpointIW 8 [5/4/2017 01:48:05; ]: > after commitMerge: _5(4.8):C35UPGMP 8 [5/4/2017 01:48:05; ]: > findForcedMerges: > segmentsToUpgrade=System.Collections.Generic.Dictionary`2[Lucene.Net.Index.SegmentCommitInfo,System.Nullable`1[System.Boolean]]IW > 8 [5/4/2017 01:48:05; ]: merge time 0 msec for 35 docsIndexUpgrader 8 > [5/4/2017 01:48:05; ]: All segments upgraded to version 4.8CMS 8 [5/4/2017 > 01:48:05; ]: merge thread: doneIW 8 [5/4/2017 01:48:05; ]: now flush at > close waitForMerges=TrueTP 8 [5/4/2017 01:48:05; ]: startDoFlushIW 8 > [5/4/2017 01:48:05; ]: start flush: applyAllDeletes=TrueIW 8 [5/4/2017 > 01:48:05; ]: index before flush _5(4.8):C35DW 8 [5/4/2017 01:48:05; ]: > startFullFlushDW 8 [5/4/2017 01:48:05; ]: anyChanges? numDocsInRam=0 > deletes=False hasTickets:False pendingChangesInFullFlush: FalseDW 8 > [5/4/2017 01:48:05; ]: finishFullFlush success=TrueIW 8 [5/4/2017 > 01:48:05; ]: apply all deletes during flushBD 8 [5/4/2017 01:48:05; ]: > applyDeletes: no deletes; skippingBD 8 [5/4/2017 01:48:05; ]: prune > sis=Lucene.Net.Index.SegmentInfos minGen=0 packetCount=0CMS 8 [5/4/2017 > 01:48:05; ]: now mergeCMS 8 [5/4/2017 01:48:05; ]: index: _5(4.8):C35CMS > 8 [5/4/2017 01:48:05; ]: no more merges pending; now returnIW 8 [5/4/2017 > 01:48:05; ]: waitForMergesIW 8 [5/4/2017 01:48:05; ]: waitForMerges doneIW > 8 [5/4/2017 01:48:05; ]: now call final commit()IW 8 [5/4/2017 01:48:05; ]: > commit: startIW 8 [5/4/2017 01:48:05; ]: commit: enter lockIW 8 [5/4/2017 > 01:48:05; ]: commit: now prepareIW 8 [5/4/2017 01:48:05; ]: prepareCommit: > flushIW 8 [5/4/2017 01:48:05; ]: index before flush _5(4.8):C35TP 8 > [5/4/2017 01:48:05; ]: startDoFlushDW 8 [5/4/2017 01:48:05; ]: > startFullFlushDW 8 [5/4/2017 01:48:05; ]: anyChanges? numDocsInRam=0 > deletes=False hasTickets:False pendingChangesInFullFlush: FalseIW 8 > [5/4/2017 01:48:05; ]: apply all deletes during flushBD 8 [5/4/2017 > 01:48:05; ]: applyDeletes: no deletes; skippingBD 8 [5/4/2017 01:48:05; ]: > prune sis=Lucene.Net.Index.SegmentInfos minGen=0 packetCount=0DW 8 > [5/4/2017 01:48:05; ]: finishFullFlush success=TrueTP 8 [5/4/2017 > 01:48:05; ]: startStartCommitIW 8 [5/4/2017 01:48:05; ]: StartCommit(): > startIW 8 [5/4/2017 01:48:05; ]: startCommit index=_5(4.8):C35 > changeCount=2TP 8 [5/4/2017 01:48:05; ]: midStartCommitTP 8 [5/4/2017 > 01:48:05; ]: midStartCommit2IW 8 [5/4/2017 01:48:05; ]: done all syncs: > _5.fdx, _5.fdt, _5_Direct_0.doc, _5_Direct_0.pos, _5_Direct_0.pay, > _5_Direct_0.tim, _5_Direct_0.tip, _5_Memory_0.ram, _5_FST41_0.doc, > _5_FST41_0.pos, _5_FST41_0.pay, _5_FST41_0.tmp, _5_Memory_1.ram, > _5_Disk_0.dvdd, _5_Disk_0.dvdm, _5_Memory_0.mdvd, _5_Memory_0.mdvm, > _5_SimpleText_0.dat, _5_Asserting_0.dvd, _5_Asserting_0.dvm, _5.nvd, > _5.nvm, _5.tvx, _5.tvd, _5.fnm, _5.siTP 8 [5/4/2017 01:48:05; ]: > midStartCommitSuccessTP 8 [5/4/2017 01:48:05; ]: finishStartCommitIW 8 > [5/4/2017 01:48:05; ]: commit: pendingCommit != nullIW 8 [5/4/2017 > 01:48:05; ]: commit: wrote segments file "segments_4"IFD 8 [5/4/2017 > 01:48:05; ]: now checkpoint "_5(4.8):C35" [1 segments ; isCommit = True]IFD > 8 [5/4/2017 01:48:05; ]: deleteCommits: now decRef commit "segments_3"IFD 8 > [5/4/2017 01:48:05; ]: delete "segments_3"IFD 8 [5/4/2017 01:48:05; ]: > delete "_0.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene42_0.dvd"IFD 8 > [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; > ]: delete "_0.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete > "_0_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.nvm"IFD 8 > [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; > ]: delete "_0.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete > "_0_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.nvd"IFD 8 > [5/4/2017 01:48:05; ]: delete "_0.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_ > 0.si"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.tim"IFD 8 > [5/4/2017 01:48:05; ]: delete "_0.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete > "_0_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_1.del"IFD 8 > [5/4/2017 01:48:05; ]: delete "_1.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete > "_1.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.doc"IFD 8 > [5/4/2017 01:48:05; ]: delete "_1_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; > ]: delete "_1_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: delete > "_1.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.tip"IFD 8 > [5/4/2017 01:48:05; ]: delete "_1_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; > ]: delete "_1.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete > "_1_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.fdx"IFD 8 > [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; > ]: delete "_1.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.si"IFD 8 > [5/4/2017 01:48:05; ]: delete "_1.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_ > 2.si"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.pos"IFD 8 > [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; > ]: delete "_2.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete > "_2_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete > "_2_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.fdx"IFD 8 > [5/4/2017 01:48:05; ]: delete "_2.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete > "_2.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.tvd"IFD 8 [5/4/2017 > 01:48:05; ]: delete "_2.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete > "_2_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete > "_2_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.nvd"IFD 8 > [5/4/2017 01:48:05; ]: delete "_2_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; > ]: delete "_3.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.fdt"IFD 8 > [5/4/2017 01:48:05; ]: delete "_3.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.tvx"IFD 8 [5/4/2017 > 01:48:05; ]: delete "_3.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.si"IFD 8 > [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; > ]: delete "_3.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete > "_3_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fdx"IFD 8 > [5/4/2017 01:48:05; ]: delete "_4.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete > "_4_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fnm"IFD 8 > [5/4/2017 01:48:05; ]: delete "_4.si"IFD 8 [5/4/2017 01:48:05; ]: delete > "_4.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_4_Lucene41_0.tip"IFD 8 > [5/4/2017 01:48:05; ]: delete "_4.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete > "_4_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: 0 msec to checkpointIW 8 > [5/4/2017 01:48:05; ]: commit: doneIW 8 [5/4/2017 01:48:05; ]: at close: > _5(4.8):C35 > at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line > 573 > at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread > flushingDWPT) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638 > at Lucene.Net.Index.DocumentsWriter.PreUpdate() in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461 > at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, > Analyzer analyzer, Term delTerm) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513 > at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, > IEnumerable`1 docs, Analyzer analyzer) in > F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562 > at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, > IEnumerable`1 docs) in > F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103 > at > Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term > id, IEnumerable`1 docs) in > F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line > 116 Unhandled Exception: System.Exception: System.NullReferenceException: > Object reference not set to an instance of an object. > at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in > F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413 > at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState > state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91 > at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState > state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line > 47 > at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84 > at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line > 573 > at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread > flushingDWPT) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638 > at Lucene.Net.Index.DocumentsWriter.PreUpdate() in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461 > at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, > Analyzer analyzer, Term delTerm) in > F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513 > at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, > IEnumerable`1 docs, Analyzer analyzer) in > F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562 > at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, > IEnumerable`1 docs) in > F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103 > at > Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term > id, IEnumerable`1 docs) in > F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line > 116 > at > Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() > in > F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line > 276 ---> System.NullReferenceException: Object reference not set to an > instance of an object. > at Lucene.Net.Util.IOUtils.ReThrow(Exception th) > at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState state) > at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState > state) > at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) > at Lucene.Net.Index.DocumentsWriterPerThread.Flush() > at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread > flushingDWPT) > at Lucene.Net.Index.DocumentsWriter.PreUpdate() > at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, > Analyzer analyzer, Term delTerm) > at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, > IEnumerable`1 docs, Analyzer analyzer) > at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, > IEnumerable`1 docs) > at > Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term > id, IEnumerable`1 docs) > at > Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() > --- End of inner exception stack trace --- > at > Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() > at System.Threading.ExecutionContext.Run(ExecutionContext > executionContext, ContextCallback callback, Object state) > at > Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() > in > F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line > 276 > WARNING: Could not find TestResult.xml. > Testing [Lucene.Net.Tests.Analysis.Common] on [net451]... > dotnet.exe test --configuration Release --framework net451 --no-build > Project > 'F:\Projects\lucenenet\src\Lucene.Net.Tests.Analysis.Common\project.json' > does not support framework: net451 > WARNING: Could not find TestResult.xml. > Testing [Lucene.Net.Tests.Analysis.Common] on [netcoreapp1.0]... > dotnet.exe test --configuration Release --framework netcoreapp1.0 > --no-build NUnit .NET Core Runner 3.4.0 Copyright (C) 2016 Charlie Poole > Runtime Environment > OS Platform: Windows > OS Version: 10.0.14393 > Runtime: win10-x64 > Test Files > > F:\Projects\lucenenet\src\Lucene.Net.Tests.Analysis.Common\bin\Release\netcoreapp1.0\Lucene.Net.Tests.Analysis.Common.dll > Terminate batch job (Y/N)? y > > And yea, I see that it is having trouble finding TestResult.xml files, > too. Perhaps that move step also needs to be made optional or taken out > entirely. > > > Thanks, > Shad Storhaug (NightOwl888) > > > > From: Wyatt Barnett [mailto:[email protected]] > Sent: Wednesday, April 5, 2017 4:43 AM > To: Shad Storhaug > Cc: Connie Yau; [email protected] > Subject: Re: API Work/Stabilization Update > > Update -- build was building and testing as of yesterday evening, it looks > like some of the build.ps1 changes blew up whatever was working there, the > build is now failing because it can't find some project.json.bak file . . . > > On Tue, Apr 4, 2017 at 4:41 PM Wyatt Barnett <[email protected] > <mailto:[email protected]>> wrote: > Hi Shad -- sorry I have been trying to get the whole thing working on > TeamCity for the last few days. Unfortunately test runs take upwards of an > hour and I have to remember to get back to it to check so progress is slow. > In any case I'm actually very close to getting the fundamentals worked out > -- see > https://teamcity.jetbrains.com/viewType.html?buildTypeId=LuceneNet_Vs2015LuceneNetPortable > for the specific build project. At this point I can get it to build and run > all the tests. The problem is reading the test results -- build.ps1 makes > the nunit XML files but it moves them and for some reason TeamCity didn't > like that. Is there any reason it was taking that copy step connie? > > Project.json vs Project.csproj is pretty immaterial to me. > > Anyhow, it does appear the build server has the pre-requisites required to > do so. As for teamcity access I would create an account at > https://teamcity.jetbrains.com and then let me know what the user / email > is and I'll see if I can get them to patch you in. > > FWIW build.bat probably doesn't help much -- we need a bit more elegant > handling of the version string, and probably want to run separate tests for > each target so we know what is failing. > > > > On Tue, Apr 4, 2017 at 1:31 PM Shad Storhaug <[email protected] > <mailto:[email protected]>> wrote: > Update > > I have made it past the hurdles with the build, and got the build working > on MyGet. But since MyGet has a build timeout of 15 minutes, there is no > way to run the tests there. > > So, now we need to get it working on TeamCity. AFAIK I don't have access > to it, so if someone could either help me out or provide access that would > be great. > > project.json files are NOT the new way (as I previously thought) - > Microsoft has already scrapped this idea. I guess back when Connie was > working on it things were still up in the air: > http://fizzylogic.nl/2016/05/11/project-json-is-going-away-and-it-s-a-good-thing/ > - and actually they still are. So, we might have to keep it this way for > now and then change it around at some later point when Microsoft officially > announces that multi-targeting support is released, then we can go back to > a single solution and single project file per project. > > We need to have the .NET Core SDK installed on the build server. Although > it is possible to download the binaries, they are more than 100MB when > unpacked. So, here is the list or prerequisites for the build server. > > PowerShell version 3.0 or higher > Git for Windows > .NET Core 1.1 with SDK Preview 2.1 build 3177 ( > https://github.com/dotnet/core/blob/master/release-notes/download-archive.md > ) > > To run the build, execute "build.bat". > > Environment Variables: > > %PackageVersion% - Entire version string including pre-release tag > %RunAllTests% - Pass "true" to bypass the 4 parameters that you would > otherwise have to set on runbuild.ps1 in order to run all of the tests for > both .NET Framework and .NET Core > > > Thanks, > Shad Storhaug (NightOwl888) > > > -----Original Message----- > From: Shad Storhaug [mailto:[email protected]<mailto: > [email protected]>] > Sent: Sunday, April 2, 2017 4:10 PM > To: [email protected]<mailto:[email protected]> > Cc: [email protected]<mailto:[email protected]>; Connie Yau > Subject: RE: API Work/Stabilization Update > > Update > > I tried running the tests with verbosity on and NUnit is misbehaving in > ways I didn't previously realize. There is apparently some resource leak > that is causing a buildup over several tests and then once it reaches a > certain threshold all of the tests will fail with an OutOfMemoryException. > For now, I am going to consider fixing verbosity low priority - instead we > should disable it for the time being and focus on the build/test runs. > > Looking through the build script there are few things of note: > > 1. It skips long running tests and tests that are marked with a timeout by > default 2. It runs in Release by default 3. Connie has listed Visual Studio > 2015 Update 3 as a prerequisite > > Clearly, this isn't what we want. We shouldn't be skipping any tests on > the build server except for those that have been manually ignored. > Verbosity is being ignored in the build script by default, which explains > why the verbosity issue is not causing the script to crash. > > I am also trying to work out if there is a way to run without Visual > Studio. The script depends on dotnet.exe, which (I think) is installed by > the .NET Core SDK. I am attempting to find out if we really need the SDK or > if the executable is all we need. Either way, I don't see this leaving the > ground without having .NET Framework 4.5.1 and (possibly) .NET Core 1.1 > installed on the server. > > I guess this leaves a few questions: > > 1. Is it safe to assume the .NET Framework 4.5.1 and .NET Core 1.1 > runtimes will be installed on the server? > 2. Can we install the .NET Core SDK on the server, or is our only option > to find out if the tools we need are portable (I can't imagine they are > not, but you never know)? > > > Thanks, > Shad Storhaug (NightOwl888) > > -----Original Message----- > From: Wyatt Barnett [mailto:[email protected]<mailto: > [email protected]>] > Sent: Saturday, April 1, 2017 7:52 AM > To: [email protected]<mailto:[email protected]> > Subject: Re: API Work/Stabilization Update > > I wanted to walk back to one thing you asked about the tests Shad: > > "Worst case, we can just turn off verbosity on the build, but it might be > helpful to leave it on if something goes wrong" > > I *think* I'm turning the verbosity off in the tests -- they should be > running in release mode. They would not even run in debug if I recall > correctly. Let me know how the switch works and I'll make sure I'm throwing > it on the build server side. > > On Fri, Mar 31, 2017 at 6:56 PM Shad Storhaug <[email protected] > <mailto:[email protected]>> wrote: > > > I don't necessarily need ownership access, but he did give me > > ownership to the https://www.nuget.org/packages/Spatial4n.Core/ package. > > Alternatively, Itamar can enter his keys on > > https://www.myget.org/feed/Security/spatial4n - he already has > ownership. > > Once the keys are added there, I will be able to push. > > > > -----Original Message----- > > From: Prescott Nasser > > [mailto:[email protected]<mailto:[email protected]>] > > Sent: Saturday, April 1, 2017 5:49 AM > > To: [email protected]<mailto:[email protected]> > > Subject: RE: API Work/Stabilization Update > > > > Access like ownership access? Paging Itamar.. > > > > > > > > -----Original Message----- > > From: Shad Storhaug > > [mailto:[email protected]<mailto:[email protected]>] > > Sent: Friday, March 31, 2017 3:38 PM > > To: [email protected]<mailto:[email protected]> > > Subject: RE: API Work/Stabilization Update > > > > BTW - the contrib NuGet package is now dead - the functionality has > > been moved into new sub-projects just like in Lucene. > > > > But, that reminds me - I still don't have access to > > https://www.nuget.org/packages/Spatial4n.Core.NTS/, so I have been > > unable to update it to the same version that > > https://www.nuget.org/packages/Spatial4n.Core/ is on. NTS is not > > required by Lucene.Net, but it is required for anyone who needs to run > the tests. > > > > -----Original Message----- > > From: Shad Storhaug > > [mailto:[email protected]<mailto:[email protected]>] > > Sent: Saturday, April 1, 2017 5:31 AM > > To: [email protected]<mailto:[email protected]> > > Subject: RE: API Work/Stabilization Update > > > > Wyatt, > > > > Great. Yea, actually after I sent my last email I was starting to > > wonder if Connie ever tested the build on a box without Visual Studio > installed. > > We might need to determine what the prerequisites for the build are so > > we know what to put on the build server. > > > > I haven't yet had a chance to determine what tests are causing the > > build to crash (although, I know I can reliably reproduce it). I plan > > on getting that straightened out shortly. Worst case, we can just turn > > off verbosity on the build, but it might be helpful to leave it on if > > something goes wrong. I'll also look into the build dependencies and the > script itself. > > > > I am just going through now and deleting all of the old 3.x source > > files that are now just causing noise in the project and resurrecting > > the tests for the support classes. But I am nearly finished. If you > > are available, I would like to try to get the beta2 release done over > > the weekend (at least to the point where it is on MyGet). We need to > > get the README and CONTRIBUTING files updated with the latest status > > (it would be helpful to sync the wiki page as well: > > https://cwiki.apache.org/confluence/display/LUCENENET/Current+Status - > > I don't think I have access for that). > > > > Several issues regarding the main readme need to be addressed: > > > > 1. There is no link to the wiki > > 2. There is no link to the issue tracker 3. It is not clear what the > > status is 4. There are no build instructions 5. There is no mention > > that we need help to finish 6. There is no link to the license 7. The > > documentation links are out of date 8. The top part of the repo > > already lists the files, do we really need to describe them again? > > > > Itamar mentioned he would be working on a new web site - I am not sure > > what the status of that is, but either way these issues are causing > > friction with anyone who is willing to help, but can't navigate > > through these hurdles. People who are used to working with GitHub, its > > issue tracker, and wiki don't find it natural to go looking somewhere > > else to these tools. For beta testing, we definitely need to make it > > crystal clear how to report bugs. > > > > Also, the "known issues" list is getting short enough so I can start > > adding them to the issue tracker. > > > > I have been using the downtime while running tests to fix up the > > documentation comments in Lucene.Net.Core, but it's still only about > > 50% done. I could use some help getting them fixed so they are at > > least visible in Visual Studio intellisense. See the latest status on > > #203. And then of course we can use them to generate new documents. > > > > > > Thanks, > > Shad Storhaug (NightOwl888) > > > > > > -----Original Message----- > > From: Wyatt Barnett > > [mailto:[email protected]<mailto:[email protected]>] > > Sent: Saturday, April 1, 2017 4:22 AM > > To: [email protected]<mailto:[email protected]> > > Subject: Re: API Work/Stabilization Update > > > > Shad -- definitely makes sense. > > > > Json files are fine -- functionally this is a bit too fancy to use > > Teamcity's automagic nuget package generation so as long as we've got > > a file to edit we are fine. > > > > Myget -> nuget works for me but that doesn't solve the key problem. I > > don't have it, maybe Prescott or Itamar know where it is kept but I > > can't claim to have ever seen it. I joined this party after the last > nuget push. > > > > It is a bit foggy but I think I ran into the nunit-console issue with > > the > > Build.ps1 script. Remember that with build servers the pre-requisites > > often need to be embedded in the project for things to work properly. > > > > Anyhow, let me know when you are in a good place with your branch to > > start slogging through getting the new build working. In the interests > > of full disclosure I'm working an event the last week and a half of > > April and will be completely out of pocket then. But I'm about otherwise. > > > > On Sat, Mar 25, 2017 at 10:04 AM Shad Storhaug > > <[email protected]<mailto:[email protected]>> > > wrote: > > > > > Wyatt, > > > > > > > We will probably want to build out .nuspec files to get all the > > > > nuget > > > stuff right for these projects -- I don't think the generation will > > > work for us to get things quite right. > > > > > > Connie has set us up to use .json files instead of .nuspec files to > > > generate the NuGet packages (the new way instead of the old way). > > > The build script Build.ps1 does it all (it even has help > > > documentation), but it is missing an option to override versioning. > > > Ideally we would be able to override the version that is in the > > > .json file with an environment variable (which you can pass from > > > TeamCity), and be able to override that on the CLI for local > > > "one-off" builds. See the build > > instructions on #191: > > > https://github.com/apache/lucenenet/pull/191 > > > > > > > Regarding the nuget key -- that plan works for me, the trick is I > > > > don't > > > have the key to add to myget. > > > > > > I don't know what order the infrastructure was setup in on your end, > > > but my thought was that if someone had previously pushed from MyGet > > > to NuGet the key is probably already configured there. But yea, you > > > would need access to MyGet to confirm. > > > > > > > I would love to start beating on that a bit but the .net core > > > > version > > > seems to want NUnit 3.5+ which needs to be added to the project to run. > > > > > > I will take a look at your pull request, but I think this is a > > > symptom of trying to run using the older tooling. The Build.ps1 > > > script already has the ability to test, and all of the tooling is > > > there to do it (I think - maybe I should do a fresh clone to be > > > sure). It does have some prerequisites, though (see #191). It builds > > > both the .NET Framework and .NET Core versions and packages them into > NuGet. > > > > > > Per #191: Hopefully Lucene.Net.sln can be removed in the future > > > because the .NET Core projects compile for .NET 4.5.1 already. > > > > > > So I think the aim is to eventually eliminate those .csproj files > > > (and for that matter .nuspec files) and use strictly .json files for > > > project configuration going forward. > > > > > > > > > Thanks, > > > Shad Storhaug (NightOwl888) > > > > > > -----Original Message----- > > > From: Wyatt Barnett > > > [mailto:[email protected]<mailto:[email protected]>] > > > Sent: Saturday, March 25, 2017 5:00 AM > > > To: [email protected]<mailto:[email protected]> > > > Subject: Re: API Work/Stabilization Update > > > > > > Shad -- the overall plan sounds good. We will probably want to build > > > out .nuspec files to get all the nuget stuff right for these > > > projects > > > -- I don't think the generation will work for us to get things quite > > right. > > > > > > Regarding the nuget key -- that plan works for me, the trick is I > > > don't have the key to add to myget. Come to think of it I don't > > > think I have the proverbial keys to the myget page either but I > > > think Martin can help us out there. > > > > > > Buffers could be the issue on the tests -- I've long suspected that > > > or I/O causing the meltdown, I just haven't been able to reproduce. > > > I would love to start beating on that a bit but the .net core > > > version seems to want NUnit 3.5+ which needs to be added to the > > > project to run. If you get that added I can start beating on the > > > test problems a > > bit more. > > > > > > Thanks for all your hard work putting this together, let me know how > > > I can help you get it out the proverbial door. > > > > > > On Fri, Mar 24, 2017 at 9:34 AM Shad Storhaug > > > <[email protected]<mailto:[email protected]>> > > > wrote: > > > > > > > Wyatt, > > > > > > > > Thanks. Actually, I was thinking this should go in a few steps > > > > instead of > > > > one: > > > > > > > > 1. Merge #203. > > > > 2. Change the pre-release label to "beta2" and work out any issues > > > > to build/push to MyGet (might take a few tries) 3. Update the > > > > README and CONTRIBUTING pages 4. Push the package to NuGet > > > > > > > > I have always just used the control panel at MyGet to push > > > > upstream to NuGet, and it is capable of storing someone's key so > > > > the person who pushes it doesn't actually need it. > > > > > > > > As far as the tests burning down are concerned, I discovered that > > > > some of them write so much "verbose" data that they overflow > > > > NUnit's buffer and cause it to crash (sometimes this even causes > > > > Visual Studio to crash locally). I think I have found all of the > > > > tests in the Core that were causing this and hard-coded them to > > > > set verbose off (with the ability to manually override), but I > > > > noticed that there are still tests in Analysis.Common that can cause > it to crash. > > > > I haven't investigated if there is a setting in NUnit to increase > > > > the buffer size, which might be a better fix, but I could probably > > > > track down the rest of the tests that are causing this before the > > merge. > > > > > > > > Thanks, > > > > Shad Storhaug (NightOwl888) > > > > > > > > -----Original Message----- > > > > From: Wyatt Barnett > > > > [mailto:[email protected]<mailto:[email protected]>] > > > > Sent: Friday, March 24, 2017 8:14 PM > > > > To: [email protected]<mailto:[email protected]> > > > > Subject: Re: API Work/Stabilization Update > > > > > > > > I'm about and happy to update when you are ready. > > > > > > > > I think the build should start working for 203 if you add the > > > > nunit-console nuget package. At least work in the sense of the > > > > build will start. I'm still chasing those build time bugbears I > > > > haven't been able to slay yet. > > > > > > > > As for getting to nuget.org<http://nuget.org> -- who has the key? > > > > I've never had access to it so I'm not sure we can update what is > out there. > > > > > > > > > > > > On Fri, Mar 24, 2017 at 09:10 Shad Storhaug > > > > <[email protected]<mailto:[email protected]>> > > > wrote: > > > > > > > > > Wyatt, Prescott, Itamar, anybody? We've almost got all of the > > > > > tests passing on #203 and would like to merge to master and > > > > > release it with the new pre-release label "beta2". > > > > > > > > > > If there is nobody available to get the build running after > > > > > merging, could someone give me access to TeamCity to work on it? > > > > > > > > > > Thanks, > > > > > Shad Storhaug (NightOwl888) > > > > > > > > > > From: Shad Storhaug > > > > > Sent: Tuesday, March 21, 2017 5:25 AM > > > > > To: '[email protected]<mailto:[email protected]>' > > > > > Subject: API Work/Stabilization Update > > > > > > > > > > I am getting very close to getting #203 merged. I wouldn't go so > > > > > far as to say that the API is finished, but the most significant > > > > > of the breaking API changes are now behind us. > > > > > > > > > > BUILD/VERSIONING > > > > > > > > > > I just wanted to be sure there is someone available to help get > > > > > the build working after the merge. I think it would be > > > > > appropriate to change the pre-release label from "beta" to > > > > > "beta2" (without resetting the build number, since that is > actually what NuGet uses). > > > > > This would be primarily because of a major breaking API change, > > > > > but also to indicate another advancement toward release. > > > > > > > > > > We should probably also get this onto NuGet as soon as possible > > > > > to > > > > > (hopefully) make it easier to recruit help to stabilize and > > > > > create some integration packages for popular Microsoft frameworks. > > > > > > > > > > KNOWN ISSUES > > > > > > > > > > > > > > > 1. The QueryParser.Flexible custom localized message > > > functionality > > > > > is currently not implemented for .NET core, so those tests are > > > > > now > > > > failing. > > > > > > > > > > 2. The implementation of Lucene.Net.Expressions currently > reads > > > > data > > > > > from the configuration file. This is not how modern libraries > > > > > are supposed to be built - instead we want any configuration to > > > > > be pushed in from the application that uses Lucene.Net. Reading > > > > > from the configuration file directly means no opportunity to use > > > > > dependency injection. There is also a namespace > > > > > Support/Configuration that can and should be removed after the > > > > > implementation is refactored to be DI-friendly (see > > > > > http://blog.ploeh.dk/2014/05/19/di-friendly-framework/). I > > > > > haven't yet worked out how the implementation was done in .NET - > > > > > in Java, the defaults were read from an embedded resource file > > > > > and could be overridden by passing in a ClassLoader (similar to > > > > > .NET's Assembly > > > > class) - if anyone has any information on how the "auto generated" > > > > C# code was generated, please share. > > > > > > > > > > 3. The Collation functionality in Analysis.Common doesn't > work > > > with > > > > > icu-dotnet, and has been excluded from compilation using the > > > > > constant FEATURE_COLLATION. I am now convinced after reading the > > > > > docs that it would be better to port the similar functionality > > > > > from Analysis.ICU because it was designed to work with icu4j and > > > > > is therefore more likely to work with icu-dotnet. > > > > > > > > > > 4. The Highlighter PostingsHighlight and VectorHighlight > > > > > functionality relies on icu-dotnet, which doesn't have a close > > > > > match for the BreakIterator in the JRE, so there are likely some > > > > > big differences in the functionality. Several hacks were put in > > > > > to make the tests pass, but these are not likely to fix all of > > > > > the issues in the > > > > wild. > > > > > > > > > > 5. There are several namespaces in Lucene.Net.Core and > > > > > Lucene.Net.Codecs that have broken documentation comments. > > > > > > > > > > 6. There are some concurrency and performance issues (as > > pointed > > > > out > > > > > by Vincent Van Den Burghe): > > > > > http://git.net/ml/general/2017-02/msg00168.html > > > > > > > > > > 7. We have around 2 dozen tests that fail during > randomization > > > > > (averaging about 17 broken per run), and 8 tests that fail > > > > > all/most of the time. > > > > > > > > > > RESOLVED ISSUES (in addition to API refactoring) > > > > > > > > > > > > > > > 1. Finished implementing the randomization of Codecs, > Culture, > > > Time > > > > > Zone, and InfoStream in the TestFramework. > > > > > > > > > > 2. Added factories for Codec, DocValuesFormat, and > > PostingsFormat > > > > so > > > > > custom implementations can be provided via dependency injection > > > > > instead of using the Java-ish NamedSPILoader class. The name > > > > > must now be provided by an attribute (or by class naming > > > > > convention) rather than via constructor, so it can be read > > > > > without creating an instance of > > > > the class. > > > > > > > > > > 3. Fixed several of the codecs in Lucene.Net.Codecs that were > > > still > > > > > not functioning (and not being tested because of the unfinished > > > > > RandomCodec class and test mocks). > > > > > > > > > > 4. Reviewed all catch blocks in Lucene.Net.Core to ensure the > > > right > > > > > type of exceptions are being caught and the right type re-thrown. > > > > > > > > > > 5. Fixed culture-sensitive comparison and sort order issues > > when > > > > > using strings in Lucene.Net.Core and Lucene.Net.Codecs. > > > > > > > > > > 6. Merged similar functionality in Support into the same > class > > > and > > > > > deleted several unused Support classes. > > > > > > > > > > 7. Made the API CLS compliant, so it now works with all .NET > > > > > languages. > > > > > > > > > > > > > > > Shad Storhaug (NightOwl888) > > > > > > > > > > > > > > > >
