Adrien, I’ve pushed some more changes and seems like I’d have to regenerate some test indexes but I’m not sure how to do that. Do you mind taking a look at this in it’s current form, and also my commits? It is all @ my fork here: https://github.com/anshumg/lucene-solr <https://github.com/anshumg/lucene-solr>
P.S: I thought it’d make more sense to do this on a feature-branch but the upgrade script wasn’t happy about that. -Anshum > On Jun 29, 2017, at 9:20 AM, Anshum Gupta <ansh...@apple.com> wrote: > > Going with your suggestions, seems like we’d be wiping out all of the > backward-codecs folder/package, is that correct ? Also, do we need to put in > anything to ensure back-combat between 6x, and 7x? > > -Anshum > > > >> On Jun 29, 2017, at 7:21 AM, Anshum Gupta <ansh...@apple.com >> <mailto:ansh...@apple.com>> wrote: >> >> Thanks Adrien, I’d want to try and do this myself as long as you can >> validate the correctness :). >> >> I’ll be working on this in a few hours and should have an update later today >> and hopefully we’d wrap it up soon. >> >> -Anshum >> >> >> >>> On Jun 28, 2017, at 10:39 AM, Adrien Grand <jpou...@gmail.com >>> <mailto:jpou...@gmail.com>> wrote: >>> >>> If you don't want to do it, I can do it tomorrow but if you'd like to give >>> it a try I'd be happy to help if you need any guidance. >>> >>> Le mer. 28 juin 2017 à 19:38, Adrien Grand <jpou...@gmail.com >>> <mailto:jpou...@gmail.com>> a écrit : >>> Hi Anshum, >>> >>> This looks like a good start to me. You would also need to remove the 6.x >>> version constants so that TestBackwardCompatibility does not think they are >>> worth testing, as well as all codecs, postings formats and doc values >>> formats that are defined in the lucene/backward-codecs module since they >>> are only about 6.x codecs. >>> >>> Le mer. 28 juin 2017 à 09:57, Anshum Gupta <ansh...@apple.com >>> <mailto:ansh...@apple.com>> a écrit : >>> Thanks for confirming that Alan, I had similar thoughts but wasn’t sure. >>> >>> I don’t want to change anything that I’m not confident about so I’m just >>> going to create remove those and commit it to my fork. If someone who’s >>> confident agrees with what I’m doing, I’ll go ahead and make those changes >>> to the upstream :). >>> >>> -Anshum >>> >>> >>> >>>> On Jun 28, 2017, at 12:54 AM, Alan Woodward <a...@flax.co.uk >>>> <mailto:a...@flax.co.uk>> wrote: >>>> >>>> We don’t need to support lucene5x codecs in 7, so you should be able to >>>> just remove those tests (and the the relevant packages from >>>> backwards-codecs too), I think? >>>> >>>> >>>>> On 28 Jun 2017, at 08:38, Anshum Gupta <ansh...@apple.com >>>>> <mailto:ansh...@apple.com>> wrote: >>>>> >>>>> I tried to move forward to see this work before automatically computing >>>>> the versions but I have about 30 odd failing test. I’ve made those >>>>> changes and pushed to my local GitHub account in case you have the time >>>>> to look: https://github.com/anshumg/lucene-solr >>>>> <https://github.com/anshumg/lucene-solr> >>>>> >>>>> Here’s the build summary if that helps: >>>>> >>>>> [junit4] Tests with failures [seed: 31C3B60E557C7E14] (first 10 out of >>>>> 31): >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testOutliers2 >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testShortRange >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewValues >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFullLongRange >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testRamBytesUsed >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testFewLargeValues >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testByteRange >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat.testLongRange >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene50.TestLucene50SegmentInfoFormat.testRandomExceptions >>>>> [junit4] - >>>>> org.apache.lucene.codecs.lucene62.TestLucene62SegmentInfoFormat.testRandomExceptions >>>>> [junit4] >>>>> [junit4] >>>>> [junit4] JVM J0: 0.56 .. 9.47 = 8.91s >>>>> [junit4] JVM J1: 0.56 .. 4.13 = 3.57s >>>>> [junit4] JVM J2: 0.56 .. 47.28 = 46.73s >>>>> [junit4] JVM J3: 0.56 .. 3.89 = 3.33s >>>>> [junit4] Execution time total: 47 seconds >>>>> [junit4] Tests summary: 8 suites, 215 tests, 30 errors, 1 failure, 24 >>>>> ignored (24 assumptions) >>>>> >>>>> >>>>> -Anshum >>>>> >>>>> >>>>> >>>>>> On Jun 27, 2017, at 4:15 AM, Adrien Grand <jpou...@gmail.com >>>>>> <mailto:jpou...@gmail.com>> wrote: >>>>>> >>>>>> The test***BackwardCompatibility cases can be removed since they make >>>>>> sure that Lucene 7 can read Lucene 6 norms, while Lucene 8 doesn't have >>>>>> to be able to read Lucene 6 norms. >>>>>> >>>>>> TestSegmentInfos needs to be adapted to the new versions, we need to >>>>>> replace 5 with 6 and 8 with 9. Maybe we should compute those numbers >>>>>> automatically based on Version.LATEST.major so that it does not require >>>>>> manual changes when moving to a new major version. That would give 5 -> >>>>>> Version.LATEST.major-2 and 8 -> Version.LATEST.major+1. >>>>>> >>>>>> I can do those changes on Thursday if you don't feel comfortable doing >>>>>> them. >>>>>> >>>>>> >>>>>> >>>>>> Le mar. 27 juin 2017 à 08:12, Anshum Gupta <ansh...@apple.com >>>>>> <mailto:ansh...@apple.com>> a écrit : >>>>>> Without making any changes at all and just bumping up the version, I hit >>>>>> these errors when running the tests: >>>>>> >>>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>>> -Dtestcase=TestSegmentInfos -Dtests.method=testIllegalCreatedVersion >>>>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>>>> -Dtests.file.encoding=US-ASCII >>>>>> [junit4] FAILURE 0.01s J0 | >>>>>> TestSegmentInfos.testIllegalCreatedVersion <<< >>>>>> [junit4] > Throwable #1: junit.framework.AssertionFailedError: >>>>>> Expected exception IllegalArgumentException but no exception was thrown >>>>>> [junit4] > at >>>>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:CE340683BE44C211]:0) >>>>>> [junit4] > at >>>>>> org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2672) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion(TestSegmentInfos.java:35) >>>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>>> -Dtestcase=TestSegmentInfos -Dtests.method=testVersionsOneSegment >>>>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>>>> -Dtests.file.encoding=US-ASCII >>>>>> [junit4] ERROR 0.00s J0 | TestSegmentInfos.testVersionsOneSegment >>>>>> <<< >>>>>> [junit4] > Throwable #1: >>>>>> org.apache.lucene.index.CorruptIndexException: segments file recorded >>>>>> indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older >>>>>> version=7.0.0 >>>>>> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>>>> [junit4] > at >>>>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:A7477EE8875F2E36]:0) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment(TestSegmentInfos.java:67) >>>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>>> [junit4] 2> NOTE: reproduce with: ant test >>>>>> -Dtestcase=TestSegmentInfos -Dtests.method=testVersionsTwoSegments >>>>>> -Dtests.seed=C818A61FA6C293A1 -Dtests.slow=true -Dtests.locale=es-PR >>>>>> -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true >>>>>> -Dtests.file.encoding=US-ASCII >>>>>> [junit4] ERROR 0.00s J0 | TestSegmentInfos.testVersionsTwoSegments >>>>>> <<< >>>>>> [junit4] > Throwable #1: >>>>>> org.apache.lucene.index.CorruptIndexException: segments file recorded >>>>>> indexCreatedVersionMajor=8 but segment=_0(7.0.0):C1 has older >>>>>> version=7.0.0 >>>>>> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(RAMInputStream(name=segments_1)))) >>>>>> [junit4] > at >>>>>> __randomizedtesting.SeedInfo.seed([C818A61FA6C293A1:4EE9CC4194FBB648]:0) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:392) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:293) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:443) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:440) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:692) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:644) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:445) >>>>>> [junit4] > at >>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments(TestSegmentInfos.java:96) >>>>>> [junit4] > at java.lang.Thread.run(Thread.java:748) >>>>>> >>>>>> On applying the patch here (https://pastebin.com/tM4Fpy1Q >>>>>> <https://pastebin.com/tM4Fpy1Q>), I end up with the following errors: >>>>>> >>>>>> [junit4] Tests with failures [seed: 5B388AB1E2BEFF87]: >>>>>> [junit4] - >>>>>> org.apache.lucene.search.similarities.TestSimilarityBase.testLengthEncodingBackwardCompatibility >>>>>> [junit4] - >>>>>> org.apache.lucene.search.similarities.TestClassicSimilarity.testNormEncodingBackwardCompatibility >>>>>> [junit4] - >>>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>>> [junit4] - >>>>>> org.apache.lucene.search.similarities.TestBM25Similarity.testLengthEncodingBackwardCompatibility >>>>>> >>>>>> Any ideas on what I’m missing here? >>>>>> >>>>>> >>>>>> -Anshum >>>>>> >>>>>> >>>>>> >>>>>>> On Jun 26, 2017, at 2:55 PM, Ryan Ernst <r...@iernst.net >>>>>>> <mailto:r...@iernst.net>> wrote: >>>>>>> >>>>>>> After changing that constant check, do you get the same test failures? >>>>>>> What are the actual failure messages? >>>>>>> >>>>>>> On Mon, Jun 26, 2017 at 10:38 AM Anshum Gupta <ans...@anshumgupta.net >>>>>>> <mailto:ans...@anshumgupta.net>> wrote: >>>>>>> Thanks Adrien, sadly, the tests aren't passing. That's the reason why I >>>>>>> didn't push the changes. I'll see if someone else can help while you're >>>>>>> away. >>>>>>> >>>>>>> -Anshum >>>>>>> >>>>>>> On Mon, Jun 26, 2017 at 9:55 AM Adrien Grand <jpou...@gmail.com >>>>>>> <mailto:jpou...@gmail.com>> wrote: >>>>>>> Your proposed change looks correct to me. As far as whether other >>>>>>> changes are required, I'm currently traveling but can look when I'm >>>>>>> back on Thursday. Feel free to push the branches if tests are passing, >>>>>>> we can fix things later? >>>>>>> >>>>>>> Le lun. 26 juin 2017 à 07:13, Anshum Gupta <ans...@anshumgupta.net >>>>>>> <mailto:ans...@anshumgupta.net>> a écrit : >>>>>>> The only throw case that I think needs changing is in SegmentInfos.java >>>>>>> L315 >>>>>>> Version luceneVersion = Version.fromBits(input.readVInt(), >>>>>>> input.readVInt(), input.readVInt()); >>>>>>> if (luceneVersion.onOrAfter(Version.LUCENE_7_0_0) == false) { >>>>>>> // TODO: should we check indexCreatedVersion instead? >>>>>>> throw new IndexFormatTooOldException(input, "this index is too old >>>>>>> (version: " + luceneVersion + ")"); >>>>>>> } >>>>>>> >>>>>>> The Lucene version here should be LUCENE_7_0_0, instead of the original >>>>>>> LUCENE_6_0_0. >>>>>>> >>>>>>> Is there anything else that's needed (and is this the correct change?). >>>>>>> >>>>>>> -Anshum >>>>>>> >>>>>>> >>>>>>> On Sun, Jun 25, 2017 at 9:45 PM Anshum Gupta <ans...@anshumgupta.net >>>>>>> <mailto:ans...@anshumgupta.net>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> I am working on creating the 7x, and 7.0 branches but I have the >>>>>>> following failing tests: >>>>>>> [junit4] Tests with failures [seed: 4FBDDCD3F96316D3]: >>>>>>> [junit4] - >>>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsTwoSegments >>>>>>> [junit4] - >>>>>>> org.apache.lucene.index.TestSegmentInfos.testIllegalCreatedVersion >>>>>>> [junit4] - >>>>>>> org.apache.lucene.index.TestSegmentInfos.testVersionsOneSegment >>>>>>> >>>>>>> I assume this is a result of me not doing anything about the following >>>>>>> TODO that got printed when I ran addVersion.py. >>>>>>> TODO >>>>>>> - Update IndexFormatTooOldException throw cases >>>>>>> >>>>>>> Can someone shed more light on what needs to be done here? >>>>>>> >>>>>>> -Anshum >>>>>> >>>>> >>>> >>> >> >
