I don't think Jan's workflow blocks Solr releases on Lucene releases. It just blocks this one feature branch merge on a Lucene release. Multiple Solr releases can happen between step 6 and step 7.
I completely agree with that being the workflow going forward with separate repos, Jan. It will unfortunately be a pain to integrate changes that affect both Lucene and Solr, but I think that's just a consequence of splitting the projects. Neither option gives us everything we want, so here are the pros and cons in my opinion. Using a snapshot lucene version - Easier to make changes to lucene and solr concurrently - Solr releases are blocked until the snapshot version being depended on is released. - Builds may break at any time, and possibly for different sets of users depending on dependency caches. Using a released lucene version - Harder to update lucene and solr concurrently - Solr can make releases independent of Lucene's release schedule - Builds are stable and consistent. Personally I think stability and the ability to own our own release schedule outweigh the benefits of being able to iterate on both projects concurrently. But it's definitely something that we should decide on as a community. On Fri, Feb 26, 2021 at 12:43 PM Mike Drob <[email protected]> wrote: > The part of this process that I really don't like is that Solr then still > becomes beholden to Lucene's release schedule. We don't know how long step > 7 takes, and will be effectively blocked from making our own releases until > that happens. > > On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl <[email protected]> wrote: > >> The developer workflow for adding something to both Lucene and Solr would >> be as any other dependency, right? >> So say we are on Luene 9.0. This is the process in my head: >> >> 1. Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to >> your local laptop (whatever command that is in gradle) >> 2. Make your Solr feature branch depend on lucene-9.1-SNAPSHOT >> instead of lucene-9.0.0 -hopefully Gradle will pick the local version over >> Apache Nexus version >> 3. Iterate 1-2 until happy >> 4. Make a Lucene PR and eventually commit the Lucene change >> 5. After next Jenkins build the feature is in Apache Nexus snapshot >> as lucene-9.1-SNAPSHOT >> 6. Now the Solr Pull Request will compile and can be tested by others >> 7. Wait until Lucene 9.1 release >> 8. Upgrade Solr's lucene dependency on 'main' >> 9. Merge Solr PR >> 10. Backporting will be a similar process, i.e. first backport and >> release in Lucene, then backport in Sol >> >> Hmm, as I wrote this list I can understand why so many features were >> added only to Solr and not to Lucene in the early days :) >> >> Jan >> >> 26. feb. 2021 kl. 14:22 skrev Gus Heck <[email protected]>: >> >> Except I just finished helping a contributor with a feature that touches >> both and I know for a fact that it was developed for his customer who was >> using solr (payload inequalities)... and have another in the works (the >> AQP).... Not being able to enhance lucene to support a feature in solr is >> an issue IMHO. >> >> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob <[email protected]> wrote: >> >>> It is possible to publish snapshots into the Apache Nexus repository. >>> That said, I think it is a bad idea for Solr to depend on Lucene snapshots >>> because that constrains the ability to do releases. Either you have to wait >>> for a Lucene release and then you can cut over, or you have to figure out >>> what changes you need to roll back. >>> >>> Features today rarely touch both fronts anyway, they usually land in >>> Lucene first and then percolate into Solr. For an easy example, we can see >>> how WAND was developed recently. >>> >>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman <[email protected]> >>> wrote: >>> >>>> Once the projects are on separate release cadences there wont be an >>>> ability to “add on both fronts” anymore. You will have to add to lucene, >>>> wait for a release, then add to Solr once Solr upgrades its lucene >>>> dependency to that new version. I dont imagine that we are going to keep >>>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in >>>> perpetuity. After it becomes possible (when lucene 9.0 is released) we >>>> should only be using released lucene versions as dependencies for every >>>> version branch in Solr. >>>> >>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck <[email protected]> wrote: >>>> >>>>> Until the first feature that wants to add something on both fronts... >>>>> Is it possible for Lucene to publish nightly snapshots? I know there is >>>>> some level of support for snapshots in central, though I don't know what >>>>> their usage policies are. If that's too restricted is there an artifact >>>>> repo controlled by the ASF that could be used? (An implementation of >>>>> Apache >>>>> Archiva?) This would have the added benefit of allowing solr to detect >>>>> when >>>>> Lucene breaks something before its released. >>>>> >>>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman < >>>>> [email protected]> wrote: >>>>> >>>>>> Hey everyone, >>>>>> >>>>>> Currently there is discussion going on, in SOLR-14762 >>>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the >>>>>> split of the lucene-solr repo into individual repos for Solr and Lucene. >>>>>> There seems to be agreement that we shouldn't wait for a Lucene release >>>>>> to >>>>>> do the split, and instead split now and release whenever that happens. >>>>>> >>>>>> The biggest issue that arises there is that Solr's master branch is >>>>>> obviously based on Lucene's master branch, since they are currently the >>>>>> same. So when the split happens, Solr master will have to depend on >>>>>> Lucene >>>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but >>>>>> that will result in inconsistent builds, depending on whatever cached >>>>>> dependencies each dev has locally. Personally, I think that will cause a >>>>>> bunch of build errors and headaches for everyone trying to maintain Solr. >>>>>> >>>>>> There is another option though. We could instead do an *alpha* >>>>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore >>>>>> Solr >>>>>> can reliably depend on a stable version of lucene until 9.0 is truly >>>>>> released. (And lucene can use a stable version of Solr, if it sees a need >>>>>> for that). There would be no guarantees for using this alpha release, and >>>>>> we don't have to advertise it at all. >>>>>> >>>>>> It's not perfect, but I think it would be preferable to depending on >>>>>> an ever-changing SNAPSHOT lucene. >>>>>> >>>>>> - Houston >>>>>> >>>>> >>>>> >>>>> -- >>>>> http://www.needhamsoftware.com (work) >>>>> http://www.the111shift.com (play) >>>>> >>>> >> >> -- >> http://www.needhamsoftware.com (work) >> http://www.the111shift.com (play) >> >> >>
