Thanks again for volunteering Uwe. I'm wondering what the status of this is? There was recently a major change in Lucene to move SpanQueries, and I think we're going to feel it in Solr. If not, it's just a matter of time before there is some other impactful change.
~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Fri, May 7, 2021 at 10:22 AM Uwe Schindler <[email protected]> wrote: > Hi, > Sure! > I will play a little bit around. If needed, I can add some property to the > lucene build to locate a "local repo" (I think we have that already). > I just have to figure out, if some local repo, copied to some Webserver > works well as a full M2 repository, especially how to handle multiple > intermediary releases in it. > > Uwe > > Am May 7, 2021 1:47:45 PM UTC schrieb David Smiley <[email protected]>: >> >> Sounds great Uwe! Can you do it please? >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> >> >> On Fri, May 7, 2021 at 1:55 AM Uwe Schindler <[email protected]> wrote: >> >>> Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only >>> manually triggered. >>> It builds the maven artifacts into a local file system repo. As post >>> build job we publish the output folder on nighties repo like the ref guide. >>> >>> The version number is passed as -DreleaseVersion system property (e.g. >>> populated by Jenkins from the build number), like we do on artifacts builds >>> on jenkins anyways. It's no snapshot anymore, but like an half official >>> build with some custom version internal to our process. >>> >>> If you want to upgrade Solr's Lucene, just press "build" on jenkins, >>> wait for it to finish, and finally copy-paste the version number from >>> Jenkins build description into gradle dependency version file. >>> >>> Uwe >>> >>> Uwe >>> >>> Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett < >>> [email protected]>: >>>> >>>> It could be a manual process if that’s what you think is easiest and it >>>> could still be on nightlies. It’s just a curl command to push something up: >>>> https://nightlies.apache.org/authoring.html. If it is a manual >>>> process, I think it would be better for it to be hosted in a place any of >>>> us could access, in case you’re busy/something happens to you and we need >>>> to pin a different snapshot. >>>> >>>> I’m not sure I see a reason why Solr couldn’t have our own Jenkins job >>>> that built Lucene once a week (or on demand, or whatever cadence that >>>> works) and pushes the snapshot to nightlies - we don’t have to use one of >>>> Lucene’s jobs, do we? >>>> >>>> I feel like we could also cascade a new Lucene snapshot build into a >>>> “Lucene validation” Solr build that verifies everything is OK with a new >>>> snapshot before updating the hosted snapshot on nightlies (although, I >>>> don’t actually know how to do that, it just seems like something that could >>>> be done). >>>> >>>> Cassandra >>>> On May 6, 2021, 10:13 AM -0500, David Smiley <[email protected]>, >>>> wrote: >>>> >>>> I asked on the Lucene dev list about possible use of the Nightlies >>>> server. We don't want to pollute nightlies on a regular basis (I think); >>>> this would be an on-demand thing. As such, I'm not sure a CI server is the >>>> best way to approach this vs a manual script publishing to an ASF personal >>>> Home space. >>>> >>>> ~ David Smiley >>>> Apache Lucene/Solr Search Developer >>>> http://www.linkedin.com/in/davidwsmiley >>>> >>>> >>>> On Mon, May 3, 2021 at 10:55 AM Cassandra Targett < >>>> [email protected]> wrote: >>>> >>>>> I’m sort of surprised no one has mentioned the >>>>> https://nightlies.apache.org/ server, which could be used for this >>>>> purpose. Jenkins can push to it, and nothing gets deleted until we decide >>>>> to delete it (or overwrite it). Solr Operator builds go there as do drafts >>>>> of the Ref Guide pre-publication (in different directories now, but we’ll >>>>> fix that eventually). >>>>> >>>>> Cassandra >>>>> On May 3, 2021, 9:15 AM -0500, David Smiley <[email protected]>, >>>>> wrote: >>>>> >>>>> RE "an external system like Maven" -- we're merely talking about >>>>> adding another JAR repo to the list of repos we already have. Heck, for >>>>> this limited purpose, we could even use >>>>> http://home.apache.org/~dsmiley/ Note that the Lucene project >>>>> already uses home dirs of some users for benchmark data. If we were >>>>> talking about adding a Maven *build* then I would totally appreciate your >>>>> concern. >>>>> >>>>> I volunteer to use my space for this purpose. >>>>> >>>>> ~ David Smiley >>>>> Apache Lucene/Solr Search Developer >>>>> http://www.linkedin.com/in/davidwsmiley >>>>> >>>>> >>>>> On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya < >>>>> [email protected]> wrote: >>>>> >>>>>> > Let’s not complicate things. >>>>>> By bringing in external systems like Maven, I think we're >>>>>> complicating things even though a straight forward way (git submodules) >>>>>> exists. >>>>>> >>>>>> On Sat, May 1, 2021 at 5:00 PM Jan Høydahl < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Sub modules is for organizing internal repos, not for pulling in >>>>>>> external deps. Let’s not complicate things. And once we switch main to >>>>>>> 10.x >>>>>>> we’d need to use pure jar dependencies in branch_9x to depend upon >>>>>>> actually >>>>>>> released and voted Lucene binaries. >>>>>>> >>>>>>> We need some tooling to smoothly work with bleeding edge Lucene >>>>>>> including local snapshot builds with not yet pushed changes to Lucene. >>>>>>> I’d >>>>>>> rather put in some work to establish such tooling or procedures. >>>>>>> >>>>>>> Jan Høydahl >>>>>>> >>>>>>> 1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya < >>>>>>> [email protected]>: >>>>>>> >>>>>>> >>>>>>> > What submodules don't solve is releases - if you're >>>>>>> > on a >>>>>>> particular unreleased Lucene version then releasing >>>>>>> > Solr would still mean you need some kind of >>>>>>> > "public" pinned Lucene release for the >>>>>>> > Mavenworld. >>>>>>> >>>>>>> >>>>>>> Can we then update the submodule to point to the release tag or sha >>>>>>> that Lucene got released from? >>>>>>> >>>>>>> On Sat, 1 May, 2021, 11:45 am Dawid Weiss, <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> > Other than literally adding the git submodule, would we do >>>>>>>> anything else to modify the gradle build so that or do we (and >>>>>>>> Jenkinsfile) >>>>>>>> have to manually do a step to install Lucene before proceeding? >>>>>>>> >>>>>>>> Technically you add a submodule and then make a composite build from >>>>>>>> Solr side. I can provide a PR that does it if you wish... This is >>>>>>>> simple. What submodules don't solve is releases - if you're on a >>>>>>>> particular unreleased Lucene version then releasing Solr would still >>>>>>>> mean you need some kind of "public" pinned Lucene release for the >>>>>>>> Maven world. If you distribute the entire thing as binaries you can >>>>>>>> just publish a snapshot of Lucene code, of course. >>>>>>>> >>>>>>>> > I think adding git submodule means that we have to add back in >>>>>>>> all the build code for it. >>>>>>>> >>>>>>>> No. The submodule is just a reference to a particular repository >>>>>>>> (and >>>>>>>> commit) - the build code and remains nearly identical as it was with >>>>>>>> Lucene. Composite builds in gradle are quite nice in that they're >>>>>>>> (almost) transparent compared to dependency references. >>>>>>>> >>>>>>>> > At which point, I'd rather just copy and commit the >>>>>>>> > code they have so we don't have to learn another git system. >>>>>>>> >>>>>>>> It's not really a different git system - it's a mechanism of >>>>>>>> interacting with multiple repositories at once. Yes, it does add >>>>>>>> some >>>>>>>> additional complexity but I think sooner or later you'll have to >>>>>>>> interact with it anyway - many people favor monorepos these days and >>>>>>>> this is a common way to assemble them from fragmented >>>>>>>> sub-repositories. >>>>>>>> >>>>>>>> > I've >>>>>>>> > heard submodules don't play nice with Jenkins, but don't have any >>>>>>>> > direct experience with them. >>>>>>>> >>>>>>>> Some tools may require an additional flag to initialize and clone >>>>>>>> sub-modules on the initial pull. Or manually. >>>>>>>> >>>>>>>> git submodule init >>>>>>>> git submodule update --recursive. >>>>>>>> >>>>>>>> To summarize - I am not pushing for submodules, I do think they're a >>>>>>>> viable alternative but pinned Maven references are simpler. The >>>>>>>> problem with maven references is that we'd need a long(er) term >>>>>>>> maven >>>>>>>> repository where such pinned references would live (without being >>>>>>>> wiped out). Perhaps infra can help here and it'd solve the problem. >>>>>>>> >>>>>>>> Dawid >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>> -- >>> Uwe Schindler >>> Achterdiek 19, 28357 Bremen >>> https://www.thetaphi.de >>> >> > -- > Uwe Schindler > Achterdiek 19, 28357 Bremen > https://www.thetaphi.de >
