To get the best version name let’s use something that’s compatible with:
https://maven.apache.org/ref/3.3.3/maven-artifact/apidocs/org/apache/maven/artifact/versioning/ComparableVersion.html Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] From: Uwe Schindler <[email protected]> Sent: Tuesday, May 25, 2021 4:09 PM To: [email protected] Cc: 'David Smiley' <[email protected]> Subject: RE: Proposal to pin the Lucene snapshot version on main Hi, as alternative to the jenkins-build-number I could also use the commit hash (jenkins has a env var for this). Setup is the same, but instead of ${BUILD_ID} I’d use ${GIT_COMMIT}, e.g. “9.0.0-prerelease${GIT_COMMIT}” or “9.0.0-prerelease+${GIT_COMMIT}” (or another char instead of “+”like in Ubuntu/Debian packages) What do you think? Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] <mailto:[email protected]> From: Uwe Schindler <[email protected] <mailto:[email protected]> > Sent: Tuesday, May 25, 2021 3:57 PM To: [email protected] <mailto:[email protected]> Cc: 'David Smiley' <[email protected] <mailto:[email protected]> > Subject: RE: Proposal to pin the Lucene snapshot version on main Sorry, I changed the folder layout a bit: https://nightlies.apache.org/solr/lucene-prereleases/5/ easiest would be to add a property in the general Gradle build: prereleaseVersion=”4” And then add snippets to add custom repo: “https://nightlies.apache.org/solr/lucene-prereleases/${prereleaseVersion}/ <https://nightlies.apache.org/solr/lucene-prereleases/prerelease$%7bprereleaseVersion%7d/> ” And use this version number for dependencies: “9.0.0-prerelease${prereleaseVersion}” ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] <mailto:[email protected]> From: Uwe Schindler <[email protected] <mailto:[email protected]> > Sent: Tuesday, May 25, 2021 3:52 PM To: [email protected] <mailto:[email protected]> Cc: 'David Smiley' <[email protected] <mailto:[email protected]> > Subject: RE: Proposal to pin the Lucene snapshot version on main Hi, it’s working, the first prerelease: You have to add this folder as a Maven Repository (so each prerelease is a separate Maven repository): https://nightlies.apache.org/solr/lucene-prereleases/prerelease4/ Then you configure the version number of lucene in your dependencies: “9.0.0-prerelease4” IMHO, easiest would be to add a property in the general Gradle build: prereleaseVersion=”4” And then add snippets to add custom repo: “https://nightlies.apache.org/solr/lucene-prereleases/prerelease${prereleaseVersion}/ <https://nightlies.apache.org/solr/lucene-prereleases/prerelease$%7bprereleaseVersion%7d/> ” And use this version number for dependencies: “9.0.0-prerelease${prereleaseVersion}” Tell me if it works! Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] <mailto:[email protected]> From: Uwe Schindler <[email protected] <mailto:[email protected]> > Sent: Tuesday, May 25, 2021 3:35 PM To: [email protected] <mailto:[email protected]> Cc: 'David Smiley' <[email protected] <mailto:[email protected]> > Subject: RE: Proposal to pin the Lucene snapshot version on main Hi, David, can you have a look at this: <https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/> https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ This job builds a local Maven repo (unsigned) on request (somebody with Jenkins access have to trigger it): <https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/> https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/ I will now change this that the above folder is publsihed as a separate folder (also including build number) to ASF’s snapshot website. Does this directory structure (below maven-local) looks fine? ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: <mailto:[email protected]> [email protected] From: Uwe Schindler < <mailto:[email protected]> [email protected]> Sent: Tuesday, May 25, 2021 3:06 PM To: <mailto:[email protected]> [email protected] Subject: RE: Proposal to pin the Lucene snapshot version on main Hi, sorry for not acting faster. I was a bit busy with new kitchen, my main work, vaccination,… The main problem was that I was not able to sign artifacts from Jenkins, but with Alan Woodward’s tip on lucene’s dev list regarding the disabled artifact signing (I would not think that we really need this here…) I am confident. Problem is now how to synchronize the whole repository (so keep older artifacts, the current Lucene jobs creates a completely new local Maven artifact repository).When you sync this to the snapshot website, older Lucene preview releases would get lost. Maybe it’s better to store it somewhere inside jenkins. An alternative is to create a separate repository and then update base URL and artifact version inside the gradle scripts. So it could be like a repository with the jenkins build number in the repo’s base URL (needs to be cahnged in the remotes of Gradle) and the version number. IMHO, this would be a 2 line change instead of a one-line change inside solr’s build.gradle, so it’s not problematic. Uwe ----- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: [email protected] <mailto:[email protected]> From: David Smiley < <mailto:[email protected]> [email protected]> Sent: Tuesday, May 25, 2021 2:01 PM To: Uwe Schindler < <mailto:[email protected]> [email protected]> Cc: <mailto:[email protected]> [email protected]; Cassandra Targett < <mailto:[email protected]> [email protected]> Subject: Re: Proposal to pin the Lucene snapshot version on main Thanks again for volunteering Uwe. I'm wondering what the status of this is? There was recently a major change in Lucene to move SpanQueries, and I think we're going to feel it in Solr. If not, it's just a matter of time before there is some other impactful change. ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Fri, May 7, 2021 at 10:22 AM Uwe Schindler < <mailto:[email protected]> [email protected]> wrote: Hi, Sure! I will play a little bit around. If needed, I can add some property to the lucene build to locate a "local repo" (I think we have that already). I just have to figure out, if some local repo, copied to some Webserver works well as a full M2 repository, especially how to handle multiple intermediary releases in it. Uwe Am May 7, 2021 1:47:45 PM UTC schrieb David Smiley < <mailto:[email protected]> [email protected]>: Sounds great Uwe! Can you do it please? ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Fri, May 7, 2021 at 1:55 AM Uwe Schindler < <mailto:[email protected]> [email protected]> wrote: Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only manually triggered. It builds the maven artifacts into a local file system repo. As post build job we publish the output folder on nighties repo like the ref guide. The version number is passed as -DreleaseVersion system property (e.g. populated by Jenkins from the build number), like we do on artifacts builds on jenkins anyways. It's no snapshot anymore, but like an half official build with some custom version internal to our process. If you want to upgrade Solr's Lucene, just press "build" on jenkins, wait for it to finish, and finally copy-paste the version number from Jenkins build description into gradle dependency version file. Uwe Uwe Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett < <mailto:[email protected]> [email protected]>: It could be a manual process if that’s what you think is easiest and it could still be on nightlies. It’s just a curl command to push something up: <https://nightlies.apache.org/authoring.html> https://nightlies.apache.org/authoring.html. If it is a manual process, I think it would be better for it to be hosted in a place any of us could access, in case you’re busy/something happens to you and we need to pin a different snapshot. I’m not sure I see a reason why Solr couldn’t have our own Jenkins job that built Lucene once a week (or on demand, or whatever cadence that works) and pushes the snapshot to nightlies - we don’t have to use one of Lucene’s jobs, do we? I feel like we could also cascade a new Lucene snapshot build into a “Lucene validation” Solr build that verifies everything is OK with a new snapshot before updating the hosted snapshot on nightlies (although, I don’t actually know how to do that, it just seems like something that could be done). Cassandra On May 6, 2021, 10:13 AM -0500, David Smiley < <mailto:[email protected]> [email protected]>, wrote: I asked on the Lucene dev list about possible use of the Nightlies server. We don't want to pollute nightlies on a regular basis (I think); this would be an on-demand thing. As such, I'm not sure a CI server is the best way to approach this vs a manual script publishing to an ASF personal Home space. ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Mon, May 3, 2021 at 10:55 AM Cassandra Targett < <mailto:[email protected]> [email protected]> wrote: I’m sort of surprised no one has mentioned the <https://nightlies.apache.org/> https://nightlies.apache.org/ server, which could be used for this purpose. Jenkins can push to it, and nothing gets deleted until we decide to delete it (or overwrite it). Solr Operator builds go there as do drafts of the Ref Guide pre-publication (in different directories now, but we’ll fix that eventually). Cassandra On May 3, 2021, 9:15 AM -0500, David Smiley < <mailto:[email protected]> [email protected]>, wrote: RE "an external system like Maven" -- we're merely talking about adding another JAR repo to the list of repos we already have. Heck, for this limited purpose, we could even use <http://home.apache.org/~dsmiley/> http://home.apache.org/~dsmiley/ Note that the Lucene project already uses home dirs of some users for benchmark data. If we were talking about adding a Maven *build* then I would totally appreciate your concern. I volunteer to use my space for this purpose. ~ David Smiley Apache Lucene/Solr Search Developer <http://www.linkedin.com/in/davidwsmiley> http://www.linkedin.com/in/davidwsmiley On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya < <mailto:[email protected]> [email protected]> wrote: > Let’s not complicate things. By bringing in external systems like Maven, I think we're complicating things even though a straight forward way (git submodules) exists. On Sat, May 1, 2021 at 5:00 PM Jan Høydahl < <mailto:[email protected]> [email protected]> wrote: Sub modules is for organizing internal repos, not for pulling in external deps. Let’s not complicate things. And once we switch main to 10.x we’d need to use pure jar dependencies in branch_9x to depend upon actually released and voted Lucene binaries. We need some tooling to smoothly work with bleeding edge Lucene including local snapshot builds with not yet pushed changes to Lucene. I’d rather put in some work to establish such tooling or procedures. Jan Høydahl 1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya < <mailto:[email protected]> [email protected]>: > What submodules don't solve is releases - if you're > on a particular unreleased Lucene version then releasing > Solr would still mean you need some kind of > "public" pinned Lucene release for the > Mavenworld. Can we then update the submodule to point to the release tag or sha that Lucene got released from? On Sat, 1 May, 2021, 11:45 am Dawid Weiss, < <mailto:[email protected]> [email protected]> wrote: > Other than literally adding the git submodule, would we do anything else to > modify the gradle build so that or do we (and Jenkinsfile) have to manually > do a step to install Lucene before proceeding? Technically you add a submodule and then make a composite build from Solr side. I can provide a PR that does it if you wish... This is simple. What submodules don't solve is releases - if you're on a particular unreleased Lucene version then releasing Solr would still mean you need some kind of "public" pinned Lucene release for the Maven world. If you distribute the entire thing as binaries you can just publish a snapshot of Lucene code, of course. > I think adding git submodule means that we have to add back in all the build > code for it. No. The submodule is just a reference to a particular repository (and commit) - the build code and remains nearly identical as it was with Lucene. Composite builds in gradle are quite nice in that they're (almost) transparent compared to dependency references. > At which point, I'd rather just copy and commit the > code they have so we don't have to learn another git system. It's not really a different git system - it's a mechanism of interacting with multiple repositories at once. Yes, it does add some additional complexity but I think sooner or later you'll have to interact with it anyway - many people favor monorepos these days and this is a common way to assemble them from fragmented sub-repositories. > I've > heard submodules don't play nice with Jenkins, but don't have any > direct experience with them. Some tools may require an additional flag to initialize and clone sub-modules on the initial pull. Or manually. git submodule init git submodule update --recursive. To summarize - I am not pushing for submodules, I do think they're a viable alternative but pinned Maven references are simpler. The problem with maven references is that we'd need a long(er) term maven repository where such pinned references would live (without being wiped out). Perhaps infra can help here and it'd solve the problem. Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: <mailto:[email protected]> [email protected] For additional commands, e-mail: <mailto:[email protected]> [email protected] -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
