Hi,
as alternative to the jenkins-build-number I could also use the commit hash
(jenkins has a env var for this). Setup is the same, but instead of ${BUILD_ID}
I’d use ${GIT_COMMIT}, e.g.
“9.0.0-prerelease${GIT_COMMIT}” or “9.0.0-prerelease+${GIT_COMMIT}” (or another
char instead of “+”like in Ubuntu/Debian packages)
What do you think?
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [email protected]
From: Uwe Schindler <[email protected]>
Sent: Tuesday, May 25, 2021 3:57 PM
To: [email protected]
Cc: 'David Smiley' <[email protected]>
Subject: RE: Proposal to pin the Lucene snapshot version on main
Sorry, I changed the folder layout a bit:
https://nightlies.apache.org/solr/lucene-prereleases/5/
easiest would be to add a property in the general Gradle build:
prereleaseVersion=”4”
And then add snippets to add custom repo:
“https://nightlies.apache.org/solr/lucene-prereleases/${prereleaseVersion}/
<https://nightlies.apache.org/solr/lucene-prereleases/prerelease$%7bprereleaseVersion%7d/>
”
And use this version number for dependencies:
“9.0.0-prerelease${prereleaseVersion}”
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [email protected] <mailto:[email protected]>
From: Uwe Schindler <[email protected] <mailto:[email protected]> >
Sent: Tuesday, May 25, 2021 3:52 PM
To: [email protected] <mailto:[email protected]>
Cc: 'David Smiley' <[email protected] <mailto:[email protected]> >
Subject: RE: Proposal to pin the Lucene snapshot version on main
Hi,
it’s working, the first prerelease:
You have to add this folder as a Maven Repository (so each prerelease is a
separate Maven repository):
https://nightlies.apache.org/solr/lucene-prereleases/prerelease4/
Then you configure the version number of lucene in your dependencies:
“9.0.0-prerelease4”
IMHO, easiest would be to add a property in the general Gradle build:
prereleaseVersion=”4”
And then add snippets to add custom repo:
“https://nightlies.apache.org/solr/lucene-prereleases/prerelease${prereleaseVersion}/
<https://nightlies.apache.org/solr/lucene-prereleases/prerelease$%7bprereleaseVersion%7d/>
”
And use this version number for dependencies:
“9.0.0-prerelease${prereleaseVersion}”
Tell me if it works!
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [email protected] <mailto:[email protected]>
From: Uwe Schindler <[email protected] <mailto:[email protected]> >
Sent: Tuesday, May 25, 2021 3:35 PM
To: [email protected] <mailto:[email protected]>
Cc: 'David Smiley' <[email protected] <mailto:[email protected]> >
Subject: RE: Proposal to pin the Lucene snapshot version on main
Hi,
David, can you have a look at this:
<https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/>
https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/
This job builds a local Maven repo (unsigned) on request (somebody with Jenkins
access have to trigger it):
<https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/>
https://ci-builds.apache.org/job/Solr/job/Lucene-prerelease-main/ws/build/maven-local/
I will now change this that the above folder is publsihed as a separate folder
(also including build number) to ASF’s snapshot website.
Does this directory structure (below maven-local) looks fine?
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: <mailto:[email protected]> [email protected]
From: Uwe Schindler < <mailto:[email protected]> [email protected]>
Sent: Tuesday, May 25, 2021 3:06 PM
To: <mailto:[email protected]> [email protected]
Subject: RE: Proposal to pin the Lucene snapshot version on main
Hi,
sorry for not acting faster. I was a bit busy with new kitchen, my main work,
vaccination,…
The main problem was that I was not able to sign artifacts from Jenkins, but
with Alan Woodward’s tip on lucene’s dev list regarding the disabled artifact
signing (I would not think that we really need this here…) I am confident.
Problem is now how to synchronize the whole repository (so keep older
artifacts, the current Lucene jobs creates a completely new local Maven
artifact repository).When you sync this to the snapshot website, older Lucene
preview releases would get lost. Maybe it’s better to store it somewhere inside
jenkins. An alternative is to create a separate repository and then update base
URL and artifact version inside the gradle scripts. So it could be like a
repository with the jenkins build number in the repo’s base URL (needs to be
cahnged in the remotes of Gradle) and the version number. IMHO, this would be a
2 line change instead of a one-line change inside solr’s build.gradle, so it’s
not problematic.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [email protected] <mailto:[email protected]>
From: David Smiley < <mailto:[email protected]> [email protected]>
Sent: Tuesday, May 25, 2021 2:01 PM
To: Uwe Schindler < <mailto:[email protected]> [email protected]>
Cc: <mailto:[email protected]> [email protected]; Cassandra Targett <
<mailto:[email protected]> [email protected]>
Subject: Re: Proposal to pin the Lucene snapshot version on main
Thanks again for volunteering Uwe. I'm wondering what the status of this is?
There was recently a major change in Lucene to move SpanQueries, and I think
we're going to feel it in Solr. If not, it's just a matter of time before
there is some other impactful change.
~ David Smiley
Apache Lucene/Solr Search Developer
<http://www.linkedin.com/in/davidwsmiley>
http://www.linkedin.com/in/davidwsmiley
On Fri, May 7, 2021 at 10:22 AM Uwe Schindler < <mailto:[email protected]>
[email protected]> wrote:
Hi,
Sure!
I will play a little bit around. If needed, I can add some property to the
lucene build to locate a "local repo" (I think we have that already).
I just have to figure out, if some local repo, copied to some Webserver works
well as a full M2 repository, especially how to handle multiple intermediary
releases in it.
Uwe
Am May 7, 2021 1:47:45 PM UTC schrieb David Smiley <
<mailto:[email protected]> [email protected]>:
Sounds great Uwe! Can you do it please?
~ David Smiley
Apache Lucene/Solr Search Developer
<http://www.linkedin.com/in/davidwsmiley>
http://www.linkedin.com/in/davidwsmiley
On Fri, May 7, 2021 at 1:55 AM Uwe Schindler < <mailto:[email protected]>
[email protected]> wrote:
Hi Cassandra. Sure, We can create a job on ASF Jenkins that is only manually
triggered.
It builds the maven artifacts into a local file system repo. As post build job
we publish the output folder on nighties repo like the ref guide.
The version number is passed as -DreleaseVersion system property (e.g.
populated by Jenkins from the build number), like we do on artifacts builds on
jenkins anyways. It's no snapshot anymore, but like an half official build with
some custom version internal to our process.
If you want to upgrade Solr's Lucene, just press "build" on jenkins, wait for
it to finish, and finally copy-paste the version number from Jenkins build
description into gradle dependency version file.
Uwe
Uwe
Am May 6, 2021 8:13:00 PM UTC schrieb Cassandra Targett <
<mailto:[email protected]> [email protected]>:
It could be a manual process if that’s what you think is easiest and it could
still be on nightlies. It’s just a curl command to push something up:
<https://nightlies.apache.org/authoring.html>
https://nightlies.apache.org/authoring.html. If it is a manual process, I think
it would be better for it to be hosted in a place any of us could access, in
case you’re busy/something happens to you and we need to pin a different
snapshot.
I’m not sure I see a reason why Solr couldn’t have our own Jenkins job that
built Lucene once a week (or on demand, or whatever cadence that works) and
pushes the snapshot to nightlies - we don’t have to use one of Lucene’s jobs,
do we?
I feel like we could also cascade a new Lucene snapshot build into a “Lucene
validation” Solr build that verifies everything is OK with a new snapshot
before updating the hosted snapshot on nightlies (although, I don’t actually
know how to do that, it just seems like something that could be done).
Cassandra
On May 6, 2021, 10:13 AM -0500, David Smiley < <mailto:[email protected]>
[email protected]>, wrote:
I asked on the Lucene dev list about possible use of the Nightlies server. We
don't want to pollute nightlies on a regular basis (I think); this would be an
on-demand thing. As such, I'm not sure a CI server is the best way to approach
this vs a manual script publishing to an ASF personal Home space.
~ David Smiley
Apache Lucene/Solr Search Developer
<http://www.linkedin.com/in/davidwsmiley>
http://www.linkedin.com/in/davidwsmiley
On Mon, May 3, 2021 at 10:55 AM Cassandra Targett <
<mailto:[email protected]> [email protected]> wrote:
I’m sort of surprised no one has mentioned the <https://nightlies.apache.org/>
https://nightlies.apache.org/ server, which could be used for this purpose.
Jenkins can push to it, and nothing gets deleted until we decide to delete it
(or overwrite it). Solr Operator builds go there as do drafts of the Ref Guide
pre-publication (in different directories now, but we’ll fix that eventually).
Cassandra
On May 3, 2021, 9:15 AM -0500, David Smiley < <mailto:[email protected]>
[email protected]>, wrote:
RE "an external system like Maven" -- we're merely talking about adding another
JAR repo to the list of repos we already have. Heck, for this limited purpose,
we could even use <http://home.apache.org/~dsmiley/>
http://home.apache.org/~dsmiley/ Note that the Lucene project already uses
home dirs of some users for benchmark data. If we were talking about adding a
Maven *build* then I would totally appreciate your concern.
I volunteer to use my space for this purpose.
~ David Smiley
Apache Lucene/Solr Search Developer
<http://www.linkedin.com/in/davidwsmiley>
http://www.linkedin.com/in/davidwsmiley
On Sun, May 2, 2021 at 12:17 AM Ishan Chattopadhyaya <
<mailto:[email protected]> [email protected]> wrote:
> Let’s not complicate things.
By bringing in external systems like Maven, I think we're complicating things
even though a straight forward way (git submodules) exists.
On Sat, May 1, 2021 at 5:00 PM Jan Høydahl <
<mailto:[email protected]> [email protected]> wrote:
Sub modules is for organizing internal repos, not for pulling in external deps.
Let’s not complicate things. And once we switch main to 10.x we’d need to use
pure jar dependencies in branch_9x to depend upon actually released and voted
Lucene binaries.
We need some tooling to smoothly work with bleeding edge Lucene including local
snapshot builds with not yet pushed changes to Lucene. I’d rather put in some
work to establish such tooling or procedures.
Jan Høydahl
1. mai 2021 kl. 08:41 skrev Ishan Chattopadhyaya <
<mailto:[email protected]> [email protected]>:
> What submodules don't solve is releases - if you're
> on a
particular unreleased Lucene version then releasing
> Solr would still mean you need some kind of
> "public" pinned Lucene release for the
> Mavenworld.
Can we then update the submodule to point to the release tag or sha that Lucene
got released from?
On Sat, 1 May, 2021, 11:45 am Dawid Weiss, < <mailto:[email protected]>
[email protected]> wrote:
> Other than literally adding the git submodule, would we do anything else to
> modify the gradle build so that or do we (and Jenkinsfile) have to manually
> do a step to install Lucene before proceeding?
Technically you add a submodule and then make a composite build from
Solr side. I can provide a PR that does it if you wish... This is
simple. What submodules don't solve is releases - if you're on a
particular unreleased Lucene version then releasing Solr would still
mean you need some kind of "public" pinned Lucene release for the
Maven world. If you distribute the entire thing as binaries you can
just publish a snapshot of Lucene code, of course.
> I think adding git submodule means that we have to add back in all the build
> code for it.
No. The submodule is just a reference to a particular repository (and
commit) - the build code and remains nearly identical as it was with
Lucene. Composite builds in gradle are quite nice in that they're
(almost) transparent compared to dependency references.
> At which point, I'd rather just copy and commit the
> code they have so we don't have to learn another git system.
It's not really a different git system - it's a mechanism of
interacting with multiple repositories at once. Yes, it does add some
additional complexity but I think sooner or later you'll have to
interact with it anyway - many people favor monorepos these days and
this is a common way to assemble them from fragmented
sub-repositories.
> I've
> heard submodules don't play nice with Jenkins, but don't have any
> direct experience with them.
Some tools may require an additional flag to initialize and clone
sub-modules on the initial pull. Or manually.
git submodule init
git submodule update --recursive.
To summarize - I am not pushing for submodules, I do think they're a
viable alternative but pinned Maven references are simpler. The
problem with maven references is that we'd need a long(er) term maven
repository where such pinned references would live (without being
wiped out). Perhaps infra can help here and it'd solve the problem.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: <mailto:[email protected]>
[email protected]
For additional commands, e-mail: <mailto:[email protected]>
[email protected]
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
