I feel responsible for part of this as I added the 'collectJarInfos' to
renovate.json and AGENTS.md. While it apparently fixed one condition at
that point, it was still flawed.
So I used Claude to analyze, and here is what came out. Take it for what it
is...
Short version: it's a build bug, the lockfiles are the
symptom, and neither resolveAndLockAll nor solrbot is really "the" culprit --
they just disagree because we ask them to run different commands.
WTF are all of these `xxxxxxCopy` entries?
------------------------------------------
They come from gradle/validation/jar-checks.gradle, in collectJarInfos:
configurations.jarValidation.extendsFrom.each {conf ->
if (excludeRules && excludeRules.size() > 0) {
conf = conf.copyRecursive() // <-- here
conf.canBeResolved = true
excludeRuleMaps.forEach {conf.exclude(it)}
}
if (conf.canBeResolved) {
queue.addAll(conf.resolvedConfiguration.firstLevelModuleDependencies)
}
}
Gradle names the result of Configuration.copyRecursive() "<source>Copy", so
compileClasspath begets compileClasspathCopy, etc. These are throwaway scratch
configurations that exist only while collectJarInfos is executing; they're
there so we can apply jarValidation's exclude rules without those excludes
leaking into the real classpaths.
The reason they end up in the lockfiles is
gradle/validation/dependencies.gradle:
dependencyLocking {
lockAllConfigurations()
}
copyRecursive() copies the source's resolution strategy, so each clone inherits
active dependency locking -- under its own name.
That also explains why only 4 lockfiles have them: the copy branch only runs
when a project puts an exclude directly on jarValidation, which is exactly
solr/core, solr/test-framework, solr/solrj-zookeeper and solr-ref-guide.
(solr/modules/ltr's "localPythonClientCopy" is a red herring -- a hand-named
configuration that merely ends in "Copy".)
1) Why did solrbot add them?
----------------------------
Because .github/renovate.json tells it to run a different command than
lucene-upgrade.md tells us to run:
"postUpgradeTasks": {
"commands": [
"./gradlew resolveAndLockAll collectJarInfos --write-locks",
...
collectJarInfos executes inside a --write-locks build, so the ephemeral clones
get their lock state persisted. dev-docs/gradle-help/dependencies.txt and
AGENTS.md matched the bot; dependency-upgrades.adoc, lucene-upgrade.md and
ui/component-development.adoc all said plain `resolveAndLockAll --write-locks`.
So we had two documented commands producing two different lockfiles.
On timing: you're right that 742c7d99c37 is where ~470 of these appeared. Worth
noting that commit was an actions/setup-java bump -- a pure GitHub-Actions
change that cannot possibly affect Gradle resolution, yet it churned ~920
lockfile lines. It was the first solrbot lockfile regeneration after
0d0a7ab6bf2 (Gradle 9.6.0, three days earlier); before that upgrade the clones
weren't getting written.
2) Why didn't resolveAndLockAll update them?
--------------------------------------------
Because it can't see them. resolveAndLockAll iterates project.configurations:
project.configurations.findAll {
it.canBeResolved && ...
}.each {it.resolve()}
The clones aren't in that container -- they're created inside collectJarInfos'
task action and thrown away when it finishes. And Gradle's unique-lockfile
writer *merges* rather than rewrites, so entries for configurations that
weren't resolved in this build are preserved verbatim. Hence the split state
you saw: the real configurations moved to 10.5.1, the ghosts stayed at 10.4.0.
That merge behaviour is also why your `updateLicenses` then died at
jar-checks.gradle:134 -- collectJarInfos resolved runtimeClasspathCopy against
lock state you'd just written for the *old* Lucene.
3) Which system is doing the wrong thing?
-----------------------------------------
The build is. Nothing ships out of those clones; they're internal scratch for
license/jar enumeration. Giving them lock state was never useful, and it's not
fixable from either end -- the bot can't stop generating them without dropping
collectJarInfos, and resolveAndLockAll structurally cannot refresh them.
Evidence that this was already actively harmful, on main today:
solr/core/gradle.lockfile pins org.slf4j:slf4j-api at BOTH 2.0.17 (the real
configurations) and 2.0.18 (testRuntimeClasspathCopy only). A version nothing
resolves to any more, sitting in a lockfile.
4) If I delete them, will solrbot add them back?
------------------------------------------------
As things stand, yes -- which is why the fix has to be in the build rather than
in the lockfiles.
I have a draft PR in https://github.com/apache/solr/pull/4794 that attempts to
fix:
- jar-checks.gradle: after copyRecursive(), call
resolutionStrategy.deactivateDependencyLocking() and
shouldResolveConsistentlyWith(source). The second half is load-bearing:
dropping locking alone could let a transitive drift away from the tested
version and silently change which JAR gets license-checked. Pinning to the
source configuration keeps the versions exact without any lock state.
- Same treatment for the two copyRecursive() sites in
gradle/solr/packaging.gradle
(runtimeLibsCopy, testRuntimeClasspathCopy) -- same latent bug, currently
invisible only because those tasks don't run during lock generation.
- Stripped the stale entries from the 4 lockfiles.
- Dropped collectJarInfos from the lock command in renovate.json,
dependencies.txt and AGENTS.md, so all six places that document this now
say the same thing: ./gradlew resolveAndLockAll --write-locks
- While in renovate.json: its fileFilters was "solr/**/gradle.lockfile",
which silently discarded bot changes to the root gradle.lockfile,
platform/gradle.lockfile and settings-gradle.lockfile. Widened.
- lucene-upgrade.md: the version-catalog snippet was still the old
"apache-lucene:*=10.3.0" ivy-era syntax. Fixed, plus a note that
updateLicenses compiles Solr (which answers your "WTF is updateLicenses
trying to compile all of Solr?" -- collectJarInfos depends on
jarValidation, which extends compileClasspath, which pulls in project
dependencies).
Verified:
- validateJarChecksums / validateJarLicenses pass with zero changes under
solr/licenses -- i.e. the JAR set is byte-for-byte the same, so
shouldResolveConsistentlyWith is holding the versions.
- Running the OLD bot command (resolveAndLockAll collectJarInfos
--write-locks) no longer reintroduces a single Copy entry.
- resolveAndLockAll --write-locks twice in a row is now a no-op diff.
- Bumped apache-lucene to 10.5.1 and re-locked: every lucene coordinate moves
cleanly, no stragglers at 10.4.0, and collectJarInfos now fails only on the
genuine API changes rather than on lock state.
- gradlew check -x test passes.
Jan
> 21. aug. 2026 kl. 23:18 skrev Chris Hostetter <[email protected]>:
>
>
> : I think we should dramatically scale back the set of classpaths (aka gradle
> : "configurations"?) in the "lockfile" to reduce problems like this and also
>
> that feels like a very tangential suggestion to the main concerns in this
> thread?
>
> Even if 'runtimeClasspath' had been the only configuration in the
> lockfiles...
>
> 1) Solrbot added a runtimeClasspathCopy to a bunch of lockfiles (why?)
>
> 2) resolveAndLockAll didn't update runtimeClasspathCopy in any file (why?)
>
> 3) which of those two systems is doing the wrong thing?
>
> 4) If I try to "fix" the issue by deleting all the runtimeClasspathCopy
> entries in the lockfiles, is solrbot going to add them back at some point?
>
>
>
> : On Thu, Aug 20, 2026 at 7:08 PM Chris Hostetter <[email protected]>
> : wrote:
> :
> : >
> : > Spelunking through git history, it looks like this SolrBot PR/commit
> : > added (almost) all of the `xxxxCopy` entries to the `gradle.lockfile`
> : > files...
> : >
> : > https://github.com/apache/solr/pull/4564
> : > main:
> : >
> https://github.com/apache/solr/commit/742c7d99c37701791f998fba9f960db469942218
> : > 10x:
> : >
> https://github.com/apache/solr/commit/4fafb78192bba240d43c9539f6dcb98fc1ee1be6
> : >
> : >
> : >
> : > hossman@slate:~/lucene/solr [j21] [main] $ git grep Copy | grep -c
> : > gradle.lockfile
> : > 471
> : > hossman@slate:~/lucene/solr [j21] [main] $ git co
> : > 742c7d99c37701791f998fba9f960db469942218~
> : > ...
> : > hossman@slate:~/lucene/solr [j21] [c851aae5c82] $ git grep Copy | grep -c
> : > gradle.lockfile
> : > 1
> : >
> : >
> : >
> : >
> : >
> : >
> : >
> : >
> : >
> : > : Date: Thu, 20 Aug 2026 15:54:23 -0700 (MST)
> : > : From: Chris Hostetter <[email protected]>
> : > : To: Solr Dev <[email protected]>
> : > : Subject: lucene-upgrade.md instructions don't seem to work -- notably:
> : > : resolveAndLockAll
> : > :
> : > :
> : > : I had a little time this afternoon so I wanted to take a quick stab at
> : > : upgrading Solr to Lucene 10.5.1 just to see what would break, and I feel
> : > like
> : > : either there is a bug in the resolveAndLockAll task or I am seriously
> : > : missunderstanding something with the instructions in lucene-upgrade.md
> : > :
> : > :
> : > :
> : > : Here's what I tried...
> : > :
> : > :
> : > : 1) clean checkout of main, review lucene-upgrade.md
> : > :
> : > : 2) right off the bat, the syntax of gradle/libs.versions.toml does not
> : > match
> : > : what lucene-upgrade.md says to modify.
> : > :
> : > : Instead of a change like this (as documented) ...
> : > :
> : > : - apache-lucene:*=10.3.0
> : > : + apache-lucene:*=10.4.0
> : > :
> : > : ...I assume this is a minor variation in how things are managed, and try
> : > : making a change like this...
> : > :
> : > : -apache-lucene = "10.4.0"
> : > : +apache-lucene = "10.5.1"
> : > :
> : > :
> : > : 3) ./gradlew resolveAndLockAll --write-locks
> : > :
> : > : At first glance this seems to work -- lots of log messages about
> : > "Persisted
> : > : dependency lock state for ..." and `git status` shows a lot of
> : > : `gradle.lockfile` files have been modified.
> : > :
> : > : At this point I move on to the next step of the instructions and get
> : > really
> : > : confused by 2 things -- but before I get to that, let me jump ahead to
> : > what I
> : > : eventually realized...
> : > :
> : > : $ git diff solr/core/gradle.lockfile | grep lucene-core
> : > :
> : >
> -org.apache.lucene:lucene-core:10.4.0=compileClasspath,compileClasspathCopy,jarValidation,runtimeClasspath,runtimeClasspathCopy,runtimeLibs,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy
> : > :
> : >
> +org.apache.lucene:lucene-core:10.4.0=compileClasspathCopy,runtimeClasspathCopy,testCompileClasspathCopy,testRuntimeClasspathCopy
> : > :
> : >
> +org.apache.lucene:lucene-core:10.5.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
> : > :
> : > : ...this seems to have happened in every `gradle.lockfile` modified, for
> : > every
> : > : lucene jar mentioned --- the `xxxxxxCopy` values (no idea what these
> : > are?) all
> : > : stay associated with 10.4.0, while the corisponding `xxxxxx` values are
> : > all
> : > : correctly changed to 10.5.1.
> : > :
> : > : WTF are all of these `xxxxxxCopy` entries?
> : > :
> : > :
> : > : But as I mentioned -- it took me a while to notice this happened, so
> : > let's
> : > : talk about the other weird thing when i tried to continue with the
> : > : instructions...
> : > :
> : > : 4) ./gradlew updateLicenses
> : > :
> : > : The first thing that confuses the hell out of me when running this
> : > command, is
> : > : that for whatever reason, it depends on `:solr:api:compileJava`
> : > :
> : > : which means it depends on many things, including
> : > `:solr:core:compileJava` --
> : > : which fails because there have been some public (but not intended for
> end
> : > : user) java API changes in lucene.
> : > :
> : > : WTF is `updateLicenses` trying to compile all of Solr?
> : > :
> : > : Fortunately the compilation problems seem pretty minor, and easy to fix,
> : > so
> : > : then I retry `./gradlew updateLicenses` ...
> : > :
> : > : ...and that's when everything goes wonky...
> : > :
> : > : 5) REDIX: ./gradlew updateLicenses
> : > :
> : > : We make it past compilation, and quickly get to...
> : > :
> : > :
> : > : ...
> : > : > Task :solr:core:collectJarInfos FAILED
> : > : ...
> : > : FAILURE: Build failed with an exception.
> : > :
> : > : * Where:
> : > : Script '/home/hossman/lucene/solr/gradle/validation/jar-checks.gradle'
> : > line:
> : > : 134
> : > :
> : > : * What went wrong:
> : > : Execution failed for task ':solr:core:collectJarInfos' (registered in
> : > script
> : > : 'gradle/validation/jar-checks.gradle').
> : > : > Could not resolve all dependencies for configuration
> : > : ':solr:core:runtimeClasspathCopy'.
> : > : > Could not resolve
> org.apache.lucene:lucene-analysis-kuromoji:10.5.1.
> : > : Required by:
> : > : project ':solr:core'
> : > : project ':solr:core' > project ':platform'
> : > : > Cannot find a version of
> : > 'org.apache.lucene:lucene-analysis-kuromoji'
> : > : that satisfies the version constraints:
> : > : Dependency path: 'root' (runtimeClasspathCopy) -->
> : > : 'org.apache.lucene:lucene-analysis-kuromoji:10.5.1'
> : > : Constraint path: 'root' (runtimeClasspathCopy) -->
> : > : 'org.apache.lucene:lucene-analysis-kuromoji:{strictly 10.4.0}' because
> : > of the
> : > : following reason: Dependency version enforced by Dependency Locking
> : > : Constraint path: 'root' (runtimeClasspathCopy) --> 'project
> : > : ':platform'' (runtimeElements) -->
> : > : 'org.apache.lucene:lucene-analysis-kuromoji:10.5.1'
> : > :
> : > : ....lots of errors like the above for every lucene jar ...
> : > :
> : > : > There are 19 more failures with identical causes.
> : > :
> : > : * Try:
> : > : > Run with :solr:core:dependencyInsight --configuration
> : > : runtimeClasspathCopy --dependency
> : > org.apache.lucene:lucene-analysis-kuromoji
> : > : to get more insight on how to solve the conflict.
> : > :
> : > : ... lots more "Run with :solr:core:dependencyInsight lines ...
> : > :
> : > : BUILD FAILED in 24s
> : > :
> : > :
> : > : 5) Try :solr:core:dependencyInsight ?
> : > :
> : > : This fails short and sweet...
> : > :
> : > : ./gradlew :solr:core:dependencyInsight --configuration
> : > runtimeClasspathCopy
> : > : --dependency org.apache.lucene:lucene-analysis-kuromoji
> : > :
> : > : FAILURE: Build failed with an exception.
> : > :
> : > : * What went wrong:
> : > : configuration 'runtimeClasspathCopy' not found in configuration
> : > container for
> : > : project ':solr:core'.
> : > :
> : > : * Try:
> : > : > Run with --stacktrace option to get the stack trace.
> : > : > Run with --info or --debug option to get more log output.
> : > : > Get more help at https://help.gradle.org.
> : > :
> : > : BUILD FAILED in 6s
> : > :
> : > :
> : > : WTF are all of these `xxxxxxCopy` entries? (REDUX)
> : > :
> : > :
> : > :
> : > :
> : > : It's at this point that i just start greping for '10.4.0' to try and
> : > figure
> : > : out what I screwed up, and realize something is very very wrong with all
> : > of
> : > : our `gradle.lockfile` files
> : > :
> : > :
> : > :
> : > :
> : > : -Hoss
> : > : http://www.lucidworks.com/
> : > :
> : >
> : > -Hoss
> : > http://www.lucidworks.com/
> : >
> : > ---------------------------------------------------------------------
> : > To unsubscribe, e-mail: [email protected]
> : > For additional commands, e-mail: [email protected]
> : >
> : >
> :
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
