Thanks Jan, AND FYI for everyone -- while Jan was looking into this, i was opening some Jiras (that Jan has already commented on) ..
https://issues.apache.org/jira/browse/SOLR-18396 https://issues.apache.org/jira/browse/SOLR-18397 : Date: Sat, 22 Aug 2026 01:22:02 +0200 : From: Jan Høydahl <[email protected]> : Reply-To: [email protected] : To: [email protected] : Subject: Re: lucene-upgrade.md instructions don't seem to work -- notably: : resolveAndLockAll : : I feel responsible for part of this as I added the 'collectJarInfos' to : renovate.json and AGENTS.md. While it apparently fixed one condition at : that point, it was still flawed. : : So I used Claude to analyze, and here is what came out. Take it for what it is... : : : Short version: it's a build bug, the lockfiles are the : symptom, and neither resolveAndLockAll nor solrbot is really "the" culprit -- : they just disagree because we ask them to run different commands. : : WTF are all of these `xxxxxxCopy` entries? : ------------------------------------------ : : They come from gradle/validation/jar-checks.gradle, in collectJarInfos: : : configurations.jarValidation.extendsFrom.each {conf -> : if (excludeRules && excludeRules.size() > 0) { : conf = conf.copyRecursive() // <-- here : conf.canBeResolved = true : excludeRuleMaps.forEach {conf.exclude(it)} : } : if (conf.canBeResolved) { : queue.addAll(conf.resolvedConfiguration.firstLevelModuleDependencies) : } : } : : Gradle names the result of Configuration.copyRecursive() "<source>Copy", so : compileClasspath begets compileClasspathCopy, etc. These are throwaway scratch : configurations that exist only while collectJarInfos is executing; they're : there so we can apply jarValidation's exclude rules without those excludes : leaking into the real classpaths. : : The reason they end up in the lockfiles is gradle/validation/dependencies.gradle: : : dependencyLocking { : lockAllConfigurations() : } : : copyRecursive() copies the source's resolution strategy, so each clone inherits : active dependency locking -- under its own name. : : That also explains why only 4 lockfiles have them: the copy branch only runs : when a project puts an exclude directly on jarValidation, which is exactly : solr/core, solr/test-framework, solr/solrj-zookeeper and solr-ref-guide. : : (solr/modules/ltr's "localPythonClientCopy" is a red herring -- a hand-named : configuration that merely ends in "Copy".) : : 1) Why did solrbot add them? : ---------------------------- : : Because .github/renovate.json tells it to run a different command than : lucene-upgrade.md tells us to run: : : "postUpgradeTasks": { : "commands": [ : "./gradlew resolveAndLockAll collectJarInfos --write-locks", : ... : : collectJarInfos executes inside a --write-locks build, so the ephemeral clones : get their lock state persisted. dev-docs/gradle-help/dependencies.txt and : AGENTS.md matched the bot; dependency-upgrades.adoc, lucene-upgrade.md and : ui/component-development.adoc all said plain `resolveAndLockAll --write-locks`. : So we had two documented commands producing two different lockfiles. : : On timing: you're right that 742c7d99c37 is where ~470 of these appeared. Worth : noting that commit was an actions/setup-java bump -- a pure GitHub-Actions : change that cannot possibly affect Gradle resolution, yet it churned ~920 : lockfile lines. It was the first solrbot lockfile regeneration after : 0d0a7ab6bf2 (Gradle 9.6.0, three days earlier); before that upgrade the clones : weren't getting written. : : 2) Why didn't resolveAndLockAll update them? : -------------------------------------------- : : Because it can't see them. resolveAndLockAll iterates project.configurations: : : project.configurations.findAll { : it.canBeResolved && ... : }.each {it.resolve()} : : The clones aren't in that container -- they're created inside collectJarInfos' : task action and thrown away when it finishes. And Gradle's unique-lockfile : writer *merges* rather than rewrites, so entries for configurations that : weren't resolved in this build are preserved verbatim. Hence the split state : you saw: the real configurations moved to 10.5.1, the ghosts stayed at 10.4.0. : : That merge behaviour is also why your `updateLicenses` then died at : jar-checks.gradle:134 -- collectJarInfos resolved runtimeClasspathCopy against : lock state you'd just written for the *old* Lucene. : : 3) Which system is doing the wrong thing? : ----------------------------------------- : : The build is. Nothing ships out of those clones; they're internal scratch for : license/jar enumeration. Giving them lock state was never useful, and it's not : fixable from either end -- the bot can't stop generating them without dropping : collectJarInfos, and resolveAndLockAll structurally cannot refresh them. : : Evidence that this was already actively harmful, on main today: : solr/core/gradle.lockfile pins org.slf4j:slf4j-api at BOTH 2.0.17 (the real : configurations) and 2.0.18 (testRuntimeClasspathCopy only). A version nothing : resolves to any more, sitting in a lockfile. : : 4) If I delete them, will solrbot add them back? : ------------------------------------------------ : : As things stand, yes -- which is why the fix has to be in the build rather than : in the lockfiles. : : I have a draft PR in https://github.com/apache/solr/pull/4794 that attempts to fix: : : - jar-checks.gradle: after copyRecursive(), call : resolutionStrategy.deactivateDependencyLocking() and : shouldResolveConsistentlyWith(source). The second half is load-bearing: : dropping locking alone could let a transitive drift away from the tested : version and silently change which JAR gets license-checked. Pinning to the : source configuration keeps the versions exact without any lock state. : - Same treatment for the two copyRecursive() sites in gradle/solr/packaging.gradle : (runtimeLibsCopy, testRuntimeClasspathCopy) -- same latent bug, currently : invisible only because those tasks don't run during lock generation. : - Stripped the stale entries from the 4 lockfiles. : - Dropped collectJarInfos from the lock command in renovate.json, : dependencies.txt and AGENTS.md, so all six places that document this now : say the same thing: ./gradlew resolveAndLockAll --write-locks : - While in renovate.json: its fileFilters was "solr/**/gradle.lockfile", : which silently discarded bot changes to the root gradle.lockfile, : platform/gradle.lockfile and settings-gradle.lockfile. Widened. : - lucene-upgrade.md: the version-catalog snippet was still the old : "apache-lucene:*=10.3.0" ivy-era syntax. Fixed, plus a note that : updateLicenses compiles Solr (which answers your "WTF is updateLicenses : trying to compile all of Solr?" -- collectJarInfos depends on : jarValidation, which extends compileClasspath, which pulls in project : dependencies). : : Verified: : - validateJarChecksums / validateJarLicenses pass with zero changes under : solr/licenses -- i.e. the JAR set is byte-for-byte the same, so : shouldResolveConsistentlyWith is holding the versions. : - Running the OLD bot command (resolveAndLockAll collectJarInfos : --write-locks) no longer reintroduces a single Copy entry. : - resolveAndLockAll --write-locks twice in a row is now a no-op diff. : - Bumped apache-lucene to 10.5.1 and re-locked: every lucene coordinate moves : cleanly, no stragglers at 10.4.0, and collectJarInfos now fails only on the : genuine API changes rather than on lock state. : - gradlew check -x test passes. : : Jan : : > 21. aug. 2026 kl. 23:18 skrev Chris Hostetter <[email protected]>: : > : > : > : I think we should dramatically scale back the set of classpaths (aka gradle : > : "configurations"?) in the "lockfile" to reduce problems like this and also : > : > that feels like a very tangential suggestion to the main concerns in this : > thread? : > : > Even if 'runtimeClasspath' had been the only configuration in the : > lockfiles... : > : > 1) Solrbot added a runtimeClasspathCopy to a bunch of lockfiles (why?) : > : > 2) resolveAndLockAll didn't update runtimeClasspathCopy in any file (why?) : > : > 3) which of those two systems is doing the wrong thing? : > : > 4) If I try to "fix" the issue by deleting all the runtimeClasspathCopy : > entries in the lockfiles, is solrbot going to add them back at some point? : > : > : > : > : On Thu, Aug 20, 2026 at 7:08 PM Chris Hostetter <[email protected]> : > : wrote: : > : : > : > : > : > Spelunking through git history, it looks like this SolrBot PR/commit : > : > added (almost) all of the `xxxxCopy` entries to the `gradle.lockfile` : > : > files... : > : > : > : > https://github.com/apache/solr/pull/4564 : > : > main: : > : > https://github.com/apache/solr/commit/742c7d99c37701791f998fba9f960db469942218 : > : > 10x: : > : > https://github.com/apache/solr/commit/4fafb78192bba240d43c9539f6dcb98fc1ee1be6 : > : > : > : > : > : > : > : > hossman@slate:~/lucene/solr [j21] [main] $ git grep Copy | grep -c : > : > gradle.lockfile : > : > 471 : > : > hossman@slate:~/lucene/solr [j21] [main] $ git co : > : > 742c7d99c37701791f998fba9f960db469942218~ : > : > ... : > : > hossman@slate:~/lucene/solr [j21] [c851aae5c82] $ git grep Copy | grep -c : > : > gradle.lockfile : > : > 1 : > : > : > : > : > : > : > : > : > : > : > : > : > : > : > : > : > : > : > : > : Date: Thu, 20 Aug 2026 15:54:23 -0700 (MST) : > : > : From: Chris Hostetter <[email protected]> : > : > : To: Solr Dev <[email protected]> : > : > : Subject: lucene-upgrade.md instructions don't seem to work -- notably: : > : > : resolveAndLockAll : > : > : : > : > : : > : > : I had a little time this afternoon so I wanted to take a quick stab at : > : > : upgrading Solr to Lucene 10.5.1 just to see what would break, and I feel : > : > like : > : > : either there is a bug in the resolveAndLockAll task or I am seriously : > : > : missunderstanding something with the instructions in lucene-upgrade.md : > : > : : > : > : : > : > : : > : > : Here's what I tried... : > : > : : > : > : : > : > : 1) clean checkout of main, review lucene-upgrade.md : > : > : : > : > : 2) right off the bat, the syntax of gradle/libs.versions.toml does not : > : > match : > : > : what lucene-upgrade.md says to modify. : > : > : : > : > : Instead of a change like this (as documented) ... : > : > : : > : > : - apache-lucene:*=10.3.0 : > : > : + apache-lucene:*=10.4.0 : > : > : : > : > : ...I assume this is a minor variation in how things are managed, and try : > : > : making a change like this... : > : > : : > : > : -apache-lucene = "10.4.0" : > : > : +apache-lucene = "10.5.1" : > : > : : > : > : : > : > : 3) ./gradlew resolveAndLockAll --write-locks : > : > : : > : > : At first glance this seems to work -- lots of log messages about : > : > "Persisted : > : > : dependency lock state for ..." and `git status` shows a lot of : > : > : `gradle.lockfile` files have been modified. : > : > : : > : > : At this point I move on to the next step of the instructions and get : > : > really : > : > : confused by 2 things -- but before I get to that, let me jump ahead to : > : > what I : > : > : eventually realized... : > : > : : > : > : $ git diff solr/core/gradle.lockfile | grep lucene-core : > : > : : > : > -org.apache.lucene:lucene-core:10.4.0=compileClasspath,compileClasspathCopy,jarValidation,runtimeClasspath,runtimeClasspathCopy,runtimeLibs,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy : > : > : : > : > +org.apache.lucene:lucene-core:10.4.0=compileClasspathCopy,runtimeClasspathCopy,testCompileClasspathCopy,testRuntimeClasspathCopy : > : > : : > : > +org.apache.lucene:lucene-core:10.5.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath : > : > : : > : > : ...this seems to have happened in every `gradle.lockfile` modified, for : > : > every : > : > : lucene jar mentioned --- the `xxxxxxCopy` values (no idea what these : > : > are?) all : > : > : stay associated with 10.4.0, while the corisponding `xxxxxx` values are : > : > all : > : > : correctly changed to 10.5.1. : > : > : : > : > : WTF are all of these `xxxxxxCopy` entries? : > : > : : > : > : : > : > : But as I mentioned -- it took me a while to notice this happened, so : > : > let's : > : > : talk about the other weird thing when i tried to continue with the : > : > : instructions... : > : > : : > : > : 4) ./gradlew updateLicenses : > : > : : > : > : The first thing that confuses the hell out of me when running this : > : > command, is : > : > : that for whatever reason, it depends on `:solr:api:compileJava` : > : > : : > : > : which means it depends on many things, including : > : > `:solr:core:compileJava` -- : > : > : which fails because there have been some public (but not intended for end : > : > : user) java API changes in lucene. : > : > : : > : > : WTF is `updateLicenses` trying to compile all of Solr? : > : > : : > : > : Fortunately the compilation problems seem pretty minor, and easy to fix, : > : > so : > : > : then I retry `./gradlew updateLicenses` ... : > : > : : > : > : ...and that's when everything goes wonky... : > : > : : > : > : 5) REDIX: ./gradlew updateLicenses : > : > : : > : > : We make it past compilation, and quickly get to... : > : > : : > : > : : > : > : ... : > : > : > Task :solr:core:collectJarInfos FAILED : > : > : ... : > : > : FAILURE: Build failed with an exception. : > : > : : > : > : * Where: : > : > : Script '/home/hossman/lucene/solr/gradle/validation/jar-checks.gradle' : > : > line: : > : > : 134 : > : > : : > : > : * What went wrong: : > : > : Execution failed for task ':solr:core:collectJarInfos' (registered in : > : > script : > : > : 'gradle/validation/jar-checks.gradle'). : > : > : > Could not resolve all dependencies for configuration : > : > : ':solr:core:runtimeClasspathCopy'. : > : > : > Could not resolve org.apache.lucene:lucene-analysis-kuromoji:10.5.1. : > : > : Required by: : > : > : project ':solr:core' : > : > : project ':solr:core' > project ':platform' : > : > : > Cannot find a version of : > : > 'org.apache.lucene:lucene-analysis-kuromoji' : > : > : that satisfies the version constraints: : > : > : Dependency path: 'root' (runtimeClasspathCopy) --> : > : > : 'org.apache.lucene:lucene-analysis-kuromoji:10.5.1' : > : > : Constraint path: 'root' (runtimeClasspathCopy) --> : > : > : 'org.apache.lucene:lucene-analysis-kuromoji:{strictly 10.4.0}' because : > : > of the : > : > : following reason: Dependency version enforced by Dependency Locking : > : > : Constraint path: 'root' (runtimeClasspathCopy) --> 'project : > : > : ':platform'' (runtimeElements) --> : > : > : 'org.apache.lucene:lucene-analysis-kuromoji:10.5.1' : > : > : : > : > : ....lots of errors like the above for every lucene jar ... : > : > : : > : > : > There are 19 more failures with identical causes. : > : > : : > : > : * Try: : > : > : > Run with :solr:core:dependencyInsight --configuration : > : > : runtimeClasspathCopy --dependency : > : > org.apache.lucene:lucene-analysis-kuromoji : > : > : to get more insight on how to solve the conflict. : > : > : : > : > : ... lots more "Run with :solr:core:dependencyInsight lines ... : > : > : : > : > : BUILD FAILED in 24s : > : > : : > : > : : > : > : 5) Try :solr:core:dependencyInsight ? : > : > : : > : > : This fails short and sweet... : > : > : : > : > : ./gradlew :solr:core:dependencyInsight --configuration : > : > runtimeClasspathCopy : > : > : --dependency org.apache.lucene:lucene-analysis-kuromoji : > : > : : > : > : FAILURE: Build failed with an exception. : > : > : : > : > : * What went wrong: : > : > : configuration 'runtimeClasspathCopy' not found in configuration : > : > container for : > : > : project ':solr:core'. : > : > : : > : > : * Try: : > : > : > Run with --stacktrace option to get the stack trace. : > : > : > Run with --info or --debug option to get more log output. : > : > : > Get more help at https://help.gradle.org. : > : > : : > : > : BUILD FAILED in 6s : > : > : : > : > : : > : > : WTF are all of these `xxxxxxCopy` entries? (REDUX) : > : > : : > : > : : > : > : : > : > : : > : > : It's at this point that i just start greping for '10.4.0' to try and : > : > figure : > : > : out what I screwed up, and realize something is very very wrong with all : > : > of : > : > : our `gradle.lockfile` files : > : > : : > : > : : > : > : : > : > : : > : > : -Hoss : > : > : http://www.lucidworks.com/ : > : > : : > : > : > : > -Hoss : > : > http://www.lucidworks.com/ : > : > : > : > --------------------------------------------------------------------- : > : > To unsubscribe, e-mail: [email protected] : > : > For additional commands, e-mail: [email protected] : > : > : > : > : > : : > : > -Hoss : > http://www.lucidworks.com/ : > : > --------------------------------------------------------------------- : > To unsubscribe, e-mail: [email protected] : > For additional commands, e-mail: [email protected] : : : : --------------------------------------------------------------------- : To unsubscribe, e-mail: [email protected] : For additional commands, e-mail: [email protected] : : -Hoss http://www.lucidworks.com/
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
