Found it.
It's a solr thing made worse by the interaction of lucene testutils and
jdk.internal.loader.URLClassPath's decision to hide anything gone wrong
when checking a URL
/*
* Checks whether the resource URL should be returned.
* Returns null on security check failure.
* Called by java.net.URLClassLoader.
*/
public static URL checkURL(URL url) {
if (url != null) {
try {
check(url);
} catch (Exception e) {
return null;
}
}
return url;
}
Yay. Fun. JDK classes swallowing exceptions silently.
At the start of this it only took me a little while to discover that there
was a security manager in play via debugging. Remembering that I saw emails
about that, I went to jira, found the ticket enabling it by default in 9.x
and eventually tracked down the name of the security policy file by reading
solr.in.sh and /bin/solr... The key issue that tripped me up is that the
tests have a *separate* security policy file, and there was pretty much no
way to know this without extensive reading of the build. Thus I got thrown
off track when
permission java.io.FilePermission
"${user.home}${/}.m2${/}repository${/}-", "read";
To solr/server/etc/security.policy had no effect. That and the fact that
no security exception was reported, led me to start chasing increasingly
improbable hypotheses. Many hours later when I went back to debugging
deeply into class loading, I found that the code was actually reading the
jar files in question, and then I finally caught it throwing a security
exception during my debugging.
It turns out that adding the above permission to
gradle/testing/randomization/policies/solr-tests.policy allows the test to
pass. [1]
I think we need to document this somewhere (or someone needs to point me to
the doc I missed, FWIW I hit this basically following the process in
dev-docs/dependency-upgrades.adoc treating lucene like a dependency, and
unaware that there is a "shortcut" mode for lucene specifically in
gradle/lucene-dev/lucene-dev-repo-composite.gradle and I find reading that
file none-to clear anyway)
That's the solr part, the lucene part is that the security exception is hit
when in org.apache.lucene.codecs.Codec$Holder.<clinit>(Codec.java:58)
when org.apache.lucene.tests.util.TestRuleSetupAndRestoreClassEnv#before
does
savedCodec = Codec.getDefault();
The error message "An SPI class of type org.apache.lucene.codecs.Codec with
name 'Lucene95' does not exist." was moderately misleading because the file
and the services files in the jar definitely did exist. This message should
vary if there is an installed security manager, maybe saying something like:
"An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene95'
does not exist. We have detected that a security manager is installed so it
is also possible that the jar containing the codec is inaccessible under
the current security policy. (Java does not throw SecurityException if this
is the case, it just ignores the jar!)" [2]
Having that explicitly called out would have been SUPER helpful.
-Gus
[1]: https://issues.apache.org/jira/browse/SOLR-16804
[2]: https://github.com/apache/lucene/issues/12300
On Mon, May 15, 2023 at 3:17 PM Michael Sokolov <msoko...@gmail.com> wrote:
> random guess - does it have something to do with modules?
>
> On Mon, May 15, 2023 at 11:14 AM Gus Heck <gus.h...@gmail.com> wrote:
> >
> > I hadn't seen that one. Thanks, I'll look at it. It already looks a bit
> confusing though since it seems to have options for pointing to a repo, but
> I appear to be pulling the jars successfully from .m2/repository already...
> (except then they don't work, so successful means I see them in the
> classpath of the relevant classloader). And if we can't deploy a valid jar
> to mavenLocal for some reason (tweaked the solr build so it sees
> mavenLocal()), (or solr can't consume such a jar) that seems like an issue
> for whichever one is breaking that.
> >
> > Debugging: The JDK appears to be attempting to load the services file
> from modules, but not seeing the lucene module. (just the jdk ones) Also it
> passes through a block that says:
> >
> > // not in a package of a module defined to this loader
> > for (URL url : findMiscResource(name)) {
> >
> > (but then iterates
> jdk.internal.loader.BuiltinClassLoader#nameToModule.values() to load things
> anyway)
> >
> > -Gus
> >
> > On Mon, May 15, 2023 at 10:54 AM Houston Putman <houstonput...@gmail.com>
> wrote:
> >>
> >> Gus, I haven't done this myself, but are you using the instructions
> provided in Solr's "gradle/lucene-dev/lucene-dev-repo-composite.gradle"?
> >>
> >> It looks like you need to specify the development lucene version
> differently than other dependencies...
> >>
> >> - Houston
> >>
> >> On Sat, May 13, 2023 at 10:14 AM Michael Sokolov <msoko...@gmail.com>
> wrote:
> >>>
> >>> doh I actually read your email and you said you already checked that -
> >>> I'm going to send out one of those "sokolov would like to retract the
> >>> previous email" emails. Does GMail even pretend to do that? I don't
> >>> know what's going on there! sorry
> >>>
> >>> On Sat, May 13, 2023 at 10:13 AM Michael Sokolov <msoko...@gmail.com>
> wrote:
> >>> >
> >>> > sorry - META-INF not WEB-INF
> >>> >
> >>> > On Sat, May 13, 2023 at 10:12 AM Michael Sokolov <msoko...@gmail.com>
> wrote:
> >>> > >
> >>> > > You are probably missing the contents of WEB-INF in your custom
> jar?
> >>> > > Roughly speaking the files in there define run-time-bound
> "services"
> >>> > > that are looked up by name by the JDK's service-loader API.
> >>> > >
> >>> > > On Sat, May 13, 2023 at 9:33 AM Gus Heck <gus.h...@gmail.com>
> wrote:
> >>> > > >
> >>> > > > Cross posting to lucene on the possibility that folks here are
> more likely to add customized lucene to Solr and recognize what I'm
> stumbling on? (zero responses on solr list)
> >>> > > >
> >>> > > > Note that the specific test that I happened to copy is not the
> issue, all tests are doing this (or at least so many tests are failing I
> can't see the ones that are passing easily).
> >>> > > >
> >>> > > > ---------- Forwarded message ---------
> >>> > > > From: Gus Heck <gus.h...@gmail.com>
> >>> > > > Date: Wed, May 10, 2023 at 6:50 PM
> >>> > > > Subject: Running 10.0 build with a custom lucene 9.5
> >>> > > > To: <dev@solr.apache.org>
> >>> > > >
> >>> > > >
> >>> > > > Lucene:
> >>> > > >
> >>> > > > I made a tweak to lucene for something I'm investigating, gave
> it a new version, deployed to mavenLocal()
> >>> > > > I have verified that the jars are built with correct
> META-INF/services files
> >>> > > >
> >>> > > > Solr:
> >>> > > >
> >>> > > > I added mavenLocal() in gradle/globals.gradle
> >>> > > > I removed the license file sha1 sigs for the default lucene &
> creates signatures for my test version
> >>> > > > I updated versions.props
> >>> > > > I updated versions.lock
> >>> > > >
> >>> > > > Now when I run individual solr tests via my ide they seem to
> pass, but virtually every test run via gradle fails with something like:
> >>> > > >
> >>> > > > org.apache.solr.embedded.TestJettySolrRunner > classMethod FAILED
> >>> > > > java.lang.ExceptionInInitializerError
> >>> > > > at
> org.apache.lucene.codecs.Codec.getDefault(Codec.java:141)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetupAndRestoreClassEnv.java:137)
> >>> > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:42)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> >>> > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> >>> > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> >>> > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > >
> >>> > > > Caused by:
> >>> > > > java.lang.IllegalArgumentException: An SPI class of type
> org.apache.lucene.codecs.Codec with name 'Lucene95' does not exist. You
> need to add the corresponding JAR file supporting this SPI to your
> classpath. The current classpath supports the following names: []
> >>> > > > at
> org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113)
> >>> > > > at
> org.apache.lucene.codecs.Codec$Holder.<clinit>(Codec.java:58)
> >>> > > > ... 19 more
> >>> > > >
> >>> > > > org.apache.solr.embedded.TestJettySolrRunner > classMethod FAILED
> >>> > > > java.lang.NullPointerException
> >>> > > > at
> java.base/java.util.Objects.requireNonNull(Objects.java:221)
> >>> > > > at
> org.apache.lucene.codecs.Codec.setDefault(Codec.java:151)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleSetupAndRestoreClassEnv.after(TestRuleSetupAndRestoreClassEnv.java:292)
> >>> > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:49)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> >>> > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> >>> > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> >>> > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> >>> > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> >>> > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > >
> >>> > > > org.apache.solr.embedded.TestJettySolrRunner > test suite's
> output saved to
> /Users/gus/projects/apache/solr/code/solr/solr/test-framework/build/test-results/test/outputs/OUTPUT-org.apache.solr.embedded.TestJettySolrRunner.txt,
> copied below:
> >>> > > > > java.lang.ExceptionInInitializerError
> >>> > > > > at
> org.apache.lucene.codecs.Codec.getDefault(Codec.java:141)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetupAndRestoreClassEnv.java:137)
> >>> > > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:42)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> >>> > > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> >>> > > > > at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> >>> > > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > >
> >>> > > > > Caused by:
> >>> > > > > java.lang.IllegalArgumentException: An SPI class of
> type org.apache.lucene.codecs.Codec with name 'Lucene95' does not exist.
> You need to add the corresponding JAR file supporting this SPI to your
> classpath. The current classpath supports the following names: []
> >>> > > > > at
> org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113)
> >>> > > > > at
> org.apache.lucene.codecs.Codec$Holder.<clinit>(Codec.java:58)
> >>> > > > > ... 19 more
> >>> > > > > java.lang.NullPointerException
> >>> > > > > at
> java.base/java.util.Objects.requireNonNull(Objects.java:221)
> >>> > > > > at
> org.apache.lucene.codecs.Codec.setDefault(Codec.java:151)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleSetupAndRestoreClassEnv.after(TestRuleSetupAndRestoreClassEnv.java:292)
> >>> > > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:49)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> >>> > > > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> >>> > > > > at
> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> >>> > > > > at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> >>> > > > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> >>> > > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2> NOTE: Mac OS X 12.6 aarch64/Azul Systems, Inc. 11.0.19
> (64-bit)/cpus=10,threads=1,free=235002360,total=268435456
> >>> > > > 2> NOTE: All tests run in this JVM: [TestJettySolrRunner]
> >>> > > > 2> NOTE: reproduce with: gradlew test --tests
> TestJettySolrRunner -Dtests.seed=2FB768B5A303764 -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> >>> > > >
> >>> > > > The only difference I've been able to determine is that the IDE
> does not run the test with a security manager, but that feels like a red
> herring since nothing is throwing security exception (or if it is, it's
> getting swallowed!).
> >>> > > >
> >>> > > > I have debugged and I can see that the classloaders have the new
> lucene jars I built on the classpath (in /Users/gus/.m2/repository as they
> should), but I haven't yet figuree out why the services are not loading.
> >>> > > >
> >>> > > > -Gus
> >>> > > >
> >>> > > >
> >>> > > > --
> >>> > > > http://www.needhamsoftware.com (work)
> >>> > > > http://www.the111shift.com (play)
> >>> > > >
> >>> > > >
> >>> > > > --
> >>> > > > http://www.needhamsoftware.com (work)
> >>> > > > http://www.the111shift.com (play)
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>>
> >
> >
> > --
> > http://www.needhamsoftware.com (work)
> > http://www.the111shift.com (play)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
