Can we commit the security policy addition independently?
On Thu, Jun 3, 2021 at 2:20 PM Michael Gibney <[email protected]>
wrote:
> Ok, I think I've got this now. jars from the mavenLocal() repo are used
> in-place by gradle, and not cached in `${gradle.lib.dir}`. This requires a
> security policy addition to
> `gradle/testing/randomization/policies/solr-tests.policy`:
>
> grant {
> // Allow reading gradle worker JAR.
> permission java.io.FilePermission "${gradle.worker.jar}", "read";
> // Allow reading from classpath JARs (resources).
> permission java.io.FilePermission "${gradle.user.home}${/}-", "read";
> + // Allow reading lucene jars from mavenLocal repository
> + permission java.io.FilePermission
> "${user.home}${/}.m2${/}repository${/}org${/}apache${/}lucene${/}-", "read";
> };
>
> This, along with the addition of the mavenLocal() repo and the change to
> "9.0.0-SNAPSHOT" lucene version in versions.props (and subsequent update of
> versions.lock file via `gradlew --write-locks`), does the trick.
>
> If I were to take a crack at documenting this workflow, any ideas on the
> best place to do so?
>
> Michael
>
>
> On Wed, Jun 2, 2021 at 9:19 AM Michael Gibney <[email protected]>
> wrote:
>
>> Thanks for digging into this! I can confirm that my
>> lucene-*-9.0.0-SNAPSHOT.jar files have META-INF/services/ with all the
>> expected codecs. I did remove the lucene subfolder from the maven
>> repo, re-ran `gradlew mavenToLocalRepo`, manually verified that the
>> expected META-INF/services/ listed all codecs, and re-ran the Solr
>> build (with `--refresh-dependencies`, and after having added the
>> `mavenLocal()` repo, changed `versions.props` to point at
>> "9.0.0-SNAPSHO", and run `gradlew --write-locks` to update the
>> versions.lock file). I'm still seeing the same exception ...
>>
>> Michael
>>
>> On Wed, Jun 2, 2021 at 8:01 AM Uwe Schindler <[email protected]> wrote:
>> >
>> > FYI,
>> >
>> > I did a recompile locally and checked the JAR files in my local Maven
>> repository ("./gradlew mavenToLocalRepo"): All contain META-INF/services
>> with all Codecs.
>> >
>> > I have no idea what goes wrong on your side. Next I will try to setup
>> mavenLocal() and a SNAPSHOT dependency, too.
>> >
>> > -----
>> > Uwe Schindler
>> > Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > https://www.thetaphi.de
>> > eMail: [email protected]
>> >
>> > > -----Original Message-----
>> > > From: Uwe Schindler <[email protected]>
>> > > Sent: Wednesday, June 2, 2021 12:40 PM
>> > > To: [email protected]
>> > > Subject: RE: Solr/Lucene joint development workflow?
>> > >
>> > > I'd suggest to remove the whole Lucene subfolder from your local
>> Maven repo.
>> > > E.g., rm -rf ~/.m2/repository/org/apache/lucene
>> > >
>> > > Then compile and deploy Lucene again.
>> > >
>> > > Uwe
>> > >
>> > > -----
>> > > Uwe Schindler
>> > > Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > > https://www.thetaphi.de
>> > > eMail: [email protected]
>> > >
>> > > > -----Original Message-----
>> > > > From: Uwe Schindler <[email protected]>
>> > > > Sent: Wednesday, June 2, 2021 12:38 PM
>> > > > To: '[email protected]' <[email protected]>
>> > > > Subject: RE: Solr/Lucene joint development workflow?
>> > > >
>> > > > Hi,
>> > > >
>> > > > this error helps. It looks like the lucene-core.jar file does not
>> have the META-
>> > > > INF/services. I wonder how Solr works!
>> > > >
>> > > > It would be good to figure out what's going on and why the JAR
>> files in your
>> > > > local repository do not have the required META-INF/services
>> subfolder.
>> > > >
>> > > > Uwe
>> > > >
>> > > > -----
>> > > > Uwe Schindler
>> > > > Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > > > https://www.thetaphi.de
>> > > > eMail: [email protected]
>> > > >
>> > > > > -----Original Message-----
>> > > > > From: Michael Gibney <[email protected]>
>> > > > > Sent: Wednesday, June 2, 2021 6:05 AM
>> > > > > To: [email protected]
>> > > > > Subject: Re: Solr/Lucene joint development workflow?
>> > > > >
>> > > > > Uwe,
>> > > > > Thanks for this advice! I followed this as closely as I could
>> > > > > (including the gradle `--refresh-dependencies` flag), and was
>> again
>> > > > > able to compile (and compileTest) against mavenLocal. But on
>> actually
>> > > > > running tests, afaict all tests fail with the same stacktrace
>> (copied
>> > > > > below). iirc this was the same error I was getting previously
>> with the
>> > > > > SNAPSHOT build (before the introduction of the prerelease
>> dependency).
>> > > > > I'd be grateful for any advice on addressing this remaining error
>> (and
>> > > > > having encountered the error, I continue to be curious whether
>> anyone
>> > > > > has this workflow or something similar working in practice).
>> > > > >
>> > > > > David, your suggestion to add "mavenLocalI()" to Gradle's repo
>> list
>> > > > > seems reasonable -- perhaps a more viable option now that the main
>> > > > > branch is pegged to specific builds (as opposed to a SNAPSHOT)?
>> I'm
>> > > > > happy to help contribute to documentation of this process (once I
>> have
>> > > > > it working!).
>> > > > >
>> > > > > One other challenge I encountered was determining which commit the
>> > > > > current prerelease build is based on. The desired workflow would
>> be to
>> > > > > update the Lucene feature branch by merging upstream changes as
>> far as
>> > > > > the commit of the prerelease build (for use as a dependency of the
>> > > > > Solr-side feature branch).
>> > > > >
>> > > > > The exception I'm getting on running tests is:
>> > > > >
>> > > > > java.lang.ExceptionInInitializerError
>> > > > > at
>> org.apache.lucene.codecs.Codec.getDefault(Codec.java:141)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetup
>> > > > > AndRestoreClassEnv.java:133)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter
>> > > > > Rule.java:42)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>> > > > > ntAdapter.java:36)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClass
>> > > > > Name.java:38)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod
>> > > > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod
>> > > > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>> > > > > ntAdapter.java:36)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>> > > > > ntAdapter.java:36)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio
>> > > > > nsRequired.java:53)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter
>> > > > > Rule.java:43)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja
>> > > > > va:44)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn
>> > > > > oreAfterMaxFailures.java:60)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest
>> > > > > Suites.java:47)
>> > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>> > > > > ntAdapter.java:36)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(
>> > > > > ThreadLeakControl.java:370)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin
>> > > > > gTask$0(ThreadLeakControl.java:826)
>> > > > > at java.base/java.lang.Thread.run(Thread.java:834)
>> > > > >
>> > > > > Caused by:
>> > > > > java.lang.IllegalArgumentException: An SPI class of type
>> > > > > org.apache.lucene.codecs.Codec with name 'Lucene90' does not
>> exist.
>> > > > > You need to add the corresponding JAR file supporting this SPI to
>> your
>> > > > > classpath. The current classpath supports the following names: []
>> > > > > at
>> > > > >
>> org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113)
>> > > > > at
>> org.apache.lucene.codecs.Codec$Holder.<clinit>(Codec.java:58)
>> > > > > ... 19 more
>> > > > >
>> > > > >
>> > > > > On Fri, May 28, 2021 at 4:03 PM David Smiley <[email protected]>
>> > > wrote:
>> > > > > >
>> > > > > > To whomever does this first, it would be great if the steps
>> could be
>> > > written
>> > > > > down carefully in more detail so that it can be shared for its
>> eventual
>> > > > inclusion
>> > > > > in /dev-docs
>> > > > > >
>> > > > > > Maybe "mavenLocalI()" should simply be added to Gradle's repo
>> list by
>> > > > > default? This will speed up some first-time builds in some
>> environments
>> > > who
>> > > > > already have lots of local Maven dependencies.
>> > > > > >
>> > > > > > ~ David Smiley
>> > > > > > Apache Lucene/Solr Search Developer
>> > > > > > http://www.linkedin.com/in/davidwsmiley
>> > > > > >
>> > > > > >
>> > > > > > On Thu, May 27, 2021 at 12:09 PM Uwe Schindler <[email protected]
>> >
>> > > > > wrote:
>> > > > > >>
>> > > > > >> Hi Michael,
>> > > > > >>
>> > > > > >> since yesterday, the Lucene dependency is no longer a snapshot
>> one, so
>> > > > > parts of your mail are no longer valid. But the worksflow is very
>> similar.
>> > > > > >>
>> > > > > >> If the Solr team wants to use a newer preview build of Lucene,
>> there is a
>> > > > > workflow using ASF Jenkins that can build a new "prerelease"
>> release of the
>> > > > > Lucene main branch and deploy it on some ASF data dump server as
>> maven
>> > > > > repository.
>> > > > > >>
>> > > > > >> The Gradle build of Solr is referring to the recent repository
>> and picks a
>> > > > > specific version (the build number of ASF jenkins).
>> > > > > >>
>> > > > > >> For joint development, I'd suggest to work like this:
>> > > > > >> - Change the dependency in the gradle versions.props file to
>> SNAPSHOT
>> > > > > (only local, don't submit this as PR)
>> > > > > >> - Add mavenLocal() as suggested in your mail (only local,
>> don't submit
>> > > this
>> > > > as
>> > > > > PR)
>> > > > > >> - develop changes in lucene and install them in local repo
>> > > > > >> - use them from solr through the snapshot dependency. BUT:
>> make sure
>> > > > the
>> > > > > snapshot is updated, this can be enforced by passing a gradlew
>> command
>> > > line
>> > > > > parameter to redownload all dependencies.
>> > > > > >> - once you are done with joint development create pull
>> requests in both
>> > > > > repositories
>> > > > > >> - Lucene's change should be merged first. Once this is done
>> and all tests
>> > > > > pass, ask a committer to trigger the Jenkins job to build a new
>> prerelease
>> > > > > >> - Update the Solr pull request and enter the new repository
>> coordinates
>> > > > > created by Jenkins (needs to be done in 2 files, see below).
>> > > > > >>
>> > > > > >> For more details how to update Lucene dependencies in Solr,
>> run:
>> > > > > >> $ gradlew :helpDeps
>> > > > > >>
>> > > > > >> Uwe
>> > > > > >>
>> > > > > >> -----
>> > > > > >> Uwe Schindler
>> > > > > >> Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > > > > >> https://www.thetaphi.de
>> > > > > >> eMail: [email protected]
>> > > > > >>
>> > > > > >> > -----Original Message-----
>> > > > > >> > From: Michael Gibney <[email protected]>
>> > > > > >> > Sent: Wednesday, May 26, 2021 8:02 PM
>> > > > > >> > To: [email protected]
>> > > > > >> > Subject: Solr/Lucene joint development workflow?
>> > > > > >> >
>> > > > > >> > I'm working on some features that involve changes to both
>> Lucene and
>> > > > > >> > Solr. Post-TLP-split, I'm wondering whether anyone has
>> recommended
>> > > > > >> > techniques to handle this kind of situation.
>> > > > > >> >
>> > > > > >> > Ideally one would work on Lucene changes, get them merged,
>> and then
>> > > > > >> > proceed with Solr development; but realistically even if
>> this were as
>> > > > > >> > straightforward in practice as it sounds in principal, there
>> are cases
>> > > > > >> > where one would still want to develop in parallel.
>> > > > > >> >
>> > > > > >> > I haven't been able to find any documented recommendation on
>> this
>> > > > > >> > subject. It's possible to have a locally built Lucene
>> snapshot (via
>> > > > > >> > `gradlew mavenToLocalRepo`); but I was only able to actually
>> _build_
>> > > > > >> > Solr against the local Lucene artifact by adding
>> `mavenLocal()` to the
>> > > > > >> > `allprojects/repositories` block in `gradle/defaults.gradle`
>> -- and I
>> > > > > >> > have yet to figure a way get the local Lucene artifact on
>> the test
>> > > > > >> > classpath (so I'm as yet unable to run Solr tests that
>> depend on
>> > > > > >> > unmerged upstream changes to Lucene).
>> > > > > >> >
>> > > > > >> > It's also possible that the partially-functional approach
>> described
>> > > > > >> > above will have to change now that Solr main depends on a
>> specific
>> > > > > >> > Lucene snapshot version.
>> > > > > >> >
>> > > > > >> > Is anybody doing something like this? Or perhaps I'm asking
>> the wrong
>> > > > > >> > question? I can think of solutions that involve setting up
>> my own
>> > > > > >> > maven repository, to which I publish my own pinned versions
>> of
>> > > Lucene,
>> > > > > >> > and refer to such pinned versions/repo as part of a given
>> Solr
>> > > > > >> > "patch". But that feels both half-baked _and_ bloated, so I
>> don't want
>> > > > > >> > to go down that road unless I feel convinced there's no
>> better
>> > > > > >> > alternative.
>> > > > > >> >
>> > > > > >> > Michael
>> > > > > >> >
>> > > > > >> >
>> ---------------------------------------------------------------------
>> > > > > >> > To unsubscribe, e-mail: [email protected]
>> > > > > >> > For additional commands, e-mail: [email protected]
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> ---------------------------------------------------------------------
>> > > > > >> To unsubscribe, e-mail: [email protected]
>> > > > > >> For additional commands, e-mail: [email protected]
>> > > > > >>
>> > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > To unsubscribe, e-mail: [email protected]
>> > > > > For additional commands, e-mail: [email protected]
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: [email protected]
>> > > For additional commands, e-mail: [email protected]
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [email protected]
>> > For additional commands, e-mail: [email protected]
>> >
>>
>
