[
https://issues.apache.org/jira/browse/LUCENE-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss updated LUCENE-9077:
--------------------------------
Description:
This task focuses on providing gradle-based build equivalent for Lucene and
Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*.
Try running the following to see an overview of helper guides concerning
typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work
on any of these, please add your name to the list. Once you have a patch/ pull
request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of
workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during
the entire build (this allows for an empty set of filtered tests at single
project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir
plugin, lockfile) and how to retrieve structured information about current
dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be
done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to
minimize odd build side-effects
* (/) Repro-line for failed tests/ runs.
* (/) add a top-level README note about building with gradle (and the required
JVM).
* (/) add an equivalent of 'validate-source-patterns'
(check-source-patterns.groovy) to precommit.
* (/) add an equivalent of 'rat-sources' to precommit.
* (/) add an equivalent of 'check-example-lucene-match-version' (solr only) to
precommit.
* (/) javadoc compilation
Hard-to-implement stuff already investigated:
* (/) (done) -*Printing console output of failed tests.* There doesn't seem
to be any way to do this in a reasonably efficient way. There are onOutput
listeners but they're slow to operate and solr tests emit *tons* of output so
it's an overkill.-
* (!) (LUCENE-9120) *Tests working with security-debug logs or other JVM-early
log output*. Gradle's test runner works by redirecting Java's stdout/ syserr so
this just won't work. Perhaps we can spin the ant-based test runner for such
corner-cases.
Of lesser importance:
* Add an equivalent of 'documentation-lint" to precommit.
* (/) add rendering of javadocs (gradlew javadoc)
* Attach javadocs to maven publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
it'll be difficult to run it sensibly because gradle doesn't offer cwd
separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution
there are minor differences in library versions and some JARs are excluded/
moved around. I didn't try to force these as everything seems to work (tests,
etc.) – perhaps these differences should be fixed in the ant build instead.
* [EOE] identify and port various "regenerate" tasks from ant builds (javacc,
precompiled automata, etc.)
* Fill in POM details in gradle/defaults-maven.gradle so that they reflect the
previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it
imports the project out of the box, without the need for any special tuning).
* Clean up dependencies, especially for Solr*: any \{ transitive = false }
should just explicitly exclude whatever they don't need (and their dependencies
currently declared explicitly should be folded). Figure out which scope to
import a dependency to.
* Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently
XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it
from a binary distribution?
*{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and
Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from
master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments
replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a
one-man effort. The more people understand and can contribute to the build, the
more healthy it will be.
was:
This task focuses on providing gradle-based build equivalent for Lucene and
Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*.
Try running the following to see an overview of helper guides concerning
typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work
on any of these, please add your name to the list. Once you have a patch/ pull
request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of
workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during
the entire build (this allows for an empty set of filtered tests at single
project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir
plugin, lockfile) and how to retrieve structured information about current
dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be
done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to
minimize odd build side-effects
* (/) Repro-line for failed tests/ runs.
* (/) add a top-level README note about building with gradle (and the required
JVM).
* (/) add an equivalent of 'validate-source-patterns'
(check-source-patterns.groovy) to precommit.
* (/) add an equivalent of 'rat-sources' to precommit.
* (/) add an equivalent of 'check-example-lucene-match-version' (solr only) to
precommit.
* (/) javadoc compilation
Hard-to-implement stuff already investigated:
* (/) (done) -*Printing console output of failed tests.* There doesn't seem
to be any way to do this in a reasonably efficient way. There are onOutput
listeners but they're slow to operate and solr tests emit *tons* of output so
it's an overkill.-
* (!) (LUCENE-9120) *Tests working with security-debug logs or other JVM-early
log output*. Gradle's test runner works by redirecting Java's stdout/ syserr so
this just won't work. Perhaps we can spin the ant-based test runner for such
corner-cases.
Of lesser importance:
* add an equivalent of 'documentation-lint" to precommit.
* add rendering of javadocs (gradlew javadoc) and attaching them to maven
publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
it'll be difficult to run it sensibly because gradle doesn't offer cwd
separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution
there are minor differences in library versions and some JARs are excluded/
moved around. I didn't try to force these as everything seems to work (tests,
etc.) – perhaps these differences should be fixed in the ant build instead.
* [EOE] identify and port various "regenerate" tasks from ant builds (javacc,
precompiled automata, etc.)
* fill in POM details in gradle/defaults-maven.gradle so that they reflect the
previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it
imports the project out of the box, without the need for any special tuning).
* *Clean up dependencies, especially for Solr*: any \{ transitive = false }
should just explicitly exclude whatever they don't need (and their dependencies
currently declared explicitly should be folded). Figure out which scope to
import a dependency to.
* Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently
XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it
from a binary distribution?
*{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and
Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from
master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments
replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a
one-man effort. The more people understand and can contribute to the build, the
more healthy it will be.
> Gradle build
> ------------
>
> Key: LUCENE-9077
> URL: https://issues.apache.org/jira/browse/LUCENE-9077
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Major
> Fix For: master (9.0)
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This task focuses on providing gradle-based build equivalent for Lucene and
> Solr (on master branch). See notes below on why this respin is needed.
> The code lives on *gradle-master* branch. It is kept with sync with *master*.
> Try running the following to see an overview of helper guides concerning
> typical workflow, testing and ant-migration helpers:
> gradlew :help
> A list of items that needs to be added or requires work. If you'd like to
> work on any of these, please add your name to the list. Once you have a
> patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.
> * (/) Apply forbiddenAPIs
> * (/) Generate hardware-aware gradle defaults for parallelism (count of
> workers and test JVMs).
> * (/) Fail the build if --tests filter is applied and no tests execute
> during the entire build (this allows for an empty set of filtered tests at
> single project level).
> * (/) Port other settings and randomizations from common-build.xml
> * (/) Configure security policy/ sandboxing for tests.
> * (/) test's console output on -Ptests.verbose=true
> * (/) add a :helpDeps explanation to how the dependency system works
> (palantir plugin, lockfile) and how to retrieve structured information about
> current dependencies of a given module (in a tree-like output).
> * (/) jar checksums, jar checksum computation and validation. This should be
> done without intermediate folders (directly on dependency sets).
> * (/) verify min. JVM version and exact gradle version on build startup to
> minimize odd build side-effects
> * (/) Repro-line for failed tests/ runs.
> * (/) add a top-level README note about building with gradle (and the
> required JVM).
> * (/) add an equivalent of 'validate-source-patterns'
> (check-source-patterns.groovy) to precommit.
> * (/) add an equivalent of 'rat-sources' to precommit.
> * (/) add an equivalent of 'check-example-lucene-match-version' (solr only)
> to precommit.
> * (/) javadoc compilation
> Hard-to-implement stuff already investigated:
> * (/) (done) -*Printing console output of failed tests.* There doesn't seem
> to be any way to do this in a reasonably efficient way. There are onOutput
> listeners but they're slow to operate and solr tests emit *tons* of output so
> it's an overkill.-
> * (!) (LUCENE-9120) *Tests working with security-debug logs or other
> JVM-early log output*. Gradle's test runner works by redirecting Java's
> stdout/ syserr so this just won't work. Perhaps we can spin the ant-based
> test runner for such corner-cases.
> Of lesser importance:
> * Add an equivalent of 'documentation-lint" to precommit.
> * (/) add rendering of javadocs (gradlew javadoc)
> * Attach javadocs to maven publications.
> * Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
> it'll be difficult to run it sensibly because gradle doesn't offer cwd
> separation for the forked test runners.
> * if you diff solr packaged distribution against ant-created distribution
> there are minor differences in library versions and some JARs are excluded/
> moved around. I didn't try to force these as everything seems to work (tests,
> etc.) – perhaps these differences should be fixed in the ant build instead.
> * [EOE] identify and port various "regenerate" tasks from ant builds
> (javacc, precompiled automata, etc.)
> * Fill in POM details in gradle/defaults-maven.gradle so that they reflect
> the previous content better (dependencies aside).
> * Add any IDE integration layers that should be added (I use IntelliJ and it
> imports the project out of the box, without the need for any special tuning).
> * Clean up dependencies, especially for Solr*: any \{ transitive = false }
> should just explicitly exclude whatever they don't need (and their
> dependencies currently declared explicitly should be folded). Figure out
> which scope to import a dependency to.
> * Add Solr packaging for docs/* (see TODO in packaging/build.gradle;
> currently XSLT...)
> * I didn't bother adding Solr dist/test-framework to packaging (who'd use it
> from a binary distribution?
>
> *{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and
> Cao Mạnh Đạt but also applies lessons learned from those two efforts:
> * *Do not try to do too many things at once*. If we deviate too far from
> master, the branch will be hard to merge.
> * *Do everything in baby-steps* and add small, independent build fragments
> replacing the old ant infrastructure.
> * *Try to engage people to run, test and contribute early*. It can't be a
> one-man effort. The more people understand and can contribute to the build,
> the more healthy it will be.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
