subject:"Gradle build"

> Huh, I thought by "cd lucene/core" would be equivalent to "-p lucene/core" 
> from the root checkout?

I don't think it is. It's a wrapper script - I'm not sure it
recognizes where it's run from.

> real0m4.802s
> user0m2.988s
> sys 0m0.189s
>
> Not much better?

Ok. In that case, it's something else. I have a strong suspicion that
gradle's test task scans through all the classes or does something
weird. You could try running with a build scan so that there are more
details on which phase actually takes those 5 seconds.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

[Adding Jerome back on CC -- not sure he's sub'd]

More below:

On Wed, Oct 20, 2021 at 10:37 AM Dawid Weiss  wrote:

> > I do!  I am now using pure Gradle/Lucene clean checkout defaults, from
> quite a while back (last time you scolded me ;) )
>
> Oh, I never scold. I politely pass over the torch of wisdom (or
> bullwaste, depending on the day). :)
>

Perfect!  I love it :)


> > But it's still slower than ant on 8.x was/is, plus it makes me pay a
> warmup penalty the first time (at defaults anyways).
>
> Part of the startup penalty is the evaluation of various scripts and
> settings that never get used. There are gradle-sque ways of avoiding
> that (lazy evaluation) but they do increase code complexity (in my
> opinion). The first-time compilation should be improved by moving
> scripts into buildSrc (or plugins). Again: this removes the clarity of
> individual aspect-based scripts (again: my opinion).
>

OK


> > so then I tested on main, with JDK15, also in cwd lucene/core: "time
> ../../gradlew test --tests
> "org.apache.lucene.index.TestIndexWriter.testGetCommitData"
> -Ptests.seed=D708CEE0862DB94C > ignored":
>
> You're running the test task in *each and every submodule* that
> declares it, then filter for a specific test case. Try this:
>

Huh, I thought by "cd lucene/core" would be equivalent to "-p lucene/core"
from the root checkout?

time ../../gradlew -p lucene/core test --tests
> "org.apache.lucene.index.TestIndexWriter.testGetCommitData"
> -Ptests.seed=D708CEE0862DB94C > ignored
>

I tried that but the "lucene/core" seems to incorrectly "stack up":

beast3:core[main]$ time ../../gradlew test -p lucene/core --tests
"org.apache.lucene.index.TestIndexWriter.testGetCommitData"
-Ptests.seed=D708CEE0862DB94C

Starting a Gradle Daemon (subsequent builds will be faster)


FAILURE: Build failed with an exception.


* What went wrong:
The specified project directory '/l/trunk/lucene/core/lucene/core' does not
exist.

I think you either "cd lucene/core" or you stay in ROOT and pass -p
lucene:core:test (or -p lucene/core?)?


> or full scoped task :lucene:core:test. Should be slightly better.
>

OK I tried that, in ROOT of the checkout "time ./gradlew lucene:core:test
--tests "org.apache.lucene.index.TestIndexWriter.testGetCommitData"
-Ptests.seed=D708CEE0862DB94C > ignore"

(discard first warmup, then:)

real0m4.514s

user0m3.090s

sys 0m0.186s


real0m4.587s

user0m3.029s

sys 0m0.186s


real0m4.802s

user0m2.988s

sys 0m0.189s

Not much better?

> Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper
> 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :)
>
I'm jealous.
>

LOL well there are even faster CPUs now I think, so I'm jealous of them!
Turtles all the way down ...


> > Also I want to thank you for migrating us to Gradle in the first place
>
> No need to thank anybody. It's fun.
>

Awesome!


> > But I really don't like waiting :)  And yes maybe I just should learn
> how to use fancy IDE debuggers instead of SOP + rerun many times ;)
>
> IntelliJ works very well for me with Lucene (especially if gradle is
> not used for compilation/ test launching). You may also look at this -
> never tried it but it looks like something your all-green terminals
> may look forward to:
>
>
> https://docs.gradle.org/current/userguide/command_line_interface.html#sec:continuous_build


Whoa, that does look interesting!  Thanks for the pointer.

Mike

Re: Gradle build optimization

2021-10-20 Thread Jerome Prinet

I'm glad to see that this initiative creates a lot of excitement!

I however need to narrow down a bit the scope of expectations as my plan is
not to go too deep into the project's logic (won't cover tests taking too
long to run for instance) but rather applying some techniques to avoid task
execution when not needed.
This will happen without involving extra complexity hopefully.

We'll probably discuss that with the PR.

So far, here is the Jira ticket, feel free to drop your comments in there
as well:
https://issues.apache.org/jira/browse/LUCENE-10195

Best,
Jerome

On Wed, Oct 20, 2021 at 4:42 PM Dawid Weiss  wrote:

> > The fully scoped one doesn't enjoy tab completion in the shell though.
> > So I always do path-based "-p lucene/core" because it is less typing
> > and less mistakes.
>
> Funny, it's exactly why I prefer -p as well.
>
> D.
>

Re: Gradle build optimization

> The fully scoped one doesn't enjoy tab completion in the shell though.
> So I always do path-based "-p lucene/core" because it is less typing
> and less mistakes.

Funny, it's exactly why I prefer -p as well.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

2021-10-20 Thread Robert Muir

On Wed, Oct 20, 2021 at 10:37 AM Dawid Weiss  wrote:
> You're running the test task in *each and every submodule* that
> declares it, then filter for a specific test case. Try this:
>
> time ../../gradlew -p lucene/core test --tests
> "org.apache.lucene.index.TestIndexWriter.testGetCommitData"
> -Ptests.seed=D708CEE0862DB94C > ignored
>
> or full scoped task :lucene:core:test. Should be slightly better.
>

The fully scoped one doesn't enjoy tab completion in the shell though.
So I always do path-based "-p lucene/core" because it is less typing
and less mistakes.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

> I do!  I am now using pure Gradle/Lucene clean checkout defaults, from quite 
> a while back (last time you scolded me ;) )

Oh, I never scold. I politely pass over the torch of wisdom (or
bullwaste, depending on the day). :)

> But it's still slower than ant on 8.x was/is, plus it makes me pay a warmup 
> penalty the first time (at defaults anyways).

Part of the startup penalty is the evaluation of various scripts and
settings that never get used. There are gradle-sque ways of avoiding
that (lazy evaluation) but they do increase code complexity (in my
opinion). The first-time compilation should be improved by moving
scripts into buildSrc (or plugins). Again: this removes the clarity of
individual aspect-based scripts (again: my opinion).

> so then I tested on main, with JDK15, also in cwd lucene/core: "time 
> ../../gradlew test --tests 
> "org.apache.lucene.index.TestIndexWriter.testGetCommitData" 
> -Ptests.seed=D708CEE0862DB94C > ignored":

You're running the test task in *each and every submodule* that
declares it, then filter for a specific test case. Try this:

time ../../gradlew -p lucene/core test --tests
"org.apache.lucene.index.TestIndexWriter.testGetCommitData"
-Ptests.seed=D708CEE0862DB94C > ignored

or full scoped task :lucene:core:test. Should be slightly better.

> Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper 
> 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :)

I'm jealous.

> Also I want to thank you for migrating us to Gradle in the first place

No need to thank anybody. It's fun.

> But I really don't like waiting :)  And yes maybe I just should learn how to 
> use fancy IDE debuggers instead of SOP + rerun many times ;)

IntelliJ works very well for me with Lucene (especially if gradle is
not used for compilation/ test launching). You may also look at this -
never tried it but it looks like something your all-green terminals
may look forward to:

https://docs.gradle.org/current/userguide/command_line_interface.html#sec:continuous_build

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

It is very possible I am holding gradlew wrong, like this:
https://i2.wp.com/www.adatosystems.com/wp-content/uploads/2019/03/wrong_tool.jpg

Mike McCandless

http://blog.mikemccandless.com


On Wed, Oct 20, 2021 at 10:21 AM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Jerome, if you are not sub'd to the Lucene dev list, then you maybe missed
> some of the replies here.  I'm CC'ing you back on here.
>
> More responses below:
>
> On Wed, Oct 20, 2021 at 8:45 AM Dawid Weiss  wrote:
>
>>
>>
>>> I was just annoyed yesterday at how long running a single test case
>>> takes.
>>>
>>
>> Use the background daemon, Mike.
>>
>
> I do!  I am now using pure Gradle/Lucene clean checkout defaults, from
> quite a while back (last time you scolded me ;) )
>
> And the defaults enable the daemon I think?  Yeah, just confirmed.
>
> But it's still slower than ant on 8.x was/is, plus it makes me pay a
> warmup penalty the first time (at defaults anyways).  I just tested 8.x in
> cwd lucene/core and JDK8: "time ant test -Dtestcase=TestIndexWriter
> -Dtestmethod=testGetCommitData > ignored" three times:
>
> real0m2.609s
>
> user0m10.583s
>
> sys 0m0.590s
>
>
> real0m2.585s
>
> user0m10.329s
>
> sys 0m0.827s
>
>
> real0m2.610s
>
> user0m10.458s
>
> sys 0m0.606s
>
>
>
> so then I tested on main, with JDK15, also in cwd lucene/core: "time
> ../../gradlew test --tests
> "org.apache.lucene.index.TestIndexWriter.testGetCommitData"
> -Ptests.seed=D708CEE0862DB94C > ignored":
>
> ** (first time discarded due to "daemon" warmup its fire up)
>
> real0m4.447s
>
> user0m2.956s
>
> sys 0m0.217s
>
>
> real0m4.399s
>
> user0m3.028s
>
> sys 0m0.233s
>
>
> real0m4.404s
>
> user0m3.021s
>
> sys 0m0.181s
>
>
> Now I know there are many differences besides ant/gradle!  Maybe JDK15's
> startup time got worse.  The random seed does not carry over.  Lucene's
> sources are different, etc.  And maybe I should care about a second or two
> :)  But I do!
>
> Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper
> 3990X), 256 GB RAM, fast SSD, 10g networking, etc. :)
>
> Also I want to thank you for migrating us to Gradle in the first place --
> I know that was a crazy amount of work!  I'm trying not to complain (too
> much).  But I really don't like waiting :)  And yes maybe I just should
> learn how to use fancy IDE debuggers instead of SOP + rerun many times ;)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>

Re: Gradle build optimization

Jerome, if you are not sub'd to the Lucene dev list, then you maybe missed
some of the replies here.  I'm CC'ing you back on here.

More responses below:

On Wed, Oct 20, 2021 at 8:45 AM Dawid Weiss  wrote:

>
>
>> I was just annoyed yesterday at how long running a single test case takes.
>>
>
> Use the background daemon, Mike.
>

I do!  I am now using pure Gradle/Lucene clean checkout defaults, from
quite a while back (last time you scolded me ;) )

And the defaults enable the daemon I think?  Yeah, just confirmed.

But it's still slower than ant on 8.x was/is, plus it makes me pay a warmup
penalty the first time (at defaults anyways).  I just tested 8.x in cwd
lucene/core and JDK8: "time ant test -Dtestcase=TestIndexWriter
-Dtestmethod=testGetCommitData > ignored" three times:

real0m2.609s

user0m10.583s

sys 0m0.590s

real0m2.585s

user0m10.329s

sys 0m0.827s

real0m2.610s

user0m10.458s

sys 0m0.606s

so then I tested on main, with JDK15, also in cwd lucene/core: "time
../../gradlew test --tests
"org.apache.lucene.index.TestIndexWriter.testGetCommitData"
-Ptests.seed=D708CEE0862DB94C > ignored":

** (first time discarded due to "daemon" warmup its fire up)

real0m4.447s

user0m2.956s

sys 0m0.217s

real0m4.399s

user0m3.028s

sys 0m0.233s

real0m4.404s

user0m3.021s

sys 0m0.181s

Now I know there are many differences besides ant/gradle!  Maybe JDK15's
startup time got worse.  The random seed does not carry over.  Lucene's
sources are different, etc.  And maybe I should care about a second or two
:)  But I do!

Also, I'm running on a 128 core crazy beast of a box (Ryzen Threadripper
3990X), 256 GB RAM, fast SSD, 10g networking, etc. :)

Also I want to thank you for migrating us to Gradle in the first place -- I
know that was a crazy amount of work!  I'm trying not to complain (too
much).  But I really don't like waiting :)  And yes maybe I just should
learn how to use fancy IDE debuggers instead of SOP + rerun many times ;)

Mike McCandless

http://blog.mikemccandless.com

Re: Gradle build optimization

Btw. this is a placeholder for some of the known bits that require solving.

https://issues.apache.org/jira/browse/LUCENE-9871

It'd be awesome to see some help from you guys at Gradle to solve the
non-trivial things:

- test runner and early/ internal JVM output (the issue I mentioned before),
- test runner test case ordering optimization (load balancing between
worker JVMs); currently the long-tail test case can slow down builds
significantly.
- support for user-local gradle.properties or computed properties there (we
currently generate this file because it'll be slightly different for every
developer, this requires a non-optimal first run),
- support for project-controlled up-to-date mechanism (versioned artifact
checksums) that is not bypassed on the first run. This is needed for
generated resources - I didn't figure out how to do it other than with ugly
hacks.

If any of these is of interest to you, I can provide more pointers, of
course. Let me know!

Dawid

On Wed, Oct 20, 2021 at 1:48 PM Jerome Prinet  wrote:

> Hi Lucene team!
> I am a software engineer at Gradle and I'd like to help you optimize
> Lucene's Gradle build.
> I have already been working on this topic and I'd like to open a PR.
> Should I start by creating a JIRA issue first?
> Please let me know how to proceed.
> Kind regards,
>
> --
>
> Jerome Prinet
> Senior Software Engineer
> Gradle
> W. gradle.com
>
>

Re: Gradle build optimization

2021-10-20 Thread Robert Muir

On Wed, Oct 20, 2021 at 8:52 AM Dawid Weiss  wrote:
>
> Can you tell us a bit more about which directions these mentioned
> optimizations apply to? If it's about build caches then it's already
> been discussed as a contribution to Solr:

Sure, something like that is definitely worthy of discussion first.

I don't understand why build caches are ever needed. If the outputs
are up-to-date with the inputs, then it should say UP-TO-DATE.

If I ask for "clean", then I really mean it, and expect the next run
to be slower.

These are simple rules for any build system (Makefile, etc). Gradle is
no exception.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

Also, run an explicit project task (or narrow down the project with
-p); if you're running a generic 'test' over all projects it may have
to run/ scan all the classes. If you provide a full command line you
used it'll be easier why it takes so long.

D.

On Wed, Oct 20, 2021 at 2:44 PM Dawid Weiss  wrote:
>
>
>>
>> I was just annoyed yesterday at how long running a single test case takes.
>
>
> Use the background daemon, Mike.
>
> D.
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

Can you tell us a bit more about which directions these mentioned
optimizations apply to? If it's about build caches then it's already
been discussed as a contribution to Solr:

https://issues.apache.org/jira/browse/SOLR-15603

Also, it'd be great to see some feedback for more fundamental issues
within gradle -- like this one:

https://issues.apache.org/jira/browse/LUCENE-9120
https://github.com/gradle/gradle/issues/11609

(this is marked as closed -- incorrectly; it's still a bug and always has been).

Dawid

On Wed, Oct 20, 2021 at 1:48 PM Jerome Prinet  wrote:
>
> Hi Lucene team!
> I am a software engineer at Gradle and I'd like to help you optimize Lucene's 
> Gradle build.
> I have already been working on this topic and I'd like to open a PR.
> Should I start by creating a JIRA issue first?
> Please let me know how to proceed.
> Kind regards,
>
> --
>
> Jerome Prinet
> Senior Software Engineer
> Gradle
> W. gradle.com
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build optimization

> I was just annoyed yesterday at how long running a single test case takes.
>

Use the background daemon, Mike.

D.

Re: Gradle build optimization

Whoa, hi Jerome, yes please :)

I was just annoyed yesterday at how long running a single test case takes.

Please create an issue and reference it (in all caps, i.e. LUCENE-X) in
the description.

Thanks,

Mike McCandless

http://blog.mikemccandless.com

On Wed, Oct 20, 2021 at 7:48 AM Jerome Prinet  wrote:

> Hi Lucene team!
> I am a software engineer at Gradle and I'd like to help you optimize
> Lucene's Gradle build.
> I have already been working on this topic and I'd like to open a PR.
> Should I start by creating a JIRA issue first?
> Please let me know how to proceed.
> Kind regards,
>
> --
>
> Jerome Prinet
> Senior Software Engineer
> Gradle
> W. gradle.com
>
>

Gradle build optimization

2021-10-20 Thread Jerome Prinet

Hi Lucene team!
I am a software engineer at Gradle and I'd like to help you optimize
Lucene's Gradle build.
I have already been working on this topic and I'd like to open a PR.
Should I start by creating a JIRA issue first?
Please let me know how to proceed.
Kind regards,

-- 

Jerome Prinet
Senior Software Engineer
Gradle
W. gradle.com

Re: package-tgz-src / package-src-tgz equivilent missing from gradle build on master?

2021-02-08 Thread Chris Hostetter

: At this major juncture in the project (switching build systems), it's worth
: re-considering what we need to produce on a release.  A -src.tar.gz seems
: anachronistic to me given both (a) GitHub source control providing a
: convenient download .tar.gz for any release "tag" --
: https://github.com/apache/lucene-solr/releases, and (b) maven central
: having -source.jar for convenience in IDE tooling.  So why bother?

Because we're required to by ASF policy...

http://www.apache.org/legal/release-policy.html#artifacts

"Every ASF release MUST contain one or more source packages, which MUST be 
sufficient for a user to build and test the release provided they have 
access to the appropriate platform and tools."

...

"As a convenience to users that might not have the appropriate tools to 
build a compiled version of the source, binary/bytecode packages MAY be 
distributed alongside official Apache releases. In all such cases, the 
binary/bytecode package MUST have the same version number as the source 
release and MUST only add binary/bytecode files that are the result of 
compiling that version of the source code release and its dependencies."


-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: package-tgz-src / package-src-tgz equivilent missing from gradle build on master?

2021-02-08 Thread David Smiley

At this major juncture in the project (switching build systems), it's worth
re-considering what we need to produce on a release.  A -src.tar.gz seems
anachronistic to me given both (a) GitHub source control providing a
convenient download .tar.gz for any release "tag" --
https://github.com/apache/lucene-solr/releases, and (b) maven central
having -source.jar for convenience in IDE tooling.  So why bother?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Feb 3, 2021 at 2:51 PM Houston Putman 
wrote:

> Thanks for getting the list of steps currently taken.
>
> These are easy to keep in the gradle task:
>
>- excluding jdk javadoc package-list files (for licensing reasons
>evidently)
>- setting chmod bits on scripts
>
>
> building Changes.html from CHANGES.txt
>>
> I'm not sure building Changes.html is a huge necessity for the source
> package. It's already done in the artifacts release, so maybe it's better
> to just leave it there?
> No other documentation sites are provided, so I find it a bit odd that
> this is. I might be missing context though.
>
> including lucene/CHANGES.txt in solr as LUCENE_CHANGES.txt
>>
>  While useful when the projects are connected, we probably won't keep this
> when the projects are split (though I could be wrong on this, it would be
> good to discuss anyways).
> If that happens before 9.0 is cut, then this is likely moot.
>
> So we could always start with a task that does a clean checkout, and the
> two easy steps mentioned above. Should be a good first pass, that we can
> change later if needed.
>
> - Houston
>
> On Wed, Feb 3, 2021 at 11:51 AM Chris Hostetter 
> wrote:
>
>>
>> : It wasn't as much of an oversight as lack of knowledge how to deal
>> : with it. I personally think the source distribution should be
>> : equivalent to a clean git checkout. If this works for you then things
>> : are relatively simple and I can do it.
>>
>> I'm not an authority here, so what works for me doesn't really matter :)
>>
>> While i would generally agree with you, I think there are some aspects of
>> how the current targets work that are worth bearing in mind since
>> deliberate decisions were made to get to this point - deliberate
>> decisions should probably made to "undo" these choices..
>>
>> * excluding jdk javadoc package-list files (for licensing reasons
>> evidently)
>> * building Changes.html from CHANGES.txt
>> * including lucnee/CHANGES.txt in solr as LUCENE_CHANGES.txt
>> * setting chmod bits on scripts
>>
>>
>> : On Wed, Feb 3, 2021 at 1:30 AM Chris Hostetter <
>> hossman_luc...@fucit.org> wrote:
>> : >
>> : >
>> : > thinking about how we (want to) build solr docker containers moving
>> : > forward (SOLR-15102) lead me to realize that on the mster branch,
>> there
>> : > doesn't seem to be any logic to build the "*-src.tgz" files.
>> : >
>> : > On branch_8x "ant package" in both the lucene & solr directories
>> handles
>> : > this by delegation to either package-tgz-src or package-src-tgz (for
>> some
>> : > reason they have diff names in each dir?) ... but there doesn't seem
>> to be
>> : > any equivilent of this logic in the gradle build.
>> : >
>> : > Was this an oversight, or is ther some other plan for where/how we
>> deal
>> : > with "src.tgz" release artifacts in 9.x ?
>> : >
>> : >
>> : > -Hoss
>> : > http://www.lucidworks.com/
>> : >
>> : > -
>> : > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> : > For additional commands, e-mail: dev-h...@lucene.apache.org
>> : >
>> :
>> : -
>> : To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> : For additional commands, e-mail: dev-h...@lucene.apache.org
>> :
>> :
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: package-tgz-src / package-src-tgz equivilent missing from gradle build on master?

2021-02-03 Thread Houston Putman

Thanks for getting the list of steps currently taken.

These are easy to keep in the gradle task:

   - excluding jdk javadoc package-list files (for licensing reasons
   evidently)
   - setting chmod bits on scripts


building Changes.html from CHANGES.txt
>
I'm not sure building Changes.html is a huge necessity for the source
package. It's already done in the artifacts release, so maybe it's better
to just leave it there?
No other documentation sites are provided, so I find it a bit odd that this
is. I might be missing context though.

including lucene/CHANGES.txt in solr as LUCENE_CHANGES.txt
>
 While useful when the projects are connected, we probably won't keep this
when the projects are split (though I could be wrong on this, it would be
good to discuss anyways).
If that happens before 9.0 is cut, then this is likely moot.

So we could always start with a task that does a clean checkout, and the
two easy steps mentioned above. Should be a good first pass, that we can
change later if needed.

- Houston

On Wed, Feb 3, 2021 at 11:51 AM Chris Hostetter 
wrote:

>
> : It wasn't as much of an oversight as lack of knowledge how to deal
> : with it. I personally think the source distribution should be
> : equivalent to a clean git checkout. If this works for you then things
> : are relatively simple and I can do it.
>
> I'm not an authority here, so what works for me doesn't really matter :)
>
> While i would generally agree with you, I think there are some aspects of
> how the current targets work that are worth bearing in mind since
> deliberate decisions were made to get to this point - deliberate
> decisions should probably made to "undo" these choices..
>
> * excluding jdk javadoc package-list files (for licensing reasons
> evidently)
> * building Changes.html from CHANGES.txt
> * including lucnee/CHANGES.txt in solr as LUCENE_CHANGES.txt
> * setting chmod bits on scripts
>
>
> : On Wed, Feb 3, 2021 at 1:30 AM Chris Hostetter 
> wrote:
> : >
> : >
> : > thinking about how we (want to) build solr docker containers moving
> : > forward (SOLR-15102) lead me to realize that on the mster branch, there
> : > doesn't seem to be any logic to build the "*-src.tgz" files.
> : >
> : > On branch_8x "ant package" in both the lucene & solr directories
> handles
> : > this by delegation to either package-tgz-src or package-src-tgz (for
> some
> : > reason they have diff names in each dir?) ... but there doesn't seem
> to be
> : > any equivilent of this logic in the gradle build.
> : >
> : > Was this an oversight, or is ther some other plan for where/how we deal
> : > with "src.tgz" release artifacts in 9.x ?
> : >
> : >
> : > -Hoss
> : > http://www.lucidworks.com/
> : >
> : > -
> : > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> : > For additional commands, e-mail: dev-h...@lucene.apache.org
> : >
> :
> : -
> : To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> : For additional commands, e-mail: dev-h...@lucene.apache.org
> :
> :
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: package-tgz-src / package-src-tgz equivilent missing from gradle build on master?

2021-02-03 Thread Chris Hostetter



: It wasn't as much of an oversight as lack of knowledge how to deal
: with it. I personally think the source distribution should be
: equivalent to a clean git checkout. If this works for you then things
: are relatively simple and I can do it.

I'm not an authority here, so what works for me doesn't really matter :)

While i would generally agree with you, I think there are some aspects of 
how the current targets work that are worth bearing in mind since 
deliberate decisions were made to get to this point - deliberate 
decisions should probably made to "undo" these choices..

* excluding jdk javadoc package-list files (for licensing reasons evidently)
* building Changes.html from CHANGES.txt
* including lucnee/CHANGES.txt in solr as LUCENE_CHANGES.txt
* setting chmod bits on scripts


: On Wed, Feb 3, 2021 at 1:30 AM Chris Hostetter  
wrote:
: >
: >
: > thinking about how we (want to) build solr docker containers moving
: > forward (SOLR-15102) lead me to realize that on the mster branch, there
: > doesn't seem to be any logic to build the "*-src.tgz" files.
: >
: > On branch_8x "ant package" in both the lucene & solr directories handles
: > this by delegation to either package-tgz-src or package-src-tgz (for some
: > reason they have diff names in each dir?) ... but there doesn't seem to be
: > any equivilent of this logic in the gradle build.
: >
: > Was this an oversight, or is ther some other plan for where/how we deal
: > with "src.tgz" release artifacts in 9.x ?
: >
: >
: > -Hoss
: > http://www.lucidworks.com/
: >
: > -
: > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
: > For additional commands, e-mail: dev-h...@lucene.apache.org
: >
: 
: -
: To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
: For additional commands, e-mail: dev-h...@lucene.apache.org
: 
: 

-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: package-tgz-src / package-src-tgz equivilent missing from gradle build on master?

2021-02-02 Thread Dawid Weiss

It wasn't as much of an oversight as lack of knowledge how to deal
with it. I personally think the source distribution should be
equivalent to a clean git checkout. If this works for you then things
are relatively simple and I can do it.

Dawid

On Wed, Feb 3, 2021 at 1:30 AM Chris Hostetter  wrote:
>
>
> thinking about how we (want to) build solr docker containers moving
> forward (SOLR-15102) lead me to realize that on the mster branch, there
> doesn't seem to be any logic to build the "*-src.tgz" files.
>
> On branch_8x "ant package" in both the lucene & solr directories handles
> this by delegation to either package-tgz-src or package-src-tgz (for some
> reason they have diff names in each dir?) ... but there doesn't seem to be
> any equivilent of this logic in the gradle build.
>
> Was this an oversight, or is ther some other plan for where/how we deal
> with "src.tgz" release artifacts in 9.x ?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

package-tgz-src / package-src-tgz equivilent missing from gradle build on master?

2021-02-02 Thread Chris Hostetter




thinking about how we (want to) build solr docker containers moving 
forward (SOLR-15102) lead me to realize that on the mster branch, there 
doesn't seem to be any logic to build the "*-src.tgz" files.


On branch_8x "ant package" in both the lucene & solr directories handles 
this by delegation to either package-tgz-src or package-src-tgz (for some 
reason they have diff names in each dir?) ... but there doesn't seem to be 
any equivilent of this logic in the gradle build.


Was this an oversight, or is ther some other plan for where/how we deal 
with "src.tgz" release artifacts in 9.x ?



-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build, where's the output?

2020-09-22 Thread Erick Erickson

SOLR-14888

I’m a bit underwater now, so probably won’t get to a PR for a while.

> On Sep 22, 2020, at 12:33 PM, Mike Drob  wrote:
> 
> It likely disappeared when I switched us to use the distribution plugin. Feel 
> free to open either a JIRA or PR for it, depending on your comfort. 
> 
> On Tue, Sep 22, 2020 at 11:11 AM Erick Erickson  
> wrote:
> We’re probably talking about different things. I have access to all the 
> folders, I think our gradle build has changed recently and we stopped 
> printing out a message for where the results of a specific task wound up.
> 
> 
> 
> Always possible it’s something else of course, I’m checking whether it’s a 
> change to our build scripts which have been changing a lot lately.
> 
> 
> 
> > On Sep 22, 2020, at 11:51 AM, Martin Gainty  wrote:
> 
> > 
> 
> > travis doc suggests user account does not have 'read access' to dependency 
> > folder
> 
> > 
> 
> > https://docs.travis-ci.com/user/languages/java/#projects-using-gradle
> 
> > 
> 
> > m-
> 
> > 
> 
> > From: Erick Erickson 
> 
> > Sent: Tuesday, September 22, 2020 10:48 AM
> 
> > To: dev@lucene.apache.org 
> 
> > Subject: Gradle build, where's the output?
> 
> >  
> 
> > The Gradle build used to print out where to find the artifacts for the 
> > “assemble” and “dev” tasks, but that disappeared sometime. Is this 
> > intentional? It really helped me the first time I tried to run Solr after 
> > building with Gradle…
> 
> > 
> 
> > Currently, “gradlew tasks” does print the location of the artifacts for the 
> > “dev” target, but not the “assemble” target, that might suffice.
> 
> > -
> 
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> 
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 
> 
> 
> 
> -
> 
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> 
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build, where's the output?

2020-09-22 Thread Mike Drob

It likely disappeared when I switched us to use the distribution plugin.
Feel free to open either a JIRA or PR for it, depending on your comfort.

On Tue, Sep 22, 2020 at 11:11 AM Erick Erickson 
wrote:

> We’re probably talking about different things. I have access to all the
> folders, I think our gradle build has changed recently and we stopped
> printing out a message for where the results of a specific task wound up.
>
>
>
> Always possible it’s something else of course, I’m checking whether it’s a
> change to our build scripts which have been changing a lot lately.
>
>
>
> > On Sep 22, 2020, at 11:51 AM, Martin Gainty  wrote:
>
> >
>
> > travis doc suggests user account does not have 'read access' to
> dependency folder
>
> >
>
> > https://docs.travis-ci.com/user/languages/java/#projects-using-gradle
>
> >
>
> > m-
>
> >
>
> > From: Erick Erickson 
>
> > Sent: Tuesday, September 22, 2020 10:48 AM
>
> > To: dev@lucene.apache.org 
>
> > Subject: Gradle build, where's the output?
>
> >
>
> > The Gradle build used to print out where to find the artifacts for the
> “assemble” and “dev” tasks, but that disappeared sometime. Is this
> intentional? It really helped me the first time I tried to run Solr after
> building with Gradle…
>
> >
>
> > Currently, “gradlew tasks” does print the location of the artifacts for
> the “dev” target, but not the “assemble” target, that might suffice.
>
> > -
>
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>
> > For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
>
>
> -
>
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
>

Re: Gradle build, where's the output?

2020-09-22 Thread Erick Erickson

We’re probably talking about different things. I have access to all the 
folders, I think our gradle build has changed recently and we stopped printing 
out a message for where the results of a specific task wound up.

Always possible it’s something else of course, I’m checking whether it’s a 
change to our build scripts which have been changing a lot lately.

> On Sep 22, 2020, at 11:51 AM, Martin Gainty  wrote:
> 
> travis doc suggests user account does not have 'read access' to dependency 
> folder
> 
> https://docs.travis-ci.com/user/languages/java/#projects-using-gradle
> 
> m-
> 
> From: Erick Erickson 
> Sent: Tuesday, September 22, 2020 10:48 AM
> To: dev@lucene.apache.org 
> Subject: Gradle build, where's the output?
>  
> The Gradle build used to print out where to find the artifacts for the 
> “assemble” and “dev” tasks, but that disappeared sometime. Is this 
> intentional? It really helped me the first time I tried to run Solr after 
> building with Gradle…
> 
> Currently, “gradlew tasks” does print the location of the artifacts for the 
> “dev” target, but not the “assemble” target, that might suffice.
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build, where's the output?

2020-09-22 Thread Martin Gainty

travis doc suggests user account does not have 'read access' to dependency 
folder

https://docs.travis-ci.com/user/languages/java/#projects-using-gradle

m-


From: Erick Erickson 
Sent: Tuesday, September 22, 2020 10:48 AM
To: dev@lucene.apache.org 
Subject: Gradle build, where's the output?

The Gradle build used to print out where to find the artifacts for the 
“assemble” and “dev” tasks, but that disappeared sometime. Is this intentional? 
It really helped me the first time I tried to run Solr after building with 
Gradle…

Currently, “gradlew tasks” does print the location of the artifacts for the 
“dev” target, but not the “assemble” target, that might suffice.
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Gradle build, where's the output?

2020-09-22 Thread Erick Erickson

The Gradle build used to print out where to find the artifacts for the 
“assemble” and “dev” tasks, but that disappeared sometime. Is this intentional? 
It really helped me the first time I tried to run Solr after building with 
Gradle…

Currently, “gradlew tasks” does print the location of the artifacts for the 
“dev” target, but not the “assemble” target, that might suffice.
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build question

2020-05-14 Thread Dawid Weiss

> WDYT about making 3g the default (I volunteer)?

I honestly don't know what consumes so much memory in the daemon -
whether it's the caches or something else. It doesn't seem the build
should need SO much memory... Feel free to bump but it'd be even
better to figure out why it needs so much and perhaps clean it up! :)

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Gradle build question

2020-05-13 Thread Erick Erickson

I’m occasionally having to bump the "org.gradle.jvmargs=-Xmx2g” setting in 
gradle.properties. I usually bump it to 4g, but have no evidence that 3g 
wouldn’t work.

WDYT about making 3g the default (I volunteer)? FWIW, I’m constantly cleaning 
everything out as I switch back and forth between ant and gradle, so I’m 
probably hitting it more often than people who can just set it once and 
forgeddaboudit because it gets regenerated often. It’s no big deal for me to 
set it when I need to but if I hit it I expect others might too.

Erick

P.S. In order to see some files I’m working on getting warnings out of, I’m 
adding “-Xmaxwarns", “1” to my local defaults-java.gradle file. I’ve 
already inadvertently checked that in once, if anyone sees this in future you 
know who to blame. Ping me and I’ll fix.
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Occasional compilation failures from gradle build

2020-05-05 Thread David Smiley

I see it often; I raised this in slack a few weeks ago.  Kevin & Mike Drob
responded.  It seems "gw clean" makes it go away.  Still, this is annoying
and points to a problem we should address.

~ David


On Tue, May 5, 2020 at 8:40 AM Erick Erickson 
wrote:

> I was doing quite a bit of building the assemble and dev tasks and never
> saw this. Of course I was bouncing between and and gradle a lot and
> habitually executed a "git clean -d -x -f”, don’t have a clue whether
> that’d be relevant, though I rather doubt it is.
>
> > On May 5, 2020, at 7:27 AM, Alan Woodward  wrote:
> >
> > Hi all,
> >
> > In about 50% of my gradle precommit runs, I get a Solr compilation error
> that looks like this:
> >
> >
> /Users/romseygeek/projects/lucene-solr/solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java:44:
> error: BlockPoolSlice is not public in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl; cannot be accessed
> from outside package
> > import
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice;
> >
> > From what I can tell, this class is designed to override classes from
> Hadoop to fix certain bugs, but it seems that gradle is sometimes putting
> the original Hadoop class files first in the class path, so the overridden
> class is not picked up.
> >
> > Is anyone else seeing this?
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Occasional compilation failures from gradle build

2020-05-05 Thread Erick Erickson

I was doing quite a bit of building the assemble and dev tasks and never saw 
this. Of course I was bouncing between and and gradle a lot and habitually 
executed a "git clean -d -x -f”, don’t have a clue whether that’d be relevant, 
though I rather doubt it is.

> On May 5, 2020, at 7:27 AM, Alan Woodward  wrote:
> 
> Hi all,
> 
> In about 50% of my gradle precommit runs, I get a Solr compilation error that 
> looks like this:
> 
> /Users/romseygeek/projects/lucene-solr/solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java:44:
>  error: BlockPoolSlice is not public in 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl; cannot be accessed 
> from outside package
> import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice;
> 
> From what I can tell, this class is designed to override classes from Hadoop 
> to fix certain bugs, but it seems that gradle is sometimes putting the 
> original Hadoop class files first in the class path, so the overridden class 
> is not picked up.  
> 
> Is anyone else seeing this?
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Occasional compilation failures from gradle build

2020-05-05 Thread Alan Woodward

Hi all,

In about 50% of my gradle precommit runs, I get a Solr compilation error that 
looks like this:

/Users/romseygeek/projects/lucene-solr/solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java:44:
 error: BlockPoolSlice is not public in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl; cannot be accessed from 
outside package
import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice;

From what I can tell, this class is designed to override classes from Hadoop to 
fix certain bugs, but it seems that gradle is sometimes putting the original 
Hadoop class files first in the class path, so the overridden class is not 
picked up.  

Is anyone else seeing this?
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [jira] [Commented] (LUCENE-9077) Gradle build

2020-04-12 Thread Erick Erickson

Outstanding! Thanks!

> On Apr 12, 2020, at 10:29 AM, Dawid Weiss (Jira)  wrote:
> 
> gradlew -p solr/packaging dev


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Changes to Lucene and Solr Javadoc folder structure for Gradle build

2020-03-14 Thread Tomoko Uchida

Hi all,

as a sub-task of Gradle build, LUCENE-9077 [1], Dawid Weiss and I are
planning to change Javadoc folder structure (hosted on lucene.apache.org
site) in accordance with gradle project structure on LUCENE-9278 [2].

Current javadoc folder structure is derived from Ant project name. e.g.:

- https://lucene.apache.org/core/8_4_1/analyzers-icu/index.html
- https://lucene.apache.org/solr/8_4_1/solr-solrj/index.html

For Gradle build, it would be desirable that the javadocs are also placed
according to gradle project structure (project path) instead of ant one, to
keep things simple to manage [3]. Hence, it will look like this:

- https://lucene.apache.org/core/9_0_0/analysis/icu/index.html
- https://lucene.apache.org/solr/9_0_0/solr/solrj/index.html

Please see LUCENE-9278 for more details and leave comments if you have any
concerns.
[1] https://issues.apache.org/jira/browse/LUCENE-9077
[2] https://issues.apache.org/jira/browse/LUCENE-9278
[3] The change was suggested at the conversation on a github pr:
https://github.com/apache/lucene-solr/pull/1304

Thanks,
Tomoko

Re: Gradle build is on master

2020-01-16 Thread Dawid Weiss

Hi Uwe,

> Can we correct it on ant's side somehow (by ignoring jetty-start-*)?

I've just committed a quick workaround so that ant-regenerate doesn't wipe
this particular file (on master) so that builds can pass.

I don't know any better solution to this. If we're looking to
switching to gradle then
I'd rather consider the gradle build the primary source of truth and
fix ant to what's
required for it to pass. Let me know what you think though.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build is on master

2020-01-16 Thread Dawid Weiss

> Thanks Dawid – great work!

Thank you all. I don't think it's particularly great, but definitely a
step forward from ant which has gotten very complex and where simple
things are difficult to achieve. Gradle is not without its own issues
(some known, some waiting to be discovered) but it is manageable I
think.

> there is a small problem with Jenkins (still running ANT build; I may add a 
> second one running the gradle build!):

Gradle on a CI will require some overrides -- I'd disable the daemon
and set the number of workers manually. Something like:

./gradlew --no-daemon --max-workers=5 ...

In the initial stage it'd be enough if you just ran precommit and
other checks, without tests:

./gradlew --no-daemon --max-workers=5 precommit check -x test

This would be helpful enough! Of course you can run tests as well but
they take approximately the same amount of time as with ant in my
experience.

> The Jenkins build does a license and JAR file check by first deleting all 
> checksum files and then regenerate all of them from the downloaded artifacts. 
> Afterwards it checks that the GIT checkout is clean. This is done to detect 
> obsolete or incorrect checksum files.

This only works for sha files, right? Because I removed quite a few
other dangling files when I implemented this for the gradle
counterpart (Gradle build does this kind of check as part of
precommit).

> With the gradle update you added the shaded jetty JAR file, but the old ANT 
> build seems to still use the old/different name. It means after checksum 
> regeneration the checkout is no longer clean:

Correct. The shaded jetty jar file is the only exception I made. It
just didn't fit the logical way those files are looked up (by their
real artifact name). In fact I ignore "start.jar" in gradle, otherwise
it'd be reported as a dangling file...

> /home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:494: Source 
> checkout is dirty (unversioned/missing files) after running tests!!! 
> Offending files:
>
> * solr/licenses/jetty-start-9.4.24.v20191120-shaded.jar.sha1

Can we correct it on ant's side somehow (by ignoring jetty-start-*)?
This file points at a real dependency and you can tell what this
dependency is while "start.jar.sha1" doesn't really mean anything (and
it isn't easy to figure out which license applies to it, etc.). There
are more oddball checksums in there that don't really match true
redistributed project dependencies [1] but this one is the most
annoying one I think.

D.

[1] 
https://github.com/apache/lucene-solr/blob/master/gradle/validation/jar-checks.gradle#L340-L368

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: Gradle build is on master

2020-01-15 Thread Uwe Schindler

Thanks Dawid – great work!

 

there is a small problem with Jenkins (still running ANT build; I may add a 
second one running the gradle build!): The Jenkins build does a license and JAR 
file check by first deleting all checksum files and then regenerate all of them 
from the downloaded artifacts. Afterwards it checks that the GIT checkout is 
clean. This is done to detect obsolete or incorrect checksum files.

 

With the gradle update you added the shaded jetty JAR file, but the old ANT 
build seems to still use the old/different name. It means after checksum 
regeneration the checkout is no longer clean:

 

/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:494: Source checkout 
is dirty (unversioned/missing files) after running tests!!! Offending files:

* solr/licenses/jetty-start-9.4.24.v20191120-shaded.jar.sha1

 

It does not seem to happen on 8.x branch so it look like caused by Gradle build.

 

See here:  
<https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/25336/console> 
https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/25336/console

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

https://www.thetaphi.de

eMail: u...@thetaphi.de

 

From: David Smiley  
Sent: Wednesday, January 15, 2020 11:24 PM
To: Solr/Lucene Dev 
Subject: Re: Gradle build is on master

 

This is a big milestone indeed!  Thanks Mark, Dawid, Erick, Dat, Mike, etc.




~ David Smiley

Apache Lucene/Solr Search Developer

http://www.linkedin.com/in/davidwsmiley

 

 

On Wed, Jan 15, 2020 at 2:00 PM Anshum Gupta mailto:ans...@anshumgupta.net> > wrote:

Thank you Dawid and everyone else !

 

On Wed, Jan 15, 2020 at 5:33 AM Dawid Weiss mailto:dawid.we...@gmail.com> > wrote:

Hello,

I've just merged the gradle-master branch so gradle build can be used
on master.

As previously mentioned, it isn't a complete replacement for ant but I
think we're getting darn close and common tasks are fairly well
covered.

Any issues or concerns: create an issue in jira or send an e-mail to
the mailing list, CCing me directly.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org 
<mailto:dev-unsubscr...@lucene.apache.org> 
For additional commands, e-mail: dev-h...@lucene.apache.org 
<mailto:dev-h...@lucene.apache.org> 




 

-- 

Anshum Gupta

Re: Gradle build is on master

2020-01-15 Thread David Smiley

This is a big milestone indeed!  Thanks Mark, Dawid, Erick, Dat, Mike, etc.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Jan 15, 2020 at 2:00 PM Anshum Gupta  wrote:

> Thank you Dawid and everyone else !
>
> On Wed, Jan 15, 2020 at 5:33 AM Dawid Weiss  wrote:
>
>> Hello,
>>
>> I've just merged the gradle-master branch so gradle build can be used
>> on master.
>>
>> As previously mentioned, it isn't a complete replacement for ant but I
>> think we're getting darn close and common tasks are fairly well
>> covered.
>>
>> Any issues or concerns: create an issue in jira or send an e-mail to
>> the mailing list, CCing me directly.
>>
>> Dawid
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>
> --
> Anshum Gupta
>

Re: Gradle build is on master

2020-01-15 Thread Anshum Gupta

Thank you Dawid and everyone else !

On Wed, Jan 15, 2020 at 5:33 AM Dawid Weiss  wrote:

> Hello,
>
> I've just merged the gradle-master branch so gradle build can be used
> on master.
>
> As previously mentioned, it isn't a complete replacement for ant but I
> think we're getting darn close and common tasks are fairly well
> covered.
>
> Any issues or concerns: create an issue in jira or send an e-mail to
> the mailing list, CCing me directly.
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
Anshum Gupta

Re: Gradle build is on master

2020-01-15 Thread Ishan Chattopadhyaya

Thanks Dawid. Big milestone reached.
Thanks Mark, Erick and everyone else who helped. I'm super excited.

On Wed, 15 Jan, 2020, 7:03 PM Dawid Weiss,  wrote:

> Hello,
>
> I've just merged the gradle-master branch so gradle build can be used
> on master.
>
> As previously mentioned, it isn't a complete replacement for ant but I
> think we're getting darn close and common tasks are fairly well
> covered.
>
> Any issues or concerns: create an issue in jira or send an e-mail to
> the mailing list, CCing me directly.
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Gradle build is on master

2020-01-15 Thread Dawid Weiss

Hello,

I've just merged the gradle-master branch so gradle build can be used
on master.

As previously mentioned, it isn't a complete replacement for ant but I
think we're getting darn close and common tasks are fairly well
covered.

Any issues or concerns: create an issue in jira or send an e-mail to
the mailing list, CCing me directly.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

> Are you thinking to have a gradle task that fails if there is mismatch 
> between ant and gradle dependencies versions?

When you run './gradlew licenses' it currently applies all the checks
against the same "licenses" folders as ant. So when you change something in ant
and not do a corresponding change to gradle dependencies, "gradlew
precommit" (or "licenses")
will fail the build.

Technically there are a few ignored entries inside the gradle validation
script (to skip things like "start.jar"); everything else is resolved
directly from
declared dependencies.

https://github.com/apache/lucene-solr/blob/gradle-master/gradle/validation/jar-checks.gradle#L340-L368

The gradle validation script is more strict than the ant build -- it
fails if there are leftover files that don't belong
anywhere, for example.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

2020-01-08 Thread Jan Høydahl

+1 The sooner the better

> This would ensure consistency in dependencies, for example
Are you thinking to have a gradle task that fails if there is mismatch between 
ant and gradle dependencies versions?

Jan

> 8. jan. 2020 kl. 15:29 skrev Dawid Weiss :
> 
> I think the gradle-master branch is already workable enough to land on master.
> 
> If you're not familiar with gradle then, once merged, run "gradlew help".
> 
> Some notes.
> 
> 1) I have been running tests on Windows and Linux, they're ok. The
> output is slightly different from ANT's but I think it's fine for
> working.
> 
> 2) The speed of compilation and running tests selectively is much
> better than ant's, especially on multicore machines.
> 
> 3) I use IntelliJ idea and the project imports into the IDE without
> any special handling. Code formatting and such may need to be adjusted
> though.
> 
> 4) Some things are incomplete (precommit does a subset of checks).
> Some are missing (regeneration tasks). Some are different (handling of
> dependencies, build output folder locations). It will take some time
> and learning to live with a new build system. I tried to provide short
> guides into selective areas (they're available as help tasks or plain
> text files under help/).
> 
> 5) If something does *not* work, let me know.
> 
> 6) It'd be nice if we had a build job somewhere on a faster machine
> that would run at least "gradlew precommit check -x test" so that
> rudimentary checks are applied, without running all the tests. This
> would ensure consistency in dependencies, for example.
> 
> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
> kept open and occasionally merged back and forth.
> 
> I have to shift more focus to my daily job but will help out and chip
> at the remaining bits, time permitting.
> 
> Dawid
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

> Maybe the README should be altered a little bit to encourage people to mess 
> around with the gradle build. Currently all the instructions in the root 
> README are ant-based.

Will do it, thanks for the suggestion.

D

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

Dawid, I think the plan is good, and thanks for working on it.

Maybe the README should be altered a little bit to encourage people to mess
around with the gradle build. Currently all the instructions in the root
README are ant-based. I am thinking just a simple one-liner that explains
it: gradle is WIP, patches welcome, whatever you want it to say.




On Wed, Jan 8, 2020 at 12:44 PM Dawid Weiss  wrote:

> > So are we going to be able to do things like EA testing?
>
> We should be able to, eventually. For now I explicitly limited the
> environment to a particular Java/ Gradle version so that
> we're all on the same page... Gradle is a much more complicated (and
> fragile) ecosystem than ant. Weird things (like the one you
> experienced, argh!) can happen out of nowhere.
>
> Perhaps I should also clarify: the ant build will still work (on
> master) until we can iron out all the problems. I just don't want to
> keep the branch separate anymore. One of the reasons for this is
> simple: I've had more feedback in the last 5 minutes than I had in the
> last three weeks. :) I am also afraid the whole effort will die if
> there is only one person effectively using it -- this is what happened
> with Mark's branch. I tried to bring it to a functional level and I
> think it's there -- I've been working with it for over a month now.
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Gradle build to land on master

> So are we going to be able to do things like EA testing?

We should be able to, eventually. For now I explicitly limited the
environment to a particular Java/ Gradle version so that
we're all on the same page... Gradle is a much more complicated (and
fragile) ecosystem than ant. Weird things (like the one you
experienced, argh!) can happen out of nowhere.

Perhaps I should also clarify: the ant build will still work (on
master) until we can iron out all the problems. I just don't want to
keep the branch separate anymore. One of the reasons for this is
simple: I've had more feedback in the last 5 minutes than I had in the
last three weeks. :) I am also afraid the whole effort will die if
there is only one person effectively using it -- this is what happened
with Mark's branch. I tried to bring it to a functional level and I
think it's there -- I've been working with it for over a month now.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

It is crazy classfiles are involved here at all. It is even crazier to be
reading classfiles of the JDK itself. I just asked to print the help...
crazy groovy

On Wed, Jan 8, 2020 at 12:27 PM Uwe Schindler  wrote:

> Hi,
>
> Sorry, this is a no-go for going to gradle. We currently want to build and
> test Lucene with all newer Java versions, including EA releases. I have no
> idea why the Gradle people are not looking into the issue of
> forward-compatibility.
>
> I had a long discussion already on the OpenJDK mailing lists about the
> classfile version issue. The problem is mainly ASM that's used for bytecode
> analysis where Gradle and Groovy depend on. If you pass ASM a classfile
> version that it does not unerstand it brings this error.
>
> I had a discussion with Remi Forax. My proposal would be:
> ClassReader should have 2 modes:
> - strict mode that enables all features (like you can add filtering on top
> of like method renames, and so pass the ClassReader (as visitor) to a
> ClassWriter. This strict mode ensures that a ClassReader then passed to
> ClassWriter produces the same class file. The requires that ASM fully
> understands the class file.
> - lenient mode: This marks the classreader/visitor as "risky". When
> passing this to ClassReader, it can open any class file, also those with
> newer class file versions. The classfile format is made in a way
> (attributes) that you can easily read over features you don't understand.
> In that case you can use a ClassReader like this to read method signatures,
> run forbiddenapi checks,... (what MOST code out there using ASM is doing!).
> The special flag on the visitor  interface is then there to protect people.
> If you pass a "lenient" class reader to a ClassWriter, the ClassWriter
> complains. So you cannot open a ClassReader in lenient way and pass  it to
> ClassWriter, because that would produce a classfile where maybe a
> significant part of stuff is missing/incorrect.
>
> In short: If there would be a non-picky ASM version / ASM mode like
> proposed before - that can just be used for "analysis" of class files,
> Gradle would be forward compatible. But Remi is too academic and also does
> not want this because he sees some "maintenance" trouble. But this is why I
> said strict mode and lenient mode. In lenient mode you cannot complain if
> your code behaves wrong when you open a class file that is newer than the
> version of ASM supports.
>
> In forbiddenapis I use hack to provide this leniency: When opening class
> files, I just patch the version number. Maybe gradle should do the same -
> IF AND ONLY IF IT DOES NOT USE IT FOR WRITING DERIVATIVE CLASS FILES:
>
> Maybe we could try to confive Gradle, Groovy and maybe Remi to take one of
> those routes. Otherwise we cannot use Gradle with CI builds.
>
> An alternative here is to go the Elasticsearch approach: Gradle and the
> Build runs with Java 11 and the Tests are running optionally with a later
> version - by pasisng a system property pointing to a TestRunner JDK.
>
> Uwe
>
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> > -Original Message-
> > From: Dawid Weiss 
> > Sent: Wednesday, January 8, 2020 6:06 PM
> > To: Lucene Dev 
> > Subject: Re: Gradle build to land on master
> >
> > > right, that file is the exact one i change. the problem is if i change
> it to 6.0.1,
> > then it wants me to buy something.
> >
> > You can't change to any other gradle version -- stick to the version
> > defined in the wrapper and java 11. It is a fragile environment -
> > switching gradle version will cause plugin incompatibilities, etc.
> > Sigh.
> >
> > We can't upgrade to gradle 6.x yet because palantir's plugin for
> > dependency management doesn't work for me then.
> >
> > Different JVM configurations may be rough, especially with newer JVMs.
> >
> > D.
> >
> >
> > > On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs 
> wrote:
> > >>
> > >> ./gradlew should try to use gradle 5.6.4 see
> > >> ./gradle/wrapper/gradle-wrapper.properties
> > >> With java 11 this all builds fine for me (can export JAVA_HOME to
> change)
> > >> Which version of java are you using? the linked issue hints at java
> 13?
> > >>
> > >> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
> > >> >
> > >> > Here is the issue: https://github.com/gradle/gradle/issues/8681
> > >> >
> > >> > I tried upgrading gradle to 6.0.1, but it seems now it wants me

Re: Gradle build to land on master

> Sorry, this is a no-go for going to gradle. We currently want to build and 
> test Lucene with all newer Java versions, including EA releases. I have no 
> idea why the Gradle people are not looking into the issue of 
> forward-compatibility.

I'm sure there are ways to run tests against an arbitrary Java version:
https://docs.gradle.org/6.0-rc-1/userguide/building_java_projects.html#sec:java_cross_compilation

It's just not something I'm concerned with at the moment. Can't do
everything at once.

> An alternative here is to go the Elasticsearch approach: Gradle and the Build 
> runs with Java 11 and the Tests are running optionally with a later version - 
> by pasisng a system property pointing to a TestRunner JDK.

Exactly. I'm sure there are ways to do it.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: Gradle build to land on master

2020-01-08 Thread Uwe Schindler

Hi,

Sorry, this is a no-go for going to gradle. We currently want to build and test 
Lucene with all newer Java versions, including EA releases. I have no idea why 
the Gradle people are not looking into the issue of forward-compatibility.

I had a long discussion already on the OpenJDK mailing lists about the 
classfile version issue. The problem is mainly ASM that's used for bytecode 
analysis where Gradle and Groovy depend on. If you pass ASM a classfile version 
that it does not unerstand it brings this error.

I had a discussion with Remi Forax. My proposal would be:
ClassReader should have 2 modes:
- strict mode that enables all features (like you can add filtering on top of 
like method renames, and so pass the ClassReader (as visitor) to a 
ClassWriter. This strict mode ensures that a ClassReader then passed to 
ClassWriter produces the same class file. The requires that ASM fully 
understands the class file.
- lenient mode: This marks the classreader/visitor as "risky". When passing 
this to ClassReader, it can open any class file, also those with newer class 
file versions. The classfile format is made in a way (attributes) that you can 
easily read over features you don't understand. In that case you can use a 
ClassReader like this to read method signatures, run forbiddenapi checks,... 
(what MOST code out there using ASM is doing!). The special flag on the visitor 
 interface is then there to protect people. If you pass a "lenient" class 
reader to a ClassWriter, the ClassWriter complains. So you cannot open a 
ClassReader in lenient way and pass  it to ClassWriter, because that would 
produce a classfile where maybe a significant part of stuff is 
missing/incorrect.

In short: If there would be a non-picky ASM version / ASM mode like proposed 
before - that can just be used for "analysis" of class files, Gradle would be 
forward compatible. But Remi is too academic and also does not want this 
because he sees some "maintenance" trouble. But this is why I said strict mode 
and lenient mode. In lenient mode you cannot complain if your code behaves 
wrong when you open a class file that is newer than the version of ASM supports.

In forbiddenapis I use hack to provide this leniency: When opening class files, 
I just patch the version number. Maybe gradle should do the same - IF AND ONLY 
IF IT DOES NOT USE IT FOR WRITING DERIVATIVE CLASS FILES:

Maybe we could try to confive Gradle, Groovy and maybe Remi to take one of 
those routes. Otherwise we cannot use Gradle with CI builds.

An alternative here is to go the Elasticsearch approach: Gradle and the Build 
runs with Java 11 and the Tests are running optionally with a later version - 
by pasisng a system property pointing to a TestRunner JDK.

Uwe

-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de

> -Original Message-
> From: Dawid Weiss 
> Sent: Wednesday, January 8, 2020 6:06 PM
> To: Lucene Dev 
> Subject: Re: Gradle build to land on master
> 
> > right, that file is the exact one i change. the problem is if i change it 
> > to 6.0.1,
> then it wants me to buy something.
> 
> You can't change to any other gradle version -- stick to the version
> defined in the wrapper and java 11. It is a fragile environment -
> switching gradle version will cause plugin incompatibilities, etc.
> Sigh.
> 
> We can't upgrade to gradle 6.x yet because palantir's plugin for
> dependency management doesn't work for me then.
> 
> Different JVM configurations may be rough, especially with newer JVMs.
> 
> D.
> 
> 
> > On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs  wrote:
> >>
> >> ./gradlew should try to use gradle 5.6.4 see
> >> ./gradle/wrapper/gradle-wrapper.properties
> >> With java 11 this all builds fine for me (can export JAVA_HOME to change)
> >> Which version of java are you using? the linked issue hints at java 13?
> >>
> >> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
> >> >
> >> > Here is the issue: https://github.com/gradle/gradle/issues/8681
> >> >
> >> > I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy
> something?
> >> >
> >> > ]$ ./gradlew help
> >> > Starting a Gradle Daemon (subsequent builds will be faster)
> >> >
> >> > FAILURE: Build failed with an exception.
> >> >
> >> > * Where:
> >> > Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle' 
> >> > line:
> 5
> >> >
> >> > * What went wrong:
> >> > An exception occurred applying plugin request [id: 
> >> > 'com.gradle.build-scan',
> version: '3.0']
> >> > > Failed to apply plugi

Re: Gradle build to land on master

So are we going to be able to do things like EA testing?

On Wed, Jan 8, 2020 at 12:18 PM Dawid Weiss  wrote:

> > Can reproduce your problem with java 13, so guess only 11 is currently
> supported
>
> For completeness: the build actually enforces gradle and Java version
> and fails if there is a mismatch. The failure with Java 13 happens way
> before this check has a chance to even run.
>
>
> https://github.com/apache/lucene-solr/blob/gradle-master/gradle/validation/check-environment.gradle
>
> D.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Gradle build to land on master

> Can reproduce your problem with java 13, so guess only 11 is currently 
> supported

For completeness: the build actually enforces gradle and Java version
and fails if there is a mismatch. The failure with Java 13 happens way
before this check has a chance to even run.

https://github.com/apache/lucene-solr/blob/gradle-master/gradle/validation/check-environment.gradle

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

Can you file a pull request to remove it? I added it initially hoping
it'd help with test browsing but it's not going to work anyway
(because Solr emits half a gig of logs).

Dawid

On Wed, Jan 8, 2020 at 4:51 PM Thomas Matthijs  wrote:
>
> Can reproduce your problem with java 13, so guess only 11 is currently 
> supported
>
> The buildscan plugin seems to be a problem
>
> https://github.com/apache/lucene-solr/blob/gradle-master/gradle/ci/buildscan.gradle
> https://docs.gradle.com/enterprise/gradle-plugin/#connecting_to_scans_gradle_com
>
> ==> "Be careful not to commit agreement to the terms of service into a
> project that may be built by others."
>
> Looks like its only for CI builds (?) build should probably be
> disabled by default
>
> Builds seems to work with java 13 & gradle 6 with it removed:
>
> diff --git a/build.gradle b/build.gradle
> index 8e90409dde7..428d7b3e93c 100644
> --- a/build.gradle
> +++ b/build.gradle
> @@ -1,10 +1,9 @@
>
>  plugins {
>id "base"
>id "com.palantir.consistent-versions" version "1.13.1"
> -  id "com.gradle.build-scan" version "3.0"
>id 'de.thetaphi.forbiddenapis' version '2.7' apply false
>  }
>
>  // Project version and main properties. Applies to all projects.
>  allprojects {
> @@ -17,13 +16,10 @@ allprojects {
>  // if the build file is incorrectly written and evaluates something
>  // eagerly).
>
>  apply from: file('gradle/generate-defaults.gradle')
>
> -// CI systems.
> -apply from: file('gradle/ci/buildscan.gradle')
> -
>  // Set up defaults and configure aspects for certain modules or functionality
>  // (java, tests)
>  apply from: file('gradle/defaults.gradle')
>  apply from: file('gradle/defaults-java.gradle')
>  apply from: file('gradle/testing/defaults-tests.gradle')
> diff --git a/gradle/validation/check-environment.gradle
> b/gradle/validation/check-environment.gradle
> index 3acfbb306d2..1d2d6e5509f 100644
> --- a/gradle/validation/check-environment.gradle
> +++ b/gradle/validation/check-environment.gradle
> @@ -3,11 +3,11 @@
>
>  import org.gradle.util.GradleVersion
>
>  configure(rootProject) {
>ext {
> -expectedGradleVersion = '5.6.4'
> +expectedGradleVersion = '6.0.1'
>  expectedJavaVersion = JavaVersion.VERSION_11
>}
>
>wrapper {
>  distributionType = Wrapper.DistributionType.ALL
> diff --git a/gradle/wrapper/gradle-wrapper.properties
> b/gradle/wrapper/gradle-wrapper.properties
> index 0ebb3108e20..1ba7206f882 100644
> --- a/gradle/wrapper/gradle-wrapper.properties
> +++ b/gradle/wrapper/gradle-wrapper.properties
> @@ -1,5 +1,5 @@
>  distributionBase=GRADLE_USER_HOME
>  distributionPath=wrapper/dists
> -distributionUrl=https\://services.gradle.org/distributions/gradle-5.6.4-all.zip
> +distributionUrl=https\://services.gradle.org/distributions/gradle-6.0.1-all.zip
>  zipStoreBase=GRADLE_USER_HOME
>  zipStorePath=wrapper/dists
>
> On Wed, 8 Jan 2020 at 16:38, Robert Muir  wrote:
> >
> > right, that file is the exact one i change. the problem is if i change it 
> > to 6.0.1, then it wants me to buy something.
> >
> > On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs  wrote:
> >>
> >> ./gradlew should try to use gradle 5.6.4 see
> >> ./gradle/wrapper/gradle-wrapper.properties
> >> With java 11 this all builds fine for me (can export JAVA_HOME to change)
> >> Which version of java are you using? the linked issue hints at java 13?
> >>
> >> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
> >> >
> >> > Here is the issue: https://github.com/gradle/gradle/issues/8681
> >> >
> >> > I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy 
> >> > something?
> >> >
> >> > ]$ ./gradlew help
> >> > Starting a Gradle Daemon (subsequent builds will be faster)
> >> >
> >> > FAILURE: Build failed with an exception.
> >> >
> >> > * Where:
> >> > Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle' 
> >> > line: 5
> >> >
> >> > * What went wrong:
> >> > An exception occurred applying plugin request [id: 
> >> > 'com.gradle.build-scan', version: '3.0']
> >> > > Failed to apply plugin [id 'com.gradle.build-scan']
> >> >> This build scan plugin is not compatible with less than Gradle 6.0.
> >> >  Please use the Gradle Enterprise plugin instead.
> >> >
> >> > On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:
> >> >>
> >> >> I tried it out just to see, here was my experience:
> >> >>
> >> >> $ git checkout gradle-master
> >> >> Switched to branch 'gradle-master'
> >> >> Your branch is up to date with 'origin/gradle-master'.
> >> >> $ ./gradlew help
> >> >> Starting a Gradle Daemon (subsequent builds will be faster)
> >> >>
> >> >> FAILURE: Build failed with an exception.
> >> >>
> >> >> * Where:
> >> >> Settings file 
> >> >> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
> >> >>
> >> >> * What went wrong:
> >> >> Could not compile settings file 
> >> >> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
> >> >> > startup failed:
> >> >>   General error

Re: Gradle build to land on master

> right, that file is the exact one i change. the problem is if i change it to 
> 6.0.1, then it wants me to buy something.

You can't change to any other gradle version -- stick to the version
defined in the wrapper and java 11. It is a fragile environment -
switching gradle version will cause plugin incompatibilities, etc.
Sigh.

We can't upgrade to gradle 6.x yet because palantir's plugin for
dependency management doesn't work for me then.

Different JVM configurations may be rough, especially with newer JVMs.

D.


> On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs  wrote:
>>
>> ./gradlew should try to use gradle 5.6.4 see
>> ./gradle/wrapper/gradle-wrapper.properties
>> With java 11 this all builds fine for me (can export JAVA_HOME to change)
>> Which version of java are you using? the linked issue hints at java 13?
>>
>> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
>> >
>> > Here is the issue: https://github.com/gradle/gradle/issues/8681
>> >
>> > I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy 
>> > something?
>> >
>> > ]$ ./gradlew help
>> > Starting a Gradle Daemon (subsequent builds will be faster)
>> >
>> > FAILURE: Build failed with an exception.
>> >
>> > * Where:
>> > Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle' 
>> > line: 5
>> >
>> > * What went wrong:
>> > An exception occurred applying plugin request [id: 
>> > 'com.gradle.build-scan', version: '3.0']
>> > > Failed to apply plugin [id 'com.gradle.build-scan']
>> >> This build scan plugin is not compatible with less than Gradle 6.0.
>> >  Please use the Gradle Enterprise plugin instead.
>> >
>> > On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:
>> >>
>> >> I tried it out just to see, here was my experience:
>> >>
>> >> $ git checkout gradle-master
>> >> Switched to branch 'gradle-master'
>> >> Your branch is up to date with 'origin/gradle-master'.
>> >> $ ./gradlew help
>> >> Starting a Gradle Daemon (subsequent builds will be faster)
>> >>
>> >> FAILURE: Build failed with an exception.
>> >>
>> >> * Where:
>> >> Settings file 
>> >> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
>> >>
>> >> * What went wrong:
>> >> Could not compile settings file 
>> >> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
>> >> > startup failed:
>> >>   General error during semantic analysis: Unsupported class file major 
>> >> version 57
>> >>
>> >>   java.lang.IllegalArgumentException: Unsupported class file major 
>> >> version 57
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
>> >> at 
>> >> org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
>> >> at 
>> >> org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:254)
>> >> at 
>> >> org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:192)
>> >>
>> >> On Wed, Jan 8, 2020 at 9:30 AM Dawid Weiss  wrote:
>> >>>
>> >>> I think the gradle-master branch is already workable enough to land on 
>> >>> master.
>> >>>
>> >>> If you're not familiar with gradle then, once merged, run "gradlew help".
>> >>>
>> >>> Some notes.
>> >>>
>> >>> 1) I have been running tests on Windows and Linux, they're ok. The
>> >>> output is slightly different from ANT's but I think it's fine for
>> >>> working.
>> >>>
>> >>> 2) The speed of compilation and running tests selectively is much
>> >>> better than ant's, especially on multicore machines.
>> >>>
>> >>> 3) I use IntelliJ idea and the project imports into the IDE without
>> >>> any special handling. Code formatting and such may need to be adjusted
>> >>> though.
>> >>>
>> >>> 4) Some things are incomplete (precommit does a subset of checks).
>> >>> Some are missing (regeneration tasks). Some are different (handling of
>> >>> dependencies, build output folder locations). It will take some time
>> >>> and learning to live with a new build system. I tried to provide short
>> >>> guides into selective areas (they're available as help tasks or plain
>> >>> text files under help/).
>> >>>
>> >>> 5) If something does *not* work, let me know.
>> >>>
>> >>> 6) It'd be nice if we had a build job somewhere on a faster machine
>> >>> that would run at least "gradlew precommit check -x test" so that
>> >>> rudimentary checks are applied, without running all the tests. This
>> >>> would ensure consistency in dependencies, for example.
>> >>>
>> >>> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
>> >>> kept open and occasionally merged back and forth.
>> >>>
>> >>> I have to shift more focus to my daily job but will help out and chip
>> >>> at the remaining bits, time permitting.
>> >>>
>> >>> Dawid
>> >>>
>> >>>

Re: Gradle build to land on master

2020-01-08 Thread Thomas Matthijs

Can reproduce your problem with java 13, so guess only 11 is currently supported

The buildscan plugin seems to be a problem

https://github.com/apache/lucene-solr/blob/gradle-master/gradle/ci/buildscan.gradle
https://docs.gradle.com/enterprise/gradle-plugin/#connecting_to_scans_gradle_com

==> "Be careful not to commit agreement to the terms of service into a
project that may be built by others."

Looks like its only for CI builds (?) build should probably be
disabled by default

Builds seems to work with java 13 & gradle 6 with it removed:

diff --git a/build.gradle b/build.gradle
index 8e90409dde7..428d7b3e93c 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1,10 +1,9 @@

 plugins {
   id "base"
   id "com.palantir.consistent-versions" version "1.13.1"
-  id "com.gradle.build-scan" version "3.0"
   id 'de.thetaphi.forbiddenapis' version '2.7' apply false
 }

 // Project version and main properties. Applies to all projects.
 allprojects {
@@ -17,13 +16,10 @@ allprojects {
 // if the build file is incorrectly written and evaluates something
 // eagerly).

 apply from: file('gradle/generate-defaults.gradle')

-// CI systems.
-apply from: file('gradle/ci/buildscan.gradle')
-
 // Set up defaults and configure aspects for certain modules or functionality
 // (java, tests)
 apply from: file('gradle/defaults.gradle')
 apply from: file('gradle/defaults-java.gradle')
 apply from: file('gradle/testing/defaults-tests.gradle')
diff --git a/gradle/validation/check-environment.gradle
b/gradle/validation/check-environment.gradle
index 3acfbb306d2..1d2d6e5509f 100644
--- a/gradle/validation/check-environment.gradle
+++ b/gradle/validation/check-environment.gradle
@@ -3,11 +3,11 @@

 import org.gradle.util.GradleVersion

 configure(rootProject) {
   ext {
-expectedGradleVersion = '5.6.4'
+expectedGradleVersion = '6.0.1'
 expectedJavaVersion = JavaVersion.VERSION_11
   }

   wrapper {
 distributionType = Wrapper.DistributionType.ALL
diff --git a/gradle/wrapper/gradle-wrapper.properties
b/gradle/wrapper/gradle-wrapper.properties
index 0ebb3108e20..1ba7206f882 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-5.6.4-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.0.1-all.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists

On Wed, 8 Jan 2020 at 16:38, Robert Muir  wrote:
>
> right, that file is the exact one i change. the problem is if i change it to 
> 6.0.1, then it wants me to buy something.
>
> On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs  wrote:
>>
>> ./gradlew should try to use gradle 5.6.4 see
>> ./gradle/wrapper/gradle-wrapper.properties
>> With java 11 this all builds fine for me (can export JAVA_HOME to change)
>> Which version of java are you using? the linked issue hints at java 13?
>>
>> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
>> >
>> > Here is the issue: https://github.com/gradle/gradle/issues/8681
>> >
>> > I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy 
>> > something?
>> >
>> > ]$ ./gradlew help
>> > Starting a Gradle Daemon (subsequent builds will be faster)
>> >
>> > FAILURE: Build failed with an exception.
>> >
>> > * Where:
>> > Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle' 
>> > line: 5
>> >
>> > * What went wrong:
>> > An exception occurred applying plugin request [id: 
>> > 'com.gradle.build-scan', version: '3.0']
>> > > Failed to apply plugin [id 'com.gradle.build-scan']
>> >> This build scan plugin is not compatible with less than Gradle 6.0.
>> >  Please use the Gradle Enterprise plugin instead.
>> >
>> > On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:
>> >>
>> >> I tried it out just to see, here was my experience:
>> >>
>> >> $ git checkout gradle-master
>> >> Switched to branch 'gradle-master'
>> >> Your branch is up to date with 'origin/gradle-master'.
>> >> $ ./gradlew help
>> >> Starting a Gradle Daemon (subsequent builds will be faster)
>> >>
>> >> FAILURE: Build failed with an exception.
>> >>
>> >> * Where:
>> >> Settings file 
>> >> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
>> >>
>> >> * What went wrong:
>> >> Could not compile settings file 
>> >> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
>> >> > startup failed:
>> >>   General error during semantic analysis: Unsupported class file major 
>> >> version 57
>> >>
>> >>   java.lang.IllegalArgumentException: Unsupported class file major 
>> >> version 57
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
>> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
>> >> at 
>> >>

Re: Gradle build to land on master

right, that file is the exact one i change. the problem is if i change it
to 6.0.1, then it wants me to buy something.

On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs  wrote:

> ./gradlew should try to use gradle 5.6.4 see
> ./gradle/wrapper/gradle-wrapper.properties
> With java 11 this all builds fine for me (can export JAVA_HOME to change)
> Which version of java are you using? the linked issue hints at java 13?
>
> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
> >
> > Here is the issue: https://github.com/gradle/gradle/issues/8681
> >
> > I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy
> something?
> >
> > ]$ ./gradlew help
> > Starting a Gradle Daemon (subsequent builds will be faster)
> >
> > FAILURE: Build failed with an exception.
> >
> > * Where:
> > Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle'
> line: 5
> >
> > * What went wrong:
> > An exception occurred applying plugin request [id:
> 'com.gradle.build-scan', version: '3.0']
> > > Failed to apply plugin [id 'com.gradle.build-scan']
> >> This build scan plugin is not compatible with less than Gradle 6.0.
> >  Please use the Gradle Enterprise plugin instead.
> >
> > On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:
> >>
> >> I tried it out just to see, here was my experience:
> >>
> >> $ git checkout gradle-master
> >> Switched to branch 'gradle-master'
> >> Your branch is up to date with 'origin/gradle-master'.
> >> $ ./gradlew help
> >> Starting a Gradle Daemon (subsequent builds will be faster)
> >>
> >> FAILURE: Build failed with an exception.
> >>
> >> * Where:
> >> Settings file
> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
> >>
> >> * What went wrong:
> >> Could not compile settings file
> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
> >> > startup failed:
> >>   General error during semantic analysis: Unsupported class file major
> version 57
> >>
> >>   java.lang.IllegalArgumentException: Unsupported class file major
> version 57
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
> >> at
> org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
> >> at
> org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:254)
> >> at
> org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:192)
> >>
> >> On Wed, Jan 8, 2020 at 9:30 AM Dawid Weiss 
> wrote:
> >>>
> >>> I think the gradle-master branch is already workable enough to land on
> master.
> >>>
> >>> If you're not familiar with gradle then, once merged, run "gradlew
> help".
> >>>
> >>> Some notes.
> >>>
> >>> 1) I have been running tests on Windows and Linux, they're ok. The
> >>> output is slightly different from ANT's but I think it's fine for
> >>> working.
> >>>
> >>> 2) The speed of compilation and running tests selectively is much
> >>> better than ant's, especially on multicore machines.
> >>>
> >>> 3) I use IntelliJ idea and the project imports into the IDE without
> >>> any special handling. Code formatting and such may need to be adjusted
> >>> though.
> >>>
> >>> 4) Some things are incomplete (precommit does a subset of checks).
> >>> Some are missing (regeneration tasks). Some are different (handling of
> >>> dependencies, build output folder locations). It will take some time
> >>> and learning to live with a new build system. I tried to provide short
> >>> guides into selective areas (they're available as help tasks or plain
> >>> text files under help/).
> >>>
> >>> 5) If something does *not* work, let me know.
> >>>
> >>> 6) It'd be nice if we had a build job somewhere on a faster machine
> >>> that would run at least "gradlew precommit check -x test" so that
> >>> rudimentary checks are applied, without running all the tests. This
> >>> would ensure consistency in dependencies, for example.
> >>>
> >>> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
> >>> kept open and occasionally merged back and forth.
> >>>
> >>> I have to shift more focus to my daily job but will help out and chip
> >>> at the remaining bits, time permitting.
> >>>
> >>> Dawid
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Gradle build to land on master

openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.1+9, mixed mode, sharing)

On Wed, Jan 8, 2020 at 10:08 AM Thomas Matthijs  wrote:

> ./gradlew should try to use gradle 5.6.4 see
> ./gradle/wrapper/gradle-wrapper.properties
> With java 11 this all builds fine for me (can export JAVA_HOME to change)
> Which version of java are you using? the linked issue hints at java 13?
>
> On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
> >
> > Here is the issue: https://github.com/gradle/gradle/issues/8681
> >
> > I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy
> something?
> >
> > ]$ ./gradlew help
> > Starting a Gradle Daemon (subsequent builds will be faster)
> >
> > FAILURE: Build failed with an exception.
> >
> > * Where:
> > Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle'
> line: 5
> >
> > * What went wrong:
> > An exception occurred applying plugin request [id:
> 'com.gradle.build-scan', version: '3.0']
> > > Failed to apply plugin [id 'com.gradle.build-scan']
> >> This build scan plugin is not compatible with less than Gradle 6.0.
> >  Please use the Gradle Enterprise plugin instead.
> >
> > On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:
> >>
> >> I tried it out just to see, here was my experience:
> >>
> >> $ git checkout gradle-master
> >> Switched to branch 'gradle-master'
> >> Your branch is up to date with 'origin/gradle-master'.
> >> $ ./gradlew help
> >> Starting a Gradle Daemon (subsequent builds will be faster)
> >>
> >> FAILURE: Build failed with an exception.
> >>
> >> * Where:
> >> Settings file
> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
> >>
> >> * What went wrong:
> >> Could not compile settings file
> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
> >> > startup failed:
> >>   General error during semantic analysis: Unsupported class file major
> version 57
> >>
> >>   java.lang.IllegalArgumentException: Unsupported class file major
> version 57
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
> >> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
> >> at
> org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
> >> at
> org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:254)
> >> at
> org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:192)
> >>
> >> On Wed, Jan 8, 2020 at 9:30 AM Dawid Weiss 
> wrote:
> >>>
> >>> I think the gradle-master branch is already workable enough to land on
> master.
> >>>
> >>> If you're not familiar with gradle then, once merged, run "gradlew
> help".
> >>>
> >>> Some notes.
> >>>
> >>> 1) I have been running tests on Windows and Linux, they're ok. The
> >>> output is slightly different from ANT's but I think it's fine for
> >>> working.
> >>>
> >>> 2) The speed of compilation and running tests selectively is much
> >>> better than ant's, especially on multicore machines.
> >>>
> >>> 3) I use IntelliJ idea and the project imports into the IDE without
> >>> any special handling. Code formatting and such may need to be adjusted
> >>> though.
> >>>
> >>> 4) Some things are incomplete (precommit does a subset of checks).
> >>> Some are missing (regeneration tasks). Some are different (handling of
> >>> dependencies, build output folder locations). It will take some time
> >>> and learning to live with a new build system. I tried to provide short
> >>> guides into selective areas (they're available as help tasks or plain
> >>> text files under help/).
> >>>
> >>> 5) If something does *not* work, let me know.
> >>>
> >>> 6) It'd be nice if we had a build job somewhere on a faster machine
> >>> that would run at least "gradlew precommit check -x test" so that
> >>> rudimentary checks are applied, without running all the tests. This
> >>> would ensure consistency in dependencies, for example.
> >>>
> >>> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
> >>> kept open and occasionally merged back and forth.
> >>>
> >>> I have to shift more focus to my daily job but will help out and chip
> >>> at the remaining bits, time permitting.
> >>>
> >>> Dawid
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Gradle build to land on master

2020-01-08 Thread Thomas Matthijs

./gradlew should try to use gradle 5.6.4 see
./gradle/wrapper/gradle-wrapper.properties
With java 11 this all builds fine for me (can export JAVA_HOME to change)
Which version of java are you using? the linked issue hints at java 13?

On Wed, 8 Jan 2020 at 16:01, Robert Muir  wrote:
>
> Here is the issue: https://github.com/gradle/gradle/issues/8681
>
> I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy 
> something?
>
> ]$ ./gradlew help
> Starting a Gradle Daemon (subsequent builds will be faster)
>
> FAILURE: Build failed with an exception.
>
> * Where:
> Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle' line: 
> 5
>
> * What went wrong:
> An exception occurred applying plugin request [id: 'com.gradle.build-scan', 
> version: '3.0']
> > Failed to apply plugin [id 'com.gradle.build-scan']
>> This build scan plugin is not compatible with less than Gradle 6.0.
>  Please use the Gradle Enterprise plugin instead.
>
> On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:
>>
>> I tried it out just to see, here was my experience:
>>
>> $ git checkout gradle-master
>> Switched to branch 'gradle-master'
>> Your branch is up to date with 'origin/gradle-master'.
>> $ ./gradlew help
>> Starting a Gradle Daemon (subsequent builds will be faster)
>>
>> FAILURE: Build failed with an exception.
>>
>> * Where:
>> Settings file '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
>>
>> * What went wrong:
>> Could not compile settings file 
>> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
>> > startup failed:
>>   General error during semantic analysis: Unsupported class file major 
>> version 57
>>
>>   java.lang.IllegalArgumentException: Unsupported class file major version 57
>> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
>> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
>> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
>> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
>> at 
>> org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
>> at 
>> org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:254)
>> at 
>> org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:192)
>>
>> On Wed, Jan 8, 2020 at 9:30 AM Dawid Weiss  wrote:
>>>
>>> I think the gradle-master branch is already workable enough to land on 
>>> master.
>>>
>>> If you're not familiar with gradle then, once merged, run "gradlew help".
>>>
>>> Some notes.
>>>
>>> 1) I have been running tests on Windows and Linux, they're ok. The
>>> output is slightly different from ANT's but I think it's fine for
>>> working.
>>>
>>> 2) The speed of compilation and running tests selectively is much
>>> better than ant's, especially on multicore machines.
>>>
>>> 3) I use IntelliJ idea and the project imports into the IDE without
>>> any special handling. Code formatting and such may need to be adjusted
>>> though.
>>>
>>> 4) Some things are incomplete (precommit does a subset of checks).
>>> Some are missing (regeneration tasks). Some are different (handling of
>>> dependencies, build output folder locations). It will take some time
>>> and learning to live with a new build system. I tried to provide short
>>> guides into selective areas (they're available as help tasks or plain
>>> text files under help/).
>>>
>>> 5) If something does *not* work, let me know.
>>>
>>> 6) It'd be nice if we had a build job somewhere on a faster machine
>>> that would run at least "gradlew precommit check -x test" so that
>>> rudimentary checks are applied, without running all the tests. This
>>> would ensure consistency in dependencies, for example.
>>>
>>> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
>>> kept open and occasionally merged back and forth.
>>>
>>> I have to shift more focus to my daily job but will help out and chip
>>> at the remaining bits, time permitting.
>>>
>>> Dawid
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build to land on master

Here is the issue: https://github.com/gradle/gradle/issues/8681

I tried upgrading gradle to 6.0.1, but it seems now it wants me to buy
something?

]$ ./gradlew help
Starting a Gradle Daemon (subsequent builds will be faster)

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/rob.muir/eclipse.workspace/lucene-solr/build.gradle'
line: 5

* What went wrong:
An exception occurred applying plugin request [id: 'com.gradle.build-scan',
version: '3.0']
> Failed to apply plugin [id 'com.gradle.build-scan']
   > This build scan plugin is not compatible with less than Gradle 6.0.
 Please use the Gradle Enterprise plugin instead.

On Wed, Jan 8, 2020 at 9:41 AM Robert Muir  wrote:

> I tried it out just to see, here was my experience:
>
> $ git checkout gradle-master
> Switched to branch 'gradle-master'
> Your branch is up to date with 'origin/gradle-master'.
> $ ./gradlew help
> Starting a Gradle Daemon (subsequent builds will be faster)
>
> FAILURE: Build failed with an exception.
>
> * Where:
> Settings file
> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'
>
> * What went wrong:
> Could not compile settings file
> '/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
> > startup failed:
>   General error during semantic analysis: Unsupported class file major
> version 57
>
>   java.lang.IllegalArgumentException: Unsupported class file major version
> 57
> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
> at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
> at
> org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
> at
> org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:254)
> at
> org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:192)
>
> On Wed, Jan 8, 2020 at 9:30 AM Dawid Weiss  wrote:
>
>> I think the gradle-master branch is already workable enough to land on
>> master.
>>
>> If you're not familiar with gradle then, once merged, run "gradlew help".
>>
>> Some notes.
>>
>> 1) I have been running tests on Windows and Linux, they're ok. The
>> output is slightly different from ANT's but I think it's fine for
>> working.
>>
>> 2) The speed of compilation and running tests selectively is much
>> better than ant's, especially on multicore machines.
>>
>> 3) I use IntelliJ idea and the project imports into the IDE without
>> any special handling. Code formatting and such may need to be adjusted
>> though.
>>
>> 4) Some things are incomplete (precommit does a subset of checks).
>> Some are missing (regeneration tasks). Some are different (handling of
>> dependencies, build output folder locations). It will take some time
>> and learning to live with a new build system. I tried to provide short
>> guides into selective areas (they're available as help tasks or plain
>> text files under help/).
>>
>> 5) If something does *not* work, let me know.
>>
>> 6) It'd be nice if we had a build job somewhere on a faster machine
>> that would run at least "gradlew precommit check -x test" so that
>> rudimentary checks are applied, without running all the tests. This
>> would ensure consistency in dependencies, for example.
>>
>> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
>> kept open and occasionally merged back and forth.
>>
>> I have to shift more focus to my daily job but will help out and chip
>> at the remaining bits, time permitting.
>>
>> Dawid
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Gradle build to land on master

I tried it out just to see, here was my experience:

$ git checkout gradle-master
Switched to branch 'gradle-master'
Your branch is up to date with 'origin/gradle-master'.
$ ./gradlew help
Starting a Gradle Daemon (subsequent builds will be faster)

FAILURE: Build failed with an exception.

* Where:
Settings file
'/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'

* What went wrong:
Could not compile settings file
'/Users/rob.muir/eclipse.workspace/lucene-solr/settings.gradle'.
> startup failed:
  General error during semantic analysis: Unsupported class file major
version 57

  java.lang.IllegalArgumentException: Unsupported class file major version
57
at groovyjarjarasm.asm.ClassReader.(ClassReader.java:184)
at groovyjarjarasm.asm.ClassReader.(ClassReader.java:166)
at groovyjarjarasm.asm.ClassReader.(ClassReader.java:152)
at groovyjarjarasm.asm.ClassReader.(ClassReader.java:273)
at
org.codehaus.groovy.ast.decompiled.AsmDecompiler.parseClass(AsmDecompiler.java:81)
at
org.codehaus.groovy.control.ClassNodeResolver.findDecompiled(ClassNodeResolver.java:254)
at
org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:192)

On Wed, Jan 8, 2020 at 9:30 AM Dawid Weiss  wrote:

> I think the gradle-master branch is already workable enough to land on
> master.
>
> If you're not familiar with gradle then, once merged, run "gradlew help".
>
> Some notes.
>
> 1) I have been running tests on Windows and Linux, they're ok. The
> output is slightly different from ANT's but I think it's fine for
> working.
>
> 2) The speed of compilation and running tests selectively is much
> better than ant's, especially on multicore machines.
>
> 3) I use IntelliJ idea and the project imports into the IDE without
> any special handling. Code formatting and such may need to be adjusted
> though.
>
> 4) Some things are incomplete (precommit does a subset of checks).
> Some are missing (regeneration tasks). Some are different (handling of
> dependencies, build output folder locations). It will take some time
> and learning to live with a new build system. I tried to provide short
> guides into selective areas (they're available as help tasks or plain
> text files under help/).
>
> 5) If something does *not* work, let me know.
>
> 6) It'd be nice if we had a build job somewhere on a faster machine
> that would run at least "gradlew precommit check -x test" so that
> rudimentary checks are applied, without running all the tests. This
> would ensure consistency in dependencies, for example.
>
> 7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
> kept open and occasionally merged back and forth.
>
> I have to shift more focus to my daily job but will help out and chip
> at the remaining bits, time permitting.
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Gradle build to land on master

I think the gradle-master branch is already workable enough to land on master.

If you're not familiar with gradle then, once merged, run "gradlew help".

Some notes.

1) I have been running tests on Windows and Linux, they're ok. The
output is slightly different from ANT's but I think it's fine for
working.

2) The speed of compilation and running tests selectively is much
better than ant's, especially on multicore machines.

3) I use IntelliJ idea and the project imports into the IDE without
any special handling. Code formatting and such may need to be adjusted
though.

4) Some things are incomplete (precommit does a subset of checks).
Some are missing (regeneration tasks). Some are different (handling of
dependencies, build output folder locations). It will take some time
and learning to live with a new build system. I tried to provide short
guides into selective areas (they're available as help tasks or plain
text files under help/).

5) If something does *not* work, let me know.

6) It'd be nice if we had a build job somewhere on a faster machine
that would run at least "gradlew precommit check -x test" so that
rudimentary checks are applied, without running all the tests. This
would ensure consistency in dependencies, for example.

7) The parallel branch (gradle-master) and issue (LUCENE-9077) will be
kept open and occasionally merged back and forth.

I have to shift more focus to my daily job but will help out and chip
at the remaining bits, time permitting.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

This may help you understand the hierarchy of how properties are
shadowing each other.
https://docs.gradle.org/current/userguide/build_environment.html

The discussion concerning project properties that people could
"override" locally concerned the settings
set in the global gradle.properties file -- this was something I
wasn't too happy with if they clashed with other projects.

For now the gradle build doesn't have any means to populate certain
properties with custom values (other than command-line). If there is a
need for it we can add it.

Dawid

On Tue, Dec 3, 2019 at 11:48 PM Erick Erickson  wrote:
>
> Thanks, I’m still untangling the various bits Mark put in and
> relying on people who actually, you know, understand
> things ;)
>
> > On Dec 3, 2019, at 5:11 PM, Dawid Weiss  wrote:
> >
> >> // the project version
> >> version=9.0.0
> >
> > You shouldn't have it in your global configuration. In fact, only
> > these make sense in there:
> >
> > org.gradle.daemon=true
> > org.gradle.workers.max = 6
> >
> > The cache and daemon settings I'd leave to the project's config.
> >
> > Dawid
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

Thanks, I’m still untangling the various bits Mark put in and 
relying on people who actually, you know, understand
things ;)

> On Dec 3, 2019, at 5:11 PM, Dawid Weiss  wrote:
> 
>> // the project version
>> version=9.0.0
> 
> You shouldn't have it in your global configuration. In fact, only
> these make sense in there:
> 
> org.gradle.daemon=true
> org.gradle.workers.max = 6
> 
> The cache and daemon settings I'd leave to the project's config.
> 
> Dawid
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr, gradle build and working with files compiled into the source tree

> It's a trade-off; let's acknowledge that the current system has its benefits.
> For example not needing to change to some other directory to run Solr.

I don't see those, to be honest. Switching directory to run the thing
doesn't seem like a big benefit and running in place... look at the
.gitignore files
and you'll see what I mean when I say it's an ugly thing to do.

> And at present the gradle assemble script will use a target path with a 
> version in the name which may make docs/instructions/scripts and habits more 
> susceptible to getting out of date / being wrong.  That could be fixed easily 
> though.

This is intentional, actually. Docs, instructions and scripts should
only point at and describe the internal content of this folder, not to
the folder name. If somebody builds from sources they will know what
they're doing (?).

> I like that you've hid it away in one place and it doesn't look hideous to me.

Trust me, it is terrible in many ways...

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

> // the project version
> version=9.0.0

You shouldn't have it in your global configuration. In fact, only
these make sense in there:

org.gradle.daemon=true
org.gradle.workers.max = 6

The cache and daemon settings I'd leave to the project's config.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

Here’s what I have in ~/.gradle/gradle.properties that seems to at least be in 
the right ballpark. Based on Mark’s “intro to the gradle build” page:


// the project version
version=9.0.0

// whether or not to use the Gradle daemon - if true, keeps the build process 
around for reuse for up to 3 hours so that startup times are removed and 
hotspot has a chance to rock - you should generally set this to true in your 
~/.gradle/gradle.properties
// we set to false so that a CI system with no setting will not use a daemon by 
default
org.gradle.daemon=true

// allows tasks to be executed in parallel, across modules
org.gradle.parallel=true

// max parallel jobs to run at once, including both tasks and tests.
// default is number of CPU cores which is often too high - you should start by 
setting it to half the number of cores, especially if you have hyperthreading
org.gradle.workers.max = 6

// number of jvms to distribute tests across in parallel
// defaults to number of CPU cores / 2 - you should just set this the same as 
org.gradle.workers.max
// NOTE: gradle does not try to balance tests across jvms yet: 
https://github.com/gradle/gradle/issues/2669
tests_jvms = 6

// enables gradles build cache, which will reuse cached build outputs when it 
can
org.gradle.caching=true

// experimental gradle feature - does not currently work with our version 
constraints plugin: 
https://github.com/palantir/gradle-consistent-versions/pull/145
// also known to have other issues and not known to really speed anything up 
anyhow
org.gradle.configureondemand=false

// how much ram the gradle daemon or process can use
org.gradle.jvmargs=-Xmx1g


> On Dec 3, 2019, at 3:57 PM, Dawid Weiss  wrote:
> 
> That's aligned with my intuition -- logical cores / 2. Sadly I don't
> see how we may compute the number of workers dynamically (you can do
> this with test forks per project but not with the overall number of
> workers). I asked about it [1]. This is perhaps one of the few
> settings you could override globally via ~/.gradle/gradle.properties
> because it really applies to your hardware (corrects the overestimated
> default by Gradle).
> 
> org.gradle.workers.max=[cpu cures]/2
> 
> D.
> 
> [1] https://discuss.gradle.org/t/set-max-workers-dynamically/34087
> 
> On Tue, Dec 3, 2019 at 9:19 PM David Smiley  wrote:
>> 
>> FWIW on a 6 year old MacBook Pro with a Quad-core i7, it seems max-workers 
>> of 2 is about right, clocking in at 21:32.  3 took 20:17; not much better.
>> 
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>> 
>> 
>> On Tue, Dec 3, 2019 at 9:13 AM Erick Erickson  
>> wrote:
>>> 
>>> David Smiley:
>>> 
>>> gradlew -p solr/packaging assemble
>>> 
>>> couple of things:
>>> 1> The place you run bin/solr from is different
>>> 2> I didn’t need to specify the -p parameter and it defaulted to 
>>> ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
>>> 
>>> Once I got over having to switch to a different dir than I was accustomed 
>>> to, I realized that by not mixing the build output with source, things are 
>>> _much_ cleaner. After a “gradlew clean”, the packaging directory only 
>>> contains a build.gradle file.
>>> 
>>>> On Dec 3, 2019, at 4:48 AM, Jan Høydahl  wrote:
>>>> 
>>>> gradlew -p solr/packaging assemble
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr, gradle build and working with files compiled into the source tree

2019-12-03 Thread David Smiley

It's a trade-off; let's acknowledge that the current system has its
benefits.  For example not needing to change to some other directory to run
Solr.  Some in-place changes (e.g. to bin/solr) are possible without
additional sync steps.  And at present the gradle assemble script will use
a target path with a version in the name which may make
docs/instructions/scripts and habits more susceptible to getting out of
date / being wrong.  That could be fixed easily though.

Thanks for doing "gradlew resolve".  "resolve" is not in my muscle memory;
"server" is, but seems similar.  Maybe some small changes could make
"graldew resolve" more straightforward.  Despite you and Erick not liking
it, I like that you've hid it away in one place and it doesn't look hideous
to me.  Any way there is no hurry on this matter; it doesn't block merging
or adopting the gradle build.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

On Tue, Dec 3, 2019 at 12:39 PM Dawid Weiss  wrote:

> > I’d favor taking out the “resolve” task as well. Thanks for considering
> it,
> > but let’s keep things architecturally sound and add back complexity only
> > when necessary.
>
> That entire folder is meant to be discarded at some point. Let's allow
> it to sit there for a bit though,
> doesn't do any harm and we'll see how the transition goes.
>
> D.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Gradle build effort (respin, please read)

That's aligned with my intuition -- logical cores / 2. Sadly I don't
see how we may compute the number of workers dynamically (you can do
this with test forks per project but not with the overall number of
workers). I asked about it [1]. This is perhaps one of the few
settings you could override globally via ~/.gradle/gradle.properties
because it really applies to your hardware (corrects the overestimated
default by Gradle).

org.gradle.workers.max=[cpu cures]/2

D.

[1] https://discuss.gradle.org/t/set-max-workers-dynamically/34087

On Tue, Dec 3, 2019 at 9:19 PM David Smiley  wrote:
>
> FWIW on a 6 year old MacBook Pro with a Quad-core i7, it seems max-workers of 
> 2 is about right, clocking in at 21:32.  3 took 20:17; not much better.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Dec 3, 2019 at 9:13 AM Erick Erickson  wrote:
>>
>> David Smiley:
>>
>> gradlew -p solr/packaging assemble
>>
>> couple of things:
>> 1> The place you run bin/solr from is different
>> 2> I didn’t need to specify the -p parameter and it defaulted to 
>> ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
>>
>> Once I got over having to switch to a different dir than I was accustomed 
>> to, I realized that by not mixing the build output with source, things are 
>> _much_ cleaner. After a “gradlew clean”, the packaging directory only 
>> contains a build.gradle file.
>>
>> > On Dec 3, 2019, at 4:48 AM, Jan Høydahl  wrote:
>> >
>> > gradlew -p solr/packaging assemble
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

2019-12-03 Thread David Smiley

FWIW on a 6 year old MacBook Pro with a Quad-core i7, it seems max-workers
of 2 is about right, clocking in at 21:32.  3 took 20:17; not much better.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Dec 3, 2019 at 9:13 AM Erick Erickson 
wrote:

> David Smiley:
>
> gradlew -p solr/packaging assemble
>
> couple of things:
> 1> The place you run bin/solr from is different
> 2> I didn’t need to specify the -p parameter and it defaulted to
> ‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.
>
> Once I got over having to switch to a different dir than I was accustomed
> to, I realized that by not mixing the build output with source, things are
> _much_ cleaner. After a “gradlew clean”, the packaging directory only
> contains a build.gradle file.
>
> > On Dec 3, 2019, at 4:48 AM, Jan Høydahl  wrote:
> >
> > gradlew -p solr/packaging assemble
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Solr, gradle build and working with files compiled into the source tree

> I’d favor taking out the “resolve” task as well. Thanks for considering it,
> but let’s keep things architecturally sound and add back complexity only
> when necessary.

That entire folder is meant to be discarded at some point. Let's allow
it to sit there for a bit though,
doesn't do any harm and we'll see how the transition goes.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr, gradle build and working with files compiled into the source tree

AFAIC, there’s no good reason whatsoever to preserve being able to run
from inside the source tree if it makes maintenance of the build system
harder. The current system grew organically over years and has its warts,
Preserving those warts just “so I can do what I’ve always done” is
unnecessary.

I’d favor taking out the “resolve” task as well. Thanks for considering it,
but let’s keep things architecturally sound and add back complexity only
when necessary.

If I can rebuild Solr quickly after a small change, then the inconvenience
of having to change directories after a clean is minor. Very minor.

Unless I’m missing something about where the project gets built, let’s
keep it clean and nuke the “resolve” target and use the package directory.

Erick

> On Dec 3, 2019, at 4:05 AM, Dawid Weiss  wrote:
> 
> David asked this question regarding the gradle build and I think it
> may be a longer debate so I start a new thread:
> 
>> What gradlew command is the equivalent of "ant server" for Solr?  I wish to 
>> use "bin/solr".
> 
> Here comes my very subjective opinion: let's get rid of the
> possibility to run various commands right on the source tree... Let me
> explain why.
> 
> I had a *very* hard time even understanding what is where in the Solr
> source tree and whether it's versioned, compiled or generated. I ended
> up using a clean repository, compiled repository and a distribution
> ZIP and diffing against each other. This can't be a sane thing.
> 
> Having compiled files in the source tree makes everything lousy - from
> complex rules in .gitignore files to extreme hacks inside build files
> (ant, gradle or otherwise) to make this copying and syncing work.
> 
> What's the alternative? Well, one of such alternatives is a
> "packaging" project that puts together the binary project
> distribution. With gradle this is lighting fast, incremental and
> always correct (files are synced from source folders and dependencies
> -- they are not copied). For example, putting a Solr distribution from
> scratch takes 2 minutes on my machine (first run) but subsequent run
> (with no changes to anything) is nearly instantaneous (5 secs).
> 
> If you remove a file or a dependency in the source and rebuild the package 
> with:
> 
> gradlew -p solr/packaging assemble
> 
> then the assembled output under:
> 
> solr/packaging/build/solr-9.0.0-SNAPSHOT
> 
> will always have up-to-date state of a true distribution. You can run
> this gradle task from an IDE as well as part of a launch configuration
> and then start solr with a debugger so that it attaches right to your
> sources...
> 
> The benefits of this are enormous. For instance you can shuffle source
> modules around and give them a more logical structure (the assembly
> takes care of where they're placed in the output). You can clearly see
> which components go where and how they're put together. Even now it's
> fairly clear how the distribution is put together, even if it's still
> not ideal:
> 
> https://github.com/apache/lucene-solr/blob/gradle-master/solr/packaging/build.gradle#L50-L92
> 
> I realize it's hard to get people out of their comfort zone (I'm a
> prime example of this) but give this packaging thing a try, I don't
> think you'll be disappointed.
> 
> This said, I did try to give people some backward compatibility with
> "resolve" target which copies over JARs to their previous source-tree
> locations...
> 
> gradlew resolve
> 
> I didn't verify it thoroughly though and the complexity of even trying
> to mimic the ant build here is hair-raising:
> 
> https://github.com/apache/lucene-solr/blob/gradle-master/gradle/ant-compat/resolve.gradle
> 
> Dawid
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

David Smiley:

gradlew -p solr/packaging assemble

couple of things:
1> The place you run bin/solr from is different
2> I didn’t need to specify the -p parameter and it defaulted to 
‘...solr/packaging/build/solr-9.0.0-SNAPSHOT', FWIW.

Once I got over having to switch to a different dir than I was accustomed to, I 
realized that by not mixing the build output with source, things are _much_ 
cleaner. After a “gradlew clean”, the packaging directory only contains a 
build.gradle file.

> On Dec 3, 2019, at 4:48 AM, Jan Høydahl  wrote:
> 
> gradlew -p solr/packaging assemble

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

Here's an example why I think toying with this gradle build is fun.
This adds a build fragment that cleans up the application of forbidden API
rules (they're applied automatically when a module consumes a certain
dependency).
Note how this is isolated from everything else, yet cross-cuts across
all modules to apply
it consistently and without too much boilerplate.

https://github.com/apache/lucene-solr/commit/6461909129c806a8c71d5e4d053061640fce4554

Dawid


D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

> Great work Dawid. Just trying out solr tests now.

A full suite of solr and lucene takes ~50 minutes... And I think it's
pretty much the same with ant now (?).

> I see that it is not very smart in parallel scheduling. But that is probably 
> something we can improve over time?

Maybe. I don't think it's going to improve things significantly
though. It'd take some deeper restructuring of the codebase to get
them to run faster.

> Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef 
> for faster runs?

Fixing the seed should be only done if you want to run two consecutive
times with exactly the same combination of components. Don't fix it
locally forever, please. Those randomizations are really effective to
discover odd stuff (even at the JVM bugs level...).

> We probably don’t need to randomize everything everywhere every time?

Perhaps some of the stuff we do statically at test suite level
(LuceneTestCase etc.) is too eager... I didn't look into this. But
overall I disagree with Mark here -- I still have faith in that
randomization of components has value to it.

> Also, I like your moving of build files from the source tree. Will soon get 
> used to gradlew -p solr/packaging assemble.
> Wonder how this will work with Admin UI changes. Hopefully you can just run a 
> new assemble and reload browser on each change.

If you wish to rerun assembly with a server running in the background
it may be tricky because some files may be locked by live processes
(applies to Windows in particular) and some files generated by the
server (logs, etc.) may be removed by the sync (since it's a sync, not
a copy). Working on "live" files served via http is indeed going to be
tricky. You could always work on those files inside the assembled
distribution and copy them over to the sources once done... not a
solution but a workaround.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 
> --max-workers=4
>
> the build time is XXX.

Yeah... you can tell I'm a non-linear e-mail writer, huh? s/XXX/11m 43s/.

For what it's worth, the "ant test" on Lucene takes an even longer
time on that machine:

BUILD SUCCESSFUL
Total time: 14 minutes 1 second

And I think it's a similar situation with Solr tests (I checked a
while ago, didn't try now).

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

2019-12-03 Thread Jan Høydahl

Great work Dawid. Just trying out solr tests now.
I see that it is not very smart in parallel scheduling. But that is probably 
something we can improve over time?
Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef for 
faster runs?
We probably don’t need to randomize everything everywhere every time?

Also, I like your moving of build files from the source tree. Will soon get 
used to gradlew -p solr/packaging assemble.
Wonder how this will work with Admin UI changes. Hopefully you can just run a 
new assemble and reload browser on each change.

Jan

> 3. des. 2019 kl. 09:35 skrev Dawid Weiss :
> 
>> gradlew -p lucene test
>> That took my machine 27 minutes!  I can see it used like 7 threads when I'd 
>> rather it use 3-4.  That's probably why.
> 
> I capped the execution of parallel tests to this
> (gradle/testing/defaults-tests.gradle):
> 
>  // Set up default parallel execution limits.
>  maxParallelForks = (int) Math.max(1,
> Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0))
> 
> but this limit applies per-project and gradle runs with N workers in
> parallel mode (where N equals the number of cores). For example, on my
> 8-core Linux machine (SSD drive) the test time for
> 
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894
> 
> is 11m 32s. If I limit max workers to 4:
> 
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 
> --max-workers=4
> 
> the build time is XXX.
> 
> It is obvious we need to fine-tune this but it's not obvious what the
> defaults should be because there are many dimensions to performance:
> I/O congestion, CPU cores, overall memory bandwidth and operating
> system (underlying filesystem implementation mostly, I believe). Also,
> as Mark pointed out before, the test runner is slightly different in
> Gradle and doesn't load-balance tests too efficiently (no work
> stealing).
> 
> Finally, all the above also doesn't mean we can only think of
> improving gradle performance and not tests themselves... Some of them
> are just (very) slow. :)
> 
> If you're willing to experiment then try to run with a varying number
> of workers (and identical seed to keep the tests the same). Then see
> which one turns out to be the best for you (and report back). I think
> it'll be half the number of cores (effectively physical cores) unless
> you have a very beefy machine when memory bandwidth and I/O comes into
> play.
> 
> gradlew --max-workers N -Ptests.seed=deadbeef
> 
>> gradlew :helpAnt
>> This failed: -- java.io.FileNotFoundException
> 
> Corrected, apologies.
> 
> Dawid
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr, gradle build and working with files compiled into the source tree

David asked this question regarding the gradle build and I think it
may be a longer debate so I start a new thread:

> What gradlew command is the equivalent of "ant server" for Solr? I wish to
> use "bin/solr".

Here comes my very subjective opinion: let's get rid of the
possibility to run various commands right on the source tree... Let me
explain why.

I had a *very* hard time even understanding what is where in the Solr
source tree and whether it's versioned, compiled or generated. I ended
up using a clean repository, compiled repository and a distribution
ZIP and diffing against each other. This can't be a sane thing.

Having compiled files in the source tree makes everything lousy - from
complex rules in .gitignore files to extreme hacks inside build files
(ant, gradle or otherwise) to make this copying and syncing work.

What's the alternative? Well, one of such alternatives is a
"packaging" project that puts together the binary project
distribution. With gradle this is lighting fast, incremental and
always correct (files are synced from source folders and dependencies
-- they are not copied). For example, putting a Solr distribution from
scratch takes 2 minutes on my machine (first run) but subsequent run
(with no changes to anything) is nearly instantaneous (5 secs).

If you remove a file or a dependency in the source and rebuild the package with:

gradlew -p solr/packaging assemble

then the assembled output under:

solr/packaging/build/solr-9.0.0-SNAPSHOT

will always have up-to-date state of a true distribution. You can run
this gradle task from an IDE as well as part of a launch configuration
and then start solr with a debugger so that it attaches right to your
sources...

The benefits of this are enormous. For instance you can shuffle source
modules around and give them a more logical structure (the assembly
takes care of where they're placed in the output). You can clearly see
which components go where and how they're put together. Even now it's
fairly clear how the distribution is put together, even if it's still
not ideal:

https://github.com/apache/lucene-solr/blob/gradle-master/solr/packaging/build.gradle#L50-L92

I realize it's hard to get people out of their comfort zone (I'm a
prime example of this) but give this packaging thing a try, I don't
think you'll be disappointed.

This said, I did try to give people some backward compatibility with
"resolve" target which copies over JARs to their previous source-tree
locations...

gradlew resolve

I didn't verify it thoroughly though and the complexity of even trying
to mimic the ant build here is hair-raising:

https://github.com/apache/lucene-solr/blob/gradle-master/gradle/ant-compat/resolve.gradle

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

> gradlew -p lucene test
> That took my machine 27 minutes!  I can see it used like 7 threads when I'd 
> rather it use 3-4.  That's probably why.

I capped the execution of parallel tests to this
(gradle/testing/defaults-tests.gradle):

  // Set up default parallel execution limits.
  maxParallelForks = (int) Math.max(1,
Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0))

but this limit applies per-project and gradle runs with N workers in
parallel mode (where N equals the number of cores). For example, on my
8-core Linux machine (SSD drive) the test time for

./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894

is 11m 32s. If I limit max workers to 4:

./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 --max-workers=4

the build time is XXX.

It is obvious we need to fine-tune this but it's not obvious what the
defaults should be because there are many dimensions to performance:
I/O congestion, CPU cores, overall memory bandwidth and operating
system (underlying filesystem implementation mostly, I believe). Also,
as Mark pointed out before, the test runner is slightly different in
Gradle and doesn't load-balance tests too efficiently (no work
stealing).

Finally, all the above also doesn't mean we can only think of
improving gradle performance and not tests themselves... Some of them
are just (very) slow. :)

If you're willing to experiment then try to run with a varying number
of workers (and identical seed to keep the tests the same). Then see
which one turns out to be the best for you (and report back). I think
it'll be half the number of cores (effectively physical cores) unless
you have a very beefy machine when memory bandwidth and I/O comes into
play.

gradlew --max-workers N -Ptests.seed=deadbeef

> gradlew :helpAnt
> This failed: -- java.io.FileNotFoundException

Corrected, apologies.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

2019-12-02 Thread David Smiley

Really impressive Dawid!  I like how you were able to organize the build
files, especially the hacks and keeping the build.gradle files relatively
straightforward.

The dependency approach is totally good with me.  It addresses the
essential problem with pure Maven which is unknowingly sucking in
transitive dependencies.  Code reviews will continue to make new/updated
dependencies visible as they are today by forcing the commit to include
checksums 'n such.

gradlew -p lucene test
That took my machine 27 minutes!  I can see it used like 7 threads when I'd
rather it use 3-4.  That's probably why.

gradlew :helpAnt
This failed: -- java.io.FileNotFoundException:
/Users/dsmiley/DevSearch/lucene-solr/help/ant-gradle.txt (No such file or
directory)seems like a simple error related to an incomplete rename or
other refactor

What gradlew command is the equivalent of "ant server" for Solr?  I wish to
use "bin/solr".

I'm +1 with you committing to master as soon as you are comfortable.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Dec 2, 2019 at 11:36 AM Dawid Weiss  wrote:

> > - run a single unit test (NOTE: one of the TODO’s is running the test
> multiple times. Gradle defaults to marking a test as “good” once it passes
> and doesn’t re-run it until there are code changes. This can be fixed)
>
> See gradlew :helpTests, Erick -- I corrected some of it today. In short:
>
> You can already run a single test multiple times with tests.iters
> (passed via -P or -D). This works from IDEs as well (because it's the
> runner's code, not gradle). Full 'beasting' mode is more difficult
> because current working directories will overlap. I have some ideas
> how this can be solved.
>
> I don't think we should force tests to re-run if nothing has changed.
> If everything is up-to-date and no properties have changed, the tests
> should skip. It really helps with incremental runs. Please note that
> "nothing has changed" is tricky with gradle because it takes all the
> properties, arguments, etc. passed to the test runner as input -- we
> pick a random test seed on each run so tests should re-run
> automatically because the seed will be different. However, if you run
> the same seed multiple times over (gradlew test -Ptests.seed=deadbeef)
> then it will indeed skip them. Finally, if you wish to force a re-run
> of any task gradle gives you many options to do so (the simplest is to
> add a forced "cleanXXX" of a task named XXX - try it: gradlew
> cleanTest test -Ptests.seed=deadbeef).
>
> I will add this explanation to the tests help file.
>
> I've warned gradle can be frustratingly complex! (but once you learn
> it it's fun ;).
>
> D.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Gradle build effort (respin, please read)

2019-12-02 Thread Dawid Weiss

> - run a single unit test (NOTE: one of the TODO’s is running the test 
> multiple times. Gradle defaults to marking a test as “good” once it passes 
> and doesn’t re-run it until there are code changes. This can be fixed)

See gradlew :helpTests, Erick -- I corrected some of it today. In short:

You can already run a single test multiple times with tests.iters
(passed via -P or -D). This works from IDEs as well (because it's the
runner's code, not gradle). Full 'beasting' mode is more difficult
because current working directories will overlap. I have some ideas
how this can be solved.

I don't think we should force tests to re-run if nothing has changed.
If everything is up-to-date and no properties have changed, the tests
should skip. It really helps with incremental runs. Please note that
"nothing has changed" is tricky with gradle because it takes all the
properties, arguments, etc. passed to the test runner as input -- we
pick a random test seed on each run so tests should re-run
automatically because the seed will be different. However, if you run
the same seed multiple times over (gradlew test -Ptests.seed=deadbeef)
then it will indeed skip them. Finally, if you wish to force a re-run
of any task gradle gives you many options to do so (the simplest is to
add a forced "cleanXXX" of a task named XXX - try it: gradlew
cleanTest test -Ptests.seed=deadbeef).

I will add this explanation to the tests help file.

I've warned gradle can be frustratingly complex! (but once you learn
it it's fun ;).

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle build effort (respin, please read)

2019-12-02 Thread Erick Erickson

First, a slight correction. I deserve very little credit here,
Dawid did the heavy lifting, a huge shout-out to him! Cheering
from the sidelines is easy ;)

I played around a little with this last night, and AFAIC let’s just
fold it into master. From my perspective, it’s important that we
be able to try using this in baby steps, falling back to ant
whenever there’s something we can’t do in Gradle (yet).

Having more people contribute now that it’s functional will speed
this process.

Last night I was able to:

- build Solr

- run a single unit test (NOTE: one of the TODO’s is running the test multiple 
times. Gradle defaults to marking a test as “good” once it passes and doesn’t 
re-run it until there are code changes. This can be fixed)

- run the full test suite

- start solr with bin/solr start… (NOTE: this is in a slightly different place 
now)

- start Solr and attach a debugger and hit breakpoints.

IOW, I can do my regular work just using Gradle.

+1 to merging with master.

About merging into 8x. Straw-man proposal: we don’t. There are going to be a 
zillion commits on this before we stop using Ant, and merging into 8x after 
they’re all done will be “interesting”. So I propose we target 9.0 for using 
Gradle rather than Ant and do _NOT_ merge any of this into 8x. If the Gradle 
build isn’t ready by the time 9.0 comes out, we can still use Ant and switch 
over later.

> On Dec 2, 2019, at 10:07 AM, Dawid Weiss  wrote:
> 
> Hi folks.
> 
> For those who prefer the short version:
> git clone https://gitbox.apache.org/repos/asf/lucene-solr.git -b gradle-master
> gradlew :help
> gradlew -p lucene test
> gradlew -p solr/packaging assemble
> 
> A longer explanation follows.
> 
> I realize many of you probably got tired of the discussions and issues
> concerning the gradle build. I don't know why this turned so awkward
> instead of exciting (which it is) but I wanted to try to reflect on
> the lessons learned and retry the effort, perhaps with a better
> outcome this time.
> 
> Erick and I spent some time trying to revive what Mark did but we
> failed -- there were just too many mixed concerns and work-in-progress
> changes to reconcile. I wanted something that would be:
> 
> - usable from the first minute,
> - simple (no custom gradle plugins, etc.),
> - divisible into smaller logical chunks so that contributions could be
> smaller and incremental; without the need to work on the whole thing
> for months,
> - backward compatible with ant (because we are forced to use it until
> everything is ported).
> 
> I took some of the things from Cao and Mark but I essentially rewrote
> the build file and split it into logical fragments that configure
> different build aspects:
> 
> https://github.com/apache/lucene-solr/blob/gradle-master/build.gradle#L18-L48
> 
> A good example of the "smaller fragment" approach is a developer aid
> to display the slowest tests at the end of the run -- it's
> self-contained and gracefully separated from anything else in the
> build:
> 
> https://github.com/apache/lucene-solr/blob/gradle-master/gradle/testing/slowest-tests-at-end.gradle
> 
> It was, in fact, possible to isolate all the sore points and problems
> present with the current ant-based build into separate files that
> provide "workarounds" or "hacks" so that gradle works on the same file
> structure as ant. All of these files indicate a potential problem with
> the build itself (dependencies on test classes, non-conventional
> folder naming, etc.) but until we get rid of ant we can't easily fix
> these:
> 
> https://github.com/apache/lucene-solr/tree/gradle-master/gradle/ant-compat
> 
> The gradle-master branch is essentially a fork from the master branch
> with gradle build files layered on top. My (most controversial)
> decision was to embrace transitive dependencies -- I agree with Mark
> that they just can't be managed manually anymore, especially for Solr.
> Gradle has vast possibilities of excluding and configuring
> dependencies in any way we like without the need to enumerate each and
> every package manually. When something is added for the first time or
> upgraded we will have a safety trigger of the lock file and jar
> checksums (I still need to add this particular section though).
> 
> The build works for me just fine: tests run, assembly works, solr
> packaging works. IntelliJ imports the project as it is, without any
> need for custom tuning.
> 
> The question is how do we proceed from here. I can merge master fairly
> often but I think it would make most sense if we folded this in to
> master _as soon as possible_ so that people start doing actual
> development using gradle (and provide real life feedback). Please
> shout out if you have something against it.

Gradle build effort (respin, please read)

2019-12-02 Thread Dawid Weiss

Hi folks.

For those who prefer the short version:
git clone https://gitbox.apache.org/repos/asf/lucene-solr.git -b gradle-master
gradlew :help
gradlew -p lucene test
gradlew -p solr/packaging assemble

A longer explanation follows.

I realize many of you probably got tired of the discussions and issues
concerning the gradle build. I don't know why this turned so awkward
instead of exciting (which it is) but I wanted to try to reflect on
the lessons learned and retry the effort, perhaps with a better
outcome this time.

Erick and I spent some time trying to revive what Mark did but we
failed -- there were just too many mixed concerns and work-in-progress
changes to reconcile. I wanted something that would be:

- usable from the first minute,
- simple (no custom gradle plugins, etc.),
- divisible into smaller logical chunks so that contributions could be
smaller and incremental; without the need to work on the whole thing
for months,
- backward compatible with ant (because we are forced to use it until
everything is ported).

I took some of the things from Cao and Mark but I essentially rewrote
the build file and split it into logical fragments that configure
different build aspects:

https://github.com/apache/lucene-solr/blob/gradle-master/build.gradle#L18-L48

A good example of the "smaller fragment" approach is a developer aid
to display the slowest tests at the end of the run -- it's
self-contained and gracefully separated from anything else in the
build:

https://github.com/apache/lucene-solr/blob/gradle-master/gradle/testing/slowest-tests-at-end.gradle

It was, in fact, possible to isolate all the sore points and problems
present with the current ant-based build into separate files that
provide "workarounds" or "hacks" so that gradle works on the same file
structure as ant. All of these files indicate a potential problem with
the build itself (dependencies on test classes, non-conventional
folder naming, etc.) but until we get rid of ant we can't easily fix
these:

https://github.com/apache/lucene-solr/tree/gradle-master/gradle/ant-compat

The gradle-master branch is essentially a fork from the master branch
with gradle build files layered on top. My (most controversial)
decision was to embrace transitive dependencies -- I agree with Mark
that they just can't be managed manually anymore, especially for Solr.
Gradle has vast possibilities of excluding and configuring
dependencies in any way we like without the need to enumerate each and
every package manually. When something is added for the first time or
upgraded we will have a safety trigger of the lock file and jar
checksums (I still need to add this particular section though).

The build works for me just fine: tests run, assembly works, solr
packaging works. IntelliJ imports the project as it is, without any
need for custom tuning.

The question is how do we proceed from here. I can merge master fairly
often but I think it would make most sense if we folded this in to
master _as soon as possible_ so that people start doing actual
development using gradle (and provide real life feedback). Please
shout out if you have something against it.

In the meantime, if you're familiar with gradle and would like to help
then here is a list of things that are missing from the build:
https://issues.apache.org/jira/browse/LUCENE-9077

Some of them are simple, some are more difficult. The list is very
likely not close to complete. All of these items require some
understanding of how gradle works. Gradle's documentation is very good
but my experience tells me gradle can be confusing as hell at the
beginning nonetheless - if you don't understand something in the
current build let me know, I'll try to help.

Silent consensus is great but we'd really love to hear from you. Try
it and let us know what you think.

(Erick &) Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

2019-11-16 Thread Michael Sokolov

I would start by looking at this:
https://docs.gradle.org/current/userguide/application_plugin.html?

On Sat, Nov 16, 2019 at 5:02 PM Erick Erickson  wrote:
>
> Hmmm, I’ll have to start looking at this then. There may be two separate 
> issues here:
>
> 1> for developers, a convenient way to just compile-and-go, i.e. starting a 
> Solr instance after a build. I certainly won’t attempt to insist that I can 
> do it the same way we did in the Ant world if there’s a better “gradle way”, 
> and it sounds like there is.
>
> 2> The packageDist task seems incomplete given that when I tried that and 
> unpacked the tarball, the usual way to start Solr was missing (no 
> solr/bin/solr script to be specific).
>
> <1> is more important IMO short term if we’re going to try to get people to 
> kick the tires in their daily work, <2> is a blocker before cutting over 
> completely some time in the future.
>
> And I’ll happily take a break from this and see if I can figure out why the 
> ant test fails for TestKoreanTokenizer. That’s what I was trying to do when I 
> got sidetracked. Figured “Well, let’s merge things from master just to 
> eliminate that possibility” and all hell broke loose.
>
> Thanks again,
> Erick
>
>
>
> > On Nov 16, 2019, at 2:02 PM, Dawid Weiss  wrote:
> >
> >> I’ve really got to disagree here when it comes to Solr. I dive into and 
> >> out of building/running Solr many times a day. Having to package it all up 
> >> then explode it just to check something and/or explore a problem is a 
> >> waste.
> >
> > It's not really like this. With gradle you can set up a packaging
> > folder (under build/) so that it is synced with any required resources
> > (and this is done only if necessary, incrementally). Creating a
> > compressed package is just the finalizing step from that location. All
> > this is lighting fast.
> >
> > I don't want to fight people who work with Solr -- I don't do much
> > development with Solr these days and like I said I don't even
> > understand why there are so many different "packaging" ant tasks and
> > what they actually do. My experience just tells me that keeping build
> > files mixed with source locations is not really convenient. Gradle
> > solves it in an elegant way (but sure - it's a subjective point of
> > view).
> >
> >> And at this point I don’t see how to start Solr at all. Even if I execute 
> >> the “packageDist” task and produce a tgz and zip file, when those are 
> >> exploded the ../solr/bin directory isn’t there, so no ../solr/bin/solr 
> >> script to start an instance etc.
> >
> > I'll take a look at it this weekend, Erick, but you need to give me some 
> > time.
> >
> >> Going through the cycle of having to package it up and explode it just to 
> >> do that is unacceptable.
> >
> > I need to take a look at the branch status vs. master first (vide your
> > remarks about tests failing). Then I need to understand how to package
> > Solr (from the various bits of ant code) and finally write this in
> > gradle. I'll let you know how far I can get this over the weekend.
> >
> > Dawid
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

Hmmm, I’ll have to start looking at this then. There may be two separate issues 
here:

1> for developers, a convenient way to just compile-and-go, i.e. starting a 
Solr instance after a build. I certainly won’t attempt to insist that I can do 
it the same way we did in the Ant world if there’s a better “gradle way”, and 
it sounds like there is.

2> The packageDist task seems incomplete given that when I tried that and 
unpacked the tarball, the usual way to start Solr was missing (no solr/bin/solr 
script to be specific).

<1> is more important IMO short term if we’re going to try to get people to 
kick the tires in their daily work, <2> is a blocker before cutting over 
completely some time in the future.

And I’ll happily take a break from this and see if I can figure out why the ant 
test fails for TestKoreanTokenizer. That’s what I was trying to do when I got 
sidetracked. Figured “Well, let’s merge things from master just to eliminate 
that possibility” and all hell broke loose.

Thanks again,
Erick

> On Nov 16, 2019, at 2:02 PM, Dawid Weiss  wrote:
> 
>> I’ve really got to disagree here when it comes to Solr. I dive into and out 
>> of building/running Solr many times a day. Having to package it all up then 
>> explode it just to check something and/or explore a problem is a waste.
> 
> It's not really like this. With gradle you can set up a packaging
> folder (under build/) so that it is synced with any required resources
> (and this is done only if necessary, incrementally). Creating a
> compressed package is just the finalizing step from that location. All
> this is lighting fast.
> 
> I don't want to fight people who work with Solr -- I don't do much
> development with Solr these days and like I said I don't even
> understand why there are so many different "packaging" ant tasks and
> what they actually do. My experience just tells me that keeping build
> files mixed with source locations is not really convenient. Gradle
> solves it in an elegant way (but sure - it's a subjective point of
> view).
> 
>> And at this point I don’t see how to start Solr at all. Even if I execute 
>> the “packageDist” task and produce a tgz and zip file, when those are 
>> exploded the ../solr/bin directory isn’t there, so no ../solr/bin/solr 
>> script to start an instance etc.
> 
> I'll take a look at it this weekend, Erick, but you need to give me some time.
> 
>> Going through the cycle of having to package it up and explode it just to do 
>> that is unacceptable.
> 
> I need to take a look at the branch status vs. master first (vide your
> remarks about tests failing). Then I need to understand how to package
> Solr (from the various bits of ant code) and finally write this in
> gradle. I'll let you know how far I can get this over the weekend.
> 
> Dawid
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

2019-11-16 Thread Dawid Weiss

> I’ve really got to disagree here when it comes to Solr. I dive into and out 
> of building/running Solr many times a day. Having to package it all up then 
> explode it just to check something and/or explore a problem is a waste.

It's not really like this. With gradle you can set up a packaging
folder (under build/) so that it is synced with any required resources
(and this is done only if necessary, incrementally). Creating a
compressed package is just the finalizing step from that location. All
this is lighting fast.

I don't want to fight people who work with Solr -- I don't do much
development with Solr these days and like I said I don't even
understand why there are so many different "packaging" ant tasks and
what they actually do. My experience just tells me that keeping build
files mixed with source locations is not really convenient. Gradle
solves it in an elegant way (but sure - it's a subjective point of
view).

> And at this point I don’t see how to start Solr at all. Even if I execute the 
> “packageDist” task and produce a tgz and zip file, when those are exploded 
> the ../solr/bin directory isn’t there, so no ../solr/bin/solr script to start 
> an instance etc.

I'll take a look at it this weekend, Erick, but you need to give me some time.

> Going through the cycle of having to package it up and explode it just to do 
> that is unacceptable.

I need to take a look at the branch status vs. master first (vide your
remarks about tests failing). Then I need to understand how to package
Solr (from the various bits of ant code) and finally write this in
gradle. I'll let you know how far I can get this over the weekend.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

bq. For most things (running tests, sha checks, etc.) the dependency JARs don't 
have to be copied to the project (and why should they be). 

Sure.

bq. The only exception is packaging final distribution. 

I’ve really got to disagree here when it comes to Solr. I dive into and out of 
building/running Solr many times a day. Having to package it all up then 
explode it just to check something and/or explore a problem is a waste.

And at this point I don’t see how to start Solr at all. Even if I execute the 
“packageDist” task and produce a tgz and zip file, when those are exploded the 
../solr/bin directory isn’t there, so no ../solr/bin/solr script to start an 
instance etc. 

So what it looks like to me is that the work has been done on running tests (a 
big job to be sure), but there’s nothing yet that’ll allow us to actually run 
Solr, either in a distro or as part of development. Perhaps making it work for 
the distro will also fix the development need. Of course I may be missing some 
obvious target, wouldn’t be the first time.

I’m open to working on Solr differently than I’m accustomed to, but I have to 
be able to make a change to Solr and then start Solr using that change 
conveniently. Going through the cycle of having to package it up and explode it 
just to do that is unacceptable.


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

2019-11-16 Thread Uwe Schindler

+1

Uwe

Am November 16, 2019 4:09:57 PM UTC schrieb Dawid Weiss :
>I don't think the details of how gradle locates (or stores) a
>particular
>dependency matter much. As long as the dependency is resolvable (and
>the
>signature matches the one initially added to the repository) we're
>fine.
>
>For most things (running tests, sha checks, etc.) the dependency JARs
>don't
>have to be copied to the project (and why should they be). The
>only exception is packaging final distribution.
>
>The way we currently handle things in ant/ ivy (copying to lib/ folders
>local to the project) is only there because of technological legacy. I
>don't think we should stick to it.
>
>Dawid
>
>On Sat, Nov 16, 2019 at 2:16 PM Martin Gainty 
>wrote:
>
>> good catch on what happens when my jar is not located in specified
>lib
>> folder?
>>
>> public repositories:
>> ==
>> all jars and model declarators should be discoverable in one of these
>> public repositories
>> maven  https://central.maven.org/
>> anthttps://repo1.maven.org/maven2/ant/
>> gradle   https://repo.maven.apache.org/maven2/
>>
>> local repositories:
>> =
>> during a build you will find jars (or redirects) being downloaded to
>local
>> repository such as
>> maven $user.home/.m2/repository
>> gradle $user.home/.gradle/caches
>> ant $user.home/.ant/lib
>>
>> but be aware gradle likes to reference module name instead of using
>gav
>> maven coords
>> so say you are compiling minecraft
>>
>> $user.home/.gradle/caches/minecraft/net/minecraftforge/forge/
>>
>> be aware the label that gradle will reference to fetch from caches is
>'minecraft'
>> but adding 'mincecraft' to gradle varies based on which ide you
>use..reference:https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project
>>
>>
>>
><https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project>
>> android - How to include module to the project? - Stack Overflow
>>
><https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project>
>> Clone the repository which you want to be included as a module.
>Provide
>> the path of your cloned repository. Now suppose if I want to include
>> Calendar module to my project.
>> stackoverflow.com
>>
>> resolving version:
>> =
>> plugins and dependency without versions in maven default to latest so
>in
>> maven land
>> com.erickson
>> erick-1.0
>> -1.1
>> 
>>  com.erickson
>> erick
>> 
>> would pull 1.1 version (latest-version) from central repository(s)
>>
>> when version not specified gradle will apparently resolves to newest
>> (latest modification date) e.g.
>> com.erickson
>>  -erick
>>    -1.0 modification date 11/16/2019
>>-1.1 modification date 11/01/2019
>> sans version gradle will auto-select the latest mod-date (version
>of
>> 1.0 version)
>>
>> as this is clear as mud i invite correction
>>
>> does this help?
>> M-
>>
>> --
>> *From:* Erick Erickson 
>> *Sent:* Friday, November 15, 2019 9:44 PM
>> *To:* dev@lucene.apache.org 
>> *Subject:* Anyone interested in the Gradle build, please comment on
>> SOLR-13915
>>
>> In a nutshell, it doesn’t look like there’s any task that populates:
>>
>> ../solr/server/solr-webapp/webapp/WEB-INF/lib/
>> ../solr/server/lib/ext/
>> ../solr/server/lib/
>>
>> with jar files. So "solr/bin/solr start” simply can’t start since it
>> defines CLASSPATH to point to them. Before I try to mimic the Ant
>build
>> that populates these, should we re-think how these are populated
>and/or
>> where they live?
>>
>> “gradlew assemble” pulls the jars down, but I sure can’t find
>anywhere
>> where that task is defined, and the Gradle javadocs say things like:
>>
>> assemble() - Method in class
>org.gradle.language.assembler.tasks.Assemble
>>
>> Assemble - Class in org.gradle.language.assembler.tasks
>> Translates Assembly language source files into object files.
>>
>> so I haven’t a clue what’s up with that task.
>>
>> Thanks,
>> Erick
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

2019-11-16 Thread Dawid Weiss

I don't think the details of how gradle locates (or stores) a particular
dependency matter much. As long as the dependency is resolvable (and the
signature matches the one initially added to the repository) we're fine.

For most things (running tests, sha checks, etc.) the dependency JARs don't
have to be copied to the project (and why should they be). The
only exception is packaging final distribution.

The way we currently handle things in ant/ ivy (copying to lib/ folders
local to the project) is only there because of technological legacy. I
don't think we should stick to it.

Dawid

On Sat, Nov 16, 2019 at 2:16 PM Martin Gainty  wrote:

> good catch on what happens when my jar is not located in specified lib
> folder?
>
> public repositories:
> ==
> all jars and model declarators should be discoverable in one of these
> public repositories
> maven  https://central.maven.org/
> anthttps://repo1.maven.org/maven2/ant/
> gradle   https://repo.maven.apache.org/maven2/
>
> local repositories:
> =
> during a build you will find jars (or redirects) being downloaded to local
> repository such as
> maven $user.home/.m2/repository
> gradle $user.home/.gradle/caches
> ant $user.home/.ant/lib
>
> but be aware gradle likes to reference module name instead of using gav
> maven coords
> so say you are compiling minecraft
>
> $user.home/.gradle/caches/minecraft/net/minecraftforge/forge/
>
> be aware the label that gradle will reference to fetch from caches is 
> 'minecraft'
> but adding 'mincecraft' to gradle varies based on which ide you 
> use..reference:https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project
>
>
> <https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project>
> android - How to include module to the project? - Stack Overflow
> <https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project>
> Clone the repository which you want to be included as a module. Provide
> the path of your cloned repository. Now suppose if I want to include
> Calendar module to my project.
> stackoverflow.com
>
> resolving version:
> =
> plugins and dependency without versions in maven default to latest so in
> maven land
> com.erickson
> erick-1.0
> -1.1
> 
>  com.erickson
> erick
> 
> would pull 1.1 version (latest-version) from central repository(s)
>
> when version not specified gradle will apparently resolves to newest
> (latest modification date) e.g.
> com.erickson
>  -erick
>-1.0 modification date 11/16/2019
>-1.1 modification date 11/01/2019
> sans version gradle will auto-select the latest mod-date (version of
> 1.0 version)
>
> as this is clear as mud i invite correction
>
> does this help?
> M-
>
> --
> *From:* Erick Erickson 
> *Sent:* Friday, November 15, 2019 9:44 PM
> *To:* dev@lucene.apache.org 
> *Subject:* Anyone interested in the Gradle build, please comment on
> SOLR-13915
>
> In a nutshell, it doesn’t look like there’s any task that populates:
>
> ../solr/server/solr-webapp/webapp/WEB-INF/lib/
> ../solr/server/lib/ext/
> ../solr/server/lib/
>
> with jar files. So "solr/bin/solr start” simply can’t start since it
> defines CLASSPATH to point to them. Before I try to mimic the Ant build
> that populates these, should we re-think how these are populated and/or
> where they live?
>
> “gradlew assemble” pulls the jars down, but I sure can’t find anywhere
> where that task is defined, and the Gradle javadocs say things like:
>
> assemble() - Method in class org.gradle.language.assembler.tasks.Assemble
>
> Assemble - Class in org.gradle.language.assembler.tasks
> Translates Assembly language source files into object files.
>
> so I haven’t a clue what’s up with that task.
>
> Thanks,
> Erick
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Anyone interested in the Gradle build, please comment on SOLR-13915

Thanks both. So the gradle build will create a large number of jar files for 
our modules and, when working late on Friday a fella can find them scattered 
about and think “the dependency jar files must be in there too”. When they’re 
not ;). Just looked again and… there are indeed no jar files for our 
dependencies in the tree after the build. The one advantage to making mistakes 
like this is that I actually _remember_ the issue ;)

Eventually we need two things:
> get enough stuff in the right place for “bin/solr start -e techproducts” to 
> work
> be able to add some libraries to a classpath and compile some custom SolrJ 
> code. For this I always point to “dist” and “dist/solrj-libs”.  

executing “ant server” let’s me do both of these things. There’s also a “dist” 
target, but I’ve never understood why we have both

OK, I’ll see if I can get _ant_ tests to run for a while, the Korean one 
doesn’t work on the gradle branch.

Erick

> On Nov 16, 2019, at 8:16 AM, Martin Gainty  wrote:
> 
> good catch on what happens when my jar is not located in specified lib folder?
> 
> public repositories:
> ==
> all jars and model declarators should be discoverable in one of these public 
> repositories
> maven  https://central.maven.org/
> anthttps://repo1.maven.org/maven2/ant/
> gradle   https://repo.maven.apache.org/maven2/
> 
> local repositories:
> =
> during a build you will find jars (or redirects) being downloaded to local 
> repository such as
> maven $user.home/.m2/repository
> gradle $user.home/.gradle/caches
> ant $user.home/.ant/lib
> 
> but be aware gradle likes to reference module name instead of using gav maven 
> coords
> so say you are compiling minecraft
> $user.home/.gradle/caches/minecraft/net/minecraftforge/forge/
> 
> be aware the label that gradle will reference to fetch from caches is 
> 'minecraft'
> 
> 
> but adding 'mincecraft' to gradle varies based on which ide you 
> use..reference:
> 
> https://stackoverflow.com/questions/53931937/how-to-include-module-to-the-project
> 
> 
> android - How to include module to the project? - Stack Overflow
> Clone the repository which you want to be included as a module. Provide the 
> path of your cloned repository. Now suppose if I want to include Calendar 
> module to my project.
> stackoverflow.com
> 
> resolving version:
> =
> plugins and dependency without versions in maven default to latest so in 
> maven land
> com.erickson
> erick-1.0
> -1.1
> 
>  com.erickson
> erick
> 
> would pull 1.1 version (latest-version) from central repository(s)
> 
> when version not specified gradle will apparently resolves to newest (latest 
> modification date) e.g.
> com.erickson
>  -erick 
>-1.0 modification date 11/16/2019
>-1.1 modification date 11/01/2019
> sans version gradle will auto-select the latest mod-date (version of 1.0 
> version)
> 
> as this is clear as mud i invite correction
> 
> does this help?
> M-
> 
> From: Erick Erickson 
> Sent: Friday, November 15, 2019 9:44 PM
> To: dev@lucene.apache.org 
> Subject: Anyone interested in the Gradle build, please comment on SOLR-13915
>  
> In a nutshell, it doesn’t look like there’s any task that populates:
> 
> ../solr/server/solr-webapp/webapp/WEB-INF/lib/
> ../solr/server/lib/ext/
> ../solr/server/lib/
> 
> with jar files. So "solr/bin/solr start” simply can’t start since it defines 
> CLASSPATH to point to them. Before I try to mimic the Ant build that 
> populates these, should we re-think how these are populated and/or where they 
> live?
> 
> “gradlew assemble” pulls the jars down, but I sure can’t find anywhere where 
> that task is defined, and the Gradle javadocs say things like:
> 
> assemble() - Method in class org.gradle.language.assembler.tasks.Assemble
>  
> Assemble - Class in org.gradle.language.assembler.tasks
> Translates Assembly language source files into object files.
> 
> so I haven’t a clue what’s up with that task.
> 
> Thanks,
> Erick
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Gradle build testing