Re: beasting tests

[Shubham Chaudhary]

I think you could try this:

./gradlew -p lucene/core beast -Ptests.dups=10 --tests
TestByteVectorSimilarityQuery

I confirmed it uses a different seed (long value) for each run by printing
the seed here

in beasting.gradle
.

- Shubham

On Wed, Apr 3, 2024 at 1:49 AM Michael Sokolov  wrote:

> oh! I overlooked tests.dups -- but it doesn't seem to be doing what I
> expected. EG I tried
>
> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
> -Ptests.dups=1000  -Ptests.multiplier=3
>
> and it completes very quickly reporting having run only 13 tests
>
> On Tue, Apr 2, 2024 at 4:14 PM Michael Sokolov  wrote:
> >
> > Is there  a convenient way to run a test multiple times with different
> > seeds? Do I need to write my own script? I feel like I used to be able
> > to do this in IntelliJ, but that option seems to have vanished, and I
> > don't see any such option in gradle testOpts either. I tried
> > -tests.iter but that seems to run the same test multiple times with
> > the same seed,
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Shubham Chaudhary]

Good to know the gradle task is using these files for help and keeping as
.txt too has its own benefits here.

- Shubham

On Fri, Feb 16, 2024 at 2:03 AM Dawid Weiss  wrote:

> One observation though: there are some docs under the help folder (that
>> too as text files) and some under dev-docs. I personally feel all these
>> should be organized into the dev-docs folder (as .md files for readability
>> on github and IDEs), since that was the first place I went to look for any
>> docs.
>>
>
> Expectations are different depending on whom you ask. I prefer those txt
> files over md files, actually - they're easier to cat to terminal/grep,
> etc. And there is a gradle task that shows them - type:
>
> gradlew help
>
> and you'll see what I mean:
>
> This is Lucene's gradle build. See some
> guidelines, ant-equivalent commands etc. under help/*; or type:
>
>   gradlew :helpWorkflow # Typical workflow commands.
>   gradlew :helpTests# Tests, filtering, beasting, etc.
>   gradlew :helpFormatting   # Code formatting conventions.
>   gradlew :helpJvms # Using alternative or EA JVM toolchains.
>   gradlew :helpDeps # Declaring, inspecting and excluding
> dependencies.
>   gradlew :helpForbiddenApis# How to add/apply rules for forbidden
> APIs.
>   gradlew :helpLocalSettings# Local settings, overrides and build
> performance tweaks.
>   gradlew :helpRegeneration # How to refresh generated and derived
> resources.
>   gradlew :helpJmh  # JMH micro-benchmarks.
>   gradlew :helpGit  # Git assistance and guides.
>   gradlew :helpIDEs # IDE support.
>   gradlew :helpPublishing   # Maven and other artifact publishing,
> signing, etc.
>
> The "dev-docs" are mostly concerned with processes and procedures; the
> help/ is, I think, more focused on development/code. But you're entitled
> to your own opinion, of course. ;)
>
> Dawid
>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Shubham Chaudhary]

I see. This makes more sense; it looks to me like it's fine to add this
info here in case anyone else ever requires it.

One observation though: there are some docs under the help folder (that too
as text files) and some under dev-docs. I personally feel all these should
be organized into the dev-docs folder (as .md files for readability on
github and IDEs), since that was the first place I went to look for any
docs.

- Shubham

On Fri, Feb 16, 2024 at 1:32 AM Dawid Weiss  wrote:

>
> There's actually quite a lot of docs related to Lucene tests (my remark
> was meant at the randomizedtesting package) - see here:
> https://github.com/apache/lucene/blob/main/help/tests.txt
>
> The tests.timeoutSuite parameter could be added/ explained there too. I'm
> not sure how much it's needed though - it's the first time anybody ever
> mentioned it. :)
>
> D.
>
> On Thu, Feb 15, 2024 at 8:55 PM Shubham Chaudhary 
> wrote:
>
>> I think this information could sit well within the dev-docs in lucene
>> i.e. "randomized testing in lucene". This would make it more discoverable
>> as well and there is already a lack of proper docs as Dawid pointed?. We
>> could add some references to docs like randomized testing core concepts
>> <https://github.com/randomizedtesting/randomizedtesting/wiki/Core-Concepts>.
>> If this idea makes sense I could try to write some doc around it and open a
>> PR. I would like to know your thoughts on this? Thanks!
>>
>> - Shubham
>>
>> On Thu, Feb 15, 2024 at 10:23 PM Dawid Weiss 
>> wrote:
>>
>>>
>>> Sorry, the docs are not the best, I know.
>>>
>>> It's documented here -
>>>
>>> https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/SysGlobals.java#L186-L197
>>>
>>> So:
>>>
>>> 1) if you pass tests.timeoutSuite=1000 this changes the default value
>>> for all classes that don't define any explicit timeout using an annotation;
>>> classes that do have an annotation,
>>> use the annotation's value,
>>> 2) if you pass tests.timeoutSuite=1000! then this overrides everything -
>>> the default value and all annotations.
>>>
>>> I vaguely recall option (2) was added specifically for nightlies which
>>> bumped the iteration multiplier - this affected tests that normally ran
>>> fairly fast
>>> but during nightly runs could run slower than anticipated.
>>>
>>> D.
>>>
>>>
>>> On Thu, Feb 15, 2024 at 3:18 PM David Smiley  wrote:
>>>
>>>> Oh; I didn't know that took precedence -- makes sense.  Hopefully a
>>>> test subclass (like SolrTestCase) could override it as well.
>>>>
>>>> On Mon, Feb 12, 2024 at 2:09 PM Dawid Weiss 
>>>> wrote:
>>>> >
>>>> >
>>>> > You can override the defaults using sysprops in your CI builds -
>>>> >
>>>> > -Ptests.timeoutSuite=1000!
>>>> >
>>>> > takes precedence over any annotations (1 second).
>>>> >
>>>> > Dawid
>>>> >
>>>> > On Mon, Feb 12, 2024 at 7:53 PM David Smiley 
>>>> wrote:
>>>> >>
>>>> >> Looking at LuceneTestCase, I see the annotation from
>>>> RandomizedTesting:
>>>> >> @TimeoutSuite(millis = 2 * TimeUnits.HOUR)
>>>> >> This matches my observations of some builds that timed out, perhaps
>>>> >> some flaky test hanging in Solr (that extends LuceneTestCase).
>>>> >> Looking at this annotation, there is further documentation that the
>>>> >> default can be set via sysprop tests.timeoutSuite.  Wouldn't doing
>>>> >> that make more sense than hard-coding this figure in LuceneTestCase?
>>>> >> For example, I'd like to have a normal/default test run have a low
>>>> >> timeout (10min?) but on a "nightly" run on CI, use much higher.  Not
>>>> 2
>>>> >> hours though; individual tests needing so much should have a
>>>> >> TimeoutSuite applied to them.
>>>> >>
>>>> >> ~ David Smiley
>>>> >> Apache Lucene/Solr Search Developer
>>>> >> http://www.linkedin.com/in/davidwsmiley
>>>> >>
>>>> >> -
>>>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>> >>
>>>>
>>>> -
>>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>>
>>>>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Shubham Chaudhary]

I think this information could sit well within the dev-docs in lucene i.e.
"randomized testing in lucene". This would make it more discoverable as
well and there is already a lack of proper docs as Dawid pointed?. We could
add some references to docs like randomized testing core concepts
.
If this idea makes sense I could try to write some doc around it and open a
PR. I would like to know your thoughts on this? Thanks!

- Shubham

On Thu, Feb 15, 2024 at 10:23 PM Dawid Weiss  wrote:

>
> Sorry, the docs are not the best, I know.
>
> It's documented here -
>
> https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/SysGlobals.java#L186-L197
>
> So:
>
> 1) if you pass tests.timeoutSuite=1000 this changes the default value for
> all classes that don't define any explicit timeout using an annotation;
> classes that do have an annotation,
> use the annotation's value,
> 2) if you pass tests.timeoutSuite=1000! then this overrides everything -
> the default value and all annotations.
>
> I vaguely recall option (2) was added specifically for nightlies which
> bumped the iteration multiplier - this affected tests that normally ran
> fairly fast
> but during nightly runs could run slower than anticipated.
>
> D.
>
>
> On Thu, Feb 15, 2024 at 3:18 PM David Smiley  wrote:
>
>> Oh; I didn't know that took precedence -- makes sense.  Hopefully a
>> test subclass (like SolrTestCase) could override it as well.
>>
>> On Mon, Feb 12, 2024 at 2:09 PM Dawid Weiss 
>> wrote:
>> >
>> >
>> > You can override the defaults using sysprops in your CI builds -
>> >
>> > -Ptests.timeoutSuite=1000!
>> >
>> > takes precedence over any annotations (1 second).
>> >
>> > Dawid
>> >
>> > On Mon, Feb 12, 2024 at 7:53 PM David Smiley 
>> wrote:
>> >>
>> >> Looking at LuceneTestCase, I see the annotation from RandomizedTesting:
>> >> @TimeoutSuite(millis = 2 * TimeUnits.HOUR)
>> >> This matches my observations of some builds that timed out, perhaps
>> >> some flaky test hanging in Solr (that extends LuceneTestCase).
>> >> Looking at this annotation, there is further documentation that the
>> >> default can be set via sysprop tests.timeoutSuite.  Wouldn't doing
>> >> that make more sense than hard-coding this figure in LuceneTestCase?
>> >> For example, I'd like to have a normal/default test run have a low
>> >> timeout (10min?) but on a "nightly" run on CI, use much higher.  Not 2
>> >> hours though; individual tests needing so much should have a
>> >> TimeoutSuite applied to them.
>> >>
>> >> ~ David Smiley
>> >> Apache Lucene/Solr Search Developer
>> >> http://www.linkedin.com/in/davidwsmiley
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Shubham Chaudhary]