Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-20.0.2) - Build # 16220 - Unstable!

[Dawid Weiss]

Must be a JVM bug somewhere (openj9). The stack trace is insane and leads
to jdk internals.

Dawid

On Thu, Apr 18, 2024 at 12:53 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/16220/
> Java: 64bit/openj9/jdk-20.0.2 -XX:-UseCompressedOops -Xgcpolicy:optthruput
>
> 1 tests failed.
> FAILED:  org.apache.lucene.analysis.br
> .TestBrazilianAnalyzer.testWithKeywordAttribute
>
> Error Message:
> java.lang.NullPointerException: Cannot enter synchronized block because
> "this.interruptLock" is null
>
> Stack Trace:
> java.lang.NullPointerException: Cannot enter synchronized block because
> "this.interruptLock" is null
> at
> __randomizedtesting.SeedInfo.seed([7FDF8C00B15A99A5:AAC4AF41A0A774AE]:0)
> at java.base@20.0.2/java.lang.Thread.threadState(Thread.java:2862)
> at java.base@20.0.2
> /java.lang.Thread.isTerminated(Thread.java:2877)
> at java.base@20.0.2
> /java.lang.Thread.getThreadGroup(Thread.java:2026)
> at java.base@20.0.2
> /java.lang.ThreadGroup.enumerate(ThreadGroup.java:448)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.Threads$1.run(Threads.java:116)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.Threads$1.run(Threads.java:113)
> at java.base@20.0.2
> /java.security.AccessController.doPrivileged(AccessController.java:692)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.Threads.doEnumerate(Threads.java:113)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.Threads.getThreads(Threads.java:106)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreads(ThreadLeakControl.java:751)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1600(ThreadLeakControl.java:59)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:489)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at app/org.apache.lucene.test_framework@9.11.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> at app/junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at app/randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at app/randomizedtesting.runner@2.8.1
> 

Re: [apache/lucene] Run failed: Run nightly: buildAndPushRelease and smokeTestRelease.py - main (df154cd)

[Dawid Weiss]

+1. I think the idea of those smoketester checks was to only have it for
actually released versions.

On Fri, Apr 5, 2024 at 1:30 PM Benjamin Trent  wrote:

> Hmm, yeah. Honestly, I am not sure what to do about this either. I am
> going to remove the 9.10.1 versioning from all branches but 9.10 (it's
> there to capture the next bugfix).
>
> I thought I was doing something helpful, but I guess I was a little too
> eager.
>
> On Fri, Apr 5, 2024 at 3:13 AM Dawid Weiss  wrote:
>
>>
>> Hi Ben,
>>
>> This fails in the smoke tester - failed last night too, so it reproduces.
>>
>> https://github.com/apache/lucene/actions/workflows/run-nightly-smoketester.yml
>>
>> I looked it up to getAllLuceneReleases in the smoke tester script, which
>> in turn lists all releases available at:
>> https://archive.apache.org/dist/lucene/java/
>>
>> version 9.10.1 isn't there so it complains with:
>> RuntimeError: tested version=9.10.1 but it was not released?
>>
>> I'm not sure how to deal with yet-unreleased minor versions myself!
>>
>> D.
>>
>> On Thu, Apr 4, 2024 at 4:20 PM Benjamin Trent 
>> wrote:
>>
>>> This seems related to us forgetting to make the back-compat indices &
>>> versions when 9.10.1 was released and me adding them later.
>>>
>>> I have since added the 9.10.1 to Version.java and version.txt in main
>>> and 9x. Now, both main and 9x have the back-compat indices (these changes
>>> were not at the same time, and cause a separate build failure noticed by
>>> Mike M.).
>>>
>>> But this failure commit on
>>> main df154cdc2288a33747edb8849509ea5c3cbf792e, contains both of my changes
>>> (the 9.10.1 version & back compat indices).
>>>
>>> Was there something else forgotten during this bugfix release that we
>>> need to address? I am very new to the back-compat and release logic in
>>> Lucene and I am eager to learn.
>>>
>>> On Thu, Apr 4, 2024 at 9:43 AM Dawid Weiss 
>>> wrote:
>>>
>>>>
>>>> https://github.com/apache/lucene/actions/runs/8548297347/job/23421799032
>>>>
>>>> This smoketester run failed with:
>>>>
>>>> > RuntimeError: tested version=9.10.1 but it was not released?
>>>>
>>>> I guess it's not a hiccup but something recent?
>>>>
>>>> On Thu, Apr 4, 2024 at 3:24 AM Dawid Weiss 
>>>> wrote:
>>>>
>>>>>
>>>>> [image: GitHub] [apache/lucene] Run nightly: buildAndPushRelease and
>>>>> smokeTestRelease.py workflow run
>>>>>
>>>>>   Run nightly: buildAndPushRelease and smokeTestRelease.py: Some jobs
>>>>> were not successful
>>>>>
>>>>> View workflow run
>>>>> <https://github.com/apache/lucene/actions/runs/8548297347>
>>>>>
>>>>> [image: Smoke test release on jdk 21, ubuntu-latest]
>>>>>
>>>>> *Run nightly: buildAndPushRelease and smokeTestRelease.py* / Smoke
>>>>> test release on jdk 21, ubuntu-latest
>>>>> Failed in 9 minutes and 22 seconds
>>>>> [image: annotations for Run nightly: buildAndPushRelease and
>>>>> smokeTestRelease.py / Smoke test release on jdk 21, ubuntu-latest] 1
>>>>> <https://github.com/apache/lucene/actions/runs/8548297347>
>>>>> [image: Smoke test release on jdk 22-ea, ubuntu-latest]
>>>>>
>>>>> *Run nightly: buildAndPushRelease and smokeTestRelease.py* / Smoke
>>>>> test release on jdk 22-ea, ubuntu-latest
>>>>> Cancelled
>>>>> [image: annotations for Run nightly: buildAndPushRelease and
>>>>> smokeTestRelease.py / Smoke test release on jdk 22-ea, ubuntu-latest] 2
>>>>> <https://github.com/apache/lucene/actions/runs/8548297347>
>>>>>
>>>>>
>>>>>
>>>>> —
>>>>> You are receiving this because you are subscribed to this thread.
>>>>> Manage your GitHub Actions notifications
>>>>> <https://github.com/settings/notifications>
>>>>>
>>>>>
>>>>> GitHub, Inc. ・88 Colin P Kelly Jr Street ・San Francisco, CA 94107
>>>>>
>>>>>
>>>>

Re: [apache/lucene] Run failed: Run nightly: buildAndPushRelease and smokeTestRelease.py - main (df154cd)

[Dawid Weiss]

Hi Ben,

This fails in the smoke tester - failed last night too, so it reproduces.
https://github.com/apache/lucene/actions/workflows/run-nightly-smoketester.yml

I looked it up to getAllLuceneReleases in the smoke tester script, which in
turn lists all releases available at:
https://archive.apache.org/dist/lucene/java/

version 9.10.1 isn't there so it complains with:
RuntimeError: tested version=9.10.1 but it was not released?

I'm not sure how to deal with yet-unreleased minor versions myself!

D.

On Thu, Apr 4, 2024 at 4:20 PM Benjamin Trent  wrote:

> This seems related to us forgetting to make the back-compat indices &
> versions when 9.10.1 was released and me adding them later.
>
> I have since added the 9.10.1 to Version.java and version.txt in main and
> 9x. Now, both main and 9x have the back-compat indices (these changes were
> not at the same time, and cause a separate build failure noticed by Mike
> M.).
>
> But this failure commit on main df154cdc2288a33747edb8849509ea5c3cbf792e,
> contains both of my changes (the 9.10.1 version & back compat indices).
>
> Was there something else forgotten during this bugfix release that we need
> to address? I am very new to the back-compat and release logic in Lucene
> and I am eager to learn.
>
> On Thu, Apr 4, 2024 at 9:43 AM Dawid Weiss  wrote:
>
>>
>> https://github.com/apache/lucene/actions/runs/8548297347/job/23421799032
>>
>> This smoketester run failed with:
>>
>> > RuntimeError: tested version=9.10.1 but it was not released?
>>
>> I guess it's not a hiccup but something recent?
>>
>> On Thu, Apr 4, 2024 at 3:24 AM Dawid Weiss 
>> wrote:
>>
>>>
>>> [image: GitHub] [apache/lucene] Run nightly: buildAndPushRelease and
>>> smokeTestRelease.py workflow run
>>>
>>>   Run nightly: buildAndPushRelease and smokeTestRelease.py: Some jobs
>>> were not successful
>>>
>>> View workflow run
>>> <https://github.com/apache/lucene/actions/runs/8548297347>
>>>
>>> [image: Smoke test release on jdk 21, ubuntu-latest]
>>>
>>> *Run nightly: buildAndPushRelease and smokeTestRelease.py* / Smoke test
>>> release on jdk 21, ubuntu-latest
>>> Failed in 9 minutes and 22 seconds
>>> [image: annotations for Run nightly: buildAndPushRelease and
>>> smokeTestRelease.py / Smoke test release on jdk 21, ubuntu-latest] 1
>>> <https://github.com/apache/lucene/actions/runs/8548297347>
>>> [image: Smoke test release on jdk 22-ea, ubuntu-latest]
>>>
>>> *Run nightly: buildAndPushRelease and smokeTestRelease.py* / Smoke test
>>> release on jdk 22-ea, ubuntu-latest
>>> Cancelled
>>> [image: annotations for Run nightly: buildAndPushRelease and
>>> smokeTestRelease.py / Smoke test release on jdk 22-ea, ubuntu-latest] 2
>>> <https://github.com/apache/lucene/actions/runs/8548297347>
>>>
>>>
>>>
>>> —
>>> You are receiving this because you are subscribed to this thread.
>>> Manage your GitHub Actions notifications
>>> <https://github.com/settings/notifications>
>>>
>>>
>>> GitHub, Inc. ・88 Colin P Kelly Jr Street ・San Francisco, CA 94107
>>>
>>>
>>

Re: beasting tests

[Dawid Weiss]

> Thanks for the explanation. It makes sense that we start with a given
> seed and then each iteration is different because it re-uses the same
> Random instance (or whatever static state?) without re-initialization?
>

It doesn't reuse the same random instance - it's not that simple - it
hierarchically forks the random seed between tests, classes, reiterations
etc., and then initializes the context's random so that runs are
predictable and repeatable even if you run a subset of tests for the given
main seed. Random instances are also not reused between threads in an
attempt to make failures reproducible (this, of course, is not fool-proof).

If you care to look, there are more documented examples with somewhat
increasing complexity here:
https://github.com/randomizedtesting/randomizedtesting/tree/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner

In particular, these talk about hierarchical seed propagation:

https://github.com/randomizedtesting/randomizedtesting/blob/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner/Test004MoreRandomness.java
https://github.com/randomizedtesting/randomizedtesting/blob/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner/Test006RepeatingTests.java

D.

Re: [apache/lucene] Run failed: Run nightly: buildAndPushRelease and smokeTestRelease.py - main (df154cd)

[Dawid Weiss]

https://github.com/apache/lucene/actions/runs/8548297347/job/23421799032

This smoketester run failed with:

> RuntimeError: tested version=9.10.1 but it was not released?

I guess it's not a hiccup but something recent?

On Thu, Apr 4, 2024 at 3:24 AM Dawid Weiss  wrote:

>
> [image: GitHub] [apache/lucene] Run nightly: buildAndPushRelease and
> smokeTestRelease.py workflow run
>
>   Run nightly: buildAndPushRelease and smokeTestRelease.py: Some jobs
> were not successful
>
> View workflow run
> <https://github.com/apache/lucene/actions/runs/8548297347>
>
> [image: Smoke test release on jdk 21, ubuntu-latest]
>
> *Run nightly: buildAndPushRelease and smokeTestRelease.py* / Smoke test
> release on jdk 21, ubuntu-latest
> Failed in 9 minutes and 22 seconds
> [image: annotations for Run nightly: buildAndPushRelease and
> smokeTestRelease.py / Smoke test release on jdk 21, ubuntu-latest] 1
> <https://github.com/apache/lucene/actions/runs/8548297347>
> [image: Smoke test release on jdk 22-ea, ubuntu-latest]
>
> *Run nightly: buildAndPushRelease and smokeTestRelease.py* / Smoke test
> release on jdk 22-ea, ubuntu-latest
> Cancelled
> [image: annotations for Run nightly: buildAndPushRelease and
> smokeTestRelease.py / Smoke test release on jdk 22-ea, ubuntu-latest] 2
> <https://github.com/apache/lucene/actions/runs/8548297347>
>
>
>
> —
> You are receiving this because you are subscribed to this thread.
> Manage your GitHub Actions notifications
> <https://github.com/settings/notifications>
>
>
> GitHub, Inc. ・88 Colin P Kelly Jr Street ・San Francisco, CA 94107
>
>

Re: beasting tests

[Dawid Weiss]

> Now I just need to understand why the test failure is no longer
> reproducing lol.
>

This is indeed the hard part!


> Also it's mildly confusing that when you specify tests.iters it prints a
> single test seed if it is actually going to use many different ones?
>

It prints a single seed because it starts from that seed (the static
initialization, that is). But each test has its own starting point derived
from the main seed and the test name (if I recall right). So when you pass
tests.iters=100 and run a single test, any random call in that test
(excluding static hooks and static blocks) should be different on each
iteration. You can try it by adding:

assumeTrue("", RandomizedTest.randomBoolean());

For example (I added it to TestSearch.java):

./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
...
:lucene:core:test (SUCCESS): 100 test(s), 52 skipped

if you modify this to assertTrue, you'll get to see the hierarchical seed
chain for each test that failed - note the first part is constant, the
second is derived for each test iteration:

./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
-Ptests.seed=deadbeef

and two example failures:

  - org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:3EDB7869EFFD5034]} (:lucene:core)
Test output:
/Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:3EDB7869EFFD5034]}" -Ptests.jvms=4
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
-XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
-Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512

  - org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:F44F3D10E8B98D27]} (:lucene:core)
Test output:
/Users/dweiss/work/lucene/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.TestSearch.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.TestSearch.testSearch
{seed=[DEADBEEF:F44F3D10E8B98D27]}" -Ptests.jvms=4
"-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC
-XX:ActiveProcessorCount=1" -Ptests.seed=deadbeef -Ptests.iters=100
-Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=512

If you'd like to repeat tests with *the same* starting seed for each test,
you need to pass the full chain, including the second part of the seed. For
example, this will fail 100 times (and not approximately 50% of the times):

./gradlew -p lucene/core test --tests TestSearch -Ptests.iters=100
-Ptests.seed=deadbeef:F44F3D10E8B98D27

It may seem a bit complicated but it really isn't... I hope!  And for 99%
of tests, you'd probably rerun with the first part of the seed and it'd be
sufficient to locate the problem.

The 'beast' task is a bit different because it physically re-launches the
test infrastructure so if you don't fix the initial seed, each started JVM
will have a different "starting" seed for static initializers and hooks.
This may matter for locale randomization, jvm issues or static initializers
that rely on randomness. But most isolated test methods should only rely on
their starting seed (not the "global starting seed").

Dawid

Re: beasting tests

[Dawid Weiss]

This section of the help file for testing explains the difference between
'beast', 'test' and various reiteration methods -

https://github.com/apache/lucene/blob/main/help/tests.txt#L89-L123

In *most* cases, tests.iters will be just as good as beasting (and much
faster). The only difference is when you want class-level settings to be
randomized differently (static initializers, for example).

D.

On Tue, Apr 2, 2024 at 10:54 PM Shubham Chaudhary 
wrote:

> I think you could try this:
>
> ./gradlew -p lucene/core beast -Ptests.dups=10 --tests
> TestByteVectorSimilarityQuery
>
> I confirmed it uses a different seed (long value) for each run by printing
> the seed here
> 
> in beasting.gradle
> 
> .
>
> - Shubham
>
> On Wed, Apr 3, 2024 at 1:49 AM Michael Sokolov  wrote:
>
>> oh! I overlooked tests.dups -- but it doesn't seem to be doing what I
>> expected. EG I tried
>>
>> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
>> -Ptests.dups=1000  -Ptests.multiplier=3
>>
>> and it completes very quickly reporting having run only 13 tests
>>
>> On Tue, Apr 2, 2024 at 4:14 PM Michael Sokolov 
>> wrote:
>> >
>> > Is there  a convenient way to run a test multiple times with different
>> > seeds? Do I need to write my own script? I feel like I used to be able
>> > to do this in IntelliJ, but that option seems to have vanished, and I
>> > don't see any such option in gradle testOpts either. I tried
>> > -tests.iter but that seems to run the same test multiple times with
>> > the same seed,
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: [JENKINS] Lucene » Lucene-NightlyTests-main - Build # 1315 - Still Unstable!

[Dawid Weiss]

I think gradle may not be able to filter out this test - it is the test's
name in JUnit and, in theory, it should be possible to filter it out, JUnit
4 is fairly basic in terms of test filtering and tools have their own
approach to this - I guess it's a corner case somewhere.

D.

On Tue, Apr 2, 2024 at 4:11 PM Michael McCandless 
wrote:

> Hmm this failure looks not great.
>
> I tried the "Reproduce with:" for one of the failures (see below) but it
> fails to run any tests at all?  Maybe because of the cool parameterized
> testing we now have for our back compat tests?  If I remove the "{...}"
> pattern then the failures do repro.
>
> ./gradlew :lucene:backward-codecs:test --tests
> "org.apache.lucene.backward_index.TestBinaryBackwardsCompatibility.testSearchOldIndex
> {Lucene-Version:9.10.1; Pattern: unsupported.%1$s-cfs.zip}" -Ptests.jvms=4
> -Ptests.jvmargs= -Ptests.seed=AED171B219\
> 72F50D -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.gui=true
> -Ptests.file.encoding=ISO-8859-1
> -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/test-data/enwiki.random.lines.txt
> -Ptests.vectorsize=256
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Apr 2, 2024 at 4:52 AM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> Build:
>> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/1315/
>>
>> 6 tests failed.
>> FAILED:
>> org.apache.lucene.backward_index.TestBinaryBackwardsCompatibility.testSearchOldIndex
>> {Lucene-Version:9.10.1; Pattern: unsupported.%1$s-cfs.zip}
>>
>> Error Message:
>> java.lang.AssertionError: Index name 9.10.1 not found:
>> unsupported.9.10.1-cfs.zip
>>
>> Stack Trace:
>> java.lang.AssertionError: Index name 9.10.1 not found:
>> unsupported.9.10.1-cfs.zip
>> at
>> __randomizedtesting.SeedInfo.seed([AED171B21972F50D:E4679B8937FD59F]:0)
>> at junit@4.13.1/org.junit.Assert.fail(Assert.java:89)
>> at junit@4.13.1/org.junit.Assert.assertTrue(Assert.java:42)
>> at junit@4.13.1/org.junit.Assert.assertNotNull(Assert.java:713)
>> at
>> org.apache.lucene.backward_index.BackwardsCompatibilityTestBase.setUp(BackwardsCompatibilityTestBase.java:145)
>> at
>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
>> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
>> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
>> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
>> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
>> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> at junit@4.13.1
>> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>> at randomizedtesting.runner@2.8.1
>> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
>> 

Re: beasting tests

[Dawid Weiss]

>
> ./gradlew -p lucene/core test --tests TestByteVectorSimilarityQuery
> -Ptests.dups=1000  -Ptests.multiplier=3
>
> and it completes very quickly reporting having run only 13 tests
>

The task is called 'beast', not 'test', Mike.

D.

Re: Lucene 10

[Dawid Weiss]

>
> [...] but Adrien I don't honestly believe anyone who is
> paying attention thinks that is what you have been doing!


+1. I wish I were procrastinating as productively!

D.

Re: [JENKINS] Lucene » Lucene-Coverage-main - Build # 1065 - Still Failing!

[Dawid Weiss]

I did turn off the security manager for coverage runs, a workaround but
better than none.

On Sun, Mar 10, 2024 at 6:11 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/1065/
>
> All tests passed
>
> Build Log:
> [...truncated 97716 lines...]
> BUILD FAILED in 10m 51s
> 318 actionable tasks: 318 executed
>
> Publishing build scan...
> https://ge.apache.org/s/3a3dhos6frshe
>
> Build step 'Invoke Gradle script' changed build result to FAILURE
> Build step 'Invoke Gradle script' marked build as failure
> Archiving artifacts
> Recording test results
> [Checks API] No suitable checks publisher found.
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: Query about the GitHub statistics for Lucene

[Dawid Weiss]

Perhaps this is what you meant by 'gh' but wanted to mention it -
https://github.com/apache/lucene/pulse/monthly

On Tue, Mar 5, 2024 at 4:34 PM Chris Hegarty
 wrote:

>
> > On 5 Mar 2024, at 13:26, Robert Muir  wrote:
> >
> > On Tue, Mar 5, 2024 at 4:50 AM Chris Hegarty
> >  wrote:
> >> It appears that there is no GH activity for 2024! Clearly this is
> incorrect. I’ve yet to track down what’s going on with this. Familiar to
> anyone here?
> >>
> >
> > Last time I looked at this, it appeared it is looking at the incorrect
> > github repositories, for example https://github.com/apache/lucene-solr
> > and not https://github.com/apache/lucene
>
> Ah, that could explain it!!
>
> I’ll try to take a look at what repo those report stats are generated
> from, and how we might be able to get that updated. Mostly for convenience,
> and also having a single source of truth.
>
> Anyway, thankfully git and GH are good enough to get the kind of basic
> stats we typically want - just that it’s not as clear when comparing to
> previously gathered stats. Well… commits are commits, and counting PRs
> should not result in different numbers, but you know ... ;-)
>
> Thanks,
> -Chris.
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [VOTE] Release PyLucene 9.10.0-rc1

[Dawid Weiss]

+1.

On Wed, Feb 21, 2024 at 10:50 PM Andi Vajda  wrote:

>
> The PyLucene 9.10.0 (rc1) release tracking the recent release of
> Apache Lucene 9.10.0 is ready.
>
> A release candidate is available from:
> https://dist.apache.org/repos/dist/dev/lucene/pylucene/9.10.0-rc1/
>
> PyLucene 9.10.0 is built with JCC 3.14, included in these release
> artifacts.
>
> Apart from the catch-up to Lucene 9.10.0, the other major new feature in
> this release candidate is that JCC can now generate a setup.py file
> instead
> of calling Setup() directly. This makes it possible to use modern Python
> packaging without falling afoul of "python setup.py install" being
> deprecated. Setup.py itself is not deprecated, only some of its associated
> commands are; see [1] for more information about this.
>
> In PyLucene's Makefile, there now is a new MODERN_PACKAGING variable,
> which
> can be set to true so that "python -m build" and "python -m pip install"
> are
> used for building and installing PyLucene.
>
> JCC 3.14 supports Python 3.3 up to Python 3.12.
> PyLucene may also be built with Python 2 but this configuration is no
> longer
> tested.
>
> Please vote to release these artifacts as PyLucene 9.10.0.
> Anyone interested in this release can and should vote !
>
> Thanks !
>
> Andi..
>
> ps: the KEYS file for PyLucene release signing is at:
> https://dist.apache.org/repos/dist/release/lucene/pylucene/KEYS
> https://dist.apache.org/repos/dist/dev/lucene/pylucene/KEYS
>
> pps: here is my +1
>
> [1]
> https://packaging.python.org/en/latest/discussions/setup-py-deprecated/
>

Re: The future of the PyLucene project

[Dawid Weiss]

Hi Andi,

This time, crickets, the voting thread has been completely quiet.
>

For me - and it's not an excuse at all - you hit winter holidays, I'm
really sorry!


> If the Lucene PMC agrees and no PyLucene users come forward, I propose the
> following:
>- shutdown the PyLucene project
>- fork JCC to my gitlab (https://gitlab.pyicu.org/main) where it can
>  get the occasional fix or improvement before being released to PyPI.
>  JCC has been distributed from PyPI forever,
>https://pypi.org/project/JCC/#history
>  so JCC users shouldn't even notice this...
>

I think open source is mostly about the community and folks coding together
for fun... And not many of us seem to be
able to help you with PyLucene development - I can't, for that matter,
because my Python is really limited.

Your plan sounds good to me. And you'd get more freedom from procedural
release
requirements at Apache too, which sounds  like an added benefit?... :)

I also hope that, regardless of the status of PyLucene and JCC, you remain
with the Lucene project.

Dawid

Re: [Vote] Bump the Lucene main branch to Java 21

[Dawid Weiss]

I'm fine with this requirement.

+1.

On Fri, Feb 23, 2024 at 12:24 PM Chris Hegarty
 wrote:

> Hi,
>
> Since the discussion on bumping the Lucene main branch to Java 21 is
> winding down, let's hold a vote on this important change.
>
> Once bumped, the next major release of Lucene (whenever that will be) will
> require a version of Java greater than or equal to Java 21.
>
> The vote will be open for at least 72 hours (and allow some additional
> time for the weekend) i.e. until 2024-02-28 12:00 UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>
> -Chris.
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Zhang Chao as Lucene committer

[Dawid Weiss]

Congratulations and welcome!

On Tue, Feb 20, 2024 at 6:28 PM Adrien Grand  wrote:

> I'm pleased to announce that Zhang Chao has accepted the PMC's
> invitation to become a committer.
>
> Chao, the tradition is that new committers introduce themselves with a
> brief bio.
>
> Congratulations and welcome!
>
> --
> Adrien
>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Dawid Weiss]

> I found that passing -Ptests.timeoutSuite=500 doesn't have any effect
> that I can see; it didn't interrupt the tests.


It's exactly what I said - without the exclamation mark, this is the
default value of the timeout unless a class has an annotation specifying
other value (which all of Lucene tests do because they inherit from
LuceneTestCase).


> I needed that trailing
> exclamation mark for it to do the interrupt.


Yep, it's an override that takes precedence over annotations.


> majority.  IMO LuceneTestCase shouldn't be declaring a default; it
> should be done in a gradle build file instead.


We could, sure. Or it could be shorter too. I don't have an opinion on this.


> Then, configuration
> for a build server (I'm thinking of Crave.io used by Solr PRs) can
> specify like 10 minutes because otherwise an unlucky build hogs that
> 96 core server for hours.  Until then, I'll use an exclamation mark
> for that server's config which isn't quite ideal but it's adequate.
>

+1.

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Dawid Weiss]

>
> One observation though: there are some docs under the help folder (that
> too as text files) and some under dev-docs. I personally feel all these
> should be organized into the dev-docs folder (as .md files for readability
> on github and IDEs), since that was the first place I went to look for any
> docs.
>

Expectations are different depending on whom you ask. I prefer those txt
files over md files, actually - they're easier to cat to terminal/grep,
etc. And there is a gradle task that shows them - type:

gradlew help

and you'll see what I mean:

This is Lucene's gradle build. See some
guidelines, ant-equivalent commands etc. under help/*; or type:

  gradlew :helpWorkflow # Typical workflow commands.
  gradlew :helpTests# Tests, filtering, beasting, etc.
  gradlew :helpFormatting   # Code formatting conventions.
  gradlew :helpJvms # Using alternative or EA JVM toolchains.
  gradlew :helpDeps # Declaring, inspecting and excluding
dependencies.
  gradlew :helpForbiddenApis# How to add/apply rules for forbidden APIs.
  gradlew :helpLocalSettings# Local settings, overrides and build
performance tweaks.
  gradlew :helpRegeneration # How to refresh generated and derived
resources.
  gradlew :helpJmh  # JMH micro-benchmarks.
  gradlew :helpGit  # Git assistance and guides.
  gradlew :helpIDEs # IDE support.
  gradlew :helpPublishing   # Maven and other artifact publishing,
signing, etc.

The "dev-docs" are mostly concerned with processes and procedures; the
help/ is, I think, more focused on development/code. But you're entitled
to your own opinion, of course. ;)

Dawid

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Dawid Weiss]

There's actually quite a lot of docs related to Lucene tests (my remark was
meant at the randomizedtesting package) - see here:
https://github.com/apache/lucene/blob/main/help/tests.txt

The tests.timeoutSuite parameter could be added/ explained there too. I'm
not sure how much it's needed though - it's the first time anybody ever
mentioned it. :)

D.

On Thu, Feb 15, 2024 at 8:55 PM Shubham Chaudhary 
wrote:

> I think this information could sit well within the dev-docs in lucene i.e.
> "randomized testing in lucene". This would make it more discoverable as
> well and there is already a lack of proper docs as Dawid pointed?. We could
> add some references to docs like randomized testing core concepts
> <https://github.com/randomizedtesting/randomizedtesting/wiki/Core-Concepts>.
> If this idea makes sense I could try to write some doc around it and open a
> PR. I would like to know your thoughts on this? Thanks!
>
> - Shubham
>
> On Thu, Feb 15, 2024 at 10:23 PM Dawid Weiss 
> wrote:
>
>>
>> Sorry, the docs are not the best, I know.
>>
>> It's documented here -
>>
>> https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/SysGlobals.java#L186-L197
>>
>> So:
>>
>> 1) if you pass tests.timeoutSuite=1000 this changes the default value for
>> all classes that don't define any explicit timeout using an annotation;
>> classes that do have an annotation,
>> use the annotation's value,
>> 2) if you pass tests.timeoutSuite=1000! then this overrides everything -
>> the default value and all annotations.
>>
>> I vaguely recall option (2) was added specifically for nightlies which
>> bumped the iteration multiplier - this affected tests that normally ran
>> fairly fast
>> but during nightly runs could run slower than anticipated.
>>
>> D.
>>
>>
>> On Thu, Feb 15, 2024 at 3:18 PM David Smiley  wrote:
>>
>>> Oh; I didn't know that took precedence -- makes sense.  Hopefully a
>>> test subclass (like SolrTestCase) could override it as well.
>>>
>>> On Mon, Feb 12, 2024 at 2:09 PM Dawid Weiss 
>>> wrote:
>>> >
>>> >
>>> > You can override the defaults using sysprops in your CI builds -
>>> >
>>> > -Ptests.timeoutSuite=1000!
>>> >
>>> > takes precedence over any annotations (1 second).
>>> >
>>> > Dawid
>>> >
>>> > On Mon, Feb 12, 2024 at 7:53 PM David Smiley 
>>> wrote:
>>> >>
>>> >> Looking at LuceneTestCase, I see the annotation from
>>> RandomizedTesting:
>>> >> @TimeoutSuite(millis = 2 * TimeUnits.HOUR)
>>> >> This matches my observations of some builds that timed out, perhaps
>>> >> some flaky test hanging in Solr (that extends LuceneTestCase).
>>> >> Looking at this annotation, there is further documentation that the
>>> >> default can be set via sysprop tests.timeoutSuite.  Wouldn't doing
>>> >> that make more sense than hard-coding this figure in LuceneTestCase?
>>> >> For example, I'd like to have a normal/default test run have a low
>>> >> timeout (10min?) but on a "nightly" run on CI, use much higher.  Not 2
>>> >> hours though; individual tests needing so much should have a
>>> >> TimeoutSuite applied to them.
>>> >>
>>> >> ~ David Smiley
>>> >> Apache Lucene/Solr Search Developer
>>> >> http://www.linkedin.com/in/davidwsmiley
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>>> >>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Dawid Weiss]

Sorry, the docs are not the best, I know.

It's documented here -
https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/SysGlobals.java#L186-L197

So:

1) if you pass tests.timeoutSuite=1000 this changes the default value for
all classes that don't define any explicit timeout using an annotation;
classes that do have an annotation,
use the annotation's value,
2) if you pass tests.timeoutSuite=1000! then this overrides everything -
the default value and all annotations.

I vaguely recall option (2) was added specifically for nightlies which
bumped the iteration multiplier - this affected tests that normally ran
fairly fast
but during nightly runs could run slower than anticipated.

D.


On Thu, Feb 15, 2024 at 3:18 PM David Smiley  wrote:

> Oh; I didn't know that took precedence -- makes sense.  Hopefully a
> test subclass (like SolrTestCase) could override it as well.
>
> On Mon, Feb 12, 2024 at 2:09 PM Dawid Weiss  wrote:
> >
> >
> > You can override the defaults using sysprops in your CI builds -
> >
> > -Ptests.timeoutSuite=1000!
> >
> > takes precedence over any annotations (1 second).
> >
> > Dawid
> >
> > On Mon, Feb 12, 2024 at 7:53 PM David Smiley  wrote:
> >>
> >> Looking at LuceneTestCase, I see the annotation from RandomizedTesting:
> >> @TimeoutSuite(millis = 2 * TimeUnits.HOUR)
> >> This matches my observations of some builds that timed out, perhaps
> >> some flaky test hanging in Solr (that extends LuceneTestCase).
> >> Looking at this annotation, there is further documentation that the
> >> default can be set via sysprop tests.timeoutSuite.  Wouldn't doing
> >> that make more sense than hard-coding this figure in LuceneTestCase?
> >> For example, I'd like to have a normal/default test run have a low
> >> timeout (10min?) but on a "nightly" run on CI, use much higher.  Not 2
> >> hours though; individual tests needing so much should have a
> >> TimeoutSuite applied to them.
> >>
> >> ~ David Smiley
> >> Apache Lucene/Solr Search Developer
> >> http://www.linkedin.com/in/davidwsmiley
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [VOTE] Release Lucene 9.10.0 RC1

[Dawid Weiss]

+1.
SUCCESS! [2:43:47.883753]

I've only ran the smoke tester.

On Wed, Feb 14, 2024 at 8:29 PM Adrien Grand  wrote:

> Please vote for release candidate 1 for Lucene 9.10.0
>
> The artifacts can be downloaded from:
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-9.10.0-RC1-rev-695c0ac84508438302cd346a812cfa2fdc5a10df
>
> You can run the smoke tester directly with this command:
>
> python3 -u dev-tools/scripts/smokeTestRelease.py \
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-9.10.0-RC1-rev-695c0ac84508438302cd346a812cfa2fdc5a10df
>
> The vote will be open for at least 72 hours i.e. until 2024-02-17 20:00
> UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>
> --
> Adrien
>

Re: @TimeoutSuite and defaults (RandomizedTesting)

[Dawid Weiss]

You can override the defaults using sysprops in your CI builds -

-Ptests.timeoutSuite=1000!

takes precedence over any annotations (1 second).

Dawid

On Mon, Feb 12, 2024 at 7:53 PM David Smiley  wrote:

> Looking at LuceneTestCase, I see the annotation from RandomizedTesting:
> @TimeoutSuite(millis = 2 * TimeUnits.HOUR)
> This matches my observations of some builds that timed out, perhaps
> some flaky test hanging in Solr (that extends LuceneTestCase).
> Looking at this annotation, there is further documentation that the
> default can be set via sysprop tests.timeoutSuite.  Wouldn't doing
> that make more sense than hard-coding this figure in LuceneTestCase?
> For example, I'd like to have a normal/default test run have a low
> timeout (10min?) but on a "nightly" run on CI, use much higher.  Not 2
> hours though; individual tests needing so much should have a
> TimeoutSuite applied to them.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [JENKINS] Lucene » Lucene-NightlyTests-9.x - Build # 821 - Still Failing!

[Dawid Weiss]

I filed this issue to fix this:
https://github.com/apache/lucene/pull/13093

On Sun, Feb 11, 2024 at 10:35 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.x/821/
>
> 6 tests failed.
> FAILED:
> org.apache.lucene.backward_index.TestBinaryBackwardsCompatibility.testReadNMinusTwoCommit
> {Lucene-Version:8.11.3; Pattern: unsupported.%1$s-cfs.zip}
>
> Error Message:
> java.lang.AssertionError: Index name 8.11.3 not found:
> unsupported.8.11.3-cfs.zip
>
> Stack Trace:
> java.lang.AssertionError: Index name 8.11.3 not found:
> unsupported.8.11.3-cfs.zip
> at
> __randomizedtesting.SeedInfo.seed([D0CD9C4C4D489263:2381AB48ADE0F9A3]:0)
> at junit@4.13.1/org.junit.Assert.fail(Assert.java:89)
> at junit@4.13.1/org.junit.Assert.assertTrue(Assert.java:42)
> at junit@4.13.1/org.junit.Assert.assertNotNull(Assert.java:713)
> at
> org.apache.lucene.backward_index.BackwardsCompatibilityTestBase.setUp(BackwardsCompatibilityTestBase.java:125)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@9.10.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> 

Re: [JENKINS] Lucene » Lucene-NightlyTests-main - Build # 1259 - Still Unstable!

[Dawid Weiss]

I've filed this issue to fix these issues:
https://github.com/apache/lucene/issues/13073

On Mon, Feb 5, 2024 at 11:25 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/1259/
>
> 2 tests failed.
> FAILED:
> org.apache.lucene.backward_index.TestIndexSortBackwardsCompatibility.testSortedIndexAddDocBlocks
> {Lucene-Version:10.0.0; Pattern: sorted.%1$s.zip}
>
> Error Message:
> java.lang.AssertionError: failed to parse "09-JAN-2010 12:59:20.000" as
> date
>
> Stack Trace:
> java.lang.AssertionError: failed to parse "09-JAN-2010 12:59:20.000" as
> date
> at
> __randomizedtesting.SeedInfo.seed([641E90643118CEF9:10BF0754DFBC4CF0]:0)
> at
> org.apache.lucene.backward_index.TestIndexSortBackwardsCompatibility.createIndex(TestIndexSortBackwardsCompatibility.java:176)
> at
> org.apache.lucene.backward_index.BackwardsCompatibilityTestBase.setUp(BackwardsCompatibilityTestBase.java:120)
> at jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> 

Re: Mutation Testing

[Dawid Weiss]

We had it, once -
https://github.com/apache/lucene/issues/5399

I don't think it survived ant -> gradle transition (in part because nobody
was looking at the outputs of the tool?).

D.

On Fri, Jan 19, 2024 at 2:30 AM Mike Drob  wrote:

> Has anybody experimented with adding mutation testing to Lucene using
> something like the PIT framework?
> https://pitest.org/
>
> Looks like there is GitHub PR integration via https://www.arcmutate.com/
>
> I’d be curious to hear if people have found it useful, performant and
> comprehensive.
>
> Mike
>

Re: Welcome Stefan Vodita as Lucene committter

[Dawid Weiss]

Welcome, Stefan!
Dawid

On Thu, Jan 18, 2024 at 4:54 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Hi Team,
>
> I'm pleased to announce that Stefan Vodita has accepted the Lucene PMC's
> invitation to become a committer!
>
> Stefan, the tradition is that new committers introduce themselves with a
> brief bio.
>
> Congratulations, welcome, and thank you for all your improvements to
> Lucene and our community,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>

Re: [JENKINS] Lucene-9.x-Linux (64bit/hotspot/jdk-17.0.9) - Build # 14758 - Failure!

[Dawid Weiss]

Build timeout:

> Task :lucene:analysis:opennlp:check
Build timed out (after 209 minutes). Marking the build as aborted.
Build timed out (after 209 minutes). Marking the build as failed.
Build was aborted

Archiving artifacts> Task :lucene:analysis:opennlp:check
Build timed out (after 209 minutes). Marking the build as aborted.
Build timed out (after 209 minutes). Marking the build as failed.
Build was aborted
Archiving artifacts


On Tue, Jan 16, 2024 at 3:04 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/14758/
> Java: 64bit/hotspot/jdk-17.0.9 -XX:-UseCompressedOops -XX:+UseShenandoahGC
>
> All tests passed
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: Lucene v9.9.1: org.apache.lucene.search.ScoreMode

[Dawid Weiss]

Can you check whether it's a dependency graph problem somehow, maybe (does
it compile outside of intellij?). Can you trim down the project to a
reproducible scenario so that we can look at it?

D.

On Sun, Jan 7, 2024 at 5:53 AM Nazerke S  wrote:

> One of the functions takes ScoreMode as an argument and this ScoreMode is
> not found from dependencies.  In Intellij, seeing 'cannot resolve symbol
> ScoreMode'.
>
>
> @Override
>
> public Weight createWeight(IndexSearcher searcher,
> org.apache.lucene.search.ScoreMode scoreMode, float boost) {...}
>
>
> Tried 'import org.apache.lucene.search.ScoreMode' but not found either
> way.
>
> On Sun, Jan 7, 2024 at 6:39 AM Marcus Eagan  wrote:
>
>> It’s there for sure, but that doesn’t mean there is no problem. Could you
>> share what you are seeing in more detail given the class certainly exists?
>>
>> Marcus Eagan
>>
>>
>>
>> On Sat, Jan 6, 2024 at 14:05 Chris Hegarty
>>  wrote:
>>
>>> Hi,
>>>
>>> I see no issue. ScoreMode is present in lucene-core-9.9.1.jar
>>>
>>> $ curl https://dlcdn.apache.org/lucene/java/9.9.1/lucene-9.9.1.tgz >
>>> lucene-9.9.1.tgz
>>>...
>>> $  $ tar -xzf  lucene-9.9.1.tgz  $ jar -tvf
>>> lucene-9.9.1/modules/lucene-core-9.9.1.jar | grep ScoreMode
>>>   1618 Wed Dec 13 11:06:00 GMT 2023
>>> org/apache/lucene/search/ScoreMode.class
>>>
>>> Or from maven
>>>
>>> $ curl
>>> https://repo1.maven.org/maven2/org/apache/lucene/lucene-core/9.9.1/lucene-core-9.9.1.jar
>>> > lucene-core-9.9.1.jar
>>>...
>>> $ jar -tvf lucene-core-9.9.1.jar | grep ScoreMode
>>> -rw-r--r--  0 0  01618 13 Dec 11:06
>>> org/apache/lucene/search/ScoreMode.class
>>>
>>> -Chris.
>>>
>>> > On 6 Jan 2024, at 12:42, Nazerke S  wrote:
>>> >
>>> > Hi,
>>> >
>>> > While I was trying to upgrade Solr to use Lucene v9.9.1, I encountered
>>> 'org.apache.lucene.search.ScoreMode' not found, getting resolve class
>>> issue.
>>> > Quickly took a look into the ScoreMode class in lucene codebase,
>>> there is no change.
>>> > Maybe it is related to lucene-core-9.9.1.jar issue where ScoreMode
>>> class is  ?
>>> >  Anyone could help with this ?
>>> >
>>> > Thanksss,
>>> >
>>> > --Nazerke
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>

Re: [JENKINS] Lucene » Lucene-NightlyTests-9.9 - Build # 18 - Unstable!

[Dawid Weiss]

Thank you, Chris!

On Sun, Dec 17, 2023 at 10:50 AM Chris Hegarty
 wrote:

> Hi Dawid
>
> > On 16 Dec 2023, at 07:36, Dawid Weiss  wrote:
> >
> >
> > This one has been fixed on branch_9x and on main - it's a broken test.
> Should we apply it on 9_9 to quiet down CI or is the branch frozen?
>
> I completed the RM tasks for the 9.9.1 release, and just merged the fix
> for this test into branch_9_9. Should be stable now. Thanks,
>
> -Chris.
>
> > Dawid
> >
> > On Sat, Dec 16, 2023 at 3:24 AM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
> > Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.9/18/
> >
> > 1 tests failed.
> > FAILED:
> org.apache.lucene.queries.function.TestSortedSetFieldSource.testSimple
> >
> > Error Message:
> > org.junit.ComparisonFailure: expected: but was:
> >
> > Stack Trace:
> > org.junit.ComparisonFailure: expected: but was:
> > at
> __randomizedtesting.SeedInfo.seed([7908DAB4CF9E2D92:41BBFE4AE86DF943]:0)
> > at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:117)
> > at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.lucene.queries.function.TestSortedSetFieldSource.testSimple(TestSortedSetFieldSource.java:59)
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> > at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> > at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> > at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> > at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36

Re: [JENKINS] Lucene » Lucene-NightlyTests-9.9 - Build # 18 - Unstable!

[Dawid Weiss]

This one has been fixed on branch_9x and on main - it's a broken test.
Should we apply it on 9_9 to quiet down CI or is the branch frozen?

Dawid

On Sat, Dec 16, 2023 at 3:24 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.9/18/
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.queries.function.TestSortedSetFieldSource.testSimple
>
> Error Message:
> org.junit.ComparisonFailure: expected: but was:
>
> Stack Trace:
> org.junit.ComparisonFailure: expected: but was:
> at
> __randomizedtesting.SeedInfo.seed([7908DAB4CF9E2D92:41BBFE4AE86DF943]:0)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:117)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.lucene.queries.function.TestSortedSetFieldSource.testSimple(TestSortedSetFieldSource.java:59)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@9.9.1-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   

Re: [VOTE] Release Lucene 9.9.1 RC1

[Dawid Weiss]

I hit TestSortedSetFieldSource assertion fixed in 12851 - this wasn't
applied to branch_9_9, I think. Not a big problem (test fix).

The second run succeeded on a slow vm.
SUCCESS! [3:34:11.606597]

+1.

Dawid

On Wed, Dec 13, 2023 at 12:55 PM Chris Hegarty
 wrote:

> Hi,
>
> Please vote for release candidate 1 for Lucene 9.9.1
>
> The artifacts can be downloaded from:
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-9.9.1-RC1-rev-eee32cbf5e072a8c9d459c349549094230038308
>
> You can run the smoke tester directly with this command:
>
> python3 -u dev-tools/scripts/smokeTestRelease.py \
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-9.9.1-RC1-rev-eee32cbf5e072a8c9d459c349549094230038308
>
> The vote will be open for at least 72 hours i.e. until 2023-12-16 12:00
> UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>
> -Chris.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Running tests with streaming console output but NOT verbose?

[Dawid Weiss]

Hi Mike,

What are the C and ! characters in the left most column?
>

Always good to add a legend, eh? It's this snippet that computes the
prefixes:

  (defValue != value ? "! " : computedValue ? "C " : "  "),

In short, "C" stands for a "computed" value. This means the value is
randomly computed from the provided (or random) tests.seed. This applies to
properties we can't modify in LuceneTestCase - like tests.file.encoding you
asked about (which can be declared at VM startup only). This is ok, since
we do want to "simulate" those odd locales and encodings.

The "!" next to a property means you have it locally overridden (either via
command line property or in gradle.properties). In your case, it's this
one, for example:


> ! tests.jvms   = 12   # (!= default: computed) Number of
> forked test JVMs
>

Dawid

Re: [VOTE] Release Lucene 9.9.1 RC1

[Dawid Weiss]

> SUCCESS! [0:14:52.296147]
>

What?!... How is that possible, Mike?

I get gpg warnings - is this normal?

File:
/tmp/smoke_lucene_9.9.1_eee32cbf5e072a8c9d459c349549094230038308/lucene.lucene-9.9.1-src.tgz.gpg.verify.log
verify trust
  GPG: gpg: WARNING: This key is not certified with a trusted signature!

Dawid

Re: Running tests with streaming console output but NOT verbose?

[Dawid Weiss]

I think I remember now... So, this is LuceneTestCase:

  /**
   * True if and only if tests are run in verbose mode. If this flag is
false tests are not expected
   * to print any messages. Enforced with {@link TestRuleLimitSysouts}.
   */
  public static final boolean VERBOSE =
systemPropertyAsBoolean("tests.verbose", false);

  /** Enables or disables dumping of {@link InfoStream} messages. */
  public static final boolean INFOSTREAM =
systemPropertyAsBoolean("tests.infostream", VERBOSE);

Note infostream is by default on when verbose is enabled. You can turn it
off independently though (-Ptests.infostream=false) or even set the default
in your gradle.properties...

So I don't think there's anything to do here? For the record, all the test
options and their defaults are displayed with:

gradlew -p lucene\core testOpts

Dawid

On Tue, Dec 12, 2023 at 8:45 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Ahh thanks for the quick explanation and temporary solution Dawid!.
> Naming is the hardest part :)
>
> I think long ago we used to call it "-Dtests.stdout=true" or so?  Not the
> greatest name tho.  Maybe "tests.liveConsoleOut"?  "tests.liveConsole"?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Dec 12, 2023 at 2:31 PM Dawid Weiss  wrote:
>
>>
>> This is actually an accidental (?) clash between Lucene's system property
>> and what's in defaults-tests.gradle.
>> You can manually prepend true || ... to the following in
>> defaults-tests.gradle.
>>
>> def verboseMode = resolvedTestOption("tests.verbose").toBoolean()
>>
>> I can't remember why it aligns with Lucene's logger. Maybe it
>> should/could be a
>> separate property? I find it difficult to come up with a reasonable name
>> though.
>>
>> D.
>>
>> On Tue, Dec 12, 2023 at 8:03 PM Michael McCandless <
>> luc...@mikemccandless.com> wrote:
>>
>>> Hi Team,
>>>
>>> This is prolly a Dawid question...
>>>
>>> Sometimes I want to run a test (like a slow Monster test), seeing its
>>> ongoing musings popping out on the console in "real time" (not buffered).
>>>
>>> I can do this today by adding "-Dtests.verbose=true" to the ./gradlew
>>> invocation that's running the test.
>>>
>>> But that also turns on LuceneTestCase.VERBOSE which sometimes produces
>>> insane amounts of mostly not helpful content.
>>>
>>> Is there any way to do the first (stream console output) without the
>>> second (mega verbosity enabled)?
>>>
>>> Thanks,
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>

Re: Running tests with streaming console output but NOT verbose?

[Dawid Weiss]

This is actually an accidental (?) clash between Lucene's system property
and what's in defaults-tests.gradle.
You can manually prepend true || ... to the following in
defaults-tests.gradle.

def verboseMode = resolvedTestOption("tests.verbose").toBoolean()

I can't remember why it aligns with Lucene's logger. Maybe it should/could
be a
separate property? I find it difficult to come up with a reasonable name
though.

D.

On Tue, Dec 12, 2023 at 8:03 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Hi Team,
>
> This is prolly a Dawid question...
>
> Sometimes I want to run a test (like a slow Monster test), seeing its
> ongoing musings popping out on the console in "real time" (not buffered).
>
> I can do this today by adding "-Dtests.verbose=true" to the ./gradlew
> invocation that's running the test.
>
> But that also turns on LuceneTestCase.VERBOSE which sometimes produces
> insane amounts of mostly not helpful content.
>
> Is there any way to do the first (stream console output) without the
> second (mega verbosity enabled)?
>
> Thanks,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>

Re: [JENKINS] Lucene » Lucene-Check-main - Build # 10815 - Unstable!

[Dawid Weiss]

It's the same as this issue -
https://github.com/apache/lucene/issues/12883

On Thu, Dec 7, 2023 at 1:30 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/10815/
>
> 4 tests failed.
> FAILED:
> org.apache.lucene.search.uhighlight.TestUnifiedHighlighterMTQ.testWildcardInFiltered
> {p0=stored,indexed,tokenized}
>
> Error Message:
> org.junit.ComparisonFailure: expected:<[This is a test].> but
> was:<[Test a one sentence document].>
>
> Stack Trace:
> org.junit.ComparisonFailure: expected:<[This is a test].> but
> was:<[Test a one sentence document].>
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:117)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.lucene.search.uhighlight.TestUnifiedHighlighterMTQ.testWildcardInFiltered(TestUnifiedHighlighterMTQ.java:487)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> 

Re: [JENKINS] Lucene-9.x-Linux (64bit/hotspot/jdk-11.0.21) - Build # 14204 - Unstable!

[Dawid Weiss]

Hi Uwe,

Apologies for the late reply.

I suggested to use this reader to some customers, but they were using Solr
> or Elasticsearch and it's not easy to implement it there. And they didn't
> want to pay the expensive Uwe.
>

>

Oh, they should know better - Mr Uwe is worth his price tag for technical
merits but also his eloquence and samurai-sword-sharp code smell
observation skills!


> How do you handle deletes. Because the main issue with those readers is
> that you can't update documents without also updating the main reader
> (although it's a fake update).
>

It's not a general search/ document storage system - we allow updates but
they're not real time and reindexing creates both new segments and
second-tier segments. It gets even more complicated because there are
multiple living Lucene commits (custom deletion policy). Long story but
works just fine!


> If this is used, have you thought of a SynchronizedMergePolicy that just
> applies the same merges in the secondary index?
>

I don't think it's applicable to us (as above), but it's an interesting
idea!

D.

>

Re: [JENKINS] Lucene-9.x-Linux (64bit/hotspot/jdk-11.0.21) - Build # 14204 - Unstable!

[Dawid Weiss]

> ParallelReader is also seldomly used, maybe we should remove support at
> some point. I don't know anybody using it, because it is very complicated
> to maintain consistent indexes. It only works with stable merge policies.
>

You do know somebody - you know me. We're using it extensively - the
scenario is for storing data derived from the main document in a separate
index, merging this data dynamically. The data can then be reindexed/
modified independently. Yes, we do use stable merge policies.

Dawid

Re: [JENKINS] Lucene » Lucene-Check-main - Build # 10678 - Unstable!

[Dawid Weiss]

Awesome, isn't it? :)

D.

On Mon, Nov 20, 2023 at 9:28 AM Adrien Grand  wrote:

> A one-in-a-million-runs test failure. I pushed a fix:
> https://github.com/apache/lucene/commit/194a500323531b66124577167006115c34dfde54
> .
>
> On Sun, Nov 19, 2023 at 10:00 PM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> Build:
>> https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/10678/
>>
>> 1 tests failed.
>> FAILED:  org.apache.lucene.util.TestPagedBytes.testDataInputOutput2
>>
>> Error Message:
>> java.lang.IllegalArgumentException: bound must be positive
>>
>> Stack Trace:
>> java.lang.IllegalArgumentException: bound must be positive
>> at
>> __randomizedtesting.SeedInfo.seed([65D9DDAC3D23F916:EB18D32A6DB237D6]:0)
>> at java.base/java.util.Random.nextInt(Random.java:322)
>> at
>> com.carrotsearch.randomizedtesting.Xoroshiro128PlusRandom.nextInt(Xoroshiro128PlusRandom.java:73)
>> at
>> com.carrotsearch.randomizedtesting.AssertingRandom.nextInt(AssertingRandom.java:87)
>> at
>> org.apache.lucene.util.TestPagedBytes.testDataInputOutput2(TestPagedBytes.java:156)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>> at
>> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
>> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
>> at 

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!

[Dawid Weiss]

Thanks Uwe!

On Tue, Nov 14, 2023 at 7:27 PM Uwe Schindler  wrote:

> Hi,
>
> For now the simplest is to disable always is an alternate JVM is used,
> just remove the second part of the first IF statement. In Main it is no
> longer relevant, as the runtime JDK is always >= 17, so it would always
> trigger the first if.
>
> I would not spend too much time, until errorprone gets its issues fixed.
> Actually they have some hacks but they only work with toolkits. Looks like
> the combination fork=true and not using toolkits breaks their construction
> of parameters to pass.
>
> Same for Solr.
> Am 14.11.2023 um 19:17 schrieb Uwe Schindler:
>
> Hi Dawid,
>
> The problem does not happen on Java 17, because errorprone is not enabled
> when the forked JDK is > Java 15. We did this because earlier versions
> worked correctly. But new versions of errorprone always fail when the JDK
> is forked while compiling.
>
> if (rootProject.usesAltJvm && rootProject.runtimeJavaVersion >
> JavaVersion.VERSION_15) {
>   skipReason = "won't work with JDK ${rootProject.runtimeJavaVersion} if
> used as alternative java toolchain"
> }
>
> if (!propertyOrDefault("validation.errorprone", isCIBuild).asBoolean()) {
>   skipReason = "skipped on builds not running inside CI environments, pass
> -Pvalidation.errorprone=true to enable"
> }
>
> So it looks like the errorprone plugin got broken by a recent upgrade. It
> now always fails when forked JDK is used. So we shold disable it in this
> case. We just did not notice, as previously it was only disabled when the
> runtime java version was > 17.
>
> Nowadays we no longer run alternate JVMs with Java 12, 13, 14, 15. We run
> with Java 11, 17, 19, 20, 21. So it is always disabled except for Java 11.
> With RUNTIME_JAVA_HOME==JAVA_HOME we never fork, but as we use OpenJ9, we
> fork an BOM.
>
> I will post a PR soon.
>
> Uwe
> Am 14.11.2023 um 19:06 schrieb Uwe Schindler:
>
> Hi Dawid,
>
> Hah, the issue happens only if you pass CI=true (this is set by CI
> systems), so errorprone is enabled. so do "export CI=true" and then build
> with that config.
>
> So it looks like a combination of errorprone enabled with Java 11 OpenJ9.
>
> Uwe
> Am 13.11.2023 um 09:09 schrieb Dawid Weiss:
>
>
> Sure, thanks. What's strange is that we don't use add-opens anywhere, I
> think (there is a mention of it I left in one of the
> comments, but nothing else across the codebase uses this directive).
>
> > Task :lucene:distribution.tests:compileTestJava
> warning: [options] --add-opens has no effect at compile time
>
>
>
> On Sun, Nov 12, 2023 at 10:56 PM Uwe Schindler  wrote:
>
>> Will check tomorrow, it's too late now.
>>
>> On Jenkins there were no windows builds with IBM and Java 11 yet:
>> https://jenkins.thetaphi.de/job/Lucene-9.x-Windows/
>> Am 12.11.2023 um 22:00 schrieb Dawid Weiss:
>>
>>
>> Hi Uwe,
>>
>> Can you reproduce this on Windows with the same JVM versions though?
>> Seems like I have exactly the same setup and yet this works for me just
>> fine. Strange.
>>
>> Dawid
>>
>> On Sun, Nov 12, 2023 at 9:52 PM Uwe Schindler  wrote:
>>
>>> This one was my first idea, too.
>>>
>>> It fails only with IBM Semeru in combination with Gradle using Temurin.
>>>
>>> I will dig tomorrow on Jenkins server and print all debug info.
>>>
>>> Uwe
>>>
>>>
>>> Am 12. November 2023 21:48:54 MEZ schrieb Dawid Weiss <
>>> dawid.we...@gmail.com>:
>>>
>>>>
>>>> I can't reproduce this though - used exactly the same JVMs (on Windows):
>>>>
>>>> > gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks
>>>> --console=plain
>>>> Generating gradle.properties
>>>> ...
>>>> > Task :altJvmWarning
>>>> NOTE: Alternative java toolchain will be used for compilation and tests:
>>>>   Project will use 11 (IBM JDK 11.0.20.1+1, home at:
>>>> c:\_tmp\jdk-11.0.20.1+1)
>>>>   Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at:
>>>> C:\_tmp\jdk-11.0.21+9)
>>>> ...
>>>> > Task :lucene:distribution.tests:compileJava NO-SOURCE
>>>> > Task :lucene:distribution.tests:classes UP-TO-DATE
>>>> > Task :lucene:distribution.tests:compileTestJava
>>>>
>>>> BUILD SUCCESSFUL in 23s
>>>> 5 actionable tasks: 5 executed
>>>>
>>>> On main branch it works, no idea why:
>>>>>
>>>>
>>>> O thought it's because of this:
>>>>
>>>> https://github.com/apache/lucene/commit/2e12a35c876a
>>>>
>>>> but I don't think so... seems to work for me on Windows on branch_9x
>>>> just fine?
>>>>
>>>> D.
>>>>
>>> --
>>> Uwe Schindler
>>> Achterdiek 19, 28357 Bremen
>>> https://www.thetaphi.de
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
>> eMail: u...@thetaphi.de
>>
>> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!

[Dawid Weiss]

Hi Uwe,

Hah, the issue happens only if you pass CI=true (this is set by CI
> systems), so errorprone is enabled. so do "export CI=true" and then build
> with that config.
>

Darn... I hate that flag. I'll take a look unless you beat me to it.

D.

>

Re: [JENKINS] Lucene-main-Linux (64bit/hotspot/jdk-17.0.9) - Build # 45536 - Failure!

[Dawid Weiss]

> P.S.: Jenkins kills jobs, if they take longer than usual it kills it (it
> has no hard limit, it takes the average time of previous runs and if one
> takes much longer it kills).
>

>From what I see here -

https://plugins.jenkins.io/build-timeout/

it should be possible to tweak jenkins to use an absolute timeout... and
even if not, provide a shell script to perhaps jps all java processes so
that there's more diagnostics?

D.

>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!

[Dawid Weiss]

Sure, thanks. What's strange is that we don't use add-opens anywhere, I
think (there is a mention of it I left in one of the
comments, but nothing else across the codebase uses this directive).

> Task :lucene:distribution.tests:compileTestJava
warning: [options] --add-opens has no effect at compile time



On Sun, Nov 12, 2023 at 10:56 PM Uwe Schindler  wrote:

> Will check tomorrow, it's too late now.
>
> On Jenkins there were no windows builds with IBM and Java 11 yet:
> https://jenkins.thetaphi.de/job/Lucene-9.x-Windows/
> Am 12.11.2023 um 22:00 schrieb Dawid Weiss:
>
>
> Hi Uwe,
>
> Can you reproduce this on Windows with the same JVM versions though? Seems
> like I have exactly the same setup and yet this works for me just fine.
> Strange.
>
> Dawid
>
> On Sun, Nov 12, 2023 at 9:52 PM Uwe Schindler  wrote:
>
>> This one was my first idea, too.
>>
>> It fails only with IBM Semeru in combination with Gradle using Temurin.
>>
>> I will dig tomorrow on Jenkins server and print all debug info.
>>
>> Uwe
>>
>>
>> Am 12. November 2023 21:48:54 MEZ schrieb Dawid Weiss <
>> dawid.we...@gmail.com>:
>>
>>>
>>> I can't reproduce this though - used exactly the same JVMs (on Windows):
>>>
>>> > gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks
>>> --console=plain
>>> Generating gradle.properties
>>> ...
>>> > Task :altJvmWarning
>>> NOTE: Alternative java toolchain will be used for compilation and tests:
>>>   Project will use 11 (IBM JDK 11.0.20.1+1, home at:
>>> c:\_tmp\jdk-11.0.20.1+1)
>>>   Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at:
>>> C:\_tmp\jdk-11.0.21+9)
>>> ...
>>> > Task :lucene:distribution.tests:compileJava NO-SOURCE
>>> > Task :lucene:distribution.tests:classes UP-TO-DATE
>>> > Task :lucene:distribution.tests:compileTestJava
>>>
>>> BUILD SUCCESSFUL in 23s
>>> 5 actionable tasks: 5 executed
>>>
>>> On main branch it works, no idea why:
>>>>
>>>
>>> O thought it's because of this:
>>>
>>> https://github.com/apache/lucene/commit/2e12a35c876a
>>>
>>> but I don't think so... seems to work for me on Windows on branch_9x
>>> just fine?
>>>
>>> D.
>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, 28357 Bremen
>> https://www.thetaphi.de
>>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!

[Dawid Weiss]

Hi Uwe,

Can you reproduce this on Windows with the same JVM versions though? Seems
like I have exactly the same setup and yet this works for me just fine.
Strange.

Dawid

On Sun, Nov 12, 2023 at 9:52 PM Uwe Schindler  wrote:

> This one was my first idea, too.
>
> It fails only with IBM Semeru in combination with Gradle using Temurin.
>
> I will dig tomorrow on Jenkins server and print all debug info.
>
> Uwe
>
>
> Am 12. November 2023 21:48:54 MEZ schrieb Dawid Weiss <
> dawid.we...@gmail.com>:
>
>>
>> I can't reproduce this though - used exactly the same JVMs (on Windows):
>>
>> > gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks
>> --console=plain
>> Generating gradle.properties
>> ...
>> > Task :altJvmWarning
>> NOTE: Alternative java toolchain will be used for compilation and tests:
>>   Project will use 11 (IBM JDK 11.0.20.1+1, home at:
>> c:\_tmp\jdk-11.0.20.1+1)
>>   Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at:
>> C:\_tmp\jdk-11.0.21+9)
>> ...
>> > Task :lucene:distribution.tests:compileJava NO-SOURCE
>> > Task :lucene:distribution.tests:classes UP-TO-DATE
>> > Task :lucene:distribution.tests:compileTestJava
>>
>> BUILD SUCCESSFUL in 23s
>> 5 actionable tasks: 5 executed
>>
>> On main branch it works, no idea why:
>>>
>>
>> O thought it's because of this:
>>
>> https://github.com/apache/lucene/commit/2e12a35c876a
>>
>> but I don't think so... seems to work for me on Windows on branch_9x just
>> fine?
>>
>> D.
>>
>>> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de
>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!

[Dawid Weiss]

I can't reproduce this though - used exactly the same JVMs (on Windows):

> gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks
--console=plain
Generating gradle.properties
...
> Task :altJvmWarning
NOTE: Alternative java toolchain will be used for compilation and tests:
  Project will use 11 (IBM JDK 11.0.20.1+1, home at:
c:\_tmp\jdk-11.0.20.1+1)
  Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at:
C:\_tmp\jdk-11.0.21+9)
...
> Task :lucene:distribution.tests:compileJava NO-SOURCE
> Task :lucene:distribution.tests:classes UP-TO-DATE
> Task :lucene:distribution.tests:compileTestJava

BUILD SUCCESSFUL in 23s
5 actionable tasks: 5 executed

On main branch it works, no idea why:
>

O thought it's because of this:

https://github.com/apache/lucene/commit/2e12a35c876a

but I don't think so... seems to work for me on Windows on branch_9x just
fine?

D.

>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!

[Dawid Weiss]

Hi Uwe,

Will dig tomorrow. Maybe Dawid has an idea? It looks like the alternate
> runtime is correctly detected, but why is Gradle passing compiler runzine
> options without -J in just this case. In Main the same works where Gradle
> runs with Java17-Temurin and J9 is used as runtime.
>

I think I know what this is - please let me verify and provide a PR, if
it's indeed that.

Dawid

Re: Welcome Patrick Zhai to the Lucene PMC

[Dawid Weiss]

Congratulations and welcome, Patrick!

Dawid

On Fri, Nov 10, 2023 at 9:05 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> I'm happy to announce that Patrick Zhai has accepted an invitation to join
> the Lucene Project Management Committee (PMC)!
>
> Congratulations Patrick, thank you for all your hard work improving
> Lucene's community and source code, and welcome aboard!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>

Re: Ascii folding

[Dawid Weiss]

Thanks Robert, Uwe - all this is enlightening. I didn't know about those
things you mentioned.

Dawid

On Sat, Nov 11, 2023 at 2:02 PM Uwe Schindler  wrote:

> Hi Dawid,
>
> the ASCII folding filter is meant to remove accents. You would like to
> have searching for visually similar characters. These are 2 different
> things.
>
> Actually Robert also has some config options, waht I generally use for
> wester european searches where some documents may contain names of people
> (Author names, titles in cyrillic or other languages) it to convert the
> tokens using ICU transliteration (use one of the ICU folding filters with
> the below config):
>
> Transliterator.getInstance("Any-Latin; NFD; [:Nonspacing Mark:] Remove;
> NFKC; CaseFold", Transliterator.FORWARD);
>
> This does convert everything to latin characters in a language-neutral way
> and then removes all accents by the trick "decompose, remove non-spacing
> mark, compose again and case-fold the result.
>
> Uwe
> Am 10.11.2023 um 19:03 schrieb Dawid Weiss:
>
>
> Hi Steve, Chris,
>
> Ok, makes sense. Thanks for the pointers. I agree the justification for
> the use of character-level normalization filters is highly
> context-dependent (for example, unsuitable when mixed languages are present
> on input).
>
> Dawid
>
> On Fri, Nov 10, 2023 at 6:58 PM Chris Hostetter 
> wrote:
>
>>
>> : Here's the unicode letter after "th":
>> : https://www.fileformat.info/info/unicode/char/0435/index.htm
>> :
>> : To my surprise, I couldn't find it in the ascii folding filter:
>> :
>> :
>> https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.java
>> :
>> : Anybody remembers whether the omission of Cyrillic characters was
>> : intentional (there is quite a few of them that are nearly identical in
>> : appearance to Latin letters).
>>
>> From the javadocs, i'm going to guess it's because the the filter focuses
>> on "Latin_characters_in_Unicode" ... and your "CYRILLIC SMALL LETTER IE"
>> isn't described as being a "(adjective) LATIN noun (WITH noun)" like all
>> of the other characters that are considered to have a direct mapping to
>> the "ASCII" / latin characters.
>>
>> If you look back at when it was added...
>>
>> https://issues.apache.org/jira/browse/LUCENE-1390
>>
>> ...the original focus was on deprecating "ISOLatin1AccentFilter" and
>> replacing it with "a more comprehensive version of this code that
>> included
>> not just ISO-Latin-1 (ISO-8859-1) but the entire Latin 1 and Latin
>> Extended A unicode blocks."  (The originally proposed name was
>> 'ISOLatinAccentFilter') ... subsequent discussion focused on adding more
>> Latin blocks.
>>
>> There was a related issue at the time which initially aimed to add a
>> more general "UnicodeNormalizationFilter" that ultimated resulted in
>> adding the "ICU" analysis classes...
>>
>> https://issues.apache.org/jira/browse/LUCENE-1343
>>
>> ..which IIUC may better handle "CYRILLIC SMALL LETTER IE" (but i haven't
>> tested that)
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>

Re: Ascii folding

[Dawid Weiss]

Hi Steve, Chris,

Ok, makes sense. Thanks for the pointers. I agree the justification for the
use of character-level normalization filters is highly context-dependent
(for example, unsuitable when mixed languages are present on input).

Dawid

On Fri, Nov 10, 2023 at 6:58 PM Chris Hostetter 
wrote:

>
> : Here's the unicode letter after "th":
> : https://www.fileformat.info/info/unicode/char/0435/index.htm
> :
> : To my surprise, I couldn't find it in the ascii folding filter:
> :
> :
> https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.java
> :
> : Anybody remembers whether the omission of Cyrillic characters was
> : intentional (there is quite a few of them that are nearly identical in
> : appearance to Latin letters).
>
> From the javadocs, i'm going to guess it's because the the filter focuses
> on "Latin_characters_in_Unicode" ... and your "CYRILLIC SMALL LETTER IE"
> isn't described as being a "(adjective) LATIN noun (WITH noun)" like all
> of the other characters that are considered to have a direct mapping to
> the "ASCII" / latin characters.
>
> If you look back at when it was added...
>
> https://issues.apache.org/jira/browse/LUCENE-1390
>
> ...the original focus was on deprecating "ISOLatin1AccentFilter" and
> replacing it with "a more comprehensive version of this code that included
> not just ISO-Latin-1 (ISO-8859-1) but the entire Latin 1 and Latin
> Extended A unicode blocks."  (The originally proposed name was
> 'ISOLatinAccentFilter') ... subsequent discussion focused on adding more
> Latin blocks.
>
> There was a related issue at the time which initially aimed to add a
> more general "UnicodeNormalizationFilter" that ultimated resulted in
> adding the "ICU" analysis classes...
>
> https://issues.apache.org/jira/browse/LUCENE-1343
>
> ..which IIUC may better handle "CYRILLIC SMALL LETTER IE" (but i haven't
> tested that)
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Ascii folding

[Dawid Weiss]

I just stumbled upon this stop word appearing in one of our indexes:

thе

Look closely. Can you see it? I doubt - I couldn't either. This is the hex
dump of that:

74 68 d0 b5

which means

thе and the

are two different things.

Here's the unicode letter after "th":
https://www.fileformat.info/info/unicode/char/0435/index.htm

To my surprise, I couldn't find it in the ascii folding filter:

https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.java

Anybody remembers whether the omission of Cyrillic characters was
intentional (there is quite a few of them that are nearly identical in
appearance to Latin letters).

Dawid

Re: [JENKINS] Lucene-9.x-MacOSX (64bit/hotspot/jdk-21) - Build # 3127 - Unstable!

[Dawid Weiss]

Branch 9x has System.currentTimeMillis in ReplicationClient.java (which the
stack trace of this failure points to) and it's a mac machine - I suspect
realtime clock issues caused this.
The main branch removed this code altogether in
https://github.com/apache/lucene/pull/12038.

Dawid

On Thu, Nov 9, 2023 at 8:35 AM Policeman Jenkins Server 
wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-MacOSX/3127/
> Java: 64bit/hotspot/jdk-21 -XX:-UseCompressedOops -XX:+UseParallelGC
>
> 2 tests failed.
> FAILED:
> org.apache.lucene.replicator.TestIndexAndTaxonomyReplicationClient.testConsistencyOnExceptions
>
> Error Message:
> java.lang.Exception: Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
> at __randomizedtesting.SeedInfo.seed([5A9E50A4ED05A741]:0)
>
>
> FAILED:
> org.apache.lucene.replicator.TestIndexAndTaxonomyReplicationClient.classMethod
>
> Error Message:
> java.lang.Exception: Suite timeout exceeded (>= 720 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 720 msec).
> at __randomizedtesting.SeedInfo.seed([5A9E50A4ED05A741]:0)
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: Bump minimum Java version requirement to 21

[Dawid Weiss]

> It's not just you - we have an internal JDK11 fork at BIG COMPANY for some
> folks that can't get off the stick.
>

 The truth is - most large companies will be reluctant to upgrade unless
they see a benefit in doing so. Here, we can offer this benefit (call it a
carrot, if you mentioned the stick, Mike) - speedups to vector routines,
new directory implementations Uwe has been working on, probably more.

I'm myself fairly conservative too but I also think that those new APIs are
probably worth the investment (and potential pain) to upgrade.

Dawid

Re: [JENKINS] Lucene » Lucene-Solr-repro - Build # 1164 - Still Failing!

[Dawid Weiss]

I agree - all of it looks very strange indeed. Seems like some of these
jobs were brought back from inactive state.

Dawid

On Fri, Nov 3, 2023 at 10:37 PM Chris Hostetter 
wrote:

>
> : I'm not even sure how it's possible but this build seems to be using
> stale
> : configuration (repo pointing at the commit that informs about project
> split
> : between Solr and Lucene). I've disabled the plan.
>
> This jenkins job stoped sending emails  ~ March of 2021.
>
> There is a thread from March of 2022 where Smiley mentioned it
> didn't survive the gradle migration, and suggested bring it back...
>
>
> https://lists.apache.org/list?d...@solr.apache.org:gte=365d:reproduceJenkinsFailures.py
>
>
> But around Oct 27, it started running again and sending emails ... possily
> because of something going horribly wrong on the jenkins machines
> (possibly related to the other git related failures that started happening
> in other jobs recently?)
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [JENKINS] Lucene » Lucene-Solr-repro - Build # 1164 - Still Failing!

[Dawid Weiss]

I'm not even sure how it's possible but this build seems to be using stale
configuration (repo pointing at the commit that informs about project split
between Solr and Lucene). I've disabled the plan.

Dawid

On Fri, Nov 3, 2023 at 10:17 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-repro/1164/
>
> [...truncated 40 lines...]
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: Welcome Guo Feng to the Lucene PMC

[Dawid Weiss]

Congratulations and welcome, Feng!
Dawid

On Tue, Oct 24, 2023 at 7:04 PM Adrien Grand  wrote:

> I'm pleased to announce that Guo Feng has accepted an invitation to join
> the Lucene PMC!
>
> Congratulations Feng, and welcome aboard!
>
> --
> Adrien
>

Re: Welcome Luca Cavanna to the Lucene PMC

[Dawid Weiss]

Congratulations, Luca!

On Fri, Oct 20, 2023 at 7:51 AM Adrien Grand  wrote:

> I'm pleased to announce that Luca Cavanna has accepted an invitation to
> join the Lucene PMC!
>
> Congratulations Luca, and welcome aboard!
>
> --
> Adrien
>

Re: Could we allow an IndexInput to read from a still writing IndexOutput?

[Dawid Weiss]

I think there is a certain beauty (of tape-backed storage flavor...) in
existing abstractions and I wouldn't change them unless absolutely
necessary (FST construction isn't the dominant cost in indexing). Also,
random seeks all over the place may be really problematic in certain
scenarios (as is opening a written-to file for reading, as Robert
mentioned).

> Failing that, our plan B is to wastefully duplicate the byte[] slices
from the already written bytes into our own private (heap resident, boo)
copy, which would use quite a bit more RAM while building the FST, and make
less minimal FSTs for a given RAM budget.

Well, this node cache doesn't have to be on heap... It can be a plain
temporary file (with full random access). It's a scratch-only structure
which you can delete after the fst is written. It does add I/O overhead but
doesn't interfere with the rest of the code in Lucene. Perhaps, instead of
changing IndexInput and IndexOutput, one could start with a plain temp file
(NIO API)?

I also think that the tradeoffs presented in graphs on the fst-node-cache
issue are not so bad at all. Yes, the FST is not minimal, but the
construction-space vs output size is quite all right to me.

Dawid

On Thu, Oct 19, 2023 at 3:50 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Hi Team,
>
> Today, Lucene's Directory abstraction does not allow opening an IndexInput
> on a file until the file is fully written and closed via IndexOutput.  We
> enforce this in tests, and some of our core Directory implementations
> demand this (e.g. caching the file's length on opening an IndexInput).
>
> Yet, most filesystems will easily allow simultaneous read/append of a
> single file.  We just don't expose this IO semantics to Lucene, but could
> we allow random-access reads with append-only writes on one file?  Is there
> a strong reason that we don't allow this?
>
> Quick TL/DR context: we are trying to enable FST compilation to write
> off-heap (directly to disk), enabling creating arbitrarily large FSTs with
> bounded heap, matching how FSTs can now be read off-heap, and it would be
> much much more RAM efficient if we could read/append the same file at once.
>
> Full gory details context: inspired by how Tantivy
>  (awesome and fast Rust search
> engine!) writes its FSTs , over
> in this issue  and PR
> ,
> we (thank you Dzung Bui / @dungba88!) are trying to fix Lucene's FST
> building to immediately stream the FST to disk, instead of buffering the
> whole thing in RAM and then writing to disk.
>
> This would allow building arbitrarily large FSTs without using up heap,
> and symmetrically matches how we can now read FSTs off-heap, plus FST
> building is already (mostly) append-only. This would also allow removing
> some of the crazy abstractions we have for writing FST bytes into RAM
> (FSTStore, BytesStore).  It would enable interesting things like a Codec
> whose term dictionary is stored entirely in an FST
>  (also inspired by Tantivy).
>
> The wrinkle is that, while the FST is building, it sometimes looks back
> and reads previously written bytes, to share suffixes and create a minimal
> (or near minimal) FST.  So if IndexInput could read those bytes, even as
> the FST is still appending to IndexOutput, it would "just work".
>
> Failing that, our plan B is to wastefully duplicate the byte[] slices from
> the already written bytes into our own private (heap resident, boo) copy,
> which would use quite a bit more RAM while building the FST, and make less
> minimal FSTs for a given RAM budget.  I haven't measured the added wasted
> RAM if we have to go this route but I fear it is sizable in practice, i.e.
> it strongly negates the whole idea of writing an FST off-heap since its
> effectively storing a possibly large portion of the FST in many duplicated
> byte[] fragments (in the NodeHash).
>
> So ... could we somehow relax Lucene's Directory semantics to allow
> opening an IndexInput on a still appending IndexOutput, since most
> filesystems are fine with this?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>

Re: Can we get rid of "Approve & Run" on GitHub PRs by new contributors (non-committers)?

[Dawid Weiss]

I filed a PR here -
https://github.com/apache/lucene/pull/12687

Dawid

On Mon, Oct 16, 2023 at 7:53 PM Dawid Weiss  wrote:

>
> It's actually as simple as adding:
>
> timeout-minutes: xyz
>
> to workflows.
>
>
> https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes
>
> I use it elsewhere for jobs on Windows because they tend to hang sometimes
> (for reasons unknown to me).
>
> Dawid
>
>
> On Mon, Oct 16, 2023 at 4:53 PM Robert Muir  wrote:
>
>> I think running the builds with a timeout is a good thing to do
>> anyway, for any CI build. I'm sure github actions has some fancy yaml
>> for that, but you can just do "timeout -k 1m 1h ./gradlew..." instead
>> of "./gradlew" too.
>>
>> On Mon, Oct 16, 2023 at 9:58 AM Michael McCandless
>>  wrote:
>> >
>> > When a non-committer (I think?) opens a PR, one of the committers must
>> notice it and click Approve & Run so the contributor can find out if
>> something broke in our automated tests/precommit/linting.
>> >
>> > This seems like a waste, and a friction in the worst possible place for
>> our community: new contributor onboarding experience.
>> >
>> > I think we have it to prevent e.g. a crypto mining bot of a PR sneaking
>> in and taking tons of resources to mine dogecoin or so?
>> >
>> > But 1) that doesn't seem to be happening so far, 2) when I hit "Approve
>> & Run" I never look closely to see if there is in fact a hidden crypto
>> miner in there, and 3) can't we just put some reasonable timeout on the
>> GitHub actions to block such abuse?
>> >
>> > Is this some sort of requirement by GitHub, or did we choose to turn on
>> this silly step?
>> >
>> > Mike McCandless
>> >
>> > http://blog.mikemccandless.com
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Can we get rid of "Approve & Run" on GitHub PRs by new contributors (non-committers)?

[Dawid Weiss]

It's actually as simple as adding:

timeout-minutes: xyz

to workflows.

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

I use it elsewhere for jobs on Windows because they tend to hang sometimes
(for reasons unknown to me).

Dawid


On Mon, Oct 16, 2023 at 4:53 PM Robert Muir  wrote:

> I think running the builds with a timeout is a good thing to do
> anyway, for any CI build. I'm sure github actions has some fancy yaml
> for that, but you can just do "timeout -k 1m 1h ./gradlew..." instead
> of "./gradlew" too.
>
> On Mon, Oct 16, 2023 at 9:58 AM Michael McCandless
>  wrote:
> >
> > When a non-committer (I think?) opens a PR, one of the committers must
> notice it and click Approve & Run so the contributor can find out if
> something broke in our automated tests/precommit/linting.
> >
> > This seems like a waste, and a friction in the worst possible place for
> our community: new contributor onboarding experience.
> >
> > I think we have it to prevent e.g. a crypto mining bot of a PR sneaking
> in and taking tons of resources to mine dogecoin or so?
> >
> > But 1) that doesn't seem to be happening so far, 2) when I hit "Approve
> & Run" I never look closely to see if there is in fact a hidden crypto
> miner in there, and 3) can't we just put some reasonable timeout on the
> GitHub actions to block such abuse?
> >
> > Is this some sort of requirement by GitHub, or did we choose to turn on
> this silly step?
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [JENKINS] Lucene » Lucene-Coverage-main - Build # 924 - Still Failing!

[Dawid Weiss]

Filed a security policy correction to allow jacoco to use a custom class
loader, here:
https://github.com/apache/lucene/pull/12684

On Sun, Oct 15, 2023 at 5:05 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/924/
>
> All tests passed
>
> Build Log:
> [...truncated 1221 lines...]
> BUILD FAILED in 31m 5s
> 320 actionable tasks: 320 executed
>
> Publishing build scan...
> https://ge.apache.org/s/fbhdgk7liizem
>
> Build step 'Invoke Gradle script' changed build result to FAILURE
> Build step 'Invoke Gradle script' marked build as failure
> Archiving artifacts
> Recording test results
> [Checks API] No suitable checks publisher found.
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-Coverage-main - Build # 921 - Failure!

[Dawid Weiss]

The convention for accessing jacoco reports has changed in gradle 8.x and
the aggregation plugin didn't work with the newer version. I've fixed this.

D.

On Thu, Oct 12, 2023 at 4:52 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/921/
>
> All tests passed
>
> Build Log:
> [...truncated 1351 lines...]
> BUILD FAILED in 31m 38s
> 280 actionable tasks: 280 executed
>
> Publishing build scan...
> https://ge.apache.org/s/yrb6zsep6sdfo
>
> Build step 'Invoke Gradle script' changed build result to FAILURE
> Build step 'Invoke Gradle script' marked build as failure
> Archiving artifacts
> Recording test results
> [Checks API] No suitable checks publisher found.
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-NightlyTests-main - Build # 1132 - Unstable!

[Dawid Weiss]

This failed because the thread limit has been exhausted.

[3241.014s][warning][os,thread] Failed to start thread "Unknown
thread" - pthread_create failed (EAGAIN) for attributes: stacksize:
1024k, guardsize: 0k, detached.
[3241.015s][warning][os,thread] Failed to start the native thread for
java.lang.Thread "LuceneTestCase-7564-thread-1

org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs > testP7 FAILED
java.lang.OutOfMemoryError: unable to create native thread:
possibly out of memory or process/resource limits reached
at 
__randomizedtesting.SeedInfo.seed([B5DBA9A0960A3ECC:1F73384DEAF58B95]:0)
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:798)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1343)
at org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:73)
at org.apache.lucene.index.TermStates.build(TermStates.java:119)
at org.apache.lucene.search.PhraseQuery$1.getStats(PhraseQuery.java:458)
at org.apache.lucene.search.PhraseWeight.(PhraseWeight.java:44)
at org.apache.lucene.search.PhraseQuery$1.(PhraseQuery.java:439)
at 
org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:439)
at 
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:893)
at 
org.apache.lucene.tests.search.AssertingIndexSearcher.createWeight(AssertingIndexSearcher.java:62)
at org.apache.lucene.search.BooleanWeight.(BooleanWeight.java:59)
at 
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:245)
at 
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:893)
at 
org.apache.lucene.tests.search.AssertingIndexSearcher.createWeight(AssertingIndexSearcher.java:62)
at 
org.apache.lucene.tests.search.QueryUtils$4.doSetNextReader(QueryUtils.java:617)
at 
org.apache.lucene.search.SimpleCollector.getLeafCollector(SimpleCollector.java:31)
at 
org.apache.lucene.search.FilterCollector.getLeafCollector(FilterCollector.java:38)
at 
org.apache.lucene.tests.search.AssertingCollector.getLeafCollector(AssertingCollector.java:54)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:748)
at 
org.apache.lucene.tests.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:79)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:547)
at 
org.apache.lucene.tests.search.QueryUtils.checkFirstSkipTo(QueryUtils.java:549)
at org.apache.lucene.tests.search.QueryUtils.check(QueryUtils.java:138)
at org.apache.lucene.tests.search.QueryUtils.check(QueryUtils.java:131)
at 
org.apache.lucene.tests.search.CheckHits.checkHitCollector(CheckHits.java:106)
at 
org.apache.lucene.tests.search.BaseExplanationTestCase.qtest(BaseExplanationTestCase.java:110)
at 
org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs.qtest(TestSimpleExplanationsWithFillerDocs.java:116)
at 
org.apache.lucene.search.TestSimpleExplanations.testP7(TestSimpleExplanations.java:87)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at 
org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at 
org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at 
org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at 
org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at 
org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at 

Re: [JENKINS] Lucene » Lucene-NightlyTests-9.x - Build # 665 - Unstable!

[Dawid Weiss]

https://ge.apache.org/s/orksynljk2yp6/tests/task/:lucene:core:test/details/org.apache.lucene.store.TestStressLockFactories/testSimpleFSLockFactory?top-execution=1

This test took 31 seconds... An extremely slow vm, perhaps? I don't know
what the default connection timeouts are... it does look weird though.

Dawid

On Thu, Aug 31, 2023 at 1:08 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Good grief -- why are we getting SocketTimeoutException in our
> LockVerifyServer's attempt to accept an incoming connection!?  These are
> all processes running on the same host ...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Aug 29, 2023 at 11:17 PM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> Build:
>> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.x/665/
>>
>> 2 tests failed.
>> FAILED:
>> org.apache.lucene.store.TestStressLockFactories.testSimpleFSLockFactory
>>
>> Error Message:
>> java.net.SocketTimeoutException: Accept timed out
>>
>> Stack Trace:
>> java.net.SocketTimeoutException: Accept timed out
>> at
>> __randomizedtesting.SeedInfo.seed([E1AD0D2AD68BA993:F325FE2A6E367AC7]:0)
>> at java.base/java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.base/java.net
>> .AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:474)
>> at java.base/java.net
>> .ServerSocket.implAccept(ServerSocket.java:565)
>> at java.base/java.net.ServerSocket.accept(ServerSocket.java:533)
>> at
>> org.apache.lucene.store.LockVerifyServer.run(LockVerifyServer.java:62)
>> at
>> org.apache.lucene.store.TestStressLockFactories.runImpl(TestStressLockFactories.java:53)
>> at
>> org.apache.lucene.store.TestStressLockFactories.testSimpleFSLockFactory(TestStressLockFactories.java:104)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>> at
>> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
>> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>> at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>> at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> 

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-17.0.5) - Build # 12560 - Unstable!

[Dawid Weiss]

The real reason for this is buried in other stack traces from
barrier-broken threads, it's this assertion:

Caused by:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([F7B4CD7A5624D5EC]:0)
at app//org.junit.Assert.fail(Assert.java:87)
at app//org.junit.Assert.assertTrue(Assert.java:42)
at app//org.junit.Assert.assertTrue(Assert.java:53)
at 
app//org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:150)
at 
java.base@17.0.5/java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:222)
at 
java.base@17.0.5/java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:364)
at 
app//org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:236)



On Mon, Aug 28, 2023 at 2:04 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/12560/
> Java: 64bit/openj9/jdk-17.0.5 -XX:+UseCompressedOops -Xgcpolicy:balanced
>
> 2 tests failed.
> FAILED:
> org.apache.lucene.index.TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom
>
> Error Message:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=1089, name=Thread-814,
> state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=1089, name=Thread-814,
> state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.BrokenBarrierException
> at __randomizedtesting.SeedInfo.seed([F7B4CD7A5624D5EC]:0)
> at
> app//org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:239)
> Caused by: java.util.concurrent.BrokenBarrierException
> at java.base@17.0.5
> /java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:252)
> at java.base@17.0.5
> /java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:364)
> at
> app//org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:236)
>
>
> FAILED:
> org.apache.lucene.index.TestIndexWriterThreadsToSegments.classMethod
>
> Error Message:
> java.lang.AssertionError: The test or suite printed 8227 bytes to stdout
> and stderr, even though the limit was set to 8192 bytes. Increase the limit
> with @Limit, ignore it completely with @SuppressSysoutChecks or run with
> -Dtests.verbose=true
>
> Stack Trace:
> java.lang.AssertionError: The test or suite printed 8227 bytes to stdout
> and stderr, even though the limit was set to 8192 bytes. Increase the limit
> with @Limit, ignore it completely with @SuppressSysoutChecks or run with
> -Dtests.verbose=true
> at __randomizedtesting.SeedInfo.seed([F7B4CD7A5624D5EC]:0)
> at
> app//org.apache.lucene.tests.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:283)
> at
> app//com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
> at
> app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
> at
> app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at
> app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
> app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at
> app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at
> app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
> app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at
> app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
> at java.base@17.0.5/java.lang.Thread.run(Thread.java:857)
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: Branchless binary search in Java?

[Dawid Weiss]

> Actually this is exactly the same for Java:
>

Yup, I know (we all know by now, I guess). People (including me) evidently
crave this low, iron-level control, while at the same time mostly try to
dodge writing any software in languages that are designed to be close to
the hardware. There is a love-hate relationship there that I often find
amusing.

D.

>

Re: Branchless binary search in Java?

[Dawid Weiss]

> Specifically, one of the fascinating Tantivy optimizations is the
> branchless binary search: https://quickwit.io/blog/search-a-sorted-block.
>

This is an interesting post, thanks for sharing, Mike. I remember when
people did such low-level tricks frequently (but on much simpler processors
and fairly consistent hardware) and it
always makes me wonder whether all the moving blocks involved here (rust,
llvm, actual hardware) make it sane - any change in any of these layers may
affect the outcome (and debugging what actually happened will be a
nightmare...). I like it though - nice intellectual exercise and some
assembly dumps for a change. ;)

D.

>

Re: [VOTE] Release PyLucene 9.7.0-rc1

[Dawid Weiss]

+1 to release. Thanks Andi.

On Thu, Jul 6, 2023 at 9:47 AM Andi Vajda  wrote:

>
> The PyLucene 9.7.0 (rc1) release tracking the recent release of
> Apache Lucene 9.7.0 is ready.
>
> A release candidate is available from:
> https://dist.apache.org/repos/dist/dev/lucene/pylucene/9.7.0-rc1/
>
> PyLucene 9.7.0 is built with JCC 3.13, included in these release artifacts.
>
> JCC 3.13 supports Python 3.3 up to Python 3.11.
> PyLucene may also be built with Python 2 but this configuration is no
> longer
> tested.
>
> Please vote to release these artifacts as PyLucene 9.7.0.
> Anyone interested in this release can and should vote !
>
> Thanks !
>
> Andi..
>
> ps: the KEYS file for PyLucene release signing is at:
> https://dist.apache.org/repos/dist/release/lucene/pylucene/KEYS
> https://dist.apache.org/repos/dist/dev/lucene/pylucene/KEYS
>
> pps: here is my +1
>

Re: Welcome Chris Hegarty to the Lucene PMC

[Dawid Weiss]

Congratulations and welcome, Chris!

Dawid

On Mon, Jun 19, 2023 at 11:53 AM Adrien Grand  wrote:

> I'm pleased to announce that Chris Hegarty has accepted an invitation to
> join the Lucene PMC!
>
> Congratulations Chris, and welcome aboard!
>
> --
> Adrien
>

Re: [JENKINS] Lucene-9.x-Linux (64bit/hotspot/jdk-11.0.15) - Build # 11132 - Unstable!

[Dawid Weiss]

I ran it quickly and it does reproduce on 9x:

gradlew :lucene:grouping:test  -Ptests.jvms=6 -Ptests.haltonfailure=false
 -Ptests.seed=48999F57C5128295 -Ptests.multiplier=3 -Ptests.badapples=false
-Ptests.gui=true -Ptests.file.encoding=UTF-8

   > java.lang.IndexOutOfBoundsException: Range [0, 0 + 332) out of
bounds for length 80
   > at
__randomizedtesting.SeedInfo.seed([48999F57C5128295:3AD5BA58747234E6]:0)
   > at
java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
   > at
java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromIndexSize(Preconditions.java:82)
   > at
java.base/jdk.internal.util.Preconditions.checkFromIndexSize(Preconditions.java:361)
   > at
java.base/java.util.Objects.checkFromIndexSize(Objects.java:411)
   > at
java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:180)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.store.ByteBuffersDataInput.readBytes(ByteBuffersDataInput.java:159)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.store.ByteBuffersIndexInput.readBytes(ByteBuffersIndexInput.java:85)
   > at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
/org.apache.lucene.tests.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:148)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1235)
   > at org.apache.lucene.core@9.7.0-SNAPSHOT
/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
   > at
org.apache.lucene.search.grouping.TermGroupFacetCollector$MV$SegmentResult.nextTerm(TermGroupFacetCollector.java:437)
   > at
org.apache.lucene.search.grouping.GroupFacetCollector.mergeSegmentResults(GroupFacetCollector.java:97)
   > at
org.apache.lucene.search.grouping.TestGroupFacetCollector.testRandom(TestGroupFacetCollector.java:429)



On Fri, Jun 16, 2023 at 7:40 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/11132/
> Java: 64bit/hotspot/jdk-11.0.15 -XX:-UseCompressedOops
> -XX:+UseConcMarkSweepGC
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.search.grouping.TestGroupFacetCollector.testRandom
>
> Error Message:
> java.lang.IndexOutOfBoundsException
>
> Stack Trace:
> java.lang.IndexOutOfBoundsException
> at
> __randomizedtesting.SeedInfo.seed([48999F57C5128295:3AD5BA58747234E6]:0)
> at java.base/java.nio.Buffer.checkBounds(Buffer.java:714)
> at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:179)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.store.ByteBuffersDataInput.readBytes(ByteBuffersDataInput.java:159)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.store.ByteBuffersIndexInput.readBytes(ByteBuffersIndexInput.java:85)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:148)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1235)
> at org.apache.lucene.core@9.7.0-SNAPSHOT
> /org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
> at
> org.apache.lucene.search.grouping.TermGroupFacetCollector$MV$SegmentResult.nextTerm(TermGroupFacetCollector.java:437)
> at
> org.apache.lucene.search.grouping.GroupFacetCollector.mergeSegmentResults(GroupFacetCollector.java:97)
> at
> org.apache.lucene.search.grouping.TestGroupFacetCollector.testRandom(TestGroupFacetCollector.java:429)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> 

Re: [JENKINS] Lucene » Lucene-Check-9.x - Build # 5577 - Unstable!

[Dawid Weiss]

Addressed by
https://github.com/apache/lucene/pull/12351

On Wed, Jun 7, 2023 at 1:20 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-9.x/5577/
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.analysis.hunspell.TestHunspell.testCustomCheckCanceledGivesPartialResult
>
> Error Message:
> java.lang.AssertionError: expected:<[apach]> but was:<[]>
>
> Stack Trace:
> java.lang.AssertionError: expected:<[apach]> but was:<[]>
> at
> __randomizedtesting.SeedInfo.seed([E65172D23B0D:A4EE450BAECC83DD]:0)
> at junit@4.13.1/org.junit.Assert.fail(Assert.java:89)
> at junit@4.13.1/org.junit.Assert.failNotEquals(Assert.java:835)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:120)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.lucene.analysis.hunspell.TestHunspell.testCustomCheckCanceledGivesPartialResult(TestHunspell.java:77)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> 

Re: [JENKINS] Lucene » Lucene-Check-9.x - Build # 5535 - Unstable!

[Dawid Weiss]

Can't reproduce this one, odd.

Dawid

On Sat, Jun 3, 2023 at 12:45 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-9.x/5535/
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.analysis.hunspell.TestHunspell.testCustomCheckCanceledGivesPartialResult
>
> Error Message:
> java.lang.AssertionError: expected:<[apach]> but was:<[]>
>
> Stack Trace:
> java.lang.AssertionError: expected:<[apach]> but was:<[]>
> at
> __randomizedtesting.SeedInfo.seed([A0ED2A18F883ED7D:E2521DFDB89D55AD]:0)
> at junit@4.13.1/org.junit.Assert.fail(Assert.java:89)
> at junit@4.13.1/org.junit.Assert.failNotEquals(Assert.java:835)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:120)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.lucene.analysis.hunspell.TestHunspell.testCustomCheckCanceledGivesPartialResult(TestHunspell.java:77)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@9.7.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> 

Re: [VOTE] Release PyLucene 9.6.0-rc1

[Dawid Weiss]

Late, but +1 to release.

Thank you Andi!

Dawid


On Tue, May 30, 2023 at 1:44 AM Andi Vajda  wrote:

>
> The PyLucene 9.6.0 (rc1) release tracking the recent release of
> Apache Lucene 9.6.0 is ready.
>
> A release candidate is available from:
> https://dist.apache.org/repos/dist/dev/lucene/pylucene/9.6.0-rc1/
>
> PyLucene 9.6.0 is built with JCC 3.13, included in these release artifacts.
>
> JCC 3.13 supports Python 3.3 up to Python 3.11.
> PyLucene may also be built with Python 2 but this configuration is no
> longer
> tested.
>
> Please vote to release these artifacts as PyLucene 9.6.0.
> Anyone interested in this release can and should vote !
>
> Thanks !
>
> Andi..
>
> ps: the KEYS file for PyLucene release signing is at:
> https://dist.apache.org/repos/dist/release/lucene/pylucene/KEYS
> https://dist.apache.org/repos/dist/dev/lucene/pylucene/KEYS
>
> pps: here is my +1
>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-17.0.5) - Build # 10811 - Unstable!

[Dawid Weiss]

>
> We should sometimes run the OpenJ9 to make it explicit that it doesn't
> work.
>
It doesn't - there are occasional hiccups that repeat from time to time and
cannot be reproduced with hotspot. We can create an umbrella issue for
these errors and just append logs there, perhaps it'll be of some use, I
don't know.

> Maybe it helps to improve their code. Unfortunately I don't know how to
best report issues like this.

Neither do I. Maybe chatgpt knows...

D.

>

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-17.0.5) - Build # 10811 - Unstable!

[Dawid Weiss]

There are frequent odd errors on J9. I think we should turn it off and not
recommend it to people, something is wrong with it.

On Thu, Jun 1, 2023 at 12:00 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Does not reproduce for me, but, it looks like it's IBM's J9: Java:
> 64bit/openj9/jdk-17.0.5 -XX:+UseCompressedOops -Xgcpolicy:balanced
>
> Maybe a J9 bug...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Thu, Jun 1, 2023 at 5:54 AM Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
>> Hmm, maybe it's a stdout/stderr issue -- the juicy failure details come
>> out in stderr:
>>
>>
>> https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/10811/testReport/junit/org.apache.lucene.index/TestIndexFileDeleter/testExcInDecRef/
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Thu, Jun 1, 2023 at 5:53 AM Michael McCandless <
>> luc...@mikemccandless.com> wrote:
>>
>>> It used to be that these build failure emails included the "Reproduce
>>> with: ...", thanks to Steve Rowe's incredible regular expression (the
>>> largest real-world regexp I have ever seen!!!).
>>>
>>> But I don't see it here anymore.  Does anyone know why/when it broke?
>>>
>>> I know I can click through to the Jenkins details and see the specifics,
>>> but, those are aggressively deleted ...
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>>
>>> On Thu, Jun 1, 2023 at 3:48 AM Policeman Jenkins Server <
>>> jenk...@thetaphi.de> wrote:
>>>
 Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/10811/
 Java: 64bit/openj9/jdk-17.0.5 -XX:+UseCompressedOops -Xgcpolicy:balanced

 1 tests failed.
 FAILED:  org.apache.lucene.index.TestIndexFileDeleter.testExcInDecRef

 Error Message:
 org.apache.lucene.store.AlreadyClosedException: this DocumentsWriter is
 closed

 Stack Trace:
 org.apache.lucene.store.AlreadyClosedException: this DocumentsWriter is
 closed
 at
 __randomizedtesting.SeedInfo.seed([909AEB548A3F7E78:79079C66FCF69985]:0)
 at
 app//org.apache.lucene.index.DocumentsWriter.ensureOpen(DocumentsWriter.java:200)
 at
 app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:429)
 at
 app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1532)
 at
 app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1817)
 at
 app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1470)
 at
 app//org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:222)
 at
 app//org.apache.lucene.index.TestIndexFileDeleter.testExcInDecRef(TestIndexFileDeleter.java:495)
 at 
 java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method)
 at java.base@17.0.5
 /jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
 at java.base@17.0.5
 /jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base@17.0.5
 /java.lang.reflect.Method.invoke(Method.java:568)
 at
 app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
 at
 app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
 at
 app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
 at
 app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
 at
 app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
 at
 app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
 at
 app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
 at
 app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
 at
 app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
 at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
 at
 app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
 app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
 at
 app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
 at
 app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
 at
 

Re: [VOTE] Dimension Limit for KNN Vectors

[Dawid Weiss]

I'm for option 3 (limit at algorithm level), with the default there tunable
via property (option 4).

I understand Robert's concerns and I'd love to contribute a faster
implementation but the reality is - I can't do it at the moment. I feel
like experiments are good though and we shouldn't just ban people from
trying - if somebody changes the (sane) default and gets burned by
performance, perhaps it'll be an itch to work on speeding things up (much
like it's already happening with Jonathan's patch).

Dawid

On Tue, May 16, 2023 at 10:50 AM Alessandro Benedetti 
wrote:

> Hi all,
> we have finalized all the options proposed by the community and we are
> ready to vote for the preferred one and then proceed with the
> implementation.
>
> *Option 1*
> Keep it as it is (dimension limit hardcoded to 1024)
> *Motivation*:
> We are close to improving on many fronts. Given the criticality of Lucene
> in computing infrastructure and the concerns raised by one of the most
> active stewards of the project, I think we should keep working toward
> improving the feature as is and move to up the limit after we can
> demonstrate improvement unambiguously.
>
> *Option 2*
> make the limit configurable, for example through a system property
> *Motivation*:
> The system administrator can enforce a limit its users need to respect
> that it's in line with whatever the admin decided to be acceptable for
> them.
> The default can stay the current one.
> This should open the doors for Apache Solr, Elasticsearch, OpenSearch, and
> any sort of plugin development
>
> *Option 3*
> Move the max dimension limit lower level to a HNSW specific
> implementation. Once there, this limit would not bind any other potential
> vector engine alternative/evolution.
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> *Option 4*
> Make it configurable and move it to an appropriate place.
> In particular, a simple Integer.getInteger("lucene.hnsw.maxDimensions",
> 1024) should be enough.
> *Motivation*:
> Both are good and not mutually exclusive and could happen in any order.
> Someone suggested to perfect what the _default_ limit should be, but I've
> not seen an argument _against_ configurability.  Especially in this way --
> a toggle that doesn't bind Lucene's APIs in any way.
>
> I'll keep this [VOTE] open for a week and then proceed to the
> implementation.
> --
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benede...@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io 
> LinkedIn  | Twitter
>  | Youtube
>  | Github
> 
>

Re: How to create a local build that targets Java 11, when building with 17?

[Dawid Weiss]

Apologies for not replying yesterday and thank you, Greg, for answering the
question.

All I could add is that it's a pretty common semantic versioning scheme
[1], although there is no rigorous method of policing it - very often it's
just relying on common sense. Lucene often does add big code changes to
what's a "minor" release in this scheme (9x) but only if they're backward
compatible. Major releases are limited to breaking the API, bumping the
minimum Java version, etc.

Dawid

[1] https://semver.org/

On Sat, May 6, 2023 at 1:53 AM Greg Miller  wrote:

> Hi Jonathan-
>
> The main branch is the tip of development, and what will eventually become
> 10.0. It can use a later version of Java, make (some)
> non-backwards-compatible API changes, etc. branch_9x tracks the latest 9.x
> release, and must run on the version of Java supported by 9.x releases,
> must be API backwards-compatible, etc. The general approach is to make
> changes against main, and then backport those changes to branch_9x in a 9.x
> friendly way if possible. Sometimes a change on main is complex enough that
> backporting in a 9.x friendly manner isn't really feasible, in which case
> the change will be released with 10.0. I'm sure I'm leaving out some
> details, but hopefully this is helpful. You may also find this reference
> useful:
> https://cwiki.apache.org/confluence/display/LUCENE/BackwardsCompatibility
>
> Cheers,
> -Greg
>
> On Fri, May 5, 2023 at 12:00 PM Jonathan Ellis  wrote:
>
>> Thanks.  What are the rules for what should go into main vs branch_9x?
>>
>> On Fri, May 5, 2023 at 1:54 PM Dawid Weiss  wrote:
>>
>>>
>>> The main branch is on Java 17, see build.gradle:
>>>
>>>   // Minimum Java version required to compile and run Lucene.
>>>   minJavaVersion = JavaVersion.VERSION_17
>>>
>>> Also, don't use the default gradle task created by convention; use this
>>> one:
>>>
>>> ./gradlew mavenToLocal
>>>
>>> it's an alias but it publishes only a subset of relevant projects, not
>>> all of them.
>>>
>>> Dawid
>>>
>>> On Fri, May 5, 2023 at 8:03 PM Jonathan Ellis  wrote:
>>>
>>>> Actually my hack doesn't work, the manifest file changes but the .class
>>>> files do not.
>>>>
>>>> On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis 
>>>> wrote:
>>>>
>>>>> `./gradlew publishToMavenLocal` gives me Java 17 class files by
>>>>> default, which surprises me since AFAIK 11 is still the minimum to run
>>>>> Lucene.
>>>>>
>>>>> I hacked it to work by editing javac.gradle
>>>>> sourceCompatibility = JavaVersion.VERSION_11
>>>>> targetCompatibility = JavaVersion.VERSION_11
>>>>>
>>>>> Is there a cleaner way to do this?
>>>>>
>>>>> --
>>>>> Jonathan Ellis
>>>>> co-founder, http://www.datastax.com
>>>>> @spyced
>>>>>
>>>>
>>>>
>>>> --
>>>> Jonathan Ellis
>>>> co-founder, http://www.datastax.com
>>>> @spyced
>>>>
>>>
>>
>> --
>> Jonathan Ellis
>> co-founder, http://www.datastax.com
>> @spyced
>>
>

Re: How to create a local build that targets Java 11, when building with 17?

[Dawid Weiss]

The main branch is on Java 17, see build.gradle:

  // Minimum Java version required to compile and run Lucene.
  minJavaVersion = JavaVersion.VERSION_17

Also, don't use the default gradle task created by convention; use this one:

./gradlew mavenToLocal

it's an alias but it publishes only a subset of relevant projects, not all
of them.

Dawid

On Fri, May 5, 2023 at 8:03 PM Jonathan Ellis  wrote:

> Actually my hack doesn't work, the manifest file changes but the .class
> files do not.
>
> On Fri, May 5, 2023 at 12:38 PM Jonathan Ellis  wrote:
>
>> `./gradlew publishToMavenLocal` gives me Java 17 class files by default,
>> which surprises me since AFAIK 11 is still the minimum to run Lucene.
>>
>> I hacked it to work by editing javac.gradle
>> sourceCompatibility = JavaVersion.VERSION_11
>> targetCompatibility = JavaVersion.VERSION_11
>>
>> Is there a cleaner way to do this?
>>
>> --
>> Jonathan Ellis
>> co-founder, http://www.datastax.com
>> @spyced
>>
>
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>

Re: [JENKINS] Lucene » Lucene-Check-main - Build # 8942 - Still Failing!

[Dawid Weiss]

I filed an infras issue to see what's causing those failures.
https://issues.apache.org/jira/browse/INFRA-24524



On Thu, Apr 27, 2023 at 3:11 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/8942/
>
> All tests passed
>
> Build Log:
> [...truncated 75 lines...]
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-Check-9.x - Build # 5171 - Unstable!

[Dawid Weiss]

Peter fixed it in 5e0761eab510f695806fdc3b51d0e9ea85497a34

On Sun, Apr 23, 2023 at 6:12 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-9.x/5171/
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.analysis.hunspell.TestHunspell.testSuggestionOrderStabilityOnDictionaryEditing
>
> Error Message:
> java.lang.AssertionError: expected:<[some_wordD, some_wordE, some_wordM,
> some_wordO]> but was:<[]>
>
> Stack Trace:
> java.lang.AssertionError: expected:<[some_wordD, some_wordE, some_wordM,
> some_wordO]> but was:<[]>
> at
> __randomizedtesting.SeedInfo.seed([DFCA1683F8DC8187:75EA94413E5E0164]:0)
> at junit@4.13.1/org.junit.Assert.fail(Assert.java:89)
> at junit@4.13.1/org.junit.Assert.failNotEquals(Assert.java:835)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:120)
> at junit@4.13.1/org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.lucene.analysis.hunspell.TestHunspell.testSuggestionOrderStabilityOnDictionaryEditing(TestHunspell.java:275)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at junit@4.13.1
> /org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.test_framework@9.6.0-SNAPSHOT
> /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at randomizedtesting.runner@2.8.1
> 

Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-17.0.5) - Build # 9891 - Unstable!

[Dawid Weiss]

openj9. Does not reproduce for me.

On Thu, Apr 20, 2023 at 4:50 AM Policeman Jenkins Server
 wrote:
>
> Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/9891/
> Java: 64bit/openj9/jdk-17.0.5 -XX:-UseCompressedOops -Xgcpolicy:metronome
>
> 1 tests failed.
> FAILED:  org.apache.lucene.misc.document.TestLazyDocument.testLazy
>
> Error Message:
> java.lang.ArrayIndexOutOfBoundsException
>
> Stack Trace:
> java.lang.ArrayIndexOutOfBoundsException
> at 
> __randomizedtesting.SeedInfo.seed([620750232F3F24D0:53DE5D719F63F07B]:0)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.BytesStore$2.readByte(BytesStore.java:459)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.store.DataInput.readVLong(DataInput.java:224)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.store.DataInput.readVLong(DataInput.java:209)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FST.readUnpackedNodeTarget(FST.java:1119)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FST.readArc(FST.java:1385)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FST.readArcByDirectAddressing(FST.java:1292)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:1325)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FST.readFirstRealTargetArc(FST.java:1178)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FST.readNextArc(FST.java:1202)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:112)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:55)
> at 
> app/org.apache.lucene.codecs@9.6.0-SNAPSHOT/org.apache.lucene.codecs.blocktreeords.OrdsBlockTreeTermsWriter$PendingBlock.append(OrdsBlockTreeTermsWriter.java:452)
> at 
> app/org.apache.lucene.codecs@9.6.0-SNAPSHOT/org.apache.lucene.codecs.blocktreeords.OrdsBlockTreeTermsWriter$PendingBlock.compileIndex(OrdsBlockTreeTermsWriter.java:419)
> at 
> app/org.apache.lucene.codecs@9.6.0-SNAPSHOT/org.apache.lucene.codecs.blocktreeords.OrdsBlockTreeTermsWriter$TermsWriter.writeBlocks(OrdsBlockTreeTermsWriter.java:616)
> at 
> app/org.apache.lucene.codecs@9.6.0-SNAPSHOT/org.apache.lucene.codecs.blocktreeords.OrdsBlockTreeTermsWriter$TermsWriter.finish(OrdsBlockTreeTermsWriter.java:917)
> at 
> app/org.apache.lucene.codecs@9.6.0-SNAPSHOT/org.apache.lucene.codecs.blocktreeords.OrdsBlockTreeTermsWriter.write(OrdsBlockTreeTermsWriter.java:258)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:172)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:135)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:310)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:392)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:492)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:671)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4194)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4168)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1322)
> at 
> app/org.apache.lucene.core@9.6.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1362)
> at 
> app//org.apache.lucene.misc.document.TestLazyDocument.createIndex(TestLazyDocument.java:82)
> at 
> java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
> at 
> java.base@17.0.5/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at 
> java.base@17.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@17.0.5/java.lang.reflect.Method.invoke(Method.java:568)
> at 
> app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at 
> app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
> at 
> 

Re: [Proposal] Remove max number of dimensions for KNN vectors

[Dawid Weiss]

> We do have a dataset built from Wikipedia in luceneutil. It comes in 100 and 
> 300 dimensional varieties and can easily enough generate large numbers of 
> vector documents from the articles data. To go higher we could concatenate 
> vectors from that and I believe the performance numbers would be plausible.

Apologies - I wasn't clear - I thought of building the 1k or 2k
vectors that would be realistic. Perhaps using glove or perhaps using
some other software but something that would reflect a true 2k
dimensional space accurately with "real" data underneath. I am not
familiar enough with the field to tell whether a simple concatenation
is a good enough simulation - perhaps it is.

I would really prefer to focus on doing this kind of assessment of
feasibility/ limitations rather than arguing back and forth. I did my
experiment a while ago and I can't really tell whether there have been
improvements in the indexing/ merging part - your email contradicts my
experience Mike, so I'm a bit intrigued and would like to revisit it.
But it'd be ideal to work with real vectors rather than a simulation.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [Proposal] Remove max number of dimensions for KNN vectors

[Dawid Weiss]

Can we set up a branch in which the limit is bumped to 2048, then have
a realistic, free data set (wikipedia sample or something) that has,
say, 5 million docs and vectors created using public data (glove
pre-trained embeddings or the like)? We then could run indexing on the
same hardware with 512, 1024 and 2048 and see what the numbers, limits
and behavior actually are.

I can help in writing this but not until after Easter.


Dawid

On Sat, Apr 8, 2023 at 11:29 PM Adrien Grand  wrote:
>
> As Dawid pointed out earlier on this thread, this is the rule for
> Apache projects: a single -1 vote on a code change is a veto and
> cannot be overridden. Furthermore, Robert is one of the people on this
> project who worked the most on debugging subtle bugs, making Lucene
> more robust and improving our test framework, so I'm listening when he
> voices quality concerns.
>
> The argument against removing/raising the limit that resonates with me
> the most is that it is a one-way door. As MikeS highlighted earlier on
> this thread, implementations may want to take advantage of the fact
> that there is a limit at some point too. This is why I don't want to
> remove the limit and would prefer a slight increase, such as 2048 as
> suggested in the original issue, which would enable most of the things
> that users who have been asking about raising the limit would like to
> do.
>
> I agree that the merge-time memory usage and slow indexing rate are
> not great. But it's still possible to index multi-million vector
> datasets with a 4GB heap without hitting OOMEs regardless of the
> number of dimensions, and the feedback I'm seeing is that many users
> are still interested in indexing multi-million vector datasets despite
> the slow indexing rate. I wish we could do better, and vector indexing
> is certainly more expert than text indexing, but it still is usable in
> my opinion. I understand how giving Lucene more information about
> vectors prior to indexing (e.g. clustering information as Jim pointed
> out) could help make merging faster and more memory-efficient, but I
> would really like to avoid making it a requirement for indexing
> vectors as it also makes this feature much harder to use.
>
> On Sat, Apr 8, 2023 at 9:28 PM Alessandro Benedetti
>  wrote:
> >
> > I am very attentive to listen opinions but I am un-convinced here and I an 
> > not sure that a single person opinion should be allowed to be detrimental 
> > for such an important project.
> >
> > The limit as far as I know is literally just raising an exception.
> > Removing it won't alter in any way the current performance for users in low 
> > dimensional space.
> > Removing it will just enable more users to use Lucene.
> >
> > If new users in certain situations will be unhappy with the performance, 
> > they may contribute improvements.
> > This is how you make progress.
> >
> > If it's a reputation thing, trust me that not allowing users to play with 
> > high dimensional space will equally damage it.
> >
> > To me it's really a no brainer.
> > Removing the limit and enable people to use high dimensional vectors will 
> > take minutes.
> > Improving the hnsw implementation can take months.
> > Pick one to begin with...
> >
> > And there's no-one paying me here, no company interest whatsoever, actually 
> > I pay people to contribute, I am just convinced it's a good idea.
> >
> >
> > On Sat, 8 Apr 2023, 18:57 Robert Muir,  wrote:
> >>
> >> I disagree with your categorization. I put in plenty of work and
> >> experienced plenty of pain myself, writing tests and fighting these
> >> issues, after i saw that, two releases in a row, vector indexing fell
> >> over and hit integer overflows etc on small datasets:
> >>
> >> https://github.com/apache/lucene/pull/11905
> >>
> >> Attacking me isn't helping the situation.
> >>
> >> PS: when i said the "one guy who wrote the code" I didn't mean it in
> >> any kind of demeaning fashion really. I meant to describe the current
> >> state of usability with respect to indexing a few million docs with
> >> high dimensions. You can scroll up the thread and see that at least
> >> one other committer on the project experienced similar pain as me.
> >> Then, think about users who aren't committers trying to use the
> >> functionality!
> >>
> >> On Sat, Apr 8, 2023 at 12:51 PM Michael Sokolov  wrote:
> >> >
> >> > What you said about increasing dimensions requiring a bigger ram buffer 
> >> > on merge is wrong. That's the point I was trying to make. Your concerns 
> >> > about merge costs are not wrong, but your conclusion that we need to 
> >> > limit dimensions is not justified.
> >> >
> >> > You complain that hnsw sucks it doesn't scale, but when I show it scales 
> >> > linearly with dimension you just ignore that and complain about 
> >> > something entirely different.
> >> >
> >> > You demand that people run all kinds of tests to prove you wrong but 
> >> > when they do, you don't listen and you won't put in the work 

Re: [Proposal] Remove max number of dimensions for KNN vectors

[Dawid Weiss]

> Can you describe your crash in more detail?

I can't. That experiment was a while ago and a quick test to see if I
could index rather large-ish USPTO (patent office) data as vectors.
Couldn't do it then.

> How much RAM?

My indexing jobs run with rather smallish heaps to give space for I/O
buffers. Think 4-8GB at most. So yes, it could have been the problem.
I recall segment merging grew slower and slower and then simply
crashed. Lucene should work with low heap requirements, even if it
slows down. Throwing ram at the indexing/ segment merging problem
is... I don't know - not elegant?

Anyway. My main point was to remind folks about how Apache works -
code is merged in when there are no vetoes. If Rob (or anybody else)
remains unconvinced, he or she can block the change. (I didn't invent
those rules).

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [Proposal] Remove max number of dimensions for KNN vectors

[Dawid Weiss]

> Ok, so what should we do then?

I don't know, Alessandro. I just wanted to point out the fact that by
Apache rules a committer's veto to a code change counts as a no-go. It
does not specify any way to "override" such a veto, perhaps counting
on disagreeing parties to resolve conflicting points of views in a
civil manner so that veto can be retracted (or a different solution
suggested).

I think Robert's point is not about a particular limit value but about
the algorithm itself - the current implementation does not scale. I
don't want to be an advocate for either side - I'm all for freedom of
choice but at the same time last time I tried indexing a few million
vectors, I couldn't get far before segment merging blew up with
OOMs...

> Did anything similar happen in the past? How was the current limit added?

I honestly don't know, you'd have to git blame or look at the mailing
list archives of the original contribution.

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [Proposal] Remove max number of dimensions for KNN vectors

[Dawid Weiss]

> Should create a VOTE thread, where we propose some values with a
> justification and we vote?
>

Technically, a vote thread won't help much if there's no full consensus - a
single veto will make the patch unacceptable for merging.
https://www.apache.org/foundation/voting.html#Veto

Dawid

Fwd: [JENKINS] Lucene » Lucene-MMAPv2-Linux (s390x big endian) - Build # 114 - Still Failing!

[Dawid Weiss]

Hi Uwe,

Do you have access to this runner? This build has been failing with this
message:

> Installation is not valid. Original error message: Command returned 
> unexpected result code: 2
  Error output:
  /home/jenkins/tools/java/latest19/bin/java: 1: Syntax error: "(" unexpected

I'm not sure whether there's anything I can do config-wise to fix it. Seems
like either the shell is different or the installation is indeed broken
somehow.

Dawid

-- Forwarded message -
From: Apache Jenkins Server 
Date: Sun, Apr 2, 2023 at 4:18 PM
Subject: [JENKINS] Lucene » Lucene-MMAPv2-Linux (s390x big endian) - Build
# 114 - Still Failing!
To: 


Build:
https://ci-builds.apache.org/job/Lucene/job/Lucene-MMAPv2-Linux%20(s390x%20big%20endian)/114/

No tests ran.

Build Log:
[...truncated 78 lines...]
BUILD FAILED in 22s
2 actionable tasks: 2 executed
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
Archiving artifacts
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
Recording test results
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files
were found. Configuration error?
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19

-
To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-MMAPv2-Linux (s390x big endian) - Build # 111 - Still Failing!

[Dawid Weiss]

Seems like something is broken on the executor machine:

  /home/jenkins/tools/java/latest19/bin/java: 1: Syntax error: "(" unexpected



On Thu, Mar 30, 2023 at 4:15 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-MMAPv2-Linux%20(s390x%20big%20endian)/111/
>
> No tests ran.
>
> Build Log:
> [...truncated 78 lines...]
> BUILD FAILED in 9s
> 2 actionable tasks: 2 executed
> Build step 'Invoke Gradle script' changed build result to FAILURE
> Build step 'Invoke Gradle script' marked build as failure
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> Archiving artifacts
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> Recording test results
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> ERROR: Step ‘Publish JUnit test result report’ failed: No test report
> files were found. Configuration error?
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
> Setting JDK_19_LATEST_HOME=/home/jenkins/tools/java/latest19
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-NightlyTests-9.5 - Build # 65 - Still Failing!

[Dawid Weiss]

Corrected the checkout folder and gradle invocation script to top-level
(instead of 'checkout'). I'm a bit confused - has somebody changed these
configurations? What happened there?

Dawid

On Thu, Mar 30, 2023 at 9:13 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.5/65/
>
> No tests ran.
>
> Build Log:
> [...truncated 26 lines...]
> FATAL: The Gradle wrapper has not been found in these directories:
> /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-9.5/checkout
> Build step 'Invoke Gradle script' marked build as failure
> Archiving artifacts
> Recording test results
> ERROR: Step ‘Publish JUnit test result report’ failed: No test report
> files were found. Configuration error?
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-NightlyTests-9.x - Build # 511 - Still Failing!

[Dawid Weiss]

FATAL: The Gradle wrapper has not been found in these directories:
/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-9.x/checkout

This has been going on for a few days - it's not intermittent. It seems
like there is no git clone under there - how's that possible?

Dawid

On Thu, Mar 30, 2023 at 3:47 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build:
> https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.x/511/
>
> No tests ran.
>
> Build Log:
> [...truncated 8 lines...]
> FATAL: The Gradle wrapper has not been found in these directories:
> /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-9.x/checkout
> Build step 'Invoke Gradle script' marked build as failure
> Archiving artifacts
> Recording test results
> ERROR: Step ‘Publish JUnit test result report’ failed: No test report
> files were found. Configuration error?
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: [JENKINS] Lucene-main-Linux (64bit/openj9/jdk-17.0.5) - Build # 40909 - Failure!

[Dawid Weiss]

Can't reproduce on hotspot. It may be due to a bug in openj9.

D.

On Tue, Mar 28, 2023 at 7:03 AM Dawid Weiss  wrote:

> Not sure why the message says otherwise but this actually failed with
> an exception that doesn't look good:
>
> org.apache.lucene.codecs.memory.TestFSTPostingsFormat > testRandom FAILED
> com.carrotsearch.randomizedtesting.UncaughtExceptionError:
> Captured an uncaught exception in thread: Thread[id=121,
> name=Thread-84, state=RUNNABLE, group=TGRP-TestFSTPostingsFormat]
>
> Caused by:
> java.lang.RuntimeException:
> java.lang.ArrayIndexOutOfBoundsException
> at __randomizedtesting.SeedInfo.seed([650F9368422763A]:0)
> at
> app//org.apache.lucene.tests.index.RandomPostingsTester$TestThread.run(RandomPostingsTester.java:1403)
>
> Caused by:
> java.lang.ArrayIndexOutOfBoundsException
> at
> app//org.apache.lucene.util.fst.BytesStore$2.readByte(BytesStore.java:459)
> at
> app//org.apache.lucene.store.DataInput.readVLong(DataInput.java:188)
> at
> app//org.apache.lucene.util.fst.FST.readUnpackedNodeTarget(FST.java:1119)
> at
> app//org.apache.lucene.util.fst.FST.readArc(FST.java:1385)
> at
>
> app//org.apache.lucene.util.fst.FST.readArcByDirectAddressing(FST.java:1292)
> at
>
> app//org.apache.lucene.util.fst.FST.readArcByDirectAddressing(FST.java:1279)
> at
> app//org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1458)
> at
> app//org.apache.lucene.util.fst.FSTEnum.doSeekExact(FSTEnum.java:577)
> at
>
> app//org.apache.lucene.util.fst.BytesRefFSTEnum.seekExact(BytesRefFSTEnum.java:83)
> at
>
> app//org.apache.lucene.codecs.memory.FSTTermsReader$TermsReader$SegmentTermsEnum.seekExact(FSTTermsReader.java:388)
> at
>
> app//org.apache.lucene.tests.index.RandomPostingsTester.testTermsOneThread(RandomPostingsTester.java:1523)
> at
>
> app//org.apache.lucene.tests.index.RandomPostingsTester$TestThread.run(RandomPostingsTester.java:1400)
>
> On Tue, Mar 28, 2023 at 2:34 AM Policeman Jenkins Server
>  wrote:
> >
> > Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/40909/
> > Java: 64bit/openj9/jdk-17.0.5 -XX:-UseCompressedOops -Xgcpolicy:metronome
> >
> > All tests passed
> >
> > -
> > To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: builds-h...@lucene.apache.org
>

Re: [JENKINS] Lucene-main-Linux (64bit/openj9/jdk-17.0.5) - Build # 40909 - Failure!

[Dawid Weiss]

Not sure why the message says otherwise but this actually failed with
an exception that doesn't look good:

org.apache.lucene.codecs.memory.TestFSTPostingsFormat > testRandom FAILED
com.carrotsearch.randomizedtesting.UncaughtExceptionError:
Captured an uncaught exception in thread: Thread[id=121,
name=Thread-84, state=RUNNABLE, group=TGRP-TestFSTPostingsFormat]

Caused by:
java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException
at __randomizedtesting.SeedInfo.seed([650F9368422763A]:0)
at 
app//org.apache.lucene.tests.index.RandomPostingsTester$TestThread.run(RandomPostingsTester.java:1403)

Caused by:
java.lang.ArrayIndexOutOfBoundsException
at
app//org.apache.lucene.util.fst.BytesStore$2.readByte(BytesStore.java:459)
at
app//org.apache.lucene.store.DataInput.readVLong(DataInput.java:188)
at
app//org.apache.lucene.util.fst.FST.readUnpackedNodeTarget(FST.java:1119)
at app//org.apache.lucene.util.fst.FST.readArc(FST.java:1385)
at
app//org.apache.lucene.util.fst.FST.readArcByDirectAddressing(FST.java:1292)
at
app//org.apache.lucene.util.fst.FST.readArcByDirectAddressing(FST.java:1279)
at
app//org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1458)
at
app//org.apache.lucene.util.fst.FSTEnum.doSeekExact(FSTEnum.java:577)
at
app//org.apache.lucene.util.fst.BytesRefFSTEnum.seekExact(BytesRefFSTEnum.java:83)
at
app//org.apache.lucene.codecs.memory.FSTTermsReader$TermsReader$SegmentTermsEnum.seekExact(FSTTermsReader.java:388)
at
app//org.apache.lucene.tests.index.RandomPostingsTester.testTermsOneThread(RandomPostingsTester.java:1523)
at
app//org.apache.lucene.tests.index.RandomPostingsTester$TestThread.run(RandomPostingsTester.java:1400)

On Tue, Mar 28, 2023 at 2:34 AM Policeman Jenkins Server
 wrote:
>
> Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/40909/
> Java: 64bit/openj9/jdk-17.0.5 -XX:-UseCompressedOops -Xgcpolicy:metronome
>
> All tests passed
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene PMC Chair Greg Miller

[Dawid Weiss]

Congratulations Greg and thank you both!

Dawid

On Mon, Mar 6, 2023 at 6:16 PM Bruno Roustant  wrote:
>
> Hello Lucene developers,
>
> Lucene Program Management Committee has elected a new chair, Greg Miller, and 
> the Board has approved.
>
> Greg, thank you for stepping up, and congratulations!
>
>
> - Bruno

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Ben Trent as Lucene committer

[Dawid Weiss]

Congratulations and welcome, Ben.

Dawid

On Fri, Jan 27, 2023 at 4:18 PM Adrien Grand  wrote:

> I'm pleased to announce that Ben Trent has accepted the PMC's
> invitation to become a committer.
>
> Ben, the tradition is that new committers introduce themselves with a
> brief bio.
>
> Congratulations and welcome!
>
> --
> Adrien
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/hotspot/jdk-18) - Build # 310 - Unstable!

[Dawid Weiss]

This failed with an OOM, internally. There's a lot of noise after that...
I'm not sure if there's anything to be done here about it.

Dawid

On Fri, Jan 20, 2023 at 5:17 PM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/310/
> Java: 64bit/hotspot/jdk-18 -XX:+UseCompressedOops -XX:+UseSerialGC
>
> 1 tests failed.
> FAILED:
> org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
>
> Error Message:
> java.lang.AssertionError: pendingNumDocs 395482 != 395476 totalMaxDoc
>
> Stack Trace:
> java.lang.AssertionError: pendingNumDocs 395482 != 395476 totalMaxDoc
> at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2436)
> at
> org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5622)
> at
> org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:5612)
> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3679)
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4042)
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4004)
> at
> org.apache.lucene.tests.index.RandomIndexWriter.getReader(RandomIndexWriter.java:493)
> at
> org.apache.lucene.tests.index.RandomIndexWriter.getReader(RandomIndexWriter.java:420)
> at
> org.apache.lucene.index.TestIndexWriterMergePolicy.stressUpdateSameDocumentWithMergeOnX(TestIndexWriterMergePolicy.java:793)
> at
> org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader(TestIndexWriterMergePolicy.java:737)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
> at java.base/java.lang.reflect.Method.invoke(Method.java:577)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> at
> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
> at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
> 

Re: [JENKINS] Lucene-9.x-Windows (64bit/openj9/jdk-17.0.5) - Build # 1693 - Failure!

[Dawid Weiss]

> copypasteshitnwaste.

I just love those long German compounds! :)

Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [lucene] branch main updated: Prevent NPEs while still handling the polar case for horizontal planes right off the pole

[Dawid Weiss]

I have multiple Windows machines, including laptops. Never had this issue
(but I stay away from WSL and such). Performance is, I'd say 25% slower
than comparable Linux machines. Something is wrong with your rig.

Dawid

On Thu, Nov 24, 2022 at 12:26 PM Karl Wright  wrote:

> My entire tool set and work environment is inside WSL.
>
> I've determined that the issue for me is the performance of the file
> system.  I had to remove the (bundled) antivirus software to get even where
> I am now.  But I have no evidence that even doing windows-native operations
> with this disk are fast.  I suspect that even though this is an SSD it's
> not a very fast one.  It did get twice as fast when I turned off the new
> Windows 11 "climate change" feature, which apparently conserves energy by
> throttling the hell out of everything, including disk access.  So maybe
> this is still being throttled to some degree and I have to figure out where.
>
> Karl
>
>
> On Thu, Nov 24, 2022 at 3:23 AM Jan Høydahl  wrote:
>
>> I’m not on Windows myself, but I think the trick is doing the git clone
>> to the WSL file system. So you may have one checkout for use with windows
>> and another for use within wsl.
>>
>> And if you’re a CLI person, there’s a GitHub cli tool ‘hub’ that is
>> handy: https://hub.github.com/
>>
>> Jan Høydahl
>>
>> 17. nov. 2022 kl. 16:49 skrev Dawid Weiss :
>>
>> I never used WSL but it does seem like the problem there:
>>
>> "As you can tell from the comparison table above, the WSL 2
>> architecture outperforms WSL 1 in several ways, with the exception of
>> performance across OS file systems, which can be addressed by storing
>> your project files on the same operating system as the tools you are
>> running to work on the project."
>>
>> https://learn.microsoft.com/en-us/windows/wsl/compare-versions
>>
>> Dawid
>>
>>
>> On Thu, Nov 17, 2022 at 1:11 PM Robert Muir  wrote:
>>
>>
>> if your machine is really 12 cores and 64GB ram but is that slow, then
>>
>> uninstall that windows shit immediately, that's horrible.
>>
>>
>> On Thu, Nov 17, 2022 at 5:46 AM Karl Wright  wrote:
>>
>>
>> Thanks - the target I was using was the complete "build" target on the
>> whole project.  This will be a valuable improvement. ;-)
>>
>>
>> I have slow network here so it is possible that the entire build was slow
>> for that reason.  The machine is a new Dell laptop, 12 cores, 64GB memory,
>> but I am running under Windows Subsystem for Linux which is a bit slower
>> than native Ubuntu.  Still, the gradlew command you gave takes many minutes
>> (of which a sizable amount is spent in :gitStatus - more than 5 minutes
>> there alone).  Anything less than 10 minutes I deem acceptable, which this
>> doesn't quite manage, but I'll live.
>>
>>
>> Karl
>>
>>
>>
>> On Thu, Nov 17, 2022 at 5:06 AM Dawid Weiss 
>> wrote:
>>
>>
>>
>> Thank you for the comment.
>>
>>
>>
>> Sorry if it came out the wrong way - I certainly didn't mean it to be
>> unkind.
>>
>>
>>
>> It took me several days just to get things set up so I was able to commit
>> again, and I did this through command-line not github.
>>
>>
>>
>> These things are not mutually exclusive - I work with command line as
>> well. You just push to your own repository (or a branch, if you don't care
>> to have your own fork on github) and then file a PR from there. If you're
>> on a slower machine - this is even better since precommit checks run for
>> you there.
>>
>>
>>
>> The full gradlew script takes over 2 hours to run now so if there's a
>> faster target I can use to determine these things in advance I'd love to
>> know what it is.
>>
>>
>>
>> Well, this is crazy long so I wonder what's happening. I'd love to help
>> but it'd be good to know what machine this is (disk, cpu, memory?) and what
>> the build command was. Without knowing these, I'd say - run the tests and
>> checks for the module you've changed only, not for everything. How long
>> does this take?
>>
>>
>> ./gradlew check -p lucene/spatial3d
>>
>>
>> It takes roughly 1 minute for me, including startup (after the daemon is
>> running in the background, it's much faster).
>>
>>
>> There are some workflow examples/ hints I left here:
>>
>> https://github.com/apache/lucene/blob/main/help/workflow.txt#L6-L22
>>
>>
>> Hope it helps,
>>
>> Dawid
>>
>>
>> -
>>
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: [JENKINS] Lucene-main-Linux (64bit/jdk-17.0.3) - Build # 38480 - Failure!

[Dawid Weiss]

Hi Karl,

I fixed the boxing issue errorprone has been complaining about after your
recent commits. Some of those validation checks (errorprone) are slow(er)
and run only on the CI (github and jenkins) so a local gradle check may not
find them all.

Also, is there any reference to the problem your commits are trying to
solve (issue number)? They just appear on the main branch without a
reference - it'll be difficult to backport them to 9x, should such backport
be needed.

I'd encourage you again to create a pull request on github (off the branch
you're working on) - makes life so much easier for everyone involved (runs
full CI checks, allows others to comment, preserves the history of commits
for that particular issue as a branch/diff).

Thanks,
Dawid

On Wed, Nov 23, 2022 at 7:32 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/38480/
> Java: 64bit/jdk-17.0.3 -XX:-UseCompressedOops -XX:+UseParallelGC
>
> All tests passed
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: IntelliJ Project Generation?

[Dawid Weiss]

> https://github.com/apache/lucene/blob/main/CONTRIBUTING.md#ide-support
>

Precisely. I also highly recommend using intellij compiler instead of full
gradle integration - it works much faster for me (but both modes work).

Dawid

Re: [lucene] 02/03: Fix longstanding bug in path bounds calculation, and hook up efficient isWithin() and distance logic

[Dawid Weiss]

If you need a temporary commit to make the random tests always pass while I
> diagnose and fix please let me know.
>

Please do, thanks.

Dawid

Re: Maven artifacts and releases

[Dawid Weiss]

> I think, if smoke tester knows the version, it could just check the staging 
> repo, too. It is just another URL!?

Yes, it's an URL.

> If the staging repository on nexus has a hash inside (I think they have), 
> maybe the release wizard places some file with the URL of the actual staging 
> repo into the folder?

I spent some time digging. I am no longer convinced direct Nexus
staging would be a good thing to do because you can have multiple
staging repos (from multiple rcs) and accidentally release the wrong
one - a step from which there's no recovery...

The staging API for Nexus is quite simple - Solr's upload-maven.sh has
almost all of it covered (the exception being profile ID retrieval).
Unsurprisingly, Nexus API is documented in a terrible way and lacks
the documentation
for the key staging method (deployByRepositoryId)...

https://oss.sonatype.org/nexus-staging-plugin/default/docs/index.html

I'll write that nexus-pushing script, no problem there. I'm just
looking at options other than bash + curl because this seems rather
crude. All the pieces are in place though, it's downhill now.


Dawid

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org