Re: Lucene 9.3.0 release

2022-07-21 Thread Ignacio Vera 
indeed, just waiting for him to do the honours and push the PR.

On Thu, Jul 21, 2022 at 12:30 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Greg fixed a test failure that impacts 9.3, and opened a PR and blocker
> issue: https://issues.apache.org/jira/browse/LUCENE-10659
>
> I think we should merge it for 9.3.0?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Jul 20, 2022 at 7:43 PM Julie Tibshirani 
> wrote:
>
>> Hello Mayya, I can share my view. Strictly speaking, we only need a new
>> codec version if the on-disk format changes for some type (like kNN vectors
>> in this case). Otherwise we can just evolve the logic in the same way we do
>> with other code, making the improvements in-place. Alessandro's PR is a
>> good example of this -- it's a refactor that happens to touch the
>> vectors reader code. It doesn't require a totally new codec version.
>>
>> With your PR for LUCENE-10592, I don't think we technically need a new
>> codec for it, because the on-disk format remains the same. But since it's
>> such a big change that deeply affects the vectors writing logic, it feels
>> convenient that we're making it in a new format (maybe feels a bit clearer/
>> safer).
>>
>> Julie
>>
>> On Wed, Jul 20, 2022 at 4:21 PM Mayya Sharipova
>>  wrote:
>>
>>> Hi Julie,
>>> thanks for raising this point, I was also thinking about it.
>>> I think it would be valuable to clarify what constitutes the format
>>> changes. Is on disk changes a necessary requirement for it? Or is it enough
>>> that codec readers and writers change their behaviour?
>>>
>>> For example, for LUCENE-10592, there are extensive changes about how
>>> codec readers and writers behave, while the format on disk stays the same.
>>> Specifically for the release 9.3, Alessandro also introduced some
>>> changes  to codec readers
>>> while the format on disk stayed the same.
>>>
>>>
>>>
>>>
>>> On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani 
>>> wrote:
>>>
 Hello everyone,

 We had preemptively created a Lucene93Codec to support the vector
 search changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592).
 However neither of these changes made it for 9.3, so there have not been
 any real format changes-- Lucene93Codec is the same as Lucene92Codec.
 Should we remove Lucene93Codec on branch_9_3 (effectively reverting
 https://github.com/apache/lucene/pull/924) ?

 Julie

 On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova
  wrote:

> Thanks Adrien, it is indeed a big change, also would be nice to see
> benchmarks after it is merged.
>
> So, Ignacio, please don't wait for LUCENE-10592, we will not be able
> to make it for tomorrow.
>
> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand 
> wrote:
>
>> While I can understand the excitement about LUCENE-10592, it's also a
>> big change, maybe we should not even try to get it in before cutting the
>> branch?
>>
>> On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova
>>  wrote:
>>
>>> Thanks for the reminder about the release, Ignacio!
>>> About LUCENE-10592
>>>   I will see
>>> what progress we can make today, and will let you know before Wednesday 
>>> at
>>> 9:00 CEST.
>>>
>>> On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov 
>>> wrote:
>>>
 Thanks for checking, but please don't wait for LUCENE-10577. It's
 not clear when that might get resolved

 On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera 
 wrote:

> Just a quick reminder I plan to cut the 9.3 branch this Wednesday
> at 9:00 CEST. Let me know if there is any issue.
>
> @Mike: I see that LUCENE-10577
>  is still
> under API discussions, Do you think it will make it to the release?
> LUCENE-10592 
> seems to have made good progress, would you think it will be ready?
> @Nick: I gave you feedback on the PR. I think it is way too risky
> to add an encoding in a rush. I recommend adding the change in the
> sandbox if you want to iterate in there or wait for another release 
> where
> there is more time to think through the encoding.
>
> Cheers,
>
> Ignacio
>
> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize 
> wrote:
>
>> I'd like to get ShapeDocValuesField in for the 9.3 release (
>> https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice
>> feature for enabling facets and aggregations over XYShape and 
>> LatLonShape
>> field types and could make for a good 9.3 geo highlight.
>>
>> Nicholas Knize,

Re: [DISCUSS] Read-only Jira after the GitHub issues migration?

2022-07-21 Thread Tomoko Uchida 
Just an update on the migration procedure.

> 2. Send a message to dev@ stating new issues should now be opened in
github
> 3. Start the migration
> I think the difference with this and what was previously described on
this thread is there would be no downtime for new issues.
I confirmed it's safe to create new issues while the migration is in
progress, so there will be no downtime for new issues.

For existing issues, I don't think it'd be practical to suspend our issue
system for two or three days, I would migrate comments on existing issues
created during migration in Jira by my GitHub account. A bit awkward, but
it'd be an acceptable workaround I think.

Tomoko


2022年7月19日(火) 23:24 Tomoko Uchida :

> > I think missing a few updates would be preferable to having 10k
> messages. Just my opinion though.
> I don't have objections. Then let's disable Jira notification on issues@
> before the script is started - maybe, issue watchers will notice that if
> there are comments from someone.
> Other opinions?
>
> 2022年7月19日(火) 23:17 Houston Putman :
>
>> I think missing a few updates would be preferable to having 10k messages.
>> Just my opinion though.
>>
>> On Tue, Jul 19, 2022 at 10:11 AM Tomoko Uchida <
>> tomoko.uchida.1...@gmail.com> wrote:
>>
>>> > 1. Make Jira read only

 At the very last step, we'll add comments saying "This was moved GitHub
 " to each Jira issue. It has to be done after the migration was
 completed.

>>>
>>> > Is this going to send 10k emails to the mailing list? I’ll need to
>>> update my filters so that these skip my inbox in that case.
>>>
>>> Yes, I will let you all know the mail template before starting the
>>> migration.
>>> Or, a Jira project admin could completely disable notifications from
>>> Jira - but this could bury real notifications (issues/comments by humans
>>> who don't recognize the migration).
>>>
>>> 2022年7月19日(火) 23:05 Tomoko Uchida :
>>>
 > 2. Send a message to dev@ stating new issues should now be opened in
 github
 > 3. Start the migration

 Maybe we can do a simulation for this.
 I plan a rehearsal that migrates whole existing issues into a test repo
 next week. Could some people help/test it (randomly open/close issues, add
 comments, etc. while the migration script is running)?


 2022年7月19日(火) 22:47 Tomoko Uchida :

> > 1. Make Jira read only
>
> At the very last step, we'll add comments saying "This was moved
> GitHub " to each Jira issue. It has to be done after the migration 
> was
> completed.
>
> > 2. Send a message to dev@ stating new issues should now be opened
> in github
> > 3. Start the migration
>
> In theory, it would be okay to me. I just wanted to avoid any risks
> during migration. Let me give time to consider/check if the migration can
> be safely done while new issues are created (by humans).
>
>
> 2022年7月19日(火) 21:56 Ryan Ernst :
>
>> > Yes, it won't be a really atomic switch
>>
>> While it may not be completely atomic, could it be closer? GitHub
>> already supports new issues, developers are just advised against opening
>> there. Could the order of events be:
>>
>> 1. Make Jira read only
>> 2. Send a message to dev@ stating new issues should now be opened in
>> github
>> 3. Start the migration
>> 4. When the migration is complete, send another message notifying
>> devs that pre-existing Jiras are now in GitHub,so they can then be
>> commented on and edited there.
>>
>> I think the difference with this and what was previously described on
>> this thread is there would be no downtime for new issues.
>>
>> On Tue, Jul 19, 2022 at 00:07 Tomoko Uchida <
>> tomoko.uchida.1...@gmail.com> wrote:
>>
>>> OK, thank you everyone for your comments/suggestions.
>>> I will ask infra to make Lucene Jira read-only after the migration
>>> is completed (if there are no explicit objections). For people who are
>>> critically affected by this change, please let me know about your
>>> inconvenience. I'll try to find acceptable solutions.
>>>
>>> > I would prefer that we make a nearly atomic switch -- up until
>>> time X we use Jira, then it goes read-only and at time X + t (t being 
>>> how
>>> long the migration takes, likely a day or two?), GitHub issues opens for
>>> business.
>>>
>>> Yes, it won't be a really atomic switch - there will be a moratorium
>>> in our issue system (where GitHub issue is not lifted yet, but a Jira
>>> snapshot is already taken). I estimate the whole migration process will
>>> take at least three days; will make a mail thread about the detailed
>>> schedule.
>>>
>>> Tomoko
>>>
>>>
>>> 2022年7月19日(火) 2:38 Gus Heck :
>>>
 I am 100% for preventing creation of new issues in Jira, new issues
 should only be

Re: [JENKINS] Lucene-main-Linux (64bit/jdk-17.0.3) - Build # 35849 - Unstable!

2022-07-21 Thread Michael McCandless 
Oh, nevermind!  I see the PR/blocker issue, thanks Greg.

EventuallyConsistentMikeException!

Mike McCandless

http://blog.mikemccandless.com


On Thu, Jul 21, 2022 at 6:28 AM Michael McCandless <
luc...@mikemccandless.com> wrote:

> Should this maybe also be backported to the 9.3.0 branch?  Did the
> original change land before that branch was cut?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Jul 20, 2022 at 5:46 PM Greg Miller  wrote:
>
>> OK, I think these test failures should now be resolved (on both main
>> and branch_9x). But I'll keep an eye on nightly builds/tests.
>>
>> Cheers,
>> -g
>>
>> On Wed, Jul 20, 2022 at 9:17 AM Greg Miller  wrote:
>> >
>> > I'll dig into this soon. Looks like a new test I recently added hit an
>> > issue. Apologies.
>> >
>> > Cheers,
>> > -g
>> >
>> > On Wed, Jul 20, 2022 at 2:32 AM Policeman Jenkins Server
>> >  wrote:
>> > >
>> > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/35849/
>> > > Java: 64bit/jdk-17.0.3 -XX:+UseCompressedOops -XX:+UseSerialGC
>> > >
>> > > 1 tests failed.
>> > > FAILED:  org.apache.lucene.search.TestDisiPriorityQueue.testRandom
>> > >
>> > > Error Message:
>> > > java.lang.IllegalArgumentException: bound must be greater than origin
>> > >
>> > > Stack Trace:
>> > > java.lang.IllegalArgumentException: bound must be greater than origin
>> > > at
>> __randomizedtesting.SeedInfo.seed([3E14AC43B544B726:4C58894C04240155]:0)
>> > > at
>> java.base/jdk.internal.util.random.RandomSupport.checkRange(RandomSupport.java:232)
>> > > at
>> java.base/java.util.random.RandomGenerator.nextInt(RandomGenerator.java:679)
>> > > at
>> org.apache.lucene.search.TestDisiPriorityQueue.testRandom(TestDisiPriorityQueue.java:47)
>> > > at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>> > > at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>> > > at
>> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
>> > > at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> > > at
>> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>> > > at
>> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> > > at
>> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> > > at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> > > at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>> > > at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>> > > at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>> > > at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>> > > at
>> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> > > at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> > > at
>> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>> > > at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> > > at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> > > at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> >

Re: Lucene 9.3.0 release

2022-07-21 Thread Michael McCandless 
Greg fixed a test failure that impacts 9.3, and opened a PR and blocker
issue: https://issues.apache.org/jira/browse/LUCENE-10659

I think we should merge it for 9.3.0?

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jul 20, 2022 at 7:43 PM Julie Tibshirani 
wrote:

> Hello Mayya, I can share my view. Strictly speaking, we only need a new
> codec version if the on-disk format changes for some type (like kNN vectors
> in this case). Otherwise we can just evolve the logic in the same way we do
> with other code, making the improvements in-place. Alessandro's PR is a
> good example of this -- it's a refactor that happens to touch the
> vectors reader code. It doesn't require a totally new codec version.
>
> With your PR for LUCENE-10592, I don't think we technically need a new
> codec for it, because the on-disk format remains the same. But since it's
> such a big change that deeply affects the vectors writing logic, it feels
> convenient that we're making it in a new format (maybe feels a bit clearer/
> safer).
>
> Julie
>
> On Wed, Jul 20, 2022 at 4:21 PM Mayya Sharipova
>  wrote:
>
>> Hi Julie,
>> thanks for raising this point, I was also thinking about it.
>> I think it would be valuable to clarify what constitutes the format
>> changes. Is on disk changes a necessary requirement for it? Or is it enough
>> that codec readers and writers change their behaviour?
>>
>> For example, for LUCENE-10592, there are extensive changes about how
>> codec readers and writers behave, while the format on disk stays the same.
>> Specifically for the release 9.3, Alessandro also introduced some
>> changes  to codec readers
>> while the format on disk stayed the same.
>>
>>
>>
>>
>> On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> We had preemptively created a Lucene93Codec to support the vector search
>>> changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592).
>>> However neither of these changes made it for 9.3, so there have not been
>>> any real format changes-- Lucene93Codec is the same as Lucene92Codec.
>>> Should we remove Lucene93Codec on branch_9_3 (effectively reverting
>>> https://github.com/apache/lucene/pull/924) ?
>>>
>>> Julie
>>>
>>> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova
>>>  wrote:
>>>
 Thanks Adrien, it is indeed a big change, also would be nice to see
 benchmarks after it is merged.

 So, Ignacio, please don't wait for LUCENE-10592, we will not be able to
 make it for tomorrow.

 On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand 
 wrote:

> While I can understand the excitement about LUCENE-10592, it's also a
> big change, maybe we should not even try to get it in before cutting the
> branch?
>
> On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova
>  wrote:
>
>> Thanks for the reminder about the release, Ignacio!
>> About LUCENE-10592
>>   I will see
>> what progress we can make today, and will let you know before Wednesday 
>> at
>> 9:00 CEST.
>>
>> On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov 
>> wrote:
>>
>>> Thanks for checking, but please don't wait for LUCENE-10577. It's
>>> not clear when that might get resolved
>>>
>>> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera 
>>> wrote:
>>>
 Just a quick reminder I plan to cut the 9.3 branch this Wednesday
 at 9:00 CEST. Let me know if there is any issue.

 @Mike: I see that LUCENE-10577
  is still
 under API discussions, Do you think it will make it to the release?
 LUCENE-10592 
 seems to have made good progress, would you think it will be ready?
 @Nick: I gave you feedback on the PR. I think it is way too risky
 to add an encoding in a rush. I recommend adding the change in the
 sandbox if you want to iterate in there or wait for another release 
 where
 there is more time to think through the encoding.

 Cheers,

 Ignacio

 On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize 
 wrote:

> I'd like to get ShapeDocValuesField in for the 9.3 release (
> https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice
> feature for enabling facets and aggregations over XYShape and 
> LatLonShape
> field types and could make for a good 9.3 geo highlight.
>
> Nicholas Knize, Ph.D., GISP
> Principal Engineer - Search  |  Amazon
> Apache Lucene PMC Member and Committer
> nkn...@apache.org
>
>
> On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera 
> wrote:
>
>> Thanks for the heads up, I am

Re: [JENKINS] Lucene-main-Linux (64bit/jdk-17.0.3) - Build # 35849 - Unstable!

2022-07-21 Thread Michael McCandless 
Should this maybe also be backported to the 9.3.0 branch?  Did the original
change land before that branch was cut?

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jul 20, 2022 at 5:46 PM Greg Miller  wrote:

> OK, I think these test failures should now be resolved (on both main
> and branch_9x). But I'll keep an eye on nightly builds/tests.
>
> Cheers,
> -g
>
> On Wed, Jul 20, 2022 at 9:17 AM Greg Miller  wrote:
> >
> > I'll dig into this soon. Looks like a new test I recently added hit an
> > issue. Apologies.
> >
> > Cheers,
> > -g
> >
> > On Wed, Jul 20, 2022 at 2:32 AM Policeman Jenkins Server
> >  wrote:
> > >
> > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/35849/
> > > Java: 64bit/jdk-17.0.3 -XX:+UseCompressedOops -XX:+UseSerialGC
> > >
> > > 1 tests failed.
> > > FAILED:  org.apache.lucene.search.TestDisiPriorityQueue.testRandom
> > >
> > > Error Message:
> > > java.lang.IllegalArgumentException: bound must be greater than origin
> > >
> > > Stack Trace:
> > > java.lang.IllegalArgumentException: bound must be greater than origin
> > > at
> __randomizedtesting.SeedInfo.seed([3E14AC43B544B726:4C58894C04240155]:0)
> > > at
> java.base/jdk.internal.util.random.RandomSupport.checkRange(RandomSupport.java:232)
> > > at
> java.base/java.util.random.RandomGenerator.nextInt(RandomGenerator.java:679)
> > > at
> org.apache.lucene.search.TestDisiPriorityQueue.testRandom(TestDisiPriorityQueue.java:47)
> > > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> > > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
> > > at
> org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
> > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > > at
> org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> > > at
> org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > > at
> org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
> > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
> > > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
> > > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
> > > at
> org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > > at
> org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > > at
> org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > > at
>

Re: Lucene 9.3.0 release

2022-07-21 Thread Michael Sokolov 
I'm sorry about this no op change that is so intrusive. Indeed it would be
better to roll it back, and I hope in the future we can find a better way
to evolve these codecs.

I would ideally handle this since I added in the first place.
Unfortunately I'm traveling until mid week next week and won't be able to
do it until then. Hopefully it is only referring a single commit?

On Wed, Jul 20, 2022, 7:21 PM Mayya Sharipova
 wrote:

> Hi Julie,
> thanks for raising this point, I was also thinking about it.
> I think it would be valuable to clarify what constitutes the format
> changes. Is on disk changes a necessary requirement for it? Or is it enough
> that codec readers and writers change their behaviour?
>
> For example, for LUCENE-10592, there are extensive changes about how codec
> readers and writers behave, while the format on disk stays the same.
> Specifically for the release 9.3, Alessandro also introduced some changes
> to codec readers while the
> format on disk stayed the same.
>
>
>
>
> On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani 
> wrote:
>
>> Hello everyone,
>>
>> We had preemptively created a Lucene93Codec to support the vector search
>> changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592).
>> However neither of these changes made it for 9.3, so there have not been
>> any real format changes-- Lucene93Codec is the same as Lucene92Codec.
>> Should we remove Lucene93Codec on branch_9_3 (effectively reverting
>> https://github.com/apache/lucene/pull/924) ?
>>
>> Julie
>>
>> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova
>>  wrote:
>>
>>> Thanks Adrien, it is indeed a big change, also would be nice to see
>>> benchmarks after it is merged.
>>>
>>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able to
>>> make it for tomorrow.
>>>
>>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand  wrote:
>>>
 While I can understand the excitement about LUCENE-10592, it's also a
 big change, maybe we should not even try to get it in before cutting the
 branch?

 On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova
  wrote:

> Thanks for the reminder about the release, Ignacio!
> About LUCENE-10592
>   I will see what
> progress we can make today, and will let you know before Wednesday at 9:00
> CEST.
>
> On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov 
> wrote:
>
>> Thanks for checking, but please don't wait for LUCENE-10577. It's not
>> clear when that might get resolved
>>
>> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera 
>> wrote:
>>
>>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday at
>>> 9:00 CEST. Let me know if there is any issue.
>>>
>>> @Mike: I see that LUCENE-10577
>>>  is still under
>>> API discussions, Do you think it will make it to the release?
>>> LUCENE-10592 
>>> seems to have made good progress, would you think it will be ready?
>>> @Nick: I gave you feedback on the PR. I think it is way too risky to
>>> add an encoding in a rush. I recommend adding the change in the sandbox 
>>> if
>>> you want to iterate in there or wait for another release where there is
>>> more time to think through the encoding.
>>>
>>> Cheers,
>>>
>>> Ignacio
>>>
>>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize 
>>> wrote:
>>>
 I'd like to get ShapeDocValuesField in for the 9.3 release (
 https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice
 feature for enabling facets and aggregations over XYShape and 
 LatLonShape
 field types and could make for a good 9.3 geo highlight.

 Nicholas Knize, Ph.D., GISP
 Principal Engineer - Search  |  Amazon
 Apache Lucene PMC Member and Committer
 nkn...@apache.org


 On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera 
 wrote:

> Thanks for the heads up, I am planning to cut the brunch middle
> next week, Wednesday July 20th.
> Let me know at the beginning of next week if there is any issue
> from your side.
>
> cheers,
>
> Ignacio
>
> On Mon, Jul 11, 2022 at 4:21 PM Michael Sokolov <
> msoko...@gmail.com> wrote:
>
>> I would like to see if we can get
>> https://issues.apache.org/jira/browse/LUCENE-10577 in. It is
>> working
>> and gives nice gains, but there is some controversy about the
>> API. If
>> we can't get it sorted out this week(?) it can certainly slip to
>> the
>> next revision. I know that
>> https://issues.apache.org/jira/browse/LUCENE-10592 is also
>> baking

Re: Lucene 9.3.0 release

2022-07-21 Thread Michael Sokolov 
OK I have an hour or so before vacation fun begins in earnest - let me
see if this is as easy as I claimed ...

On Wed, Jul 20, 2022 at 7:21 PM Mayya Sharipova
 wrote:
>
> Hi Julie,
> thanks for raising this point, I was also thinking about it.
> I think it would be valuable to clarify what constitutes the format changes. 
> Is on disk changes a necessary requirement for it? Or is it enough that codec 
> readers and writers change their behaviour?
>
> For example, for LUCENE-10592, there are extensive changes about how codec 
> readers and writers behave, while the format on disk stays the same.
> Specifically for the release 9.3, Alessandro also introduced some changes to 
> codec readers while the format on disk stayed the same.
>
>
>
>
> On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani  wrote:
>>
>> Hello everyone,
>>
>> We had preemptively created a Lucene93Codec to support the vector search 
>> changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592). However 
>> neither of these changes made it for 9.3, so there have not been any real 
>> format changes-- Lucene93Codec is the same as Lucene92Codec. Should we 
>> remove Lucene93Codec on branch_9_3 (effectively reverting 
>> https://github.com/apache/lucene/pull/924) ?
>>
>> Julie
>>
>> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova 
>>  wrote:
>>>
>>> Thanks Adrien, it is indeed a big change, also would be nice to see 
>>> benchmarks after it is merged.
>>>
>>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able to 
>>> make it for tomorrow.
>>>
>>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand  wrote:

 While I can understand the excitement about LUCENE-10592, it's also a big 
 change, maybe we should not even try to get it in before cutting the 
 branch?

 On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova 
  wrote:
>
> Thanks for the reminder about the release, Ignacio!
> About LUCENE-10592  I will see what progress we can make today, and will 
> let you know before Wednesday at 9:00 CEST.
>
> On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov  
> wrote:
>>
>> Thanks for checking, but please don't wait for LUCENE-10577. It's not 
>> clear when that might get resolved
>>
>> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera  wrote:
>>>
>>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday at 
>>> 9:00 CEST. Let me know if there is any issue.
>>>
>>> @Mike: I see that LUCENE-10577 is still under API discussions, Do you 
>>> think it will make it to the release? LUCENE-10592  seems to have made 
>>> good progress, would you think it will be ready?
>>> @Nick: I gave you feedback on the PR. I think it is way too risky to 
>>> add an encoding in a rush. I recommend adding the change in the sandbox 
>>> if you want to iterate in there or wait for another release where there 
>>> is more time to think through the encoding.
>>>
>>> Cheers,
>>>
>>> Ignacio
>>>
>>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize  wrote:

 I'd like to get ShapeDocValuesField in for the 9.3 release 
 (https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice 
 feature for enabling facets and aggregations over XYShape and 
 LatLonShape field types and could make for a good 9.3 geo highlight.

 Nicholas Knize, Ph.D., GISP
 Principal Engineer - Search  |  Amazon
 Apache Lucene PMC Member and Committer
 nkn...@apache.org


 On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera  wrote:
>
> Thanks for the heads up, I am planning to cut the brunch middle next 
> week, Wednesday July 20th.
> Let me know at the beginning of next week if there is any issue from 
> your side.
>
> cheers,
>
> Ignacio
>
> On Mon, Jul 11, 2022 at 4:21 PM Michael Sokolov  
> wrote:
>>
>> I would like to see if we can get
>> https://issues.apache.org/jira/browse/LUCENE-10577 in. It is working
>> and gives nice gains, but there is some controversy about the API. If
>> we can't get it sorted out this week(?) it can certainly slip to the
>> next revision. I know that
>> https://issues.apache.org/jira/browse/LUCENE-10592 is also baking and
>> has a PR that seems to be progressing rapidly.
>>
>> On Mon, Jul 11, 2022 at 10:03 AM Ignacio Vera  
>> wrote:
>> >
>> > Hello!
>> >
>> > Lucene 9.2.0 was released around 2 months ago and we are 
>> > accumulating a good bunch of new features, optimizations and bug 
>> > fixes. Would there be support for releasing Lucene 9.3 soon?
>> >
>> > I am happy being the release manager. I did not see any issues 
>> > marked "blocker", but please let me know if

Re: Lucene 9.3.0 release

2022-07-21 Thread Michael McCandless 
Super, thanks Ignacio.  And thanks for volunteering as RM!

Mike McCandless

http://blog.mikemccandless.com


On Thu, Jul 21, 2022 at 6:32 AM Ignacio Vera  wrote:

> indeed, just waiting for him to do the honours and push the PR.
>
> On Thu, Jul 21, 2022 at 12:30 PM Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
>> Greg fixed a test failure that impacts 9.3, and opened a PR and blocker
>> issue: https://issues.apache.org/jira/browse/LUCENE-10659
>>
>> I think we should merge it for 9.3.0?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Wed, Jul 20, 2022 at 7:43 PM Julie Tibshirani 
>> wrote:
>>
>>> Hello Mayya, I can share my view. Strictly speaking, we only need a new
>>> codec version if the on-disk format changes for some type (like kNN vectors
>>> in this case). Otherwise we can just evolve the logic in the same way we do
>>> with other code, making the improvements in-place. Alessandro's PR is a
>>> good example of this -- it's a refactor that happens to touch the
>>> vectors reader code. It doesn't require a totally new codec version.
>>>
>>> With your PR for LUCENE-10592, I don't think we technically need a new
>>> codec for it, because the on-disk format remains the same. But since it's
>>> such a big change that deeply affects the vectors writing logic, it feels
>>> convenient that we're making it in a new format (maybe feels a bit clearer/
>>> safer).
>>>
>>> Julie
>>>
>>> On Wed, Jul 20, 2022 at 4:21 PM Mayya Sharipova
>>>  wrote:
>>>
 Hi Julie,
 thanks for raising this point, I was also thinking about it.
 I think it would be valuable to clarify what constitutes the format
 changes. Is on disk changes a necessary requirement for it? Or is it enough
 that codec readers and writers change their behaviour?

 For example, for LUCENE-10592, there are extensive changes about how
 codec readers and writers behave, while the format on disk stays the same.
 Specifically for the release 9.3, Alessandro also introduced some
 changes  to codec readers
 while the format on disk stayed the same.




 On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani 
 wrote:

> Hello everyone,
>
> We had preemptively created a Lucene93Codec to support the vector
> search changes (Mike's work in LUCENE-10577, Mayya's work in 
> LUCENE-10592).
> However neither of these changes made it for 9.3, so there have not been
> any real format changes-- Lucene93Codec is the same as Lucene92Codec.
> Should we remove Lucene93Codec on branch_9_3 (effectively reverting
> https://github.com/apache/lucene/pull/924) ?
>
> Julie
>
> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova
>  wrote:
>
>> Thanks Adrien, it is indeed a big change, also would be nice to see
>> benchmarks after it is merged.
>>
>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able
>> to make it for tomorrow.
>>
>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand 
>> wrote:
>>
>>> While I can understand the excitement about LUCENE-10592, it's also
>>> a big change, maybe we should not even try to get it in before cutting 
>>> the
>>> branch?
>>>
>>> On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova
>>>  wrote:
>>>
 Thanks for the reminder about the release, Ignacio!
 About LUCENE-10592
   I will see
 what progress we can make today, and will let you know before 
 Wednesday at
 9:00 CEST.

 On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov 
 wrote:

> Thanks for checking, but please don't wait for LUCENE-10577. It's
> not clear when that might get resolved
>
> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera 
> wrote:
>
>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday
>> at 9:00 CEST. Let me know if there is any issue.
>>
>> @Mike: I see that LUCENE-10577
>>  is still
>> under API discussions, Do you think it will make it to the release?
>> LUCENE-10592 
>> seems to have made good progress, would you think it will be ready?
>> @Nick: I gave you feedback on the PR. I think it is way too risky
>> to add an encoding in a rush. I recommend adding the change in the
>> sandbox if you want to iterate in there or wait for another release 
>> where
>> there is more time to think through the encoding.
>>
>> Cheers,
>>
>> Ignacio
>>
>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize 
>> wrote:
>>
>>> I'd like to get ShapeDocValuesField

Re: Lucene 9.3.0 release

2022-07-21 Thread Michael Sokolov 
OK, I reverted the commit that introduced 93 codec on branch_9x and
branch_9_3. I guess it should probably get reverted on main too, but
since we have PRs that depend on it I wasn't sure if maybe we'd want
to rename to 94 instead of reverting? Anyway either way is fine by me.

On Wed, Jul 20, 2022 at 7:21 PM Mayya Sharipova
 wrote:
>
> Hi Julie,
> thanks for raising this point, I was also thinking about it.
> I think it would be valuable to clarify what constitutes the format changes. 
> Is on disk changes a necessary requirement for it? Or is it enough that codec 
> readers and writers change their behaviour?
>
> For example, for LUCENE-10592, there are extensive changes about how codec 
> readers and writers behave, while the format on disk stays the same.
> Specifically for the release 9.3, Alessandro also introduced some changes to 
> codec readers while the format on disk stayed the same.
>
>
>
>
> On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani  wrote:
>>
>> Hello everyone,
>>
>> We had preemptively created a Lucene93Codec to support the vector search 
>> changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592). However 
>> neither of these changes made it for 9.3, so there have not been any real 
>> format changes-- Lucene93Codec is the same as Lucene92Codec. Should we 
>> remove Lucene93Codec on branch_9_3 (effectively reverting 
>> https://github.com/apache/lucene/pull/924) ?
>>
>> Julie
>>
>> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova 
>>  wrote:
>>>
>>> Thanks Adrien, it is indeed a big change, also would be nice to see 
>>> benchmarks after it is merged.
>>>
>>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able to 
>>> make it for tomorrow.
>>>
>>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand  wrote:

 While I can understand the excitement about LUCENE-10592, it's also a big 
 change, maybe we should not even try to get it in before cutting the 
 branch?

 On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova 
  wrote:
>
> Thanks for the reminder about the release, Ignacio!
> About LUCENE-10592  I will see what progress we can make today, and will 
> let you know before Wednesday at 9:00 CEST.
>
> On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov  
> wrote:
>>
>> Thanks for checking, but please don't wait for LUCENE-10577. It's not 
>> clear when that might get resolved
>>
>> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera  wrote:
>>>
>>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday at 
>>> 9:00 CEST. Let me know if there is any issue.
>>>
>>> @Mike: I see that LUCENE-10577 is still under API discussions, Do you 
>>> think it will make it to the release? LUCENE-10592  seems to have made 
>>> good progress, would you think it will be ready?
>>> @Nick: I gave you feedback on the PR. I think it is way too risky to 
>>> add an encoding in a rush. I recommend adding the change in the sandbox 
>>> if you want to iterate in there or wait for another release where there 
>>> is more time to think through the encoding.
>>>
>>> Cheers,
>>>
>>> Ignacio
>>>
>>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize  wrote:

 I'd like to get ShapeDocValuesField in for the 9.3 release 
 (https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice 
 feature for enabling facets and aggregations over XYShape and 
 LatLonShape field types and could make for a good 9.3 geo highlight.

 Nicholas Knize, Ph.D., GISP
 Principal Engineer - Search  |  Amazon
 Apache Lucene PMC Member and Committer
 nkn...@apache.org


 On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera  wrote:
>
> Thanks for the heads up, I am planning to cut the brunch middle next 
> week, Wednesday July 20th.
> Let me know at the beginning of next week if there is any issue from 
> your side.
>
> cheers,
>
> Ignacio
>
> On Mon, Jul 11, 2022 at 4:21 PM Michael Sokolov  
> wrote:
>>
>> I would like to see if we can get
>> https://issues.apache.org/jira/browse/LUCENE-10577 in. It is working
>> and gives nice gains, but there is some controversy about the API. If
>> we can't get it sorted out this week(?) it can certainly slip to the
>> next revision. I know that
>> https://issues.apache.org/jira/browse/LUCENE-10592 is also baking and
>> has a PR that seems to be progressing rapidly.
>>
>> On Mon, Jul 11, 2022 at 10:03 AM Ignacio Vera  
>> wrote:
>> >
>> > Hello!
>> >
>> > Lucene 9.2.0 was released around 2 months ago and we are 
>> > accumulating a good bunch of new features, optimizations and bug 
>> > fixes. Would there be support for

Re: Lucene 9.3.0 release

2022-07-21 Thread Mayya Sharipova 
Thanks Michael for reverting the changes.
I think we should do the reversion also on the main branch, and later
introduce Lucene94 Codecs; otherwise I think it would be more difficult for
us to apply changes from main to branch_9x if they diverge so much.

On Thu, Jul 21, 2022 at 9:18 AM Michael Sokolov  wrote:

> OK, I reverted the commit that introduced 93 codec on branch_9x and
> branch_9_3. I guess it should probably get reverted on main too, but
> since we have PRs that depend on it I wasn't sure if maybe we'd want
> to rename to 94 instead of reverting? Anyway either way is fine by me.
>
> On Wed, Jul 20, 2022 at 7:21 PM Mayya Sharipova
>  wrote:
> >
> > Hi Julie,
> > thanks for raising this point, I was also thinking about it.
> > I think it would be valuable to clarify what constitutes the format
> changes. Is on disk changes a necessary requirement for it? Or is it enough
> that codec readers and writers change their behaviour?
> >
> > For example, for LUCENE-10592, there are extensive changes about how
> codec readers and writers behave, while the format on disk stays the same.
> > Specifically for the release 9.3, Alessandro also introduced some
> changes to codec readers while the format on disk stayed the same.
> >
> >
> >
> >
> > On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani 
> wrote:
> >>
> >> Hello everyone,
> >>
> >> We had preemptively created a Lucene93Codec to support the vector
> search changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592).
> However neither of these changes made it for 9.3, so there have not been
> any real format changes-- Lucene93Codec is the same as Lucene92Codec.
> Should we remove Lucene93Codec on branch_9_3 (effectively reverting
> https://github.com/apache/lucene/pull/924) ?
> >>
> >> Julie
> >>
> >> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova <
> mayya.sharip...@elastic.co.invalid> wrote:
> >>>
> >>> Thanks Adrien, it is indeed a big change, also would be nice to see
> benchmarks after it is merged.
> >>>
> >>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able
> to make it for tomorrow.
> >>>
> >>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand 
> wrote:
> 
>  While I can understand the excitement about LUCENE-10592, it's also a
> big change, maybe we should not even try to get it in before cutting the
> branch?
> 
>  On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova <
> mayya.sharip...@elastic.co.invalid> wrote:
> >
> > Thanks for the reminder about the release, Ignacio!
> > About LUCENE-10592  I will see what progress we can make today, and
> will let you know before Wednesday at 9:00 CEST.
> >
> > On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov 
> wrote:
> >>
> >> Thanks for checking, but please don't wait for LUCENE-10577. It's
> not clear when that might get resolved
> >>
> >> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera 
> wrote:
> >>>
> >>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday
> at 9:00 CEST. Let me know if there is any issue.
> >>>
> >>> @Mike: I see that LUCENE-10577 is still under API discussions, Do
> you think it will make it to the release? LUCENE-10592  seems to have made
> good progress, would you think it will be ready?
> >>> @Nick: I gave you feedback on the PR. I think it is way too risky
> to add an encoding in a rush. I recommend adding the change in the sandbox
> if you want to iterate in there or wait for another release where there is
> more time to think through the encoding.
> >>>
> >>> Cheers,
> >>>
> >>> Ignacio
> >>>
> >>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize 
> wrote:
> 
>  I'd like to get ShapeDocValuesField in for the 9.3 release (
> https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice feature
> for enabling facets and aggregations over XYShape and LatLonShape field
> types and could make for a good 9.3 geo highlight.
> 
>  Nicholas Knize, Ph.D., GISP
>  Principal Engineer - Search  |  Amazon
>  Apache Lucene PMC Member and Committer
>  nkn...@apache.org
> 
> 
>  On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera 
> wrote:
> >
> > Thanks for the heads up, I am planning to cut the brunch middle
> next week, Wednesday July 20th.
> > Let me know at the beginning of next week if there is any issue
> from your side.
> >
> > cheers,
> >
> > Ignacio
> >
> > On Mon, Jul 11, 2022 at 4:21 PM Michael Sokolov <
> msoko...@gmail.com> wrote:
> >>
> >> I would like to see if we can get
> >> https://issues.apache.org/jira/browse/LUCENE-10577 in. It is
> working
> >> and gives nice gains, but there is some controversy about the
> API. If
> >> we can't get it sorted out this week(?) it can certainly slip
> to the
> >> next revision. I know that
> >>

Re: [JENKINS] Lucene-main-Linux (64bit/jdk-17.0.3) - Build # 35849 - Unstable!

2022-07-21 Thread Greg Miller 
Ha, yes-- it's now patched on 9.3 (in addition to 9x / main). Thanks Mike!

Cheers,
-Greg

On Thu, Jul 21, 2022 at 3:31 AM Michael McCandless
 wrote:
>
> Oh, nevermind!  I see the PR/blocker issue, thanks Greg.
>
> EventuallyConsistentMikeException!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Thu, Jul 21, 2022 at 6:28 AM Michael McCandless 
>  wrote:
>>
>> Should this maybe also be backported to the 9.3.0 branch?  Did the original 
>> change land before that branch was cut?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Wed, Jul 20, 2022 at 5:46 PM Greg Miller  wrote:
>>>
>>> OK, I think these test failures should now be resolved (on both main
>>> and branch_9x). But I'll keep an eye on nightly builds/tests.
>>>
>>> Cheers,
>>> -g
>>>
>>> On Wed, Jul 20, 2022 at 9:17 AM Greg Miller  wrote:
>>> >
>>> > I'll dig into this soon. Looks like a new test I recently added hit an
>>> > issue. Apologies.
>>> >
>>> > Cheers,
>>> > -g
>>> >
>>> > On Wed, Jul 20, 2022 at 2:32 AM Policeman Jenkins Server
>>> >  wrote:
>>> > >
>>> > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/35849/
>>> > > Java: 64bit/jdk-17.0.3 -XX:+UseCompressedOops -XX:+UseSerialGC
>>> > >
>>> > > 1 tests failed.
>>> > > FAILED:  org.apache.lucene.search.TestDisiPriorityQueue.testRandom
>>> > >
>>> > > Error Message:
>>> > > java.lang.IllegalArgumentException: bound must be greater than origin
>>> > >
>>> > > Stack Trace:
>>> > > java.lang.IllegalArgumentException: bound must be greater than origin
>>> > > at 
>>> > > __randomizedtesting.SeedInfo.seed([3E14AC43B544B726:4C58894C04240155]:0)
>>> > > at 
>>> > > java.base/jdk.internal.util.random.RandomSupport.checkRange(RandomSupport.java:232)
>>> > > at 
>>> > > java.base/java.util.random.RandomGenerator.nextInt(RandomGenerator.java:679)
>>> > > at 
>>> > > org.apache.lucene.search.TestDisiPriorityQueue.testRandom(TestDisiPriorityQueue.java:47)
>>> > > at 
>>> > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
>>> > > Method)
>>> > > at 
>>> > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>>> > > at 
>>> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
>>> > > at 
>>> > > org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
>>> > > at 
>>> > > org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> > > at 
>>> > > org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>>> > > at 
>>> > > org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>>> > > at 
>>> > > org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>>> > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
>>> > > at 
>>> > > org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>>> > > at 
>>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> > > at 
>>> > > org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>>> > > at 
>>> > >

Re: New branch and feature freeze for Lucene 9.3.0

2022-07-21 Thread Greg Miller 
Ack, thanks Ignacio for the quick approval. Just merged the fix onto
the 9.3 branch.

Cheers,
-Greg

On Thu, Jul 21, 2022 at 12:32 AM Ignacio Vera  wrote:
>
> Hi Greg,
>
> Yes please fix the test in branch 9.3, I have approved the PR.
>
> On Thu, Jul 21, 2022 at 12:08 AM Greg Miller  wrote:
>>
>> Thanks Ignacio! I just created
>> https://issues.apache.org/jira/browse/LUCENE-10659 as a proposed
>> blocker for 9.3. It's a small bug fix for a unit test I recently
>> introduced on the 9x branch (one of the last things to get pulled into
>> the 9.3 candidate). I think we ought to fix this test before cutting a
>> 9.3 release. There's a PR associated with the issue already (the
>> change is already patched into main/branch_9x).
>>
>> Cheers,
>> -Greg
>>
>> On Wed, Jul 20, 2022 at 4:42 AM Ignacio Vera  wrote:
>> >
>> > Please find here the draft for the release highlights. I have probably 
>> > missed things that should be included so please feel free to add them.
>> >
>> > https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=217391905
>> >
>> > On Wed, Jul 20, 2022 at 10:17 AM Ignacio Vera  wrote:
>> >>
>> >> NOTICE:
>> >>
>> >>
>> >> Branch branch_9_3 has been cut and versions updated to 9.4 on stable 
>> >> branch.
>> >>
>> >>
>> >> Please observe the normal rules:
>> >>
>> >>
>> >> * No new features may be committed to the branch.
>> >>
>> >> * Documentation patches, build patches and serious bug fixes may be
>> >>
>> >>   committed to the branch. However, you should submit all patches you
>> >>
>> >>   want to commit to Jira first to give others the chance to review
>> >>
>> >>   and possibly vote against the patch. Keep in mind that it is our
>> >>
>> >>   main intention to keep the branch as stable as possible.
>> >>
>> >> * All patches that are intended for the branch should first be committed
>> >>
>> >>   to the unstable branch, merged into the stable branch, and then into
>> >>
>> >>   the current release branch.
>> >>
>> >> * Normal unstable and stable branch development may continue as usual.
>> >>
>> >>   However, if you plan to commit a big change to the unstable branch
>> >>
>> >>   while the branch feature freeze is in effect, think twice: can't the
>> >>
>> >>   addition wait a couple more days? Merges of bug fixes into the branch
>> >>
>> >>   may become more difficult.
>> >>
>> >> * Only Jira issues with Fix version 9.3 and priority "Blocker" will delay
>> >>
>> >>   a release candidate build.
>> >>
>> >>
>> >>
>> >> The only step missing is to add the Jenkins job on the release branch 
>> >> which is something I don't really know how to do it, hopefully someone 
>> >> can help here.
>> >>
>> >>
>> >> I am planning to build the first RC next Monday if there are no issues.
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene 9.3.0 release

2022-07-21 Thread Mayya Sharipova 
Just for the update, Michael, I went ahead and applied your reverted
changes to the `main` branch as well. I will introduce the new Lucene 94
codec.

On Thu, Jul 21, 2022 at 11:08 AM Mayya Sharipova 
wrote:

> Thanks Michael for reverting the changes.
> I think we should do the reversion also on the main branch, and later
> introduce Lucene94 Codecs; otherwise I think it would be more difficult for
> us to apply changes from main to branch_9x if they diverge so much.
>
> On Thu, Jul 21, 2022 at 9:18 AM Michael Sokolov 
> wrote:
>
>> OK, I reverted the commit that introduced 93 codec on branch_9x and
>> branch_9_3. I guess it should probably get reverted on main too, but
>> since we have PRs that depend on it I wasn't sure if maybe we'd want
>> to rename to 94 instead of reverting? Anyway either way is fine by me.
>>
>> On Wed, Jul 20, 2022 at 7:21 PM Mayya Sharipova
>>  wrote:
>> >
>> > Hi Julie,
>> > thanks for raising this point, I was also thinking about it.
>> > I think it would be valuable to clarify what constitutes the format
>> changes. Is on disk changes a necessary requirement for it? Or is it enough
>> that codec readers and writers change their behaviour?
>> >
>> > For example, for LUCENE-10592, there are extensive changes about how
>> codec readers and writers behave, while the format on disk stays the same.
>> > Specifically for the release 9.3, Alessandro also introduced some
>> changes to codec readers while the format on disk stayed the same.
>> >
>> >
>> >
>> >
>> > On Wed, Jul 20, 2022 at 4:22 PM Julie Tibshirani 
>> wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> We had preemptively created a Lucene93Codec to support the vector
>> search changes (Mike's work in LUCENE-10577, Mayya's work in LUCENE-10592).
>> However neither of these changes made it for 9.3, so there have not been
>> any real format changes-- Lucene93Codec is the same as Lucene92Codec.
>> Should we remove Lucene93Codec on branch_9_3 (effectively reverting
>> https://github.com/apache/lucene/pull/924) ?
>> >>
>> >> Julie
>> >>
>> >> On Tue, Jul 19, 2022 at 11:02 AM Mayya Sharipova <
>> mayya.sharip...@elastic.co.invalid> wrote:
>> >>>
>> >>> Thanks Adrien, it is indeed a big change, also would be nice to see
>> benchmarks after it is merged.
>> >>>
>> >>> So, Ignacio, please don't wait for LUCENE-10592, we will not be able
>> to make it for tomorrow.
>> >>>
>> >>> On Tue, Jul 19, 2022 at 10:38 AM Adrien Grand 
>> wrote:
>> 
>>  While I can understand the excitement about LUCENE-10592, it's also
>> a big change, maybe we should not even try to get it in before cutting the
>> branch?
>> 
>>  On Tue, Jul 19, 2022 at 4:09 PM Mayya Sharipova <
>> mayya.sharip...@elastic.co.invalid> wrote:
>> >
>> > Thanks for the reminder about the release, Ignacio!
>> > About LUCENE-10592  I will see what progress we can make today, and
>> will let you know before Wednesday at 9:00 CEST.
>> >
>> > On Tue, Jul 19, 2022 at 7:12 AM Michael Sokolov 
>> wrote:
>> >>
>> >> Thanks for checking, but please don't wait for LUCENE-10577. It's
>> not clear when that might get resolved
>> >>
>> >> On Mon, Jul 18, 2022, 10:42 AM Ignacio Vera 
>> wrote:
>> >>>
>> >>> Just a quick reminder I plan to cut the 9.3 branch this Wednesday
>> at 9:00 CEST. Let me know if there is any issue.
>> >>>
>> >>> @Mike: I see that LUCENE-10577 is still under API discussions, Do
>> you think it will make it to the release? LUCENE-10592  seems to have made
>> good progress, would you think it will be ready?
>> >>> @Nick: I gave you feedback on the PR. I think it is way too risky
>> to add an encoding in a rush. I recommend adding the change in the sandbox
>> if you want to iterate in there or wait for another release where there is
>> more time to think through the encoding.
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Ignacio
>> >>>
>> >>> On Wed, Jul 13, 2022 at 2:09 AM Nicholas Knize 
>> wrote:
>> 
>>  I'd like to get ShapeDocValuesField in for the 9.3 release (
>> https://issues.apache.org/jira/browse/LUCENE-10654). It's a nice feature
>> for enabling facets and aggregations over XYShape and LatLonShape field
>> types and could make for a good 9.3 geo highlight.
>> 
>>  Nicholas Knize, Ph.D., GISP
>>  Principal Engineer - Search  |  Amazon
>>  Apache Lucene PMC Member and Committer
>>  nkn...@apache.org
>> 
>> 
>>  On Tue, Jul 12, 2022 at 2:50 PM Ignacio Vera 
>> wrote:
>> >
>> > Thanks for the heads up, I am planning to cut the brunch middle
>> next week, Wednesday July 20th.
>> > Let me know at the beginning of next week if there is any issue
>> from your side.
>> >
>> > cheers,
>> >
>> > Ignacio
>> >
>> > On Mon, Jul 11, 2022 at 4:21 PM Michael Sokolov <
>> msoko...@gmail.com> wrote:
>> >>
>> >> I would like to see if

Re: New branch and feature freeze for Lucene 9.3.0

2022-07-21 Thread Ignacio Vera 
Hi Greg,

Yes please fix the test in branch 9.3, I have approved the PR.

On Thu, Jul 21, 2022 at 12:08 AM Greg Miller  wrote:

> Thanks Ignacio! I just created
> https://issues.apache.org/jira/browse/LUCENE-10659 as a proposed
> blocker for 9.3. It's a small bug fix for a unit test I recently
> introduced on the 9x branch (one of the last things to get pulled into
> the 9.3 candidate). I think we ought to fix this test before cutting a
> 9.3 release. There's a PR associated with the issue already (the
> change is already patched into main/branch_9x).
>
> Cheers,
> -Greg
>
> On Wed, Jul 20, 2022 at 4:42 AM Ignacio Vera  wrote:
> >
> > Please find here the draft for the release highlights. I have probably
> missed things that should be included so please feel free to add them.
> >
> >
> https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=217391905
> >
> > On Wed, Jul 20, 2022 at 10:17 AM Ignacio Vera  wrote:
> >>
> >> NOTICE:
> >>
> >>
> >> Branch branch_9_3 has been cut and versions updated to 9.4 on stable
> branch.
> >>
> >>
> >> Please observe the normal rules:
> >>
> >>
> >> * No new features may be committed to the branch.
> >>
> >> * Documentation patches, build patches and serious bug fixes may be
> >>
> >>   committed to the branch. However, you should submit all patches you
> >>
> >>   want to commit to Jira first to give others the chance to review
> >>
> >>   and possibly vote against the patch. Keep in mind that it is our
> >>
> >>   main intention to keep the branch as stable as possible.
> >>
> >> * All patches that are intended for the branch should first be committed
> >>
> >>   to the unstable branch, merged into the stable branch, and then into
> >>
> >>   the current release branch.
> >>
> >> * Normal unstable and stable branch development may continue as usual.
> >>
> >>   However, if you plan to commit a big change to the unstable branch
> >>
> >>   while the branch feature freeze is in effect, think twice: can't the
> >>
> >>   addition wait a couple more days? Merges of bug fixes into the branch
> >>
> >>   may become more difficult.
> >>
> >> * Only Jira issues with Fix version 9.3 and priority "Blocker" will
> delay
> >>
> >>   a release candidate build.
> >>
> >>
> >>
> >> The only step missing is to add the Jenkins job on the release branch
> which is something I don't really know how to do it, hopefully someone can
> help here.
> >>
> >>
> >> I am planning to build the first RC next Monday if there are no issues.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>