Re: Q: 9.x upgrade to hppc 0.9.1

[Mike Drob]

What is the cost of maintaining the fork? I don’t feel it’s fair to you
Dawid, if we were to expect you to port over any changes made to hppc
upstream.

Mike

On Sun, May 26, 2024 at 3:59 PM Dawid Weiss  wrote:

> If we increase the hppc fork to 23 classes and 14 test classes, then we
>> can remove the hppc dependency from all modules.
>> Do we agree that we should
>> - Increase the fork size
>> - Move it to oal.internal
>> - Remove the hppc dependency from everywhere
>>
>
> Yes, I think it's the safest way to go and it's also the cleanest - keeps
> the implementation details private and doesn't clash with anything out
> there. Dropping an existing dependency shouldn't be a problem, I think.
>
>
>> Dawid, for the size of hppc, I counted the number of files with
>> find . -type f | wc -l
>> in hppc/build/generated/main
>>
>
> Oh, ok. Many of these are a bit esoteric (even though we don't generate
> all combinations). Taking what's needed sounds reasonable to me - and it
> shouldn't be that much, really.
>
> D.
>
>
>>
>> Le dim. 26 mai 2024 à 21:52, Dawid Weiss  a
>> écrit :
>>
>>>
>>> Hi Bruno,
>>>
>>> Currently the hppc fork in Lucene is composed of 15 classes and 8 test
 classes.
 Forking everything in hppc would mean 525 classes and 193 test classes.
 I'm not sure we want to fork all hppc?

>>>
>>> My superficial analysis hinted at far fewer classes but I'll take a look
>>> tomorrow, had a busy day today.
>>>
>>>
 +1 to moving the hppc fork to oal.internal.

>>>
>>> Yes, I think it's a good idea to move it and hide it, at least for the
>>> module system.
>>>
>>> D.
>>>
>>>

Mutation Testing

[Mike Drob]

Has anybody experimented with adding mutation testing to Lucene using
something like the PIT framework?
https://pitest.org/

Looks like there is GitHub PR integration via https://www.arcmutate.com/

I’d be curious to hear if people have found it useful, performant and
comprehensive.

Mike

Re: Squash vs merge of PRs

[Mike Drob]

We all agree on using Java though, and using a specific version, and even
the style output from gradle tidy. Is that nanny state or community
consensus?

On Sat, Nov 4, 2023 at 7:29 AM Robert Muir  wrote:

> example of a nanny state IMO, trying to dictate what git commands to
> use, or what editor to use. Maybe this works for you in your corporate
> hellholes, but I think some folks have a bit of a power issue, are
> accustomed to dictacting this stuff to their employees and so on, but
> this is open-source. I don't report to you, i dont use the editor you
> tell me, or the git commands you tell me.
>
> On Sat, Nov 4, 2023 at 8:21 AM Uwe Schindler  wrote:
> >
> > Hi,
> >
> > I just wanted to give your attention to the following discussion:
> > https://github.com/apache/lucene/pull/12737#issuecomment-1793426911
> >
> >  From my knowledge the Lucene (and Solr) community decided a while back
> > to disable merging and only allow squashig of PRs. Robert always did
> > this, but because of a one-time problem with two branches he was working
> > on in parallel, he suddenly changed his mind and did merges on his own,
> > not sqashing the branch and pushing to ASF Git.
> >
> > I am also not a fan of removing all history, but especially for heavy
> > committing branches like the given PR, I think we should invite our
> > committers to also adhere to community standards everyone else
> > practices. I would agree with merging those branches if all commit
> > messages in the branch would be well-formed with issue ID or PR number,
> > but in the above case you get a history of random commits which is no
> > longer linear and not easy readable.
> >
> > What do others think?
> >
> > Uwe
> >
> > --
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: LeafCollector#finish idempotency?

[Mike Drob]

Not sure if you saw this, Greg, but Alex ran into a similar question
recently from Solr.
https://lists.apache.org/thread/1gs3nsv1mcns1czdtdnqyz84f31tqm2x

On Mon, Oct 9, 2023 at 10:47 AM Adrien Grand  wrote:

> Hi Greg,
>
> I agree that LeafCollector implementations should be able to assume that
> finish() only gets called once. The test framework already makes this
> assumption:
> https://github.com/apache/lucene/blob/dfff1e635805ffc61dd6029a8060e2635bfcbdb9/lucene/test-framework/src/java/org/apache/lucene/tests/search/AssertingLeafCollector.java#L95-L100
> .
>
> On Mon, Oct 9, 2023 at 5:38 PM Greg Miller  wrote:
>
>> Hey folks-
>>
>> I'm curious if anyone has thoughts around idempotency concerns related to
>> the LeafCollector#finish API added in GH#12380
>> . My expectation would be
>> that LeafCollector implementations should be able to assume #finish will
>> only get called once. In fact, it looks like FacetsCollector is already
>> making that assumption.
>>
>> Is this inline with other folks' expectations? If so, I'm going to, 1)
>> address a small bug related to drill-sideways that results #finish being
>> called multiple times on one of the collectors, and 2) propose some
>> additional javadoc on LeafCollector#finish clarifying this.
>>
>> Make sense?
>>
>> Cheers,
>> -Greg
>>
>
>
> --
> Adrien
>

JavaDoc generated with -noindex

[Mike Drob]

Why is our javadoc currently generated with -noindex? I did some digging
and found that we set that back in LUCENE-3977 to save 10MB, and then added
a property to re-enable it in LUCENE-4237, but I think that got lost in the
gradle migration.

While the index might have been useless at the time, it now powers the
javadoc search box, see a demo at https://youtu.be/VrI6rJNO2x4?t=925 -- The
full spec is described at
https://docs.oracle.com/en/java/javase/17/docs/specs/javadoc/javadoc-search-spec.html


I think this would be a useful thing to include, at least for releases.
WDYT?

Mike

Re: Main website not building

[Mike Drob]

In .gitattributes add a line
asf.yaml merge=ours

On Tue, Nov 22, 2022 at 11:54 AM Uwe Schindler  wrote:

> Hi I fixed this.
>
> This was caused because due to maerging from main->production the fix of
> .asf.yaml was merged, too (this was caused by repairing the problems from
> lucenepy with a duplicate master branch). So the production branch was
> building but was deployed to Staging.
>
> I tried to exclude asf.yaml from any merging, but there is no way to put a
> "sticky" bit on it. If anybody has an idea how to tell git: never touch
> asf.aml while merging, tell me!
>
> Uwe
> Am 22.11.2022 um 18:37 schrieb Adrien Grand:
>
> Hello,
>
> I've managed to make changes to the website for 9.4.2 and they are
> correctly reflected on lucene.staged.apache.org. However pushing to the
> `production` branch doesn't seem to trigger a build on
> https://ci2.apache.org/#/builders/3 and I'm not seeing the production
> website getting updated either.
>
> Is someone familiar with how the build hooks with git pushes and could
> give me pointers to debug why pushes to the production branch are not
> triggering builds?
>
> --
> Adrien
>
> --
> Uwe SchindlerAchterdiek 19, D-28357 Bremen 
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>

Re: [DISCUSS] Read-only Jira after the GitHub issues migration?

[Mike Drob]

On Tue, Jul 19, 2022 at 8:48 AM Tomoko Uchida 
wrote:

> > 1. Make Jira read only
>
> At the very last step, we'll add comments saying "This was moved GitHub
> " to each Jira issue. It has to be done after the migration was
> completed.
>

Is this going to send 10k emails to the mailing list? I’ll need to update
my filters so that these skip my inbox in that case.


> > 2. Send a message to dev@ stating new issues should now be opened in
> github
> > 3. Start the migration
>
> In theory, it would be okay to me. I just wanted to avoid any risks during
> migration. Let me give time to consider/check if the migration can be
> safely done while new issues are created (by humans).
>
>
> 2022年7月19日(火) 21:56 Ryan Ernst :
>
>> > Yes, it won't be a really atomic switch
>>
>> While it may not be completely atomic, could it be closer? GitHub already
>> supports new issues, developers are just advised against opening there.
>> Could the order of events be:
>>
>> 1. Make Jira read only
>> 2. Send a message to dev@ stating new issues should now be opened in
>> github
>> 3. Start the migration
>> 4. When the migration is complete, send another message notifying devs
>> that pre-existing Jiras are now in GitHub,so they can then be commented on
>> and edited there.
>>
>> I think the difference with this and what was previously described on
>> this thread is there would be no downtime for new issues.
>>
>> On Tue, Jul 19, 2022 at 00:07 Tomoko Uchida 
>> wrote:
>>
>>> OK, thank you everyone for your comments/suggestions.
>>> I will ask infra to make Lucene Jira read-only after the migration is
>>> completed (if there are no explicit objections). For people who are
>>> critically affected by this change, please let me know about your
>>> inconvenience. I'll try to find acceptable solutions.
>>>
>>> > I would prefer that we make a nearly atomic switch -- up until time X
>>> we use Jira, then it goes read-only and at time X + t (t being how long the
>>> migration takes, likely a day or two?), GitHub issues opens for business.
>>>
>>> Yes, it won't be a really atomic switch - there will be a moratorium in
>>> our issue system (where GitHub issue is not lifted yet, but a Jira snapshot
>>> is already taken). I estimate the whole migration process will take at
>>> least three days; will make a mail thread about the detailed schedule.
>>>
>>> Tomoko
>>>
>>>
>>> 2022年7月19日(火) 2:38 Gus Heck :
>>>
 I am 100% for preventing creation of new issues in Jira, new issues
 should only be created in one system at any one time. I feel that existing
 issues should be completed in their original system for continuity, and
 anticipate that in any case Jira will mean readable in perpetuity. The
 copying of old issues to github as a convenience for users so they aren't
 forced to look at 2 places also sounds good. Raising the standard for what
 we consider a stale issue and closing out things in Jira faster to get to a
 one system situation sooner also seems good.

 Things I think we should strive to avoid:
 1) An issue in Jira that is unresolved and duplicated (possibly
 resolved) in github... possibly leading to someone wasting time repeating a
 solution or giving up thinking there isn't a solution etc.
 2) Any issues for which the discussion is split across systems and thus
 it would be easy to miss part of the discussion and/or not have the issue
 come up in searches that are relevant to that issue.

 Also, a common pattern for me is to throw an issue ticket number that I
 have noted somewhere (i.e LUCENE-12345) into google and browse to the
 ticket if it comes up directly or to a mail archive result which has a link
 to the Jira. This is faster than searching in jira itself because I can
 always get to google in a single keystroke (new tab).  Sadly this is
 unlikely to work with github which does not put a project moniker on the
 issue id. Not sure how many others do this but if it's common I wonder if
 we can auto-insert something of the sort into github tickets so that mail
 archives from the tickets are similarly searchable? Like LUCENE-G12345 for
 github ticket #12345? The two key things that make this useful are the
 searchability of the ID in google and the fact that ticket mails often have
 a link to the ticket which the archive sites will render as a hyperlink.

 -Gus

 On Mon, Jul 18, 2022 at 11:12 AM David Smiley 
 wrote:

> I suppose someone bent on not using GitHub could also email the patch
> to the dev list, starting a thread around it.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sun, Jul 17, 2022 at 9:14 AM Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
>> Hi Team,
>>
>> Thanks to Tomoko's amazing hard work (
>> 

Re: Missing tags for recent 8.x Solr releases

[Mike Drob]

+dev@solr
-dev@lucene to bcc

Hi Nemo,

The canonical location of our tags for the 8.x release line is
https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=tags

The tags for 9.x (and future) will all be at
https://gitbox.apache.org/repos/asf?p=solr.git;a=tags as you have found.
This is the unfortunate result of some continued split development.

We may be able to copy the 8.x tags to the solr repo as well, but I'm not
certain what our options are.

Mike

On Mon, Jul 11, 2022 at 12:10 AM Nemo  wrote:

>
> Hi,
>
> It appears that recent 8.x Solr releases (post 8.8.1) haven't been
> tagged on the Git repository[1]
>
> Is this an intentional release process change? Seeing as how the 9.0.0
> release was correctly tagged, could the missing releases be backtagged
> with the correct dates? The list of untagged releases is quite small:
>
> 8.8.2, 8.9.0, 8.10.0, 8.10.1
> 8.11.0, 8.11.1, 8.11.2
>
> Context: Writing this on behalf of the endoflife.date project, where we
> recently[2] added a endoflife.date/solr page. To track new releases
> automatically, we track tags on upstream repositories. In this case,
> tracking the repo would get us 8.8.1 as the latest release instead of
> 8.11.2. We could also track Docker images, but they seem to lag behind
> by a few days, and wouldn't give us good enough coverage.
>
> Please keep me in cc for replies.
>
> Thanks,
> Nemo
>
> [1]: https://gitbox.apache.org/repos/asf?p=solr.git;a=tags.
> [2]: https://github.com/endoflife-date/endoflife.date/pull/1389
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[ANNOUNCE] Apache Lucene 8.11.2 released

[Mike Drob]

The Lucene PMC is pleased to announce the release of Apache Lucene 8.11.2.

Apache Lucene is a high-performance, full-featured text search engine library 
written entirely in Java. It is a technology suitable for nearly any 
application that requires full-text search, especially cross-platform.

This release contains numerous bug fixes, optimizations, and improvements, some 
of which are highlighted below. The release is available for immediate download 
at:

  

### Lucene 8.11.2 Release Highlights:

Bug fixes

* LUCENE-10564: Make sure SparseFixedBitSet#or updates ramBytesUsed.
* LUCENE-10477: Highlighter: WeightedSpanTermExtractor.extractWeightedSpanTerms 
to Query#rewrite
  multiple times if necessary.

Optimizations

* LUCENE-10481: FacetsCollector will not request scores if it does not use them.

Please read CHANGES.txt for a full list of changes:

  

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Deleting old RCs

[Mike Drob]

Can we delete all of these? Do we still need them? Is there any reason to
keep them? I guess they don't hurt anything by existing...

$ svn ls https://dist.apache.org/repos/dist/dev/lucene/
lucene-9.0.0-RC1-rev-903ee94dc50643299c15dfa954410f3ee4d62075/
lucene-9.0.0-RC2-rev-95072f3b71e67e308d71a6149323bf7dd04c8f75/
lucene-9.0.0-RC3-rev-1ddce848cf3d5067efcafc6569d5f8203e56af0b/
lucene-9.1.0-RC1-rev-a6114b532a273e370528675d551d3ddfa02f4679/
lucene-9.2.0-RC1-rev-978eef5459c7683038ddcca4ec56e4baa63715d0/
lucene-solr-7.3.0-RC1-reveb8a5a882f879a51389b5d43f74f3aceac9e68c9/
lucene-solr-7.6.0-RC1-rev2d4435162774ad43b66ce0e7847bf8c1558e20a9/
lucene-solr-7.7.3-RC1-rev1a0d2a901dfec93676b0fe8be425101ceb754b85/
lucene-solr-8.1.0-RC1-reve5839fb416083fcdaeedfb1e329a9fdaa29fdc50/
lucene-solr-8.3.0-RC1-revd796eca84dbabe3ae9b3c27afc01ef3bee35acb1/
lucene-solr-8.8.0-RC2-revb10659f0fc18b58b90929cfdadde94544d202c4a/

Thanks,
Mike

[RESULT] [VOTE] Release Lucene/Solr 8.11.2 RC2

[Mike Drob]

It's been >72h since the vote was initiated and the result is:

+1  6  (6 binding)
 0  1
-1  0

This vote has PASSED

On Mon, Jun 13, 2022 at 12:05 PM Mike Drob  wrote:

> Please vote for release candidate 2 for Lucene/Solr 8.11.2
>
> The artifacts can be downloaded from:
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC2-rev17dee71932c683e345508113523e764c3e4c80fa
>
> You can run the smoke tester directly with this command:
>
> python3 -u dev-tools/scripts/smokeTestRelease.py \
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC2-rev17dee71932c683e345508113523e764c3e4c80fa
>
> The vote will be open for at least 72 hours i.e. until 2022-06-16 20:00 UTC
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
> 
>

[VOTE] Release Lucene/Solr 8.11.2 RC2

[Mike Drob]

Please vote for release candidate 2 for Lucene/Solr 8.11.2

The artifacts can be downloaded from:
https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC2-rev17dee71932c683e345508113523e764c3e4c80fa

You can run the smoke tester directly with this command:

python3 -u dev-tools/scripts/smokeTestRelease.py \
https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC2-rev17dee71932c683e345508113523e764c3e4c80fa

The vote will be open for at least 72 hours i.e. until 2022-06-16 20:00 UTC

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Here is my +1

Re: [VOTE] Release Lucene/Solr 8.11.2 RC1

[Mike Drob]

This RC did not receive enough votes to pass, I've fixed the bug pointed
out by Houston and will be moving on to RC2. Thanks!

On Sun, Jun 12, 2022 at 2:57 PM Mike Drob  wrote:

> Thanks for finding that, Houston! It was an issue during backporting that
> I've corrected. I'll respin and put up a new RC with the fix.
>
> On Sat, Jun 11, 2022 at 11:21 AM Houston Putman 
> wrote:
>
>> +0
>>
>> SUCCESS! [1:02:38.547629]
>>
>> I saw this in the example logs during the smoketester:
>>
>>> ps: Invalid process id: i��\r\001
>>> Waiting up to 180 seconds to see Solr running on port 8983 [/]
>>> Started Solr server on port 8983 (pid=16758). Happy searching!
>>>
>> This seems related to SOLR-16191
>> <https://issues.apache.org/jira/browse/SOLR-16191>, which is included in
>> this release.
>> Not exactly sure what went wrong, but the example still passed?
>>
>> - Houston
>>
>> On Wed, Jun 8, 2022 at 8:50 PM Mike Drob  wrote:
>>
>>> to: dev@lucene, dev@solr
>>>
>>> Please vote for release candidate 1 for Lucene/Solr 8.11.2
>>>
>>> The artifacts can be downloaded from:
>>>
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC1-reva9ed1e5fccbd1a84c78194a1329a7e1a3032ffc6
>>>
>>> You can run the smoke tester directly with this command:
>>>
>>> python3 -u dev-tools/scripts/smokeTestRelease.py \
>>>
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC1-reva9ed1e5fccbd1a84c78194a1329a7e1a3032ffc6
>>>
>>> Please see draft release notes (and edit as appropriate) at
>>> https://cwiki.apache.org/confluence/display/LUCENE/ReleaseNote8_11_2
>>> https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote8_11_2
>>>
>>> The vote will be open for at least 72 hours not including weekend i.e.
>>> until 2022-06-14 01:00 UTC.
>>>
>>> [ ] +1  approve
>>> [ ] +0  no opinion
>>> [ ] -1  disapprove (and reason why)
>>>
>>> Here is my +1
>>>
>>

Re: [VOTE] Release Lucene/Solr 8.11.2 RC1

[Mike Drob]

Thanks for finding that, Houston! It was an issue during backporting that
I've corrected. I'll respin and put up a new RC with the fix.

On Sat, Jun 11, 2022 at 11:21 AM Houston Putman  wrote:

> +0
>
> SUCCESS! [1:02:38.547629]
>
> I saw this in the example logs during the smoketester:
>
>> ps: Invalid process id: i��\r\001
>> Waiting up to 180 seconds to see Solr running on port 8983 [/]
>> Started Solr server on port 8983 (pid=16758). Happy searching!
>>
> This seems related to SOLR-16191
> <https://issues.apache.org/jira/browse/SOLR-16191>, which is included in
> this release.
> Not exactly sure what went wrong, but the example still passed?
>
> - Houston
>
> On Wed, Jun 8, 2022 at 8:50 PM Mike Drob  wrote:
>
>> to: dev@lucene, dev@solr
>>
>> Please vote for release candidate 1 for Lucene/Solr 8.11.2
>>
>> The artifacts can be downloaded from:
>>
>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC1-reva9ed1e5fccbd1a84c78194a1329a7e1a3032ffc6
>>
>> You can run the smoke tester directly with this command:
>>
>> python3 -u dev-tools/scripts/smokeTestRelease.py \
>>
>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC1-reva9ed1e5fccbd1a84c78194a1329a7e1a3032ffc6
>>
>> Please see draft release notes (and edit as appropriate) at
>> https://cwiki.apache.org/confluence/display/LUCENE/ReleaseNote8_11_2
>> https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote8_11_2
>>
>> The vote will be open for at least 72 hours not including weekend i.e.
>> until 2022-06-14 01:00 UTC.
>>
>> [ ] +1  approve
>> [ ] +0  no opinion
>> [ ] -1  disapprove (and reason why)
>>
>> Here is my +1
>>
>

[VOTE] Release Lucene/Solr 8.11.2 RC1

[Mike Drob]

to: dev@lucene, dev@solr

Please vote for release candidate 1 for Lucene/Solr 8.11.2

The artifacts can be downloaded from:
https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC1-reva9ed1e5fccbd1a84c78194a1329a7e1a3032ffc6

You can run the smoke tester directly with this command:

python3 -u dev-tools/scripts/smokeTestRelease.py \
https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC1-reva9ed1e5fccbd1a84c78194a1329a7e1a3032ffc6

Please see draft release notes (and edit as appropriate) at
https://cwiki.apache.org/confluence/display/LUCENE/ReleaseNote8_11_2
https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote8_11_2

The vote will be open for at least 72 hours not including weekend i.e.
until 2022-06-14 01:00 UTC.

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Here is my +1

Re: Bugfix release Lucene/Solr 8.11.2

[Mike Drob]

Yes, please fix and backport SOLR-16227, it looks almost ready from the
conversation on the PR. I will plan to do the first RC on Monday if we can
get the backport completed today.

On Thu, Jun 2, 2022 at 9:18 AM kiran chitturi 
wrote:

> Hi Mike,
>
> I found a new issue in Solr SQL (SOLR-16227) and have a fix for it (
> https://github.com/apache/solr/pull/887). Can you wait on the release for
> 8.11.2 till I can backport this?
>
> Thank you,
> Kiran.
>
> On Tue, May 31, 2022 at 9:21 AM Mike Drob  wrote:
>
>> Howdy folks, now that Lucene 9.2 has wrapped and we're past the holiday
>> weekend in the United States, I'd like to take a look at getting this
>> rolling by the end of the week. I see an open PR for
>> backporting LUCENE-10236 but it doesn't look like anything else would
>> really be waiting at the moment.
>>
>> I will plan to build a release candidate on Thursday (sooner if
>> LUCENE-10236 is committed, later if somebody else shouts that they have
>> other issues).
>>
>> Thanks!
>>
>> On Tue, May 24, 2022 at 3:48 PM Jan Høydahl 
>> wrote:
>>
>>> I bumped Jackson in https://issues.apache.org/jira/browse/SOLR-16213 and
>>> also backported to 8_11. Wdyt?
>>>
>>> Jan
>>>
>>> 18. mai 2022 kl. 15:22 skrev Gus Heck :
>>>
>>> SOLR-16194 is in and ported to 8.11,.2
>>>
>>> On Wed, May 18, 2022 at 7:12 AM Jan Høydahl 
>>> wrote:
>>>
>>>> I was pinged on https://issues.apache.org/jira/browse/SOLR-16019 because
>>>> I have an in-flight PR with a backport. I'll complete and merge that PR.
>>>>
>>>> Jan
>>>>
>>>>
>>>> 13. mai 2022 kl. 01:03 skrev Mike Drob :
>>>>
>>>> To: dev@lucene, dev@solr
>>>>
>>>> NOTICE:
>>>>
>>>> I am planning on preparing a bugfix release from branch branch_8_11
>>>> (likely mid next week)
>>>>
>>>> Please observe the normal rules for committing to this branch:
>>>>
>>>> * Before committing to the branch, reply to this thread and argue
>>>>   why the fix needs backporting and how long it will take.
>>>> ** If you're backporting stuff this week still or over the weekend,
>>>> then skip
>>>> the bit about how long it will take.
>>>> * All issues accepted for backporting should be marked with 8.11.2
>>>>   in JIRA, and issues that should delay the release must be marked as
>>>> Blocker
>>>> * All patches that are intended for the branch should first be committed
>>>>   to the unstable branch, merged into the stable branch, and then into
>>>>   the current release branch.
>>>> * Only Jira issues with Fix version 8.11.2 and priority "Blocker" will
>>>> delay
>>>>   a release candidate build.
>>>>
>>>> Also, please observe that since 9.0 already exists, there cannot be any
>>>> index format breaking changes. It really should only be bug fixes that have
>>>> already been verified on the 9x branch.
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>>
>>>>
>>>
>>> --
>>> http://www.needhamsoftware.com (work)
>>> http://www.the111shift.com (play)
>>>
>>>
>>>

Re: Bugfix release Lucene/Solr 8.11.2

[Mike Drob]

Howdy folks, now that Lucene 9.2 has wrapped and we're past the holiday
weekend in the United States, I'd like to take a look at getting this
rolling by the end of the week. I see an open PR for
backporting LUCENE-10236 but it doesn't look like anything else would
really be waiting at the moment.

I will plan to build a release candidate on Thursday (sooner if
LUCENE-10236 is committed, later if somebody else shouts that they have
other issues).

Thanks!

On Tue, May 24, 2022 at 3:48 PM Jan Høydahl  wrote:

> I bumped Jackson in https://issues.apache.org/jira/browse/SOLR-16213 and
> also backported to 8_11. Wdyt?
>
> Jan
>
> 18. mai 2022 kl. 15:22 skrev Gus Heck :
>
> SOLR-16194 is in and ported to 8.11,.2
>
> On Wed, May 18, 2022 at 7:12 AM Jan Høydahl  wrote:
>
>> I was pinged on https://issues.apache.org/jira/browse/SOLR-16019 because
>> I have an in-flight PR with a backport. I'll complete and merge that PR.
>>
>> Jan
>>
>>
>> 13. mai 2022 kl. 01:03 skrev Mike Drob :
>>
>> To: dev@lucene, dev@solr
>>
>> NOTICE:
>>
>> I am planning on preparing a bugfix release from branch branch_8_11
>> (likely mid next week)
>>
>> Please observe the normal rules for committing to this branch:
>>
>> * Before committing to the branch, reply to this thread and argue
>>   why the fix needs backporting and how long it will take.
>> ** If you're backporting stuff this week still or over the weekend, then
>> skip
>> the bit about how long it will take.
>> * All issues accepted for backporting should be marked with 8.11.2
>>   in JIRA, and issues that should delay the release must be marked as
>> Blocker
>> * All patches that are intended for the branch should first be committed
>>   to the unstable branch, merged into the stable branch, and then into
>>   the current release branch.
>> * Only Jira issues with Fix version 8.11.2 and priority "Blocker" will
>> delay
>>   a release candidate build.
>>
>> Also, please observe that since 9.0 already exists, there cannot be any
>> index format breaking changes. It really should only be bug fixes that have
>> already been verified on the 9x branch.
>>
>> Thanks,
>> Mike
>>
>>
>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
>
>

Re: [VOTE] Migration to GitHub issue from Jira (LUCENE-10557)

[Mike Drob]

+0

There’s other people who have thought about this much more than I have, but
I wouldn’t want my inaction to impact the increase 15 vote threshold.

Mike Drob

On Mon, May 30, 2022 at 10:40 AM Tomoko Uchida 
wrote:

> Hi everyone!
>
> As we had previous discussion thread [1], I propose migration to GitHub
> issue from Jira.
> It'd be technically possible (see [2] for details) and I think it'd be
> good for the project - not only for welcoming new developers who are not
> familiar with Jira, but also for improving the experiences of long-term
> committers/contributors by consolidating the conversation platform.
>
> You can see a short summary of the discussion, some stats on current Jira
> issues, and a draft migration plan in [2].
> Please review [2] if you haven't seen it and vote for this proposal.
>
> The vote will be open until 2022-06-06 16:00 UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>
> *IMPORTANT NOTE*
> I set a local protocol for this vote.
> There are 95 committers on this project [3] - the vote will be effective
> if it successfully gains more than 15% of voters (>= 15) from committers
> (including PMC members). This means, that although only PMC member votes
> are counted for the final result, the votes from all committers are
> important to make the vote result effective.
>
> If there are less than 15 votes at 2022-06-06 16:00 UTC, I will expand the
> term to 2022-06-13 16:00 UTC. If this fails to get sufficient voters after
> the expanded time limit, I'll cancel this vote regardless of the result.
> But why do I set such an extra bar? My fear is that if such things are
> decided by the opinions of a few members, the result shouldn't yield a good
> outcome for the future. It isn't my goal to just pass the vote [4].
>
> [1] https://lists.apache.org/thread/78wj0vll73sct065m5jjm4z8gqb5yffk
> [2] https://issues.apache.org/jira/browse/LUCENE-10557
> [3] https://projects.apache.org/committee.html?lucene
> [4] I'm sorry for being overly cautious, but I have never met in person or
> virtually any of the committers (with a very few exceptions), therefore
> cannot assess if the vote result is reliable or not unless there is certain
> explicit feedback.
>
>
> Tomoko
>

Bugfix release Lucene/Solr 8.11.2

[Mike Drob]

To: dev@lucene, dev@solr

NOTICE:

I am planning on preparing a bugfix release from branch branch_8_11 (likely
mid next week)

Please observe the normal rules for committing to this branch:

* Before committing to the branch, reply to this thread and argue
  why the fix needs backporting and how long it will take.
** If you're backporting stuff this week still or over the weekend, then
skip
the bit about how long it will take.
* All issues accepted for backporting should be marked with 8.11.2
  in JIRA, and issues that should delay the release must be marked as
Blocker
* All patches that are intended for the branch should first be committed
  to the unstable branch, merged into the stable branch, and then into
  the current release branch.
* Only Jira issues with Fix version 8.11.2 and priority "Blocker" will delay
  a release candidate build.

Also, please observe that since 9.0 already exists, there cannot be any
index format breaking changes. It really should only be bug fixes that have
already been verified on the 9x branch.

Thanks,
Mike

Re: spotless targets

[Mike Drob]

Save a copy of your gradle.properties and then rerun. There are add opens
options in the jvm opts we generate that were not there before

On Wed, Apr 6, 2022 at 3:46 PM Michael Sokolov  wrote:

> OK, this also happens with Oracle's JDK17. Now I'm confused
>
> On Wed, Apr 6, 2022 at 4:28 PM Michael Sokolov  wrote:
> >
> > Hi, locally I failed to run spotlessCheck/spotlessApply on main (10.x)
> > branch. I assume it's because of a JVM difference; here's the error:
> >
> >
> > Step 'google-java-format' found problem in
> > 'lucene/core/src/java/module-info.java':
> > null
> > java.lang.reflect.InvocationTargetException
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > [giant gradle stack trace omitted]
> >
> > Caused by: java.lang.IllegalAccessError: class
> > com.google.googlejavaformat.java.JavaInput (in unnamed module
> > @0x1430dc06) cannot access class
> > com.sun.tools.javac.parser.Tokens$TokenKind (in module jdk.compiler)
> > because module jdk.compiler does not export com.sun.tools.javac.parser
> > to unnamed module @0x1430dc06
> > at
> com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:349)
> > at
> com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
> > at
> com.google.googlejavaformat.java.JavaInput.(JavaInput.java:276)
> > at
> com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
> > at
> com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
> > at
> com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:233)
> >
> > It seems like a no-no for Google to be relying on Sun packages, but IDK
> >
> > I tried changing
> >
> > >  id 'com.diffplug.spotless' version "6.4.2" apply false
> >
> > in build.gradle
> >
> > and
> >
> > > googleJavaFormat('1.15.0')
> >
> > in spotless.gradle
> >
> >  (the newest versions), because why not, but it didn't seem to help.
> >
> > Any idea how to fix? If it's a problem with my JDK, is there a thing I
> > can ask to have changed in that? Re-export the com.sun.tools.javac
> > package I guess?
> >
> > -Mike
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: gradlew check failure

[Mike Drob]

Is there a way to check for these missing module exports early and fail
with a more informative message?

On Mon, Jan 24, 2022 at 7:42 AM Alan Woodward  wrote:

> Hey Joel,
>
> The fix for this is to delete the gradle.properties file in the root
> directory and stop any daemons before running gradle check again.  The
> build will regenerate the gradle.properties file with some module exports
> that work around this problem in the formatter.
>
> - A
>
> On 24 Jan 2022, at 13:33, Joel Bernstein  wrote:
>
> Hi.
>
> I'm getting the following gradlew check failure with Java 17 on the lucene
> main branch:
>
> Caused by: java.lang.IllegalAccessError: class
> com.google.googlejavaformat.java.JavaInput (in unnamed module @0x3d6a6107)
> cannot access class com.sun.tools.javac.parser.Tokens$TokenKind (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.parser to unnamed module @0x3d6a6107
> at
> com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:349)
> at
> com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
> at
> com.google.googlejavaformat.java.JavaInput.(JavaInput.java:276)
> at
> com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
> at
> com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
> at
> com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:233)
> ... 142 more
>
> Is there a step I'm missing in the setup process?
>
> Thanks,
> Joel
>
>
>

Re: [VOTE] Release Lucene/Solr 8.11.1 RC1

[Mike Drob]

Fast track please

On Wed, Dec 15, 2021 at 6:34 PM Gus Heck  wrote:

> fast track please :)
>
> On Wed, Dec 15, 2021 at 7:23 PM Anshum Gupta 
> wrote:
>
>> Fast-track please :)
>>
>> On Wed, Dec 15, 2021 at 4:19 PM Jan Høydahl 
>> wrote:
>>
>>> Given the votes so far (11 binding +1) I'm also positive to publish
>>> tomorrow, and not wait for Friday.
>>> The release voting rules are three or more +1 votes and more +1 votes
>>> than -1 votes, so for the vote to fail we'd need more than 11 -1's from now
>>> :)
>>>
>>> If I see at least 3 more of you in favor (reply with "FAST-TRACK
>>> PLEASE") and no justified vetoes, then I can make it happen on Thursday
>>> afternoon UTC!
>>>
>>> Jan
>>>
>>> 15. des. 2021 kl. 22:57 skrev Ishan Chattopadhyaya <
>>> ichattopadhy...@gmail.com>:
>>>
>>> I think we should publish, release and announce asap, not waiting for
>>> 72h or the MVN propogation.
>>>
>>> On Thu, 16 Dec, 2021, 2:40 am Anshum Gupta, 
>>> wrote:
>>>
 +1 (binding)

 Smoke tester is happy.

 SUCCESS! [1:03:13.162577]

 Also tested out a sample search/indexing app.

 On Tue, Dec 14, 2021 at 6:36 AM Jan Høydahl 
 wrote:

> Please vote for release candidate 1 for Lucene/Solr 8.11.1
>
> The artifacts can be downloaded from:
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.1-RC1-rev0b002b11819df70783e83ef36b42ed1223c14b50
>
> You can run the smoke tester directly (from a fresh branch_8_11
> checkout), with this command:
>
> python3 -u dev-tools/scripts/smokeTestRelease.py \
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.1-RC1-rev0b002b11819df70783e83ef36b42ed1223c14b50
>
> The vote will be open for at least 72 hours i.e. until 2021-12-17
> 15:00 UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>
> SUCCESS! [0:54:56.979538]
>
> NOTE: You must run the smoke tester from latest commit on branch_8_11,
> since my surname contains a unicode-character, needing a fix in the gpg
> command ran by the smoketester.
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

 --
 Anshum Gupta

>>>
>>>
>>
>> --
>> Anshum Gupta
>>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: [VOTE] Release Lucene/Solr 8.11.1 RC1

[Mike Drob]

+1 (binding)

ran smoke tester - unit tests passed the first time but timed out
downloading artifacts from maven. reran a second time, modifying the smoke
test script to not run solr tests (again) and the script passed.

started up a solr server from the unpacked download and verified it against
a few log4j injections, server responded appropriately each time

manually verified a few of the other bugs having been fixed going into
8.11.1

SUCCESS! [0:35:37.290016]

On Tue, Dec 14, 2021 at 7:24 PM Timothy Potter  wrote:

> +1 (binding) ~ just ran smoke tester this time
>
> SUCCESS! [1:16:20.247006]
>
> On Tue, Dec 14, 2021 at 7:36 AM Jan Høydahl  wrote:
> >
> > Please vote for release candidate 1 for Lucene/Solr 8.11.1
> >
> > The artifacts can be downloaded from:
> >
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.1-RC1-rev0b002b11819df70783e83ef36b42ed1223c14b50
> >
> > You can run the smoke tester directly (from a fresh branch_8_11
> checkout), with this command:
> >
> > python3 -u dev-tools/scripts/smokeTestRelease.py \
> >
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.1-RC1-rev0b002b11819df70783e83ef36b42ed1223c14b50
> >
> > The vote will be open for at least 72 hours i.e. until 2021-12-17 15:00
> UTC.
> >
> > [ ] +1  approve
> > [ ] +0  no opinion
> > [ ] -1  disapprove (and reason why)
> >
> > Here is my +1
> >
> > SUCCESS! [0:54:56.979538]
> >
> > NOTE: You must run the smoke tester from latest commit on branch_8_11,
> since my surname contains a unicode-character, needing a fix in the gpg
> command ran by the smoketester.
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Log4j < 2.15.0 may still be vulnerable even if -Dlog4j2.formatMsgNoLookups=true is set

[Mike Drob]

The MDC Patterns used by solr are for the collection, shard, replica, core
and node names, and a potential trace id. All of those are restricted to
alphanumeric, no special characters like $ or { needed for the injection.
And trying to access a collection that didn’t exist Returns 404 without
logging.

Upgrading is always going to be more complete, but I think we’re still ok
for now, at least until the next iteration of this attack surfaces.



On Tue, Dec 14, 2021 at 3:37 PM solr  wrote:

> Only setting -Dlog4j2.formatMsgNoLookups=true might not be enough to
> mitigate the log4j vulnerability.
>
> See https://github.com/kmindi/log4shell-vulnerable-app
> “So even with LOG4J_FORMAT_MSG_NO_LOOKUPS true version 2.14.1 of log4j is
> vulnerable when using ThreadContextMap in PatternLayout.”
>
> ThreadContext.put(key, value) is used under the hood by MDC.  I’m not sure
> wether any user-input is actually stored in MDC in SOLR.
>
>
> Probably this should be updated:
> https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228
>
> And maybe consider releasing patch releases for other versions than 8.11
> as well which includes log4j 2.16.0?
>
>
>
> Regards,
>
>
> Fredrik
>
>
> --
> Fredrik Rødland   Cell:+47 99 21 98 17
> Maisen Pedersens vei 1Twitter: @fredrikr
> NO-1363 Høvik, NORWAY flickr:  http://www.flickr.com/fmmr/
> http://rodland.no about.me http://about.me/fmr
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Lucene/Solr 8.11.1 release

[Mike Drob]

There's been a log4j 2.16 release as well, should we pick that one up if
your next attempt fails?

https://logging.apache.org/log4j/2.x/changes-report.html#a2.16.0

On Mon, Dec 13, 2021 at 5:58 PM Jan Høydahl  wrote:

> Currently running 4th attempt at making buildAndPushRelease.py happy.
> Always different tests failing...
>
> Jan
>
> 13. des. 2021 kl. 21:00 skrev David Smiley :
>
> Looks good; thanks Jan!
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Dec 13, 2021 at 9:34 AM Jan Høydahl  wrote:
>
>> Including Lucene dev as well.
>>
>> As I see no Lucene level bug fixes for 8.11.1, I have prepared an "empty"
>> release announcement:
>> https://cwiki.apache.org/confluence/display/LUCENE/ReleaseNote8_11_1
>> Please edit as you see fit.
>>
>> The Solr announcement is slightly updated, proof-read welcome
>> https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote8_11_1
>>
>> There are now 18 CHANGES entries for Solr:
>> https://github.com/apache/lucene-solr/blob/branch_8_11/solr/CHANGES.txt
>>
>> Jan
>>
>> > 13. des. 2021 kl. 02:24 skrev Jan Høydahl :
>> >
>> > There seems to be no open blockers for 8.11.1, so I'll proceed with RC1
>> soon.
>> > Shout out if you want me to wait for a specific important bugfix.
>> >
>> > Please also review the Release Notes at
>> https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote8_11_1
>> >
>> > Jan
>> >
>> >> 8. des. 2021 kl. 02:48 skrev Timothy Potter :
>> >>
>> >> agreed! thanks for stepping up to be the RM Jan ;-)
>> >>
>> >> On Tue, Dec 7, 2021 at 6:05 PM Jan Høydahl 
>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Solr has 13 bug fixes lined up in branch_8_11 already. Lucene has no
>> changes.
>> >>> Now that Lucene 9.0 is out the door (congrats!), let's do the 8.11.1
>> release.
>> >>>
>> >>> I volunteer as RM and propose a first RC on Monday 13th. That should
>> allow some time to merge any last bug fixes both for Lucene and Solr.
>> >>> Please feel free to backport bug fixes with your own judgement (no
>> new features please). If you are uncertain whether a backport is "safe",
>> please raise a question here.
>> >>>
>> >>> I'll re-enable Jenkins for branch_8_11 now. Commits that fix or
>> @BadApples unstable tests highly appreciated.
>> >>>
>> >>> Jan
>> >>>
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
>> >>> For additional commands, e-mail: dev-h...@solr.apache.org
>> >>>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
>> >> For additional commands, e-mail: dev-h...@solr.apache.org
>> >>
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>

Re: Strange error in Solr with Lucene 9.0.0

[Mike Drob]

Sorry, meant to add that the reproduction is before the update to lucene
9.0.0 release. But I'm also able to reproduce it after the upgrade, like
Uwe noted.

On Wed, Dec 8, 2021 at 12:53 PM Mike Drob  wrote:

> I was just able to reproduce this on my local macOS machine with
>
> gradlew test --tests TestRandomDVFaceting.testRandomFaceting
> -Dtests.seed=3B93BA61C91F26D4 -Dtests.slow=true -Dtests.locale=uz-Latn
> -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
>
> on commit 0287458f836
>
> with openjdk version "11.0.13"
>
> On Wed, Dec 8, 2021 at 12:30 PM Robert Muir  wrote:
>
>> I also generally tried to "beast" this
>> SimpleFacetsTest.testRangeFacetFilterVsDocValuesRandom without any
>> seed at all (JDK 17/linux), but I didn't manage to provoke any
>> failures.
>>
>> On Wed, Dec 8, 2021 at 1:08 PM Robert Muir  wrote:
>> >
>> > I checked out the commit hash and re-ran the test with the same seed,
>> > it doesn't fail. I only ran the test twice, once with
>> > TieredStopAtLevel=1 and once without that.
>> >
>> > Given that the stacktrace looks "impossible", i suspect this might be
>> > an openjdk issue? I assume this test usually reproduces failures for
>> > given random seeds?
>> >
>> > On Wed, Dec 8, 2021 at 12:44 PM Robert Muir  wrote:
>> > >
>> > > Uwe, it looks a little crazy:
>> > >
>> > > we've got asserts here that "index" is in bounds and certainly not -1
>> > > right before the method call!
>> > >
>> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
>> > >
>> > > On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler 
>> wrote:
>> > > >
>> > > > I updated the dependencies of Solr's main branch to Lucene 9.0
>> > > >
>> > > > On Mac and Linux, the following error occurs sometimes:
>> > > >
>> https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
>> > > >
>> ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
>> > > >
>> https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
>> > > > ache.solr/TestRandomDVFaceting/testRandomFaceting/
>> > > >
>> > > > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
>> 1" at
>> > > >
>> org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
>> > > > r.java:161)
>> > > >
>> > > > This looks like a bug (or is Solr not using something with docvalues
>> > > > correctly, but an AIOOBE should not happen)!
>> > > >
>> > > > When searching through the failure mails, it looks like this was
>> already
>> > > > there with the Lucene preview from summer 2021.
>> > > >
>> > > > Uwe
>> > > >
>> > > > -
>> > > > Uwe Schindler
>> > > > Achterdiek 19, D-28357 Bremen
>> > > > https://www.thetaphi.de
>> > > > eMail: u...@thetaphi.de
>> > > >
>> > > >
>> > > >
>> > > >
>> -
>> > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > > > For additional commands, e-mail: dev-h...@lucene.apache.org
>> > > >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Strange error in Solr with Lucene 9.0.0

[Mike Drob]

I was just able to reproduce this on my local macOS machine with

gradlew test --tests TestRandomDVFaceting.testRandomFaceting
-Dtests.seed=3B93BA61C91F26D4 -Dtests.slow=true -Dtests.locale=uz-Latn
-Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true
-Dtests.file.encoding=UTF-8

on commit 0287458f836

with openjdk version "11.0.13"

On Wed, Dec 8, 2021 at 12:30 PM Robert Muir  wrote:

> I also generally tried to "beast" this
> SimpleFacetsTest.testRangeFacetFilterVsDocValuesRandom without any
> seed at all (JDK 17/linux), but I didn't manage to provoke any
> failures.
>
> On Wed, Dec 8, 2021 at 1:08 PM Robert Muir  wrote:
> >
> > I checked out the commit hash and re-ran the test with the same seed,
> > it doesn't fail. I only ran the test twice, once with
> > TieredStopAtLevel=1 and once without that.
> >
> > Given that the stacktrace looks "impossible", i suspect this might be
> > an openjdk issue? I assume this test usually reproduces failures for
> > given random seeds?
> >
> > On Wed, Dec 8, 2021 at 12:44 PM Robert Muir  wrote:
> > >
> > > Uwe, it looks a little crazy:
> > >
> > > we've got asserts here that "index" is in bounds and certainly not -1
> > > right before the method call!
> > >
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java#L1123-L1125
> > >
> > > On Wed, Dec 8, 2021 at 12:33 PM Uwe Schindler  wrote:
> > > >
> > > > I updated the dependencies of Solr's main branch to Lucene 9.0
> > > >
> > > > On Mac and Linux, the following error occurs sometimes:
> > > >
> https://jenkins.thetaphi.de/job/Solr-main-Linux/2070/testReport/junit/org.ap
> > > >
> ache.solr.request/SimpleFacetsTest/testRangeFacetFilterVsDocValuesRandom/
> > > >
> https://jenkins.thetaphi.de/job/Solr-main-MacOSX/517/testReport/junit/org.ap
> > > > ache.solr/TestRandomDVFaceting/testRandomFaceting/
> > > >
> > > > "ArrayIndexOutOfBoundsException: Index -1 out of bounds for length
> 1" at
> > > >
> org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReade
> > > > r.java:161)
> > > >
> > > > This looks like a bug (or is Solr not using something with docvalues
> > > > correctly, but an AIOOBE should not happen)!
> > > >
> > > > When searching through the failure mails, it looks like this was
> already
> > > > there with the Lucene preview from summer 2021.
> > > >
> > > > Uwe
> > > >
> > > > -
> > > > Uwe Schindler
> > > > Achterdiek 19, D-28357 Bremen
> > > > https://www.thetaphi.de
> > > > eMail: u...@thetaphi.de
> > > >
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > > > For additional commands, e-mail: dev-h...@lucene.apache.org
> > > >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Maven publication with the Gradle build

[Mike Drob]

I believe maven/nexus natively allows publishing an RC to the staging repo
and then manually promoting to a release. Apache infra has this dual set up
for us already.

Would gradle be able to hook into that process?

On Mon, Dec 6, 2021 at 9:15 AM Dawid Weiss  wrote:

> Hi Adrien,
>
> So. From gradle's point of view I don't think it's possible to reuse
> exactly the same files as were present in the RC candidate.
> Technically artifacts are a result of the build (tasks) - they can't
> be taken arbitrarily from disk. Or rather: it may be possible but will
> require terrible hacks.
>
> I suggested to Jan that we could instead package the artifacts as
> Sonatype Nexus's "distribution bundle" - a ZIP file with all the
> information (poms, checksums, jars) that should be staged. This
> requires a manual upload of this ZIP file but otherwise allows
> publishing exact identical files that were part of the RC.
>
> If we only care about releasing from the same git hash (but with
> rebuilt artifacts - so some things in manifests may change!) then
> indeed it's doable via the command Jan mentioned.
>
> I don't think there exists a third way of doing this (?).
>
> I can help add a task that will assemble a Sonatype Nexus bundle which
> you'll be able to download to Nexus. It should ultimately be part of
> the release candidate artifacts - this would make it explicit what
> will get uploaded.
>
> Dawid
>
> On Mon, Dec 6, 2021 at 2:19 PM Adrien Grand  wrote:
> >
> > Hello,
> >
> > The release wizard still suggests using Ant for Maven publication:
> >
> >   cd ~/.lucene-releases/9.0.0/lucene
> >   ant clean stage-maven-artifacts \
> >
>  
> -Dmaven.dist.dir=~/.lucene-releases/9.0.0/RC4/dist/lucene-9.0.0-RC4-rev-0b18b3b965cedaf5eb129aa41243a44c83ca826d/lucene/maven
> > \
> >   -Dm2.repository.id=apache.releases.https \
> >   -Dm2.repository.url=
> https://repository.apache.org/service/local/staging/deploy/maven2
> >
> > The Gradle build has a `mavenToApacheReleases` task that seems to do
> > what I want, but I can't find how to tell it to use the JARs of RC4
> > rather than those produced by `gradlew assembleRelease`. Can someone
> > help me with this?
> >
> > --
> > Adrien
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Closing GitHub PRs

[Mike Drob]

I understand the frustrations around closing somebody’s PR as stale, but I
also think that there is value in informing the contributors I this is
never getting solved/fixed/looked at, if this is still important please go
over there instead.

On Mon, Nov 29, 2021 at 1:55 PM Robert Muir  wrote:

> On Mon, Nov 29, 2021 at 2:49 PM Michael McCandless
>  wrote:
> >
> > Could we maybe instead bulk-add a comment explaining the split and how
> to take the PR forwards if someone (in the future) has itch/time?
> >
> > I know we humans love to clean things up, but I think leaving such
> "unclean" things open serves an important purpose.  They all had importance
> to at least one person at one point in time, and likely many of them are
> still relevant if they piqued someones curiosity to dig back into them.
> Closing them makes them harder to find for the future developer.
> >
> > I'm sure some of them are already resolved/duplicates too.  If only we
> could divine which are which.
> >
> >
>
> +1, I'd rather not auto-close PRs. I'm always frustrated by this when
> I see it in other trackers. Is there a rush to close these for some
> reason?
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Closing GitHub PRs

[Mike Drob]

We currently have almost 300 open PRs against the "master" branch in the
old lucene-solr repo.

https://github.com/apache/lucene-solr/pulls?q=is%3Apr+is%3Aopen+base%3Amaster

I think we should close all of them (possibly with a comment pointing
people to the main branch in the lucene or solr repos).

I do not know if there is an automated way to do this, or a way to do this
without spamming everybody's email lists. If there is consensus on this
clean up work then I'll take some time to investigate how to get this done
and/or chat with ASF Infra about it.

Thanks,
Mike

Re: Warnings from gradle rat task

[Mike Drob]

On Sat, Nov 27, 2021 at 9:48 AM Dawid Weiss  wrote:

> MG>I suspect you have an ancient javadoc binary in classpath
>> MG>but there is no way of knowing without looking at contents of
>> antlib.xml
>>
>>
> Looking at ant code it seems like it tried to reload tasks for some
> reason and class loaders are different hence the warning. We don't have
> javadoc on classpath - gradle does. I suspect it isolates tasks from each
> other somehow... not really worth looking that deep into, I guess.
>
> D.
>

I think I looked into using the gradle rat plugin when we were doing the
migrations but it was missing features. I filed an issue for them and I
think they fixed it since then but I never went back to check if that would
be enough for us to switch over. Maybe worth investigating, but also
possibly not worth the efforts.

Mike

>

Re: 8.11 release candidate

[Mike Drob]

After additional discussion on
https://issues.apache.org/jira/browse/SOLR-15762 I think there is a small
patch that would be nice to include for 8.11 if there is still time. Thanks

On Fri, Nov 5, 2021 at 5:05 AM Adrien Grand  wrote:

> All blockers are now addressed, I'll proceed with building a release
> candidate.
>
> On Thu, Nov 4, 2021 at 6:51 PM Adrien Grand  wrote:
>
>> Thanks Jan!
>>
>> On Thu, Nov 4, 2021 at 2:32 PM Jan Høydahl  wrote:
>>
>>> I added a PR https://github.com/apache/lucene-solr/pull/2603 for
>>> SOLR-14438, that I had in the workings. I think it will do...
>>>
>>> Jan
>>>
>>> 4. nov. 2021 kl. 10:34 skrev Adrien Grand :
>>>
>>> Joel recommended not treating SOLR-15762 as a blocker, so we are left
>>> with the following issues as blockers:
>>>  - SOLR-15706, which should hopefully be addressed soon.
>>>  - SOLR-14438, which hasn't had activity for a long time. Is someone
>>> looking into it? Can someone comment on whether it should actually be
>>> treated as a blocker for 8.11?
>>>
>>> On Wed, Nov 3, 2021 at 7:32 PM Adrien Grand  wrote:
>>>
 Jason's above issue isn't fixed yet but it looks like we should be able
 to have a fix in the very near future.

 However we seem to still have these two other open Solr blockers for
 8.11:
  - SOLR-14438 : Mention
 decodeAES in LICENSE.txt
  - SOLR-15762 : 
 IllegalStateException:
 Recursive update thrown when executing complex Join queries

 Can someone who is more familiar with Solr than I am help confirm that
 we should indeed treat these two issues as blockers?

 On Tue, Nov 2, 2021 at 9:45 PM Adrien Grand  wrote:

> Hi Jason,
>
> This looks bad indeed. Please feel free to backport to 8.11 and update
> this thread when you are done?
>
> On Tue, Nov 2, 2021 at 8:24 PM Jason Gerlowski 
> wrote:
>
>> Hey Adrien,
>>
>> A pretty bad bug in Solr backups came in last week or so: SOLR-15706.
>> If it's not too late I'd like to try to get it into 8.11.  Would that
>> be ok?
>>
>> I'm just digging into things now so I don't have a complete
>> understanding yet, but I'm optimistic it won't long delay your
>> timeline for cutting the RC.
>>
>> Best,
>>
>> Jason
>>
>> On Tue, Nov 2, 2021 at 1:31 PM Adrien Grand 
>> wrote:
>> >
>> > Hello,
>> >
>> > Assuming CI is green and no blockers have been raised until then, I
>> plan to build the first release candidate for Lucene/Solr 8.11 on 
>> Thursday
>> November 4th.
>> >
>> > Please let me know if you are aware of any blocker that we should
>> address before building the first RC.
>> >
>> > --
>> > Adrien
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>
> --
> Adrien
>


 --
 Adrien

>>>
>>>
>>> --
>>> Adrien
>>>
>>>
>>>
>>
>> --
>> Adrien
>>
>
>
> --
> Adrien
>

Re: Bump minimum Java version to 17 on main (10.0)

[Mike Drob]

We'll be going to Java 18 or 19 as a minimum for MMapDirectory using the
new Panama APIs once those stabilize, right?

We could probably benefit today some from record classes, but I'm not sure
how much of a hint those are to the runtime VM for optimizations or if it
is entirely a source code syntactic sugar thing.

On Wed, Nov 3, 2021 at 11:53 AM Robert Muir  wrote:

> On Wed, Nov 3, 2021 at 12:51 PM Robert Muir  wrote:
> > >
> > > It will be released, eventually, right? In six months, a year maybe?
> Then it's people like me who would be affected: we use Lucene internally
> and this one dependency would push the entire stack to Java 17. I wouldn't
> mind this at all if there was something super-attractive there to gain but
> all the benefits of Java 17 to me at the moment lie at source-code level...
> I just don't see much benefit at runtime. So I personally don't feel the
> need to upgrade - this applies to Lucene, my own software and even my shell
> default...
> >
>
> Separately, but if you want to stay on old stuff, then stay on old
> stuff (e.g. stay on lucene 9 with java 11). This is how people do it
> with all other software. If you are happy with java 11, then you
> "personally don't feel the need to upgrade" and you can just stay on
> lucene 9, or lucene 8, or lucene 7, or whatever!
>
> But there should be a place where we can move forwards: I think the
> 'main' branch is appropriate.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Fwd: 8.11 and 9.0 feature freeze

[Mike Drob]

Adding Solr dev for 8.11 info.

-- Forwarded message -
From: Adrien Grand 
Date: Tue, Nov 2, 2021 at 8:08 AM
Subject: 8.11 and 9.0 feature freeze
To: Lucene Dev 


Hello all,

I just created branches in preparation for the upcoming 8.11 and 9.0
releases. Here's how branches map to Lucene versions now:
 - main: Lucene 10.0
 - branch_9x: Lucene 9.1
 - branch_9_0: Lucene 9.0
 - branch_8_11: Lucene/Solr 8.11
 - branch_8x: Shouldn't be used anymore, there will NOT be a 8.12 release.

8.11 and 9.0 are feature-frozen, so please be careful to only backport safe
bug fixes to these branches as of now. In case of doubt, please let me know.

-- 
Adrien

Re: Bugfix release Lucene/Solr 8.10.1

[Mike Drob]

Hi Mayya, I just committed SOLR-15269 to the release branch, a very minor
dependency update. My apologies for not coming and checking in the thread
first. Please let me know if this is ok, I can revert if you need me to.
Thanks for handling the release!

On Sun, Oct 10, 2021 at 9:04 PM Mayya Sharipova
 wrote:

> Thank you for the suggestion, Jan, I will follow it.
>
> On Sun, Oct 10, 2021 at 5:27 PM Jan Høydahl  wrote:
>
>> Remember that the releaseWizard on 8x repo is not prepared with correct
>> commands to update the 'main' repos properly. So each time it asks you to
>> do stuff on 'master'/unstable branch you must instead manually adapt andd
>> perform those steps on the new apache/lucene and apache/solr repos.
>>
>> Jan
>>
>> 10. okt. 2021 kl. 13:45 skrev Mayya Sharipova :
>>
>> Hi Houston,
>> Thank you for bringing this to my attention, I've corrected the 8.10.1
>> changelog for all the branches.
>>
>> Thanks a lot.
>>
>> On Fri, Oct 8, 2021 at 11:44 PM Houston Putman 
>> wrote:
>>
>>> Hey Mayya,
>>>
>>> I think there are some issues with the lucene changelog entries in
>>> 8.10.1 across various branches (branch_8_10, branch_8x, and main).
>>> I don't see LUCENE-10119, and LUCENE-10126 is marked as an 8.11.0 entry
>>> in at least branch_8x.
>>>
>>> - Houston
>>>
>>> On Thu, Oct 7, 2021 at 9:32 PM Mayya Sharipova <
>>> mayya.sharip...@elastic.co.invalid> wrote:
>>>
 Hello everyone,

 I am now preparing for a bugfix release from branch branch_8_10

 Please observe the normal rules for committing to this branch:

 * Before committing to the branch, reply to this thread and argue
   why the fix needs backporting and how long it will take.
 * All issues accepted for backporting should be marked with 8.10.1
   in JIRA, and issues that should delay the release must be marked as
 Blocker
 * All patches that are intended for the branch should first be committed
   to the unstable branch, merged into the stable branch, and then into
   the current release branch.
 * Only Jira issues with Fix version 8.10.1 and priority "Blocker" will
 delay
   a release candidate build.

  -
 Currently we have the following  committed for 8.10.1
 LUCENE-10119: Do not set single sort with search after
 LUCENE-10126: Fix competitive iterator wrongly skip docs
 SOLR-15572: Fix a test failure caused by simplistic line parsing.
 SOLR-15665: Move polling logic under main

 -
 We have the following bug fixes already resolved in 8.11.
 Does anyone from the list below want to backport their fixes for 8.10.1
 release as well?  Please let me know by Monday

 * LUCENE-10110: MultiCollector now handles single leaf collector that
 wants to skip low-scoring hits but the combined score mode doesn't allow
 it. (Jim Ferenczi)
 * LUCENE-10111: Missing calculating the bytes used of DocsWithFieldSet
 in NormValuesWriter. (Lu Xugang)
 * LUCENE-10116: Missing calculating the bytes used of
 DocsWithFieldSetand currentValues in SortedSetDocValuesWriter. (Lu Xugang)
 * LUCENE-10070 Skip deleted docs when accumulating facet counts for all
 docs. (Ankur Goel, Greg Miller)
 * LUCENE-10134: ConcurrentSortedSetDocValuesFacetCounts shouldn't share
 liveDocs Bits across threads. (Ankur Goel)
 * SOLR-15626: Config-read permission does not allow access to
 /solr/admin/configs?action=LIST  (Jon Senchyna)


 Thanks.



>>

Re: 8.10.1 Patch release?

[Mike Drob]

It feels weird to say that I’m against releases, but generally I feel like
bug fix releases should be scoped either for a regression discovered in
that release or for rapid security fixes. Otherwise, what’s the harm in
waiting for the next release train?

Obviously any committee is free to create a release candidate on any
commit, and if there are three PMC members in support then a release can
happen, but I don’t want to be putting pressure on ourselves where we are
constantly in the middle of a release cycle.

Or waiting a month and doing 8.11 seems fine too?

On Wed, Oct 6, 2021 at 8:17 AM Adrien Grand  wrote:

> Mike, it's unclear to me if you are suggesting waiting before doing a
> 8.10.1 release? On my end I'm good with doing a 8.10.1 release now, we
> could still do a 8.10.2 release later in case we find new bugs?
>
> On Tue, Oct 5, 2021 at 10:41 PM Mayya Sharipova
>  wrote:
>
>> No, the bug is not new and was present in the previous versions as well,
>> but was discovered quite recently.
>>
>> On Tue, Oct 5, 2021 at 3:54 PM Mike Drob  wrote:
>>
>>> Is the bug new in 8.10? If it affects older versions as well then I feel
>>> like 8.10.1 might be less urgent.
>>>
>>> Mike
>>>
>>> On Tue, Oct 5, 2021 at 2:14 PM Adrien Grand  wrote:
>>>
>>>> +1 to a 8.10.1 patch release
>>>>
>>>> On Tue, Oct 5, 2021 at 2:03 AM Mayya Sharipova
>>>>  wrote:
>>>>
>>>>> Thanks for the update, Robert.  Would be nice to have these  bug fixes
>>>>> as well.
>>>>>
>>>>> On Mon, Oct 4, 2021 at 7:56 PM Robert Muir  wrote:
>>>>>
>>>>>> FYI Looks like there are already six items currently listed under
>>>>>> "Bugfixes" for 8.11.0, so those could be candidates for the patch
>>>>>> release.
>>>>>>
>>>>>> Bug Fixes
>>>>>> -
>>>>>> * LUCENE-10110: MultiCollector now handles single leaf collector that
>>>>>> wants to skip low-scoring hits
>>>>>> but the combined score mode doesn't allow it. (Jim Ferenczi)
>>>>>> * LUCENE-10111: Missing calculating the bytes used of DocsWithFieldSet
>>>>>> in NormValuesWriter.
>>>>>> (Lu Xugang)
>>>>>> * LUCENE-10116: Missing calculating the bytes used of DocsWithFieldSet
>>>>>> and currentValues in SortedSetDocValuesWriter.
>>>>>> (Lu Xugang)
>>>>>> * LUCENE-10070 Skip deleted docs when accumulating facet counts for
>>>>>> all docs. (Ankur Goel, Greg Miller)
>>>>>> * LUCENE-10126: Sort optimization with a chunked bulk scorer
>>>>>> can wrongly skip documents (Nhat Nguyen, Mayya Sharipova)
>>>>>> * LUCENE-10134: ConcurrentSortedSetDocValuesFacetCounts shouldn't
>>>>>> share liveDocs Bits across threads.
>>>>>> (Ankur Goel)
>>>>>>
>>>>>> On Mon, Oct 4, 2021 at 7:46 PM Mayya Sharipova 
>>>>>> wrote:
>>>>>> >
>>>>>> > Hello everyone!
>>>>>> > Thank you, Timothy, for the recent 8.10 release.
>>>>>> >
>>>>>> > I wonder if we are ok to do a 8.10.1 patch release and do it fairly
>>>>>> soon? this week?
>>>>>> > Nhat fixed a bad bug where "sort with after" on a numeric field can
>>>>>> incorrectly miss documents. This bug only manifests when sort 
>>>>>> optimization
>>>>>> on numeric fields is explicitly enabled.
>>>>>> >
>>>>>> > Given that we are not sure when the next minor release will be, it
>>>>>> would be useful to have a patch release.  If nobody is opposed to it,  I
>>>>>> can volunteer to be the Release Manager.
>>>>>> >
>>>>>> > Thanks.
>>>>>>
>>>>>> -
>>>>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>>>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Adrien
>>>>
>>>
>
> --
> Adrien
>

Re: 8.10.1 Patch release?

[Mike Drob]

Is the bug new in 8.10? If it affects older versions as well then I feel
like 8.10.1 might be less urgent.

Mike

On Tue, Oct 5, 2021 at 2:14 PM Adrien Grand  wrote:

> +1 to a 8.10.1 patch release
>
> On Tue, Oct 5, 2021 at 2:03 AM Mayya Sharipova
>  wrote:
>
>> Thanks for the update, Robert.  Would be nice to have these  bug fixes as
>> well.
>>
>> On Mon, Oct 4, 2021 at 7:56 PM Robert Muir  wrote:
>>
>>> FYI Looks like there are already six items currently listed under
>>> "Bugfixes" for 8.11.0, so those could be candidates for the patch
>>> release.
>>>
>>> Bug Fixes
>>> -
>>> * LUCENE-10110: MultiCollector now handles single leaf collector that
>>> wants to skip low-scoring hits
>>> but the combined score mode doesn't allow it. (Jim Ferenczi)
>>> * LUCENE-10111: Missing calculating the bytes used of DocsWithFieldSet
>>> in NormValuesWriter.
>>> (Lu Xugang)
>>> * LUCENE-10116: Missing calculating the bytes used of DocsWithFieldSet
>>> and currentValues in SortedSetDocValuesWriter.
>>> (Lu Xugang)
>>> * LUCENE-10070 Skip deleted docs when accumulating facet counts for
>>> all docs. (Ankur Goel, Greg Miller)
>>> * LUCENE-10126: Sort optimization with a chunked bulk scorer
>>> can wrongly skip documents (Nhat Nguyen, Mayya Sharipova)
>>> * LUCENE-10134: ConcurrentSortedSetDocValuesFacetCounts shouldn't
>>> share liveDocs Bits across threads.
>>> (Ankur Goel)
>>>
>>> On Mon, Oct 4, 2021 at 7:46 PM Mayya Sharipova  wrote:
>>> >
>>> > Hello everyone!
>>> > Thank you, Timothy, for the recent 8.10 release.
>>> >
>>> > I wonder if we are ok to do a 8.10.1 patch release and do it fairly
>>> soon? this week?
>>> > Nhat fixed a bad bug where "sort with after" on a numeric field can
>>> incorrectly miss documents. This bug only manifests when sort optimization
>>> on numeric fields is explicitly enabled.
>>> >
>>> > Given that we are not sure when the next minor release will be, it
>>> would be useful to have a patch release.  If nobody is opposed to it,  I
>>> can volunteer to be the Release Manager.
>>> >
>>> > Thanks.
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>
>
> --
> Adrien
>

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

That was a bad backport from main, and I was mainly paying attention to the
main jenkins tests. Apologies about that oversight. Please see PR
https://github.com/apache/lucene-solr/pull/2578

On Mon, Sep 20, 2021 at 2:21 PM Uwe Schindler  wrote:

> Hi,
>
> This test also fails on Jenkins all the time. In all branches and on all
> platforms. All the time, it's definitely a regression.
>
> Uwe
>
> Am 20. September 2021 19:13:56 UTC schrieb Timothy Potter <
> thelabd...@gmail.com>:
>>
>> Started building the RC1 again today and the smoke tester failed. The
>> culprit was: org.apache.solr.search.TestFiltering.testRandomFiltering
>>
>> Re-ran that test from the RC checkout and it failed again:
>>
>>[junit4]   2> 5490 ERROR
>> (TEST-TestFiltering.testRandomFiltering-seed#[9A85A1D74D8AACF9]) [
>> ] o.a.s.SolrTestCaseJ4 REQUEST FAILED:
>> facet.query=*:*={!key%3DmultiSelect+ex%3Dt}*:*={!key%3DfacetQuery+cache%3Dfalse}+val_i:2+val_i:4={!+cost%3D7+tag%3Dt}-_query_:"{!frange+v%3Dval_i+l%3D2+u%3D5}"=true=xml
>>[junit4]   2> 5491 INFO
>> (TEST-TestFiltering.testRandomFiltering-seed#[9A85A1D74D8AACF9]) [
>> ] o.a.s.SolrTestCaseJ4 ###Ending testRandomFiltering
>>[junit4]   2> NOTE: reproduce with: ant test
>> -Dtestcase=TestFiltering -Dtests.method=testRandomFiltering
>> -Dtests.seed=9A85A1D74D8AACF9 -Dtests.slow=true -Dtests.badapples=true
>> -Dtests.locale=mgh -Dtests.timezone=Iceland -Dtests.asserts=true
>> -Dtests.file.encoding=US-ASCII
>>[junit4] FAILURE 0.82s | TestFiltering.testRandomFiltering <<<
>>[junit4]> Throwable #1: java.lang.AssertionError: should have 
>> unwrapped
>>[junit4]> at
>> __randomizedtesting.SeedInfo.seed([9A85A1D74D8AACF9:85E60212A8ADECF0]:0)
>>[junit4]> at
>> org.apache.solr.search.SolrIndexSearcher.getAndCacheDocSet(SolrIndexSearcher.java:862)
>>[junit4]> at
>> org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:824)
>>[junit4]> at
>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1367)
>>[junit4]> at
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:596)
>>[junit4]> at
>> org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1511)
>>[junit4]> at
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:390)
>>[junit4]> at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:368)
>>[junit4]> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)
>>[junit4]> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2637)
>>
>> Looking at the stats for this test, it seems like it started failing
>> more consistently over the past week or so:
>> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.search.TestFiltering.testRandomFiltering
>>
>> I beasted it and 3/10 failed:
>>
>>   [beaster] Tests with failures [seed: A5F8AAEF7994FE2B] (first 3 out of 10):
>>   [beaster]   - org.apache.solr.search.TestFiltering.testRandomFiltering
>>   [beaster]   - org.apache.solr.search.TestFiltering.testRandomFiltering
>>   [beaster]   - org.apache.solr.search.TestFiltering.testRandomFiltering
>>
>> I could just mark it as a BadApple and move on, but wanted to see if
>> anyone had any ideas about this test flakiness?
>>
>> Cheers,
>> Tim
>>
>> On Fri, Sep 17, 2021 at 4:06 PM Mike Drob  wrote:
>>
>>>
>>>  Can we move discussion about the implementation to the JIRA issue or the 
>>> PR?
>>>
>>>  I'm not a lawyer, so not playing with the GPL fire is the easiest way for 
>>> me to avoid getting burned. The regex I have is pretty straightforward, I 
>>> do not feel like it is a great cause for alarm.
>>>
>>>  On Fri, Sep 17, 2021 at 4:18 PM Ishan Chattopadhyaya 
>>>  wrote:
>>>
>>>>
>>>>  Given that we don't ship the code or binaries that involve that python 
>>>> library, do we need to care about the license? I'm skeptical of hand 
>>>> rolled regex and would rather favour either of the libraries Jan 
>>>> mentioned. Just my two cents.
>>>>
>>>>  On Sat, 18 Sep, 2021, 12:02 am Mike Drob,  wrote:
>>>>
>>>>>
>>>>>  The second library you linked, Jan, is AGPL. Thank you for continuing to 
>&

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

Can we move discussion about the implementation to the JIRA issue or the PR?

I'm not a lawyer, so not playing with the GPL fire is the easiest way for
me to avoid getting burned. The regex I have is pretty straightforward, I
do not feel like it is a great cause for alarm.

On Fri, Sep 17, 2021 at 4:18 PM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Given that we don't ship the code or binaries that involve that python
> library, do we need to care about the license? I'm skeptical of hand rolled
> regex and would rather favour either of the libraries Jan mentioned. Just
> my two cents.
>
> On Sat, 18 Sep, 2021, 12:02 am Mike Drob,  wrote:
>
>> The second library you linked, Jan, is AGPL. Thank you for continuing to
>> look for alternatives.
>>
>> I have some regular expressions cooked up locally that I think will let
>> us read the split lines going forward, and will put up the patch shortly.
>>
>> On Fri, Sep 17, 2021 at 7:45 AM Yuval Paz 
>> wrote:
>>
>>> Not sure if this is something can be changed easily, but if the problem
>>> is caused by some parsers don't know how to parse line wrapping in the
>>> middle of the Hash why not moving the hash completely to the new line (the
>>> specification allow new line at any point in the value)?
>>>
>>> The commit hash + date comes out to be exactly 71 bytes (including the
>>> space at the start), and it should be a constant size, and by the time the
>>> version will reach 48 bytes we all be probably dead
>>>
>>> On Fri, Sep 17, 2021, 2:47 PM Robert Muir  wrote:
>>>
>>>> Sure, but that package is archived/read-only, GPLv3. with 3 watchers
>>>> and 1 star.
>>>>
>>>> On Fri, Sep 17, 2021 at 4:27 AM Jan Høydahl 
>>>> wrote:
>>>> >
>>>> > Let's just follow the spec and move on.
>>>> >
>>>> > Just tested this python package, which has no problem parsing the
>>>> problematic manifest https://pypi.org/project/jarmanifest/
>>>> >
>>>> > >>> manifest.getAttributes("/tmp/lucene-manifest.mf")
>>>> > [{'implementationversion': '9.0.0-SNAPSHOT
>>>> de45b68c909815ce5ea7b6b9e1a2ce3375b6334d [snapshot build, details
>>>> omitted]'}]
>>>> >
>>>> > Jan
>>>> >
>>>> > 17. sep. 2021 kl. 09:32 skrev Dawid Weiss :
>>>> >
>>>> >
>>>> > We could do a few things to keep everyone happy -
>>>> >
>>>> > 1) keep abbreviated hash in the Implementat-Version and use a
>>>> separate manifest entry to store a full hash.
>>>> > 2) use a longer version for git show (abbrev=num) so that the chance
>>>> of collisions in the future is minimized. It's still not a full hash but a
>>>> > long(er) forced prefix.
>>>> >
>>>> > D.
>>>> >
>>>> > On Fri, Sep 17, 2021 at 12:21 AM Chris Hostetter <
>>>> hossman_luc...@fucit.org> wrote:
>>>> >>
>>>> >>
>>>> >> : I was referring to doing this with languages other than java.
>>>> >> :
>>>> >> : I'm also assuming that exceeding this limit is going to cause
>>>> indirect
>>>> >> : hassles for users of lucene, e.g. breaking various security /
>>>> supply
>>>> >> : chain tools. We know a lot of these are total crap but people in
>>>> the
>>>> >> : corporate world have to suffer under them.
>>>> >>
>>>> >> Just to be clear -- our 'Implementation-Version:' has been exceeding
>>>> the
>>>> >> 72 byte "single line" limit for a LONG time -- worrying about how
>>>> >> "security / supply chain" tools will handle parsing that line now
>>>> seems
>>>> >> kind of silly...
>>>> >>
>>>> >> If tools can't handle a line wrap in the 8.10 jars, then they haven't
>>>> >> been able to handle the line wrap since we switched from svn to git
>>>> (when
>>>> >> the Implementation Version values switched from being based svn
>>>> version#
>>>> >> to git sha)
>>>> >>
>>>> >> The *ONLY* thing that's new here is where in the value the line wrap
>>>> >> happens (with 8.10.0 it happens in the middle of the SHA) and that
>>>> our
>&

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

 at 3:03 AM Dawid Weiss 
>> wrote:
>> >> : >
>> >> : > Jar command doesn't have it, true. But it's fairly trivial to do,
>> even
>> >> : > with an inline snippet like this one?
>> >> : >
>> >> : > public class PrintManifest {
>> >> : >   public static void main(String[] jars) throws IOException {
>> >> : > for (var jar : jars) {
>> >> : >   var manifest = new
>> JarFile(Paths.get(jar).toFile()).getManifest();
>> >> : >   var attrs = manifest.getMainAttributes();
>> >> : >   System.out.println(jar + ": " +
>> attrs.getValue("Implementation-Version"));
>> >> : > }
>> >> : >   }
>> >> : > }
>> >> : >
>> >> : > I have this in my lucene-core-9.0.0-SNAPSHOT.jar:
>> >> : >
>> >> : > Implementation-Version: 9.0.0-SNAPSHOT
>> de45b68c909815ce5ea7b6b9e1a2ce337
>> >> : >  5b6334d [snapshot build, details omitted]
>> >> : >
>> >> : > and running:
>> >> : >
>> >> : > java PrintManifest.java lucene-core-9.0.0-SNAPSHOT.jar
>> >> : >
>> >> : > shows:
>> >> : >
>> >> : > lucene-core-9.0.0-SNAPSHOT.jar: 9.0.0-SNAPSHOT
>> >> : > de45b68c909815ce5ea7b6b9e1a2ce3375b6334d [snapshot build, details
>> >> : > omitted]
>> >> : >
>> >> : > This seems easier to me than trying to remember and keep the
>> length of
>> >> : > that line shorter than an arbitrary limit.
>> >> : >
>> >> : > Dawid
>> >> : >
>> >> : >
>> >> : > On Wed, Sep 15, 2021 at 9:46 PM Robert Muir 
>> wrote:
>> >> : > >
>> >> : > > But its irrelevant that is "valid" when virtually no tools match
>> it.
>> >> : > >
>> >> : > > In other words, I'd agree with you if the "jar" command had some
>> >> : > > ability to read these manifests and print stuff to stdout, e.g.
>> if
>> >> : > > there was ANY interop at all here.
>> >> : > >
>> >> : > > But there isn't. So IMO it makes no sense to cause confusion and
>> chaos
>> >> : > > by adding an unnecessarily long git commit hash.
>> >> : > >
>> >> : > > On Wed, Sep 15, 2021 at 3:26 PM Dawid Weiss <
>> dawid.we...@gmail.com> wrote:
>> >> : > > >
>> >> : > > >
>> >> : > > > This is valid manifest line-breaking though... Can we read the
>> manifest properly on the smoke tester side somehow (for example, run a Java
>> process that reads and extracts the required attribute)? This way we
>> wouldn't care about the implementation details of how manifest wraps the
>> lines (or escapes characters).
>> >> : > > >
>> >> : > > > D.
>> >> : > > >
>> >> : > > > On Wed, Sep 15, 2021 at 8:46 PM Mike Drob 
>> wrote:
>> >> : > > >>
>> >> : > > >> The benchmark jar has the info we need… sort of. When I built
>> it, it has:
>> >> : > > >>
>> >> : > > >> Implementation-Version: 8.10.0
>> 75a5061d3715cc5d93c4cbe4f1fa62bf035eea1
>> >> : > > >>  1 - mdrob - 2021-09-15 11:40:36
>> >> : > > >>
>> >> : > > >>
>> >> : > > >> and it’s looking for Implementation-Version: 8.10.0
>> 75a5061d3715cc5d93c4cbe4f1fa62bf035eea11 on one line.
>> >> : > > >>
>> >> : > > >> Because 8.10 is a character longer than 8.9, we happen to
>> wrap the last character of the git commit sha. From the manifest spec:
>> >> : > > >>
>> >> : > > >> No line may be longer than 72 bytes (not characters), in its
>> UTF8-encoded form.
>> >> : > > >> If a value would make the initial line longer than this, it
>> should be continued
>> >> : > > >> on extra lines (each starting with a single SPACE).
>> >> : > > >>
>> >> : > > >> And we were already teetering on the edge of that limit.
>> We'll run into this problem again in a few years when we try to release
>> version 10.0.0, so solving it now has practical benefits down the line.
>>

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

>Can we read the manifest properly on the smoke tester side somehow

This would be my option #3?


These are apparently going to need different solutions for ant and gradle,
so we'll handle them separately.

Created https://issues.apache.org/jira/browse/LUCENE-10107 for 8.x and
https://issues.apache.org/jira/browse/LUCENE-10108 for 9.x+

On Wed, Sep 15, 2021 at 2:25 PM Dawid Weiss  wrote:

>
> This is valid manifest line-breaking though... Can we read the manifest
> properly on the smoke tester side somehow (for example, run a Java process
> that reads and extracts the required attribute)? This way we wouldn't care
> about the implementation details of how manifest wraps the lines (or
> escapes characters).
>
> D.
>
> On Wed, Sep 15, 2021 at 8:46 PM Mike Drob  wrote:
>
>> The benchmark jar has the info we need… sort of. When I built it, it has:
>>
>> Implementation-Version: 8.10.0 75a5061d3715cc5d93c4cbe4f1fa62bf035eea1
>>  1 - mdrob - 2021-09-15 11:40:36
>>
>>
>> and it’s looking for Implementation-Version: 8.10.0
>> 75a5061d3715cc5d93c4cbe4f1fa62bf035eea11 on one line.
>>
>> Because 8.10 is a character longer than 8.9, we happen to wrap the last
>> character of the git commit sha. From the manifest spec:
>>
>> No line may be longer than 72 bytes (not characters), in its UTF8-encoded 
>> form.
>> If a value would make the initial line longer than this, it should be 
>> continued
>> on extra lines (each starting with a single SPACE).
>>
>> And we were already teetering on the edge of that limit. We'll run into
>> this problem again in a few years when we try to release version 10.0.0, so
>> solving it now has practical benefits down the line.
>>
>> There's a few options that I can come up with -
>> 1. Use the short-hash when we generate the jar
>> 2. Use the short-hash when we check the contents in the smoke test
>> 3. Do some line join magic in the smoke test.
>>
>> I'm leaning towards number 1 as I feel that would still be unique enough
>> for our needs, but would like to hear from others as well.
>>
>> On Wed, Sep 15, 2021 at 9:46 AM Timothy potter 
>> wrote:
>>
>>> can someone also please look into that benchmark jar issue?
>>>
>>> Sent from my iPhone
>>>
>>> On Sep 15, 2021, at 9:44 AM, Nhat Nguyen 
>>> wrote:
>>>
>>> 
>>> Thanks Mayya and Mike! I will backport it to the 8.10 branch.
>>>
>>> On Wed, Sep 15, 2021 at 10:12 AM Mike Drob  wrote:
>>>
>>>> I think since Tim is out on vacation, it's probably not too late. That
>>>> looks like a good fix to have, do we know how long the bug has been 
>>>> present?
>>>>
>>>> On Wed, Sep 15, 2021 at 7:56 AM Mayya Sharipova
>>>>  wrote:
>>>>
>>>>> Hello everyone,
>>>>> We have discovered a bug and fixed a bug in Lucene sort optimization
>>>>> <https://github.com/apache/lucene/pull/300> (LUCENE-10106) and would
>>>>> like to merge it to Lucene 8.10 if it is not too late.
>>>>> I apologize for the inconvenience, the bug was discovered just
>>>>> yesterday.
>>>>>
>>>>> On Tue, Sep 14, 2021 at 9:26 PM Timothy Potter 
>>>>> wrote:
>>>>>
>>>>>> Ahem ... unfortunately there will not be an 8.10 RC this week. I'm
>>>>>> headed out on vacation tomorrow, back at keys on Monday, Sept 20
>>>>>> unless someone else wants to pick up the RM duties before then?
>>>>>>
>>>>>> After failing the test suite at various places and other weirdness
>>>>>> like .asc files not getting created, I finally got to the smoke test
>>>>>> part, which is now failing with:
>>>>>>
>>>>>>   File
>>>>>> "/Users/tjp/.lucene-releases/8.10.0/lucene-solr/dev-tools/scripts/smokeTestRelease.py",
>>>>>> line 176, in checkJARMetaData
>>>>>> raise RuntimeError('%s is missing "%s" inside its
>>>>>> META-INF/MANIFEST.MF (wrong git revision?)' % \
>>>>>> RuntimeError: JAR file
>>>>>>
>>>>>> "/Users/tjp/.lucene-releases/8.10.0/RC1/smoketest/unpack/lucene-8.10.0/benchmark/lucene-benchmark-8.10.0.jar"
>>>>>> is missing "Implementation-Version: 8.10.0
>>>>>> ecf5c747e6df418dd05a18af327c20051f0584d7" inside its
>>>>>> META-INF/MANIFEST.MF (wrong git revision?)

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

The benchmark jar has the info we need… sort of. When I built it, it has:

Implementation-Version: 8.10.0 75a5061d3715cc5d93c4cbe4f1fa62bf035eea1
 1 - mdrob - 2021-09-15 11:40:36


and it’s looking for Implementation-Version: 8.10.0
75a5061d3715cc5d93c4cbe4f1fa62bf035eea11 on one line.

Because 8.10 is a character longer than 8.9, we happen to wrap the last
character of the git commit sha. From the manifest spec:

No line may be longer than 72 bytes (not characters), in its UTF8-encoded form.
If a value would make the initial line longer than this, it should be continued
on extra lines (each starting with a single SPACE).

And we were already teetering on the edge of that limit. We'll run into
this problem again in a few years when we try to release version 10.0.0, so
solving it now has practical benefits down the line.

There's a few options that I can come up with -
1. Use the short-hash when we generate the jar
2. Use the short-hash when we check the contents in the smoke test
3. Do some line join magic in the smoke test.

I'm leaning towards number 1 as I feel that would still be unique enough
for our needs, but would like to hear from others as well.

On Wed, Sep 15, 2021 at 9:46 AM Timothy potter  wrote:

> can someone also please look into that benchmark jar issue?
>
> Sent from my iPhone
>
> On Sep 15, 2021, at 9:44 AM, Nhat Nguyen 
> wrote:
>
> 
> Thanks Mayya and Mike! I will backport it to the 8.10 branch.
>
> On Wed, Sep 15, 2021 at 10:12 AM Mike Drob  wrote:
>
>> I think since Tim is out on vacation, it's probably not too late. That
>> looks like a good fix to have, do we know how long the bug has been present?
>>
>> On Wed, Sep 15, 2021 at 7:56 AM Mayya Sharipova
>>  wrote:
>>
>>> Hello everyone,
>>> We have discovered a bug and fixed a bug in Lucene sort optimization
>>> <https://github.com/apache/lucene/pull/300> (LUCENE-10106) and would
>>> like to merge it to Lucene 8.10 if it is not too late.
>>> I apologize for the inconvenience, the bug was discovered just
>>> yesterday.
>>>
>>> On Tue, Sep 14, 2021 at 9:26 PM Timothy Potter 
>>> wrote:
>>>
>>>> Ahem ... unfortunately there will not be an 8.10 RC this week. I'm
>>>> headed out on vacation tomorrow, back at keys on Monday, Sept 20
>>>> unless someone else wants to pick up the RM duties before then?
>>>>
>>>> After failing the test suite at various places and other weirdness
>>>> like .asc files not getting created, I finally got to the smoke test
>>>> part, which is now failing with:
>>>>
>>>>   File
>>>> "/Users/tjp/.lucene-releases/8.10.0/lucene-solr/dev-tools/scripts/smokeTestRelease.py",
>>>> line 176, in checkJARMetaData
>>>> raise RuntimeError('%s is missing "%s" inside its
>>>> META-INF/MANIFEST.MF (wrong git revision?)' % \
>>>> RuntimeError: JAR file
>>>>
>>>> "/Users/tjp/.lucene-releases/8.10.0/RC1/smoketest/unpack/lucene-8.10.0/benchmark/lucene-benchmark-8.10.0.jar"
>>>> is missing "Implementation-Version: 8.10.0
>>>> ecf5c747e6df418dd05a18af327c20051f0584d7" inside its
>>>> META-INF/MANIFEST.MF (wrong git revision?)
>>>>
>>>> FWIW, I verified that the other Lucene JAR files have this line in
>>>> them, such as core:
>>>>
>>>> Manifest-Version: 1.0
>>>> Ant-Version: Apache Ant 1.9.15
>>>> Created-By: 1.8.0_265-b01 (AppleJDK-8.0.265.1.1)
>>>> Extension-Name: org.apache.lucene
>>>> Specification-Title: Lucene Search Engine: core
>>>> Specification-Version: 8.10.0
>>>> Specification-Vendor: The Apache Software Foundation
>>>> Implementation-Title: org.apache.lucene
>>>> Implementation-Version: 8.10.0 ecf5c747e6df418dd05a18af327c20051f0584d
>>>>  7 - tjp - 2021-09-14 19:08:42
>>>> Implementation-Vendor: The Apache Software Foundation
>>>> X-Compile-Source-JDK: 8
>>>> X-Compile-Target-JDK: 8
>>>> Multi-Release: true
>>>>
>>>> On Tue, Sep 14, 2021 at 1:21 PM Ishan Chattopadhyaya
>>>>  wrote:
>>>> >
>>>> > All the best, this is the worst step.
>>>> >
>>>> > On Tue, 14 Sep, 2021, 10:47 pm Timothy Potter, 
>>>> wrote:
>>>> >>
>>>> >> Building RC1 now ... stay tuned.
>>>> >>
>>>> >> On Thu, Sep 9, 2021 at 2:30 PM Timothy Potter 
>>>> wrote:
>>>> >> >
>>>> >> > 

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

I think since Tim is out on vacation, it's probably not too late. That
looks like a good fix to have, do we know how long the bug has been present?

On Wed, Sep 15, 2021 at 7:56 AM Mayya Sharipova
 wrote:

> Hello everyone,
> We have discovered a bug and fixed a bug in Lucene sort optimization
> <https://github.com/apache/lucene/pull/300> (LUCENE-10106) and would like
> to merge it to Lucene 8.10 if it is not too late.
> I apologize for the inconvenience, the bug was discovered just yesterday.
>
> On Tue, Sep 14, 2021 at 9:26 PM Timothy Potter 
> wrote:
>
>> Ahem ... unfortunately there will not be an 8.10 RC this week. I'm
>> headed out on vacation tomorrow, back at keys on Monday, Sept 20
>> unless someone else wants to pick up the RM duties before then?
>>
>> After failing the test suite at various places and other weirdness
>> like .asc files not getting created, I finally got to the smoke test
>> part, which is now failing with:
>>
>>   File
>> "/Users/tjp/.lucene-releases/8.10.0/lucene-solr/dev-tools/scripts/smokeTestRelease.py",
>> line 176, in checkJARMetaData
>> raise RuntimeError('%s is missing "%s" inside its
>> META-INF/MANIFEST.MF (wrong git revision?)' % \
>> RuntimeError: JAR file
>>
>> "/Users/tjp/.lucene-releases/8.10.0/RC1/smoketest/unpack/lucene-8.10.0/benchmark/lucene-benchmark-8.10.0.jar"
>> is missing "Implementation-Version: 8.10.0
>> ecf5c747e6df418dd05a18af327c20051f0584d7" inside its
>> META-INF/MANIFEST.MF (wrong git revision?)
>>
>> FWIW, I verified that the other Lucene JAR files have this line in
>> them, such as core:
>>
>> Manifest-Version: 1.0
>> Ant-Version: Apache Ant 1.9.15
>> Created-By: 1.8.0_265-b01 (AppleJDK-8.0.265.1.1)
>> Extension-Name: org.apache.lucene
>> Specification-Title: Lucene Search Engine: core
>> Specification-Version: 8.10.0
>> Specification-Vendor: The Apache Software Foundation
>> Implementation-Title: org.apache.lucene
>> Implementation-Version: 8.10.0 ecf5c747e6df418dd05a18af327c20051f0584d
>>  7 - tjp - 2021-09-14 19:08:42
>> Implementation-Vendor: The Apache Software Foundation
>> X-Compile-Source-JDK: 8
>> X-Compile-Target-JDK: 8
>> Multi-Release: true
>>
>> On Tue, Sep 14, 2021 at 1:21 PM Ishan Chattopadhyaya
>>  wrote:
>> >
>> > All the best, this is the worst step.
>> >
>> > On Tue, 14 Sep, 2021, 10:47 pm Timothy Potter, 
>> wrote:
>> >>
>> >> Building RC1 now ... stay tuned.
>> >>
>> >> On Thu, Sep 9, 2021 at 2:30 PM Timothy Potter 
>> wrote:
>> >> >
>> >> > Thanks for the update Mike!
>> >> >
>> >> > I'm backporting SOLR-15620 right now and am cooking up a quick PR for
>> >> > SOLR-15621, which looks like an easy win for the issue Cassandra
>> >> > reported on Slack earlier today.
>> >> >
>> >> > Cheers,
>> >> > Tim
>> >> >
>> >> > On Thu, Sep 9, 2021 at 11:32 AM Mike Drob  wrote:
>> >> > >
>> >> > > Hi Tim, I'm still working on SOLR-1, the code and benchmarking
>> >> > > both look pretty good, but I've got a few last unit tests that I
>> need
>> >> > > to chase down. Hopefully taken care of by today or tomorrow, I'll
>> be
>> >> > > sure to keep you updated though.
>> >> > >
>> >> > >
>> >> > > On Thu, Sep 9, 2021 at 11:39 AM Timothy Potter <
>> thelabd...@gmail.com> wrote:
>> >> > > >
>> >> > > > I found https://issues.apache.org/jira/browse/SOLR-15620 while
>> testing
>> >> > > > the schema designer. I haven't built the RC yet, so going to see
>> if I
>> >> > > > can get this in today.
>> >> > > >
>> >> > > > On Tue, Sep 7, 2021 at 12:36 PM Timothy Potter <
>> thelabd...@apache.org> wrote:
>> >> > > > >
>> >> > > > > NOTICE:
>> >> > > > >
>> >> > > > > Branch branch_8_10 has been cut and versions updated to 8.11
>> on stable branch.
>> >> > > > >
>> >> > > > > Please observe the normal rules:
>> >> > > > >
>> >> > > > > * No new features may be committed to the branch.
>> >> > > > >
>> >> > > > > * Document

Java 11/17 Version Matrix

[Mike Drob]

Hi Devs,

What are our thoughts on Java 11 and 17 version compatibility going forward
for Lucene 9? Will we support both? If so, would Java 11 support likely
continue for the entire 9.x release line?

Is there a JIRA tracking this?

Thanks,
Mike

Re: New branch and feature freeze for Lucene/Solr 8.10.0

[Mike Drob]

Hi Tim, I'm still working on SOLR-1, the code and benchmarking
both look pretty good, but I've got a few last unit tests that I need
to chase down. Hopefully taken care of by today or tomorrow, I'll be
sure to keep you updated though.


On Thu, Sep 9, 2021 at 11:39 AM Timothy Potter  wrote:
>
> I found https://issues.apache.org/jira/browse/SOLR-15620 while testing
> the schema designer. I haven't built the RC yet, so going to see if I
> can get this in today.
>
> On Tue, Sep 7, 2021 at 12:36 PM Timothy Potter  wrote:
> >
> > NOTICE:
> >
> > Branch branch_8_10 has been cut and versions updated to 8.11 on stable 
> > branch.
> >
> > Please observe the normal rules:
> >
> > * No new features may be committed to the branch.
> >
> > * Documentation patches, build patches and serious bug fixes may be
> >   committed to the branch. However, you should submit all patches you
> >   want to commit to Jira first to give others the chance to review
> >   and possibly vote against the patch. Keep in mind that it is our
> >   main intention to keep the branch as stable as possible.
> >
> > * All patches that are intended for the branch should first be committed
> >   to the unstable branch, merged into the stable branch, and then into
> >   the current release branch.
> >
> > * Normal unstable and stable branch development may continue as usual.
> >   However, if you plan to commit a big change to the unstable branch
> >   while the branch feature freeze is in effect, think twice: can't the
> >   addition wait a couple more days? Merges of bug fixes into the branch
> >   may become more difficult.
> >
> > * Only Jira issues with Fix version 8.10 and priority "Blocker" will delay
> >   a release candidate build.
> > 
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Gradle HELP

[Mike Drob]

I believe it is one of the built in gradle tasks available to us.
https://docs.gradle.org/current/userguide/dependency_locking.html

On Wed, Aug 18, 2021 at 3:22 PM Michael Sokolov  wrote:
>
> I am trying to get Lucene to build in a very "special" build system
> based on Gradle, and I am stumbling across all kinds of things I don't
> really understand  (which is most of everything about gradle at this
> point). Cookie cutter cargo cultism is getting me pretty far, but one
> thing I don't understand: we have this in the precommit task:
>
> // Root-level validation tasks.
> dependsOn ":verifyLocks"
> dependsOn ":versionsPropsAreSorted"
> dependsOn ":checkWorkingCopyClean"
>
> What is the "verifyLocks" task? I can't seem to find it defined anywhere
>
> -Mike
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 9.0 release

[Mike Drob]

Looks like just LUCENE-9334 remains?

On Wed, Apr 14, 2021 at 10:18 PM Julie Tibshirani  wrote:
>
> Hello everyone! I will pick up LUCENE-9908.
>
>
> I had marked LUCENE-9583 as a blocker but I'm on board with removing its 
> blocker status given we can make changes later. I hope to come back to the 
> issue soon with some ideas.
>
>
> Julie
>
>
> On Wed, Apr 14, 2021 at 12:42 PM Adrien Grand  wrote:
>>
>> I agree that we can remove the blocker status from LUCENE-9583 and take 
>> advantage of the fact that these new APIs are experimental to improve things 
>> later.
>>
>> For the renaming issue, maybe we could just make vectors plural for now for 
>> consistency and revisit other options later.
>>
>> On Wed, Apr 14, 2021 at 8:21 PM Michael Sokolov  wrote:
>>>
>>> Thanks Adrien; I plan to tackle LUCENE-9905.
>>>
>>>  I don't have ideas about how to move forward on LUCENE-9583; I spent
>>> significant amount of time trying various permutations on that API,
>>> and what we have was the best compromise I could find at the time, so
>>> I'm not sure I agree it's a Blocker, yet I'm open to improvements.
>>> Maybe Julie will propose something?
>>>
>>> There is also a vector-related renaming issue Tomoko had opened, which
>>> I thought was marked Blocker, but I guess no longer is. Previously I
>>> had hoped to get some strong consensus, but that proved challenging.
>>> Given that, I'm OK leaving things as-is, marking these apis
>>> @experimental and potentially revisiting naming issues later, eg once
>>> we have a second vector ANN implementation.
>>>
>>> On Wed, Apr 14, 2021 at 11:07 AM Adrien Grand  wrote:
>>> >
>>> > Hi Mike,
>>> >
>>> > Here's what I know about the remaining blockers:
>>> >
>>> > LUCENE-9908 - Move VectorValues#search to VectorReader and LeafReader
>>> > This was discussed on the mailing list and it looks like there was 
>>> > agreement on making that change. If someone has cycles and can take it, 
>>> > please go ahead, otherwise I'll try to allocate some time to it. I'm 
>>> > expecting this change to be rather straightforward.
>>> >
>>> > LUCENE-9905 - Revise approach to specifying NN algorithm
>>> > This is a change to how we've been thinking about configuring the ANN 
>>> > algorithm. I don't know if someone plans to work on it.
>>> >
>>> > LUCENE-9583 - How should we expose VectorValues.RandomAccess
>>> > We'd like to get rid of this sub interface, but I'm not the best person 
>>> > to comment on how much work this needs. Maybe Mike S or Julie can give 
>>> > more info.
>>> >
>>> > LUCENE-9334 - Require consistency between data-structures on a per-field 
>>> > basis
>>> > Mayya has been working on this one and it's very close.
>>> >
>>> > LUCENE-9047 - Directory APIs should be little endian
>>> > Ignacio and Julie have been working on this one and it is close as well.
>>> >
>>> >
>>> > On Tue, Apr 13, 2021 at 10:59 PM Mike Drob  wrote:
>>> >>
>>> >> Michael, did you get a chance to mark the issues you were thinking of as 
>>> >> blockers?
>>> >>
>>> >> Adrien, I see that the remaining open blockers look mostly like your 
>>> >> open issues. Two of them have recent activity, but LUCENE-9047 would 
>>> >> need to be brought back to the lucene repo. Is this an accurate view of 
>>> >> the state of things?
>>> >>
>>> >> Now that I'm done with 8.8.2, I would love to see how we can continue to 
>>> >> make headway on 9.0!
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Mar 29, 2021 at 3:25 PM Michael Sokolov  
>>> >> wrote:
>>> >>>
>>> >>> There has been some discussion around a few code visibility and naming
>>> >>> issues related to "VectorFormat" as it is called today. I'd like to
>>> >>> get that sorted out before 9.0 - I'll hunt up the ticket(s) and mark
>>> >>> as blockers
>>> >>>
>>> >>> On Sun, Mar 28, 2021 at 11:02 AM Adrien Grand  wrote:
>>> >>> >
>>> >>> > Hello Jan,
>>> >>> >
>>> >>> > The list of blockers should be mostly up-to-date: 
>>> >

Re: Welcome Mayya Sharipova to the Lucene PMC

[Mike Drob]

Welcome!

On Mon, Jun 28, 2021 at 7:21 AM Erik Hatcher  wrote:

> Ditto - congrats and a huge welcome!
>
> Erik
>
>
> > On Jun 28, 2021, at 9:16 AM, Robert Muir  wrote:
> >
> > I am pleased to announce that Mayya has accepted an invitation to join
> > the Lucene PMC!
> >
> > Congratulations, and welcome aboard!
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Propose changing the "dist" layout

[Mike Drob]

We can have modules, but why do they need to be in an additional folder
deep? Why not just have langid next to solrj and core? Contrib to me
connotes experimental or unsupported, which these things are decidedly not.

On Fri, Jun 11, 2021 at 2:59 PM David Smiley  wrote:

> The contrib folder is just a folder of modules -- optional plugins for
> solr-core.  IMO we should simply rename "contrib" to "modules".  I think
> the only non-module in there is the prometheus exporter which could move
> out.  Mike, I'm not sure if you have a different notion of what "module"
> is?  I believe most of us would be happy to move away from "contrib"
> wording, anyway.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jun 11, 2021 at 3:03 PM Mike Drob  wrote:
>
>> I think related to this, I would like to see some "contibs" moved out
>> from the contrib folder and into proper modules. Right now the
>> definition of contrib seems to be anything that isn't core or solrj,
>> but maybe there is room for a backup module that has gcs and s3 and
>> hdfs all under it. LangId is already mentioned in our ref guide but we
>> pretend like it is always present and don't think of it as a contrib.
>> We kind of think of contrib as optional extra stuff, so maybe we call
>> the things what they are - plugins and extensions? Then we don't have
>> to think as hard about why certain things are showing up in which lib
>> folders.
>>
>> Also, minor benefit, I would then be able to type c instead of
>> having to type cor to disambiguate from con in my terminal.
>>
>> On Fri, Jun 11, 2021 at 8:09 AM David Smiley  wrote:
>> >
>> > I believe we can do a fair amount of re-organization pertaining to
>> Jetty without losing the Jetty configuration that I think is valuable to
>> users who want to tweak something.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>> >
>> > On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl 
>> wrote:
>> >>
>> >> +1 to a cleanup here for 9.0. As clean and neat organization as
>> possible. Perhaps rename "dist" -> "lib"?
>> >>
>> >> I wish we could get rid of the server (jetty) folder altogether, and
>> move everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/".
>> But that ties into custom boot-class, getting rid of web.xml and building
>> Jetty context in Java code.. I'm willing to help here if others also want
>> to go this direction. This would further hide Jetty as an impl detail and
>> let us organize stuff more freely.
>> >>
>> >> Jan
>> >>
>> >> 11. jun. 2021 kl. 13:29 skrev David Smiley :
>> >>
>> >> Bumping this conversation up, based on recent communication.  I have
>> yet to take action but really any of us can.
>> >>
>> >> ~ David Smiley
>> >> Apache Lucene/Solr Search Developer
>> >> http://www.linkedin.com/in/davidwsmiley
>> >>
>> >>
>> >> On Mon, Nov 23, 2020 at 8:48 AM David Smiley 
>> wrote:
>> >>>
>> >>> I'll proceed on this with lazy consensus.  I suspect most of us don't
>> care, unsurprisingly since I doubt anyone has any fondness for the "dist"
>> folder.
>> >>>
>> >>> ~ David Smiley
>> >>> Apache Lucene/Solr Search Developer
>> >>> http://www.linkedin.com/in/davidwsmiley
>> >>>
>> >>>
>> >>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson <
>> erickerick...@gmail.com> wrote:
>> >>>>
>> >>>> Well, Solr has grown “organically” so some things just _are_, like
>> sunrises and plagues ;)
>> >>>>
>> >>>> On a serious note, AFAIC rearrange as you see fit. I wonder how much
>> of this is left over from the war days? Anything that’s lasted through all
>> the transformations Solr has is bound to need cleaning up betimes.
>> >>>>
>> >>>> How would it relate to splitting Solr off into its own TLP? On the
>> surface, I’d guess the two efforts would be orthogonal, I mention it just
>> in case rearranging the layout would make that task easier or harder...
>> >>>>
>> >>>> > On Nov 15, 2020, at 12:18 AM, David Smiley 
>> wrote:
>> >>>> >
>> >>>&g

Re: Propose changing the "dist" layout

[Mike Drob]

I think related to this, I would like to see some "contibs" moved out
from the contrib folder and into proper modules. Right now the
definition of contrib seems to be anything that isn't core or solrj,
but maybe there is room for a backup module that has gcs and s3 and
hdfs all under it. LangId is already mentioned in our ref guide but we
pretend like it is always present and don't think of it as a contrib.
We kind of think of contrib as optional extra stuff, so maybe we call
the things what they are - plugins and extensions? Then we don't have
to think as hard about why certain things are showing up in which lib
folders.

Also, minor benefit, I would then be able to type c instead of
having to type cor to disambiguate from con in my terminal.

On Fri, Jun 11, 2021 at 8:09 AM David Smiley  wrote:
>
> I believe we can do a fair amount of re-organization pertaining to Jetty 
> without losing the Jetty configuration that I think is valuable to users who 
> want to tweak something.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jun 11, 2021 at 8:01 AM Jan Høydahl  wrote:
>>
>> +1 to a cleanup here for 9.0. As clean and neat organization as possible. 
>> Perhaps rename "dist" -> "lib"?
>>
>> I wish we could get rid of the server (jetty) folder altogether, and move 
>> everything from server/solr-webapp/webapp/WEB-INF/lib to "lib/deps/". But 
>> that ties into custom boot-class, getting rid of web.xml and building Jetty 
>> context in Java code.. I'm willing to help here if others also want to go 
>> this direction. This would further hide Jetty as an impl detail and let us 
>> organize stuff more freely.
>>
>> Jan
>>
>> 11. jun. 2021 kl. 13:29 skrev David Smiley :
>>
>> Bumping this conversation up, based on recent communication.  I have yet to 
>> take action but really any of us can.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Mon, Nov 23, 2020 at 8:48 AM David Smiley  wrote:
>>>
>>> I'll proceed on this with lazy consensus.  I suspect most of us don't care, 
>>> unsurprisingly since I doubt anyone has any fondness for the "dist" folder.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Sun, Nov 15, 2020 at 7:31 AM Erick Erickson  
>>> wrote:

 Well, Solr has grown “organically” so some things just _are_, like 
 sunrises and plagues ;)

 On a serious note, AFAIC rearrange as you see fit. I wonder how much of 
 this is left over from the war days? Anything that’s lasted through all 
 the transformations Solr has is bound to need cleaning up betimes.

 How would it relate to splitting Solr off into its own TLP? On the 
 surface, I’d guess the two efforts would be orthogonal, I mention it just 
 in case rearranging the layout would make that task easier or harder...

 > On Nov 15, 2020, at 12:18 AM, David Smiley  wrote:
 >
 > I've been doing a bit of dependency work in one of our contribs, and 
 > observing more closely than usual exactly what we produce in the 
 > distribution layout (result of gradlew assemble).  There are some tricks 
 > Dawid did in gradle/solr/packaging.gradle to pull off this stunt to keep 
 > things as they have been for many years.  The distribution layout is 
 > awkward, I think.  We produce this "dist" folder at the top level that 
 > has every JAR this project produces, *even contribs*.  But why?  I think 
 > contribs should keep to themselves.  It's ridiculous that /contribs/ltr/ 
 > is empty except for a README.txt... IMO it ought to have the JAR in a 
 > "lib" subdirectory there mixed with its dependencies (LTR has none but 
 > others sure do).  Today, each contrib's JAR is in "/dist".  And what 
 > about SolrJ?  I think SolrJ is important enough that it deserves its 
 > very own top-level directory "solrj", and like the contribs, with a 
 > "lib" alongside it.  Maybe Solrj's optional dependencies could be in a 
 > lib-optional dir next to it or lib/opt/ (beneath it).  Then... we don't 
 > need "dist" at all.  It contains the solr-core JAR but this is 
 > redundant.  Furthermore, the server webapp could be configured to add 
 > the SolrJ libs so that we don't need to redundantly put any of them in 
 > the distribution.  There might be some duplicated jars overall, but not 
 > many.  Logging libs might be explicitly excluded so that they are only 
 > in one spot.  (Logging in Java is a mess)
 >
 > WDYT?
 >
 > ~ David Smiley
 > Apache Lucene/Solr Search Developer
 > http://www.linkedin.com/in/davidwsmiley


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Test Framework - Zookeeper Watch Limits

[Mike Drob]

Oops, sent to the wrong dev list, my apologies!

On Mon, May 17, 2021 at 12:06 PM Mike Drob  wrote:
>
> Hi Devs,
>
> Do folks use the Zookeeper watch limits? What are they useful for
> debugging? Can somebody give me an example of when they have been
> helpful to you?
>
> The main reason I am asking is because I am looking at switching our
> TestZookeeper to use the new 3.7.0 ZooKeeperServerEmbedded feature and
> it would greatly simplify my task if I didn't need to worry about the
> watch limits.
>
> It's believable that they add value, but they also look like the logic
> hasn't been touched since 2014 by the mighty Tim Potter.
>
> Most of the times that I see them come up in the test output, it's
> because something else failed anyway and we had lots of other watches
> not get cleaned up, and the object release tracker went haywire as
> well.
>
> Maybe Mark has some stuff on reference branch that makes these more
> constrained as well?
>
> Hoping somebody knows the historical context and use case for these!
>
> Mike

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Test Framework - Zookeeper Watch Limits

[Mike Drob]

Hi Devs,

Do folks use the Zookeeper watch limits? What are they useful for
debugging? Can somebody give me an example of when they have been
helpful to you?

The main reason I am asking is because I am looking at switching our
TestZookeeper to use the new 3.7.0 ZooKeeperServerEmbedded feature and
it would greatly simplify my task if I didn't need to worry about the
watch limits.

It's believable that they add value, but they also look like the logic
hasn't been touched since 2014 by the mighty Tim Potter.

Most of the times that I see them come up in the test output, it's
because something else failed anyway and we had lots of other watches
not get cleaned up, and the object release tracker went haywire as
well.

Maybe Mark has some stuff on reference branch that makes these more
constrained as well?

Hoping somebody knows the historical context and use case for these!

Mike

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 9.0 release

[Mike Drob]

Michael, did you get a chance to mark the issues you were thinking of as
blockers?

Adrien, I see that the remaining open blockers look mostly like your open
issues. Two of them have recent activity, but LUCENE-9047 would need to be
brought back to the lucene repo. Is this an accurate view of the state of
things?

Now that I'm done with 8.8.2, I would love to see how we can continue to
make headway on 9.0!



On Mon, Mar 29, 2021 at 3:25 PM Michael Sokolov  wrote:

> There has been some discussion around a few code visibility and naming
> issues related to "VectorFormat" as it is called today. I'd like to
> get that sorted out before 9.0 - I'll hunt up the ticket(s) and mark
> as blockers
>
> On Sun, Mar 28, 2021 at 11:02 AM Adrien Grand  wrote:
> >
> > Hello Jan,
> >
> > The list of blockers should be mostly up-to-date:
> https://issues.apache.org/jira/browse/LUCENE-9661?jql=project%3D%22Lucene%20-%20Core%22%20and%20priority%3DBlocker%20and%20fixVersion%3D%22main%20(9.0)%22
> .
> >
> > On Sat, Mar 27, 2021 at 7:21 PM Jan Høydahl 
> wrote:
> >>
> >> Hi,
> >>
> >> Where are we at with the Lucene 9.0 release planning?
> >>
> >> The git split is largely done. Not sure about the build.
> >> Let's update the umbrella issue
> https://issues.apache.org/jira/browse/LUCENE-9375 for known remaining
> cleanup tasks.
> >> The one on that list is releaseWizard, but as Adrien says there are
> also other scripts that need updating.
> >>
> >> Jan
> >>
> >> 13. jan. 2021 kl. 15:10 skrev Adrien Grand :
> >>
> >> +1 to start planning 9.0.
> >>
> >> Since you mentioned the Gradle build, I believe that we still need to
> migrate some of the release tooling from Ant to Gradle, e.g.
> dev-tools/scripts/addBackcompatIndexes.py. These scripts are not easy to
> test without actually doing a release so the 9.0 RM might have some
> debugging to do.
> >>
> >>
> >> On Mon, Dec 28, 2020 at 7:17 PM Michael Sokolov 
> wrote:
> >>>
> >>> Hi everyone, as we head into a new year full of optimism, is it time
> >>> to start discussing the next major release? We released 8.0 on Jun 18,
> >>> 2019, over 18 months ago. Since then we've switched to a gradle-based
> >>> build. We have added vector-valued fields and an HNSW neighbor search
> >>> algorithm for them.  At the same time Solr has been getting a major
> >>> overhaul which should justify a release, I think? IIRC there was talk
> >>> of making 9.0 be the first release of Solr as its own TLP. Is it time
> >>> to start planning for that now?
> >>>
> >>> -Mike
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>>
> >>
> >>
> >> --
> >> Adrien
> >>
> >>
> >
> >
> > --
> > Adrien
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[ANNOUNCE] Apache Lucene 8.8.2 released

[Mike Drob]

The Lucene PMC is pleased to announce the release of Apache Lucene 8.8.2.

Apache Lucene is a high-performance, full-featured text search engine
library written entirely in Java. It is a technology suitable for nearly
any application that requires full-text search, especially cross-platform.

This release contains three bug fixes. The release is available for
immediate download at:

  

### Lucene 8.8.2 Release Highlights:

 * LUCENE-9870: Fix Circle2D intersectsLine t-value (distance) range clamp
 * LUCENE-9744: NPE on a degenerate query in
MinimumShouldMatchIntervalsSource$MinimumMatchesIterator.getSubMatches().
 * LUCENE-9762: DoubleValuesSource.fromQuery (also used by
FunctionScoreQuery.boostByQuery) could throw an exception when the query
implements TwoPhaseIterator and when the score is requested repeatedly

Please read CHANGES.txt for a full list of changes:

  


Note: The Apache Software Foundation uses an extensive mirroring network for
distributing releases. It is possible that the mirror you are using may not
have
replicated the release yet. If that is the case, please try another mirror.
This also applies to Maven access.

[RESULT] [VOTE] Release Lucene/Solr 8.8.2 RC1

[Mike Drob]

It's been >72h since the vote was initiated and the result is:

+1  5 (Mike Drob, Tim Potter, Anshum Gupta, Bruno Roustant, Ignacio Vera)
 0  2 (Uwe Schindler, Jan Høydahl)
-1  0

This vote has PASSED

[VOTE] Release Lucene/Solr 8.8.2 RC1

[Mike Drob]

Please vote for release candidate 1 for Lucene/Solr 8.8.2

The artifacts can be downloaded from:
https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.2-RC1-reva92a05e195b775b30ca410bc0a26e8e79e7b3bfb

You can run the smoke tester directly with this command:

python3 -u dev-tools/scripts/smokeTestRelease.py \
https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.2-RC1-reva92a05e195b775b30ca410bc0a26e8e79e7b3bfb

The vote will be open until 2021-04-12 00:00 UTC. I will tally votes on
Monday morning.

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

Here is my +1

Re: Welcome Peter Gromov as Lucene committer

[Mike Drob]

Welcome!

On Tue, Apr 6, 2021 at 1:06 PM Dawid Weiss  wrote:

> Congratulations and welcome, Peter!
>
> Dawid
>
> On Tue, Apr 6, 2021 at 7:48 PM Robert Muir  wrote:
> >
> > I'm pleased to announce that Peter Gromov has accepted the PMC's
> invitation to become a committer.
> >
> > Peter, the tradition is that new committers introduce themselves with a
> brief bio.
> >
> > Congratulations and welcome!
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Bugfix release Lucene/Solr 8.8.2

[Mike Drob]

I am going to start building the 8.8.2 RCs and running some tests locally
before the vote. Please hold on further commits to branch_8_8, or let me
know that I need to restart the builds if we find something critical in the
next few hours.

On Tue, Mar 30, 2021 at 6:26 AM Alan Woodward  wrote:

> LUCENE-9744 and LUCENE-9762 are both backported.
>
> On 30 Mar 2021, at 09:03, Ignacio Vera  wrote:
>
> Hi Mike,
>
> LUCENE-9870 has been backported, Thanks!
>
> On Mon, Mar 29, 2021 at 10:32 PM Cassandra Targett 
> wrote:
>
>> Oh, I see what you meant! I actually just redid the changes from scratch
>> instead of trying to backport the 9.0 changes as a patch, so no worries on
>> that front. Thanks for thinking of me there though!
>>
>> I just committed some changes for morelikethis.adoc that I decided I’d
>> like to backport into branch_8_8 since we’re re-publishing…it’s docs-only,
>> though.
>> On Mar 29, 2021, 2:25 PM -0500, Mike Drob , wrote:
>>
>> Thanks, Cassandra. When I skimmed the initial changeset, I saw that there
>> were changes to build.gradle - I assumed that there would have to be some
>> changes to build.xml in the 8x branch, but I didn't investigate further on
>> how they would differ. Was just trying to be helpful and give you a heads
>> up about it, but looks like I caused more confusion than help!
>>
>> On Mon, Mar 29, 2021 at 1:31 PM Cassandra Targett 
>> wrote:
>>
>>> I pushed up the URL changes I mentioned.
>>>
>>> Not sure what you mean, Mike, by “gradle->ant translation”? The build
>>> process is different in branch_8x but it’s documented in the
>>> how-to-contribute page in the ref guide that talks about contributing and
>>> building, etc.
>>> On Mar 29, 2021, 12:55 PM -0500, Mike Drob , wrote:
>>>
>>>
>>> Cassandra - yes, I plan to republish the ref guide, your expertise is
>>> absolutely appreciated. It looks like there will also be some gradle->ant
>>> translation. Was this mostly find-and-replace or was there more order to it?
>>>
>>> On Mon, Mar 29, 2021 at 12:32 PM Alan Woodward >> <https://mailto:romseyg...@gmail.com/>> wrote:
>>>
>>> Hi Mike,
>>>
>>> I’d like to back port LUCENE-9744 (which fixes an NPE in intervals
>>> queries) and LUCENE-9762 (which fixes a bug in QueryValuesSource).
>>>
>>> Thanks, Alan
>>>
>>> On 29 Mar 2021, at 09:47, Ignacio Vera >> <https://mailto:iver...@gmail.com/>> wrote:
>>>
>>> I'd like to backport Lucene-9870 which is a bug in distance queries that
>>> causes no matches along indexed lines and polygon edges. This fix
>>> only touches one class tho so very low risk.
>>>
>>> lucene
>>>
>>> On Sat, Mar 27, 2021 at 3:24 PM Mike Drob >> <https://mailto:md...@mdrob.com/>> wrote:
>>>
>>> Ishan,
>>>
>>> Thank you for bringing this up. I’m comfortable delaying an extra week
>>> to accommodate the multitude of holidays (Holi, Passover, others) coming up.
>>>
>>> I will adjust my schedule to start the vote Tuesday, Apr 6.
>>>
>>> Please make sure that all back ports are appropriately marked with
>>> fixVersion in Jira and have corresponding CHANGES entries.
>>>
>>> Mike
>>>
>>> On Fri, Mar 26, 2021 at 11:11 PM Ishan Chattopadhyaya <
>>> ichattopadhy...@gmail.com <https://mailto:ichattopadhy...@gmail.com/>>
>>> wrote:
>>>
>>> Hi Mike,
>>>
>>> I wish to get https://issues.apache.org/jira/browse/SOLR-15288 in, but
>>> will likely be able to wrap up by 2 April or so (on vacation right now due
>>> to the festival of Holi)
>>>
>>> Regards,
>>> Ishan
>>>
>>> On Sat, 27 Mar, 2021, 7:41 am Mike Drob, >> <https://mailto:md...@mdrob.com/>> wrote:
>>>
>>> I am now preparing for a bugfix release from branch branch_8_8
>>>
>>> I plan to have the RC built and vote started on Tuesday, Mar 30. If you
>>> have small, low risk bug fixes to backport before then, please do so using
>>> your best judgement.
>>>
>>> Please observe the normal rules for committing to this branch:
>>>
>>> * Before committing to the branch, reply to this thread and argue
>>>   why the fix needs backporting and how long it will take.
>>> * All issues accepted for backporting should be marked with 8.8.2
>>>   in JIRA, and issues that should delay the release must be marked as
>>> Blocker
>>> * All patches that are intended for the branch should first be committed
>>>   to the unstable branch, merged into the stable branch, and then into
>>>   the current release branch.
>>> * Only Jira issues with Fix version 8.8.2 and priority "Blocker" will
>>> delay
>>>   a release candidate build.
>>>
>>> Thanks,
>>> Mike
>>>
>>>
>>>
>

Re: Bugfix release Lucene/Solr 8.8.2

[Mike Drob]

Thanks, Cassandra. When I skimmed the initial changeset, I saw that there
were changes to build.gradle - I assumed that there would have to be some
changes to build.xml in the 8x branch, but I didn't investigate further on
how they would differ. Was just trying to be helpful and give you a heads
up about it, but looks like I caused more confusion than help!

On Mon, Mar 29, 2021 at 1:31 PM Cassandra Targett 
wrote:

> I pushed up the URL changes I mentioned.
>
> Not sure what you mean, Mike, by “gradle->ant translation”? The build
> process is different in branch_8x but it’s documented in the
> how-to-contribute page in the ref guide that talks about contributing and
> building, etc.
> On Mar 29, 2021, 12:55 PM -0500, Mike Drob , wrote:
>
>
> Cassandra - yes, I plan to republish the ref guide, your expertise is
> absolutely appreciated. It looks like there will also be some gradle->ant
> translation. Was this mostly find-and-replace or was there more order to it?
>
> On Mon, Mar 29, 2021 at 12:32 PM Alan Woodward  <https://mailto:romseyg...@gmail.com>> wrote:
>
> Hi Mike,
>
> I’d like to back port LUCENE-9744 (which fixes an NPE in intervals
> queries) and LUCENE-9762 (which fixes a bug in QueryValuesSource).
>
> Thanks, Alan
>
> On 29 Mar 2021, at 09:47, Ignacio Vera  <https://mailto:iver...@gmail.com>> wrote:
>
> I'd like to backport Lucene-9870 which is a bug in distance queries that
> causes no matches along indexed lines and polygon edges. This fix
> only touches one class tho so very low risk.
>
> lucene
>
> On Sat, Mar 27, 2021 at 3:24 PM Mike Drob  <https://mailto:md...@mdrob.com>> wrote:
>
> Ishan,
>
> Thank you for bringing this up. I’m comfortable delaying an extra week to
> accommodate the multitude of holidays (Holi, Passover, others) coming up.
>
> I will adjust my schedule to start the vote Tuesday, Apr 6.
>
> Please make sure that all back ports are appropriately marked with
> fixVersion in Jira and have corresponding CHANGES entries.
>
> Mike
>
> On Fri, Mar 26, 2021 at 11:11 PM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com <https://mailto:ichattopadhy...@gmail.com>>
> wrote:
>
> Hi Mike,
>
> I wish to get https://issues.apache.org/jira/browse/SOLR-15288 in, but
> will likely be able to wrap up by 2 April or so (on vacation right now due
> to the festival of Holi)
>
> Regards,
> Ishan
>
> On Sat, 27 Mar, 2021, 7:41 am Mike Drob,  <https://mailto:md...@mdrob.com>> wrote:
>
> I am now preparing for a bugfix release from branch branch_8_8
>
> I plan to have the RC built and vote started on Tuesday, Mar 30. If you
> have small, low risk bug fixes to backport before then, please do so using
> your best judgement.
>
> Please observe the normal rules for committing to this branch:
>
> * Before committing to the branch, reply to this thread and argue
>   why the fix needs backporting and how long it will take.
> * All issues accepted for backporting should be marked with 8.8.2
>   in JIRA, and issues that should delay the release must be marked as
> Blocker
> * All patches that are intended for the branch should first be committed
>   to the unstable branch, merged into the stable branch, and then into
>   the current release branch.
> * Only Jira issues with Fix version 8.8.2 and priority "Blocker" will delay
>   a release candidate build.
>
> Thanks,
> Mike
>
>
>

Re: Bugfix release Lucene/Solr 8.8.2

[Mike Drob]

I feel kind of weird about this one, is there a slightly less minimal patch
that is still reasonably safe and would let us continue to avoid
problematic terminology?

On Mon, Mar 29, 2021 at 1:18 PM Tomás Fernández Löbbe 
wrote:

> Mike,
> I'd like to backport SOLR-15216 (UI Fix)
>
> On Mon, Mar 29, 2021 at 10:55 AM Mike Drob  wrote:
>
>> Ignacio, Alan - I have looked at the patches and these should be safe to
>> backport and useful in a bugfix. Please go ahead and commit, and update
>> CHANGES entry as well.
>>
>> Cassandra - yes, I plan to republish the ref guide, your expertise is
>> absolutely appreciated. It looks like there will also be some gradle->ant
>> translation. Was this mostly find-and-replace or was there more order to it?
>>
>> On Mon, Mar 29, 2021 at 12:32 PM Alan Woodward 
>> wrote:
>>
>>> Hi Mike,
>>>
>>> I’d like to back port LUCENE-9744 (which fixes an NPE in intervals
>>> queries) and LUCENE-9762 (which fixes a bug in QueryValuesSource).
>>>
>>> Thanks, Alan
>>>
>>> On 29 Mar 2021, at 09:47, Ignacio Vera  wrote:
>>>
>>> I'd like to backport Lucene-9870 which is a bug in distance queries that
>>> causes no matches along indexed lines and polygon edges. This fix
>>> only touches one class tho so very low risk.
>>> lucene
>>>
>>> On Sat, Mar 27, 2021 at 3:24 PM Mike Drob  wrote:
>>>
>>>> Ishan,
>>>>
>>>> Thank you for bringing this up. I’m comfortable delaying an extra week
>>>> to accommodate the multitude of holidays (Holi, Passover, others) coming 
>>>> up.
>>>>
>>>> I will adjust my schedule to start the vote Tuesday, Apr 6.
>>>>
>>>> Please make sure that all back ports are appropriately marked with
>>>> fixVersion in Jira and have corresponding CHANGES entries.
>>>>
>>>> Mike
>>>>
>>>> On Fri, Mar 26, 2021 at 11:11 PM Ishan Chattopadhyaya <
>>>> ichattopadhy...@gmail.com> wrote:
>>>>
>>>>> Hi Mike,
>>>>>
>>>>> I wish to get https://issues.apache.org/jira/browse/SOLR-15288 in,
>>>>> but will likely be able to wrap up by 2 April or so (on vacation right now
>>>>> due to the festival of Holi)
>>>>>
>>>>> Regards,
>>>>> Ishan
>>>>>
>>>>> On Sat, 27 Mar, 2021, 7:41 am Mike Drob,  wrote:
>>>>>
>>>>>> I am now preparing for a bugfix release from branch branch_8_8
>>>>>>
>>>>>> I plan to have the RC built and vote started on Tuesday, Mar 30. If
>>>>>> you have small, low risk bug fixes to backport before then, please do so
>>>>>> using your best judgement.
>>>>>>
>>>>>> Please observe the normal rules for committing to this branch:
>>>>>>
>>>>>> * Before committing to the branch, reply to this thread and argue
>>>>>>   why the fix needs backporting and how long it will take.
>>>>>> * All issues accepted for backporting should be marked with 8.8.2
>>>>>>   in JIRA, and issues that should delay the release must be marked as
>>>>>> Blocker
>>>>>> * All patches that are intended for the branch should first be
>>>>>> committed
>>>>>>   to the unstable branch, merged into the stable branch, and then into
>>>>>>   the current release branch.
>>>>>> * Only Jira issues with Fix version 8.8.2 and priority "Blocker" will
>>>>>> delay
>>>>>>   a release candidate build.
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>
>>>

Re: Bugfix release Lucene/Solr 8.8.2

[Mike Drob]

Ignacio, Alan - I have looked at the patches and these should be safe to
backport and useful in a bugfix. Please go ahead and commit, and update
CHANGES entry as well.

Cassandra - yes, I plan to republish the ref guide, your expertise is
absolutely appreciated. It looks like there will also be some gradle->ant
translation. Was this mostly find-and-replace or was there more order to it?

On Mon, Mar 29, 2021 at 12:32 PM Alan Woodward  wrote:

> Hi Mike,
>
> I’d like to back port LUCENE-9744 (which fixes an NPE in intervals
> queries) and LUCENE-9762 (which fixes a bug in QueryValuesSource).
>
> Thanks, Alan
>
> On 29 Mar 2021, at 09:47, Ignacio Vera  wrote:
>
> I'd like to backport Lucene-9870 which is a bug in distance queries that
> causes no matches along indexed lines and polygon edges. This fix
> only touches one class tho so very low risk.
> lucene
>
> On Sat, Mar 27, 2021 at 3:24 PM Mike Drob  wrote:
>
>> Ishan,
>>
>> Thank you for bringing this up. I’m comfortable delaying an extra week to
>> accommodate the multitude of holidays (Holi, Passover, others) coming up.
>>
>> I will adjust my schedule to start the vote Tuesday, Apr 6.
>>
>> Please make sure that all back ports are appropriately marked with
>> fixVersion in Jira and have corresponding CHANGES entries.
>>
>> Mike
>>
>> On Fri, Mar 26, 2021 at 11:11 PM Ishan Chattopadhyaya <
>> ichattopadhy...@gmail.com> wrote:
>>
>>> Hi Mike,
>>>
>>> I wish to get https://issues.apache.org/jira/browse/SOLR-15288 in, but
>>> will likely be able to wrap up by 2 April or so (on vacation right now due
>>> to the festival of Holi)
>>>
>>> Regards,
>>> Ishan
>>>
>>> On Sat, 27 Mar, 2021, 7:41 am Mike Drob,  wrote:
>>>
>>>> I am now preparing for a bugfix release from branch branch_8_8
>>>>
>>>> I plan to have the RC built and vote started on Tuesday, Mar 30. If you
>>>> have small, low risk bug fixes to backport before then, please do so using
>>>> your best judgement.
>>>>
>>>> Please observe the normal rules for committing to this branch:
>>>>
>>>> * Before committing to the branch, reply to this thread and argue
>>>>   why the fix needs backporting and how long it will take.
>>>> * All issues accepted for backporting should be marked with 8.8.2
>>>>   in JIRA, and issues that should delay the release must be marked as
>>>> Blocker
>>>> * All patches that are intended for the branch should first be committed
>>>>   to the unstable branch, merged into the stable branch, and then into
>>>>   the current release branch.
>>>> * Only Jira issues with Fix version 8.8.2 and priority "Blocker" will
>>>> delay
>>>>   a release candidate build.
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>
>

Re: Bugfix release Lucene/Solr 8.8.2

[Mike Drob]

Ishan,

Thank you for bringing this up. I’m comfortable delaying an extra week to
accommodate the multitude of holidays (Holi, Passover, others) coming up.

I will adjust my schedule to start the vote Tuesday, Apr 6.

Please make sure that all back ports are appropriately marked with
fixVersion in Jira and have corresponding CHANGES entries.

Mike

On Fri, Mar 26, 2021 at 11:11 PM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Hi Mike,
>
> I wish to get https://issues.apache.org/jira/browse/SOLR-15288 in, but
> will likely be able to wrap up by 2 April or so (on vacation right now due
> to the festival of Holi)
>
> Regards,
> Ishan
>
> On Sat, 27 Mar, 2021, 7:41 am Mike Drob,  wrote:
>
>> I am now preparing for a bugfix release from branch branch_8_8
>>
>> I plan to have the RC built and vote started on Tuesday, Mar 30. If you
>> have small, low risk bug fixes to backport before then, please do so using
>> your best judgement.
>>
>> Please observe the normal rules for committing to this branch:
>>
>> * Before committing to the branch, reply to this thread and argue
>>   why the fix needs backporting and how long it will take.
>> * All issues accepted for backporting should be marked with 8.8.2
>>   in JIRA, and issues that should delay the release must be marked as
>> Blocker
>> * All patches that are intended for the branch should first be committed
>>   to the unstable branch, merged into the stable branch, and then into
>>   the current release branch.
>> * Only Jira issues with Fix version 8.8.2 and priority "Blocker" will
>> delay
>>   a release candidate build.
>>
>> Thanks,
>> Mike
>>
>

Bugfix release Lucene/Solr 8.8.2

[Mike Drob]

I am now preparing for a bugfix release from branch branch_8_8

I plan to have the RC built and vote started on Tuesday, Mar 30. If you
have small, low risk bug fixes to backport before then, please do so using
your best judgement.

Please observe the normal rules for committing to this branch:

* Before committing to the branch, reply to this thread and argue
  why the fix needs backporting and how long it will take.
* All issues accepted for backporting should be marked with 8.8.2
  in JIRA, and issues that should delay the release must be marked as
Blocker
* All patches that are intended for the branch should first be committed
  to the unstable branch, merged into the stable branch, and then into
  the current release branch.
* Only Jira issues with Fix version 8.8.2 and priority "Blocker" will delay
  a release candidate build.

Thanks,
Mike

Welcome Bruno to the Apache Lucene PMC

[Mike Drob]

I am pleased to announce that Bruno has accepted an invitation to join the
Lucene PMC!

Congratulations, and welcome aboard!

Mike

Re: Repository fork (master) about to happen (Wednesday)

[Mike Drob]

Can we provide a sequence of git commands for folks to run? Or will the
official guidance be to create new local clones of each repo?

On Mon, Mar 8, 2021 at 12:18 PM David Smiley  wrote:

> Yeah, I agree with Jan -- don't rename the GitHub repo.  It's going to be
> painful no matter what and a rename doesn't seem appropriate.
>
> I am curious as to the status of /solr code being buildable without
> /lucene.  The steps above at #2 say for each project to remove the other
> side.  Is Solr ready?  Where will Solr get the Lucene binaries?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Mar 8, 2021 at 12:53 PM Jan Høydahl  wrote:
>
>> Hi,
>>
>> There are 324 open PRs. Some numbers:
>> Number updated last month: 40
>> Number not touched last 4 months: 249 (77%)
>> With LUCENE in title: 93
>> With SOLR in title: 181
>>
>> It would be nice to auto migrate but some times you just have to face
>> changes and do some extra work :)
>> Given how easy it is to create a new PR in the new repo based on the
>> existing PR branch, I say we just clearly document how to do it, and let
>> the ~50 PRs that are actually being worked on be re-created. PR author can
>> add a link to the old one to reference review comments that cannot be
>> carried over.
>>
>> It would be misleading to just rename to either solr or lucene. Much
>> better to leave old repo there with a README notice that people need to
>> clone the new repo(s) or update their remotes.
>>
>> Jan
>>
>>
>> > 8. mar. 2021 kl. 17:21 skrev Uwe Schindler :
>> >
>> > Hi again,
>> >
>> > we can maybe "improve" the situation a bit: On Github you can (with
>> Admin/Ownership rights) rename a project. So my suggestion:
>> >
>> > - Check pull requests and count how many affect solr and how many
>> affect Lucene.
>> > - In cooperation with Infra rename the Github project
>> ("apache/lucene-solr.git") to "apache/lucene.git" (if more pull requests
>> affect Lucene) or "apache/solr.git" (if more are Solr). The PRs will
>> survive the rename. Also the old GitHub URL will redirect to the renamed
>> one. The other project should be created as a fork - of course without PRs.
>> >
>> > We can only do this in cooperation with Apache Infra stuff, because we
>> can' change the Github repo settings or rename them using the Github UI.
>> >
>> > Uwe
>> >
>> > -
>> > Uwe Schindler
>> > Achterdiek 19, D-28357 Bremen
>> > https://www.thetaphi.de
>> > eMail: u...@thetaphi.de
>> >
>> >> -Original Message-
>> >> From: Uwe Schindler 
>> >> Sent: Monday, March 8, 2021 5:16 PM
>> >> To: dev@lucene.apache.org
>> >> Subject: RE: Repository fork (master) about to happen (Wednesday)
>> >>
>> >> I think the problem was what happens with the PR in Githubs user
>> interface.
>> >>
>> >> This question was asked many times, answer is simple: NO you can't
>> move over
>> >> Pull requests to different repositories on Github! It's also
>> impossible to export
>> >> and reimport them. People have to recreate them.
>> >>
>> >> Dawid is correct: You can merge the pull request also into another
>> rlocal
>> >> repository, but this is impossible with the Github UI. So basically,
>> you have to
>> >> read the email that comes in with the Pull Request that lists the link
>> to the
>> >> branch and patch. Then use git command line (or Tortoise) and
>> copypaste the
>> >> URL there as "source branch" for the merge. Then you execute the merge,
>> >> squash and commit/push.
>> >>
>> >> Uwe
>> >>
>> >> -
>> >> Uwe Schindler
>> >> Achterdiek 19, D-28357 Bremen
>> >> https://www.thetaphi.de
>> >> eMail: u...@thetaphi.de
>> >>
>> >>> -Original Message-
>> >>> From: Dawid Weiss 
>> >>> Sent: Monday, March 8, 2021 3:25 PM
>> >>> To: Lucene Dev 
>> >>> Subject: Re: Repository fork (master) about to happen (Wednesday)
>> >>>
>>  What happens to open PRs?
>> >>>
>> >>> A pull request on github is essentially a diff between two commits.
>> >>> Existing PRs have to be placed on top of the new development branch.
>> >>> Note these repositories are (initially) identical with lucene-solr so
>> >>> if somebody clones the solr repo and the solr-lucene repo, they can
>> >>> create the same PR over at the new project (pointing at the new main
>> >>> branch as a reference).
>> >>>
>> >>> Dawid
>> >>>
>> >>> -
>> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >>> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>>
>> 

Re: Separate Solr build: help with the remaining last mile needed.

[Mike Drob]

I created a PR for the tests https://github.com/apache/lucene-solr/pull/2462

On Fri, Mar 5, 2021 at 1:43 PM Dawid Weiss  wrote:

> This can be done later too, it's not crucial - these files will be
> just ignored for compilation immediately after the split.
>
> D.
>
> On Fri, Mar 5, 2021 at 8:11 PM Mike Drob  wrote:
> >
> > I think I can make TestICUCollationField work without depending on
> Lucene test sources. The other two will be a bit trickier.
> >
> > On Fri, Mar 5, 2021 at 1:56 AM Dawid Weiss 
> wrote:
> >>
> >> > ./gradlew -Dskip.lucene=true assemble check -x test -x documentation
> >> > -x checkBrokenLinks -x checkLocalJavadocLinksSite
> >>
> >> I made Solr documentation compile last night. Some of the generated
> >> links point at void (and thus the broken links checker fails) but I
> >> think it's enough to make the repository split and then clean up
> >> what's needed in each corresponding repository.
> >>
> >> So... this Sunday? Prior to that, we'd have to disable CI build
> >> services currently pointing at the master branch so that they don't
> >> start failing (I'll remove all content from master).
> >>
> >> Dawid
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Separate Solr build: help with the remaining last mile needed.

[Mike Drob]

I think I can make TestICUCollationField work without depending on Lucene
test sources. The other two will be a bit trickier.

On Fri, Mar 5, 2021 at 1:56 AM Dawid Weiss  wrote:

> > ./gradlew -Dskip.lucene=true assemble check -x test -x documentation
> > -x checkBrokenLinks -x checkLocalJavadocLinksSite
>
> I made Solr documentation compile last night. Some of the generated
> links point at void (and thus the broken links checker fails) but I
> think it's enough to make the repository split and then clean up
> what's needed in each corresponding repository.
>
> So... this Sunday? Prior to that, we'd have to disable CI build
> services currently pointing at the master branch so that they don't
> start failing (I'll remove all content from master).
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Review request - New Solr website

[Mike Drob]

I noticed that the Ref Guide is missing from the new site. Is this
something that we will have to figure out how to restore after the site
goes live?

On Mon, Mar 1, 2021 at 2:56 AM Jan Høydahl  wrote:

> Hi,
>
> I have been working on https://issues.apache.org/jira/browse/SOLR-14499
> to prepare the separate website for Solr.
> I believe the work is practically done, and would like a broader review
> before I actually publish the changes.
>
> The staging site which will eventually be solr.apache.org is at
> https://lucene-solrtlp.staged.apache.org/
> The staging site which shows the lucene site without Solr is at
> https://lucene-new.staged.apache.org/
>
> Any feedback is welcome, here or in the JIRA issue. I intend to publish
> the new sites in a couple of days.
>
> Jan
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Proposal for the Lucene Dependency after git repo split

[Mike Drob]

The part of this process that I really don't like is that Solr then still
becomes beholden to Lucene's release schedule. We don't know how long step
7 takes, and will be effectively blocked from making our own releases until
that happens.

On Fri, Feb 26, 2021 at 8:51 AM Jan Høydahl  wrote:

> The developer workflow for adding something to both Lucene and Solr would
> be as any other dependency, right?
> So say we are on Luene 9.0. This is the process in my head:
>
>1. Adapt Lucene as needed, and "mvn install" lucene-9.1-SNAPSHOT to
>your local laptop (whatever command that is in gradle)
>2. Make your Solr feature branch depend on lucene-9.1-SNAPSHOT instead
>of lucene-9.0.0 -hopefully Gradle will pick the local version over Apache
>Nexus version
>3. Iterate 1-2 until happy
>4. Make a Lucene PR and eventually commit the Lucene change
>5. After next Jenkins build the feature is in Apache Nexus snapshot as
>lucene-9.1-SNAPSHOT
>6. Now the Solr Pull Request will compile and can be tested by others
>7. Wait until Lucene 9.1 release
>8. Upgrade Solr's lucene dependency on 'main'
>9. Merge Solr PR
>10. Backporting will be a similar process, i.e. first backport and
>release in Lucene, then backport in Sol
>
> Hmm, as I wrote this list I can understand why so many features were added
> only to Solr and not to Lucene in the early days :)
>
> Jan
>
> 26. feb. 2021 kl. 14:22 skrev Gus Heck :
>
> Except I just finished helping a contributor with a feature that touches
> both and I know for a fact  that it was developed for his customer who was
> using solr (payload inequalities)... and have another in the works (the
> AQP) Not being able to enhance lucene to support a feature in solr is
> an issue IMHO.
>
> On Thu, Feb 25, 2021 at 6:05 PM Mike Drob  wrote:
>
>> It is possible to publish snapshots into the Apache Nexus repository.
>> That said, I think it is a bad idea for Solr to depend on Lucene snapshots
>> because that constrains the ability to do releases. Either you have to wait
>> for a Lucene release and then you can cut over, or you have to figure out
>> what changes you need to roll back.
>>
>> Features today rarely touch both fronts anyway, they usually land in
>> Lucene first and then percolate into Solr. For an easy example, we can see
>> how WAND was developed recently.
>>
>> On Thu, Feb 25, 2021 at 5:02 PM Houston Putman 
>> wrote:
>>
>>> Once the projects are on separate release cadences there wont be an
>>> ability to “add on both fronts” anymore. You will have to add to lucene,
>>> wait for a release, then add to Solr once Solr upgrades its lucene
>>> dependency to that new version. I dont imagine that we are going to keep
>>> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
>>> perpetuity. After it becomes possible (when lucene 9.0 is released) we
>>> should only be using released lucene versions as dependencies for every
>>> version branch in Solr.
>>>
>>> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck  wrote:
>>>
>>>> Until the first feature that wants to add something on both fronts...
>>>> Is it possible for Lucene to publish nightly snapshots? I know there is
>>>> some level of support for snapshots in central, though I don't know what
>>>> their usage policies are. If that's too restricted is there an artifact
>>>> repo controlled by the ASF that could be used? (An implementation of Apache
>>>> Archiva?) This would have the added benefit of allowing solr to detect when
>>>> Lucene breaks something before its released.
>>>>
>>>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman 
>>>> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> Currently there is discussion going on, in SOLR-14762
>>>>> <https://issues.apache.org/jira/browse/SOLR-14762>, regarding the
>>>>> split of the lucene-solr repo into individual repos for Solr and Lucene.
>>>>> There seems to be agreement that we shouldn't wait for a Lucene release to
>>>>> do the split, and instead split now and release whenever that happens.
>>>>>
>>>>> The biggest issue that arises there is that Solr's master branch is
>>>>> obviously based on Lucene's master branch, since they are currently the
>>>>> same. So when the split happens, Solr master will have to depend on Lucene
>>>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>>>&g

Re: Proposal for the Lucene Dependency after git repo split

[Mike Drob]

It is possible to publish snapshots into the Apache Nexus repository. That
said, I think it is a bad idea for Solr to depend on Lucene snapshots
because that constrains the ability to do releases. Either you have to wait
for a Lucene release and then you can cut over, or you have to figure out
what changes you need to roll back.

Features today rarely touch both fronts anyway, they usually land in Lucene
first and then percolate into Solr. For an easy example, we can see how
WAND was developed recently.

On Thu, Feb 25, 2021 at 5:02 PM Houston Putman 
wrote:

> Once the projects are on separate release cadences there wont be an
> ability to “add on both fronts” anymore. You will have to add to lucene,
> wait for a release, then add to Solr once Solr upgrades its lucene
> dependency to that new version. I dont imagine that we are going to keep
> Solr master/main, or even 8x, 9x, etc, depending on Lucene snaphsots in
> perpetuity. After it becomes possible (when lucene 9.0 is released) we
> should only be using released lucene versions as dependencies for every
> version branch in Solr.
>
> On Thu, Feb 25, 2021 at 5:49 PM Gus Heck  wrote:
>
>> Until the first feature that wants to add something on both fronts... Is
>> it possible for Lucene to publish nightly snapshots? I know there is some
>> level of support for snapshots in central, though I don't know what
>> their usage policies are. If that's too restricted is there an artifact
>> repo controlled by the ASF that could be used? (An implementation of Apache
>> Archiva?) This would have the added benefit of allowing solr to detect when
>> Lucene breaks something before its released.
>>
>> On Thu, Feb 25, 2021 at 4:50 PM Houston Putman 
>> wrote:
>>
>>> Hey everyone,
>>>
>>> Currently there is discussion going on, in SOLR-14762
>>> , regarding the split
>>> of the lucene-solr repo into individual repos for Solr and Lucene. There
>>> seems to be agreement that we shouldn't wait for a Lucene release to do the
>>> split, and instead split now and release whenever that happens.
>>>
>>> The biggest issue that arises there is that Solr's master branch is
>>> obviously based on Lucene's master branch, since they are currently the
>>> same. So when the split happens, Solr master will have to depend on Lucene
>>> 9.0-SNAPSHOT. We can have solr merely depend on the lucene snapshot, but
>>> that will result in inconsistent builds, depending on whatever cached
>>> dependencies each dev has locally. Personally, I think that will cause a
>>> bunch of build errors and headaches for everyone trying to maintain Solr.
>>>
>>> There is another option though. We could instead do an *alpha*
>>> "release" of lucene-solr 9.0 right before the repo is split. Therefore Solr
>>> can reliably depend on a stable version of lucene until 9.0 is truly
>>> released. (And lucene can use a stable version of Solr, if it sees a need
>>> for that). There would be no guarantees for using this alpha release, and
>>> we don't have to advertise it at all.
>>>
>>> It's not perfect, but I think it would be preferable to depending on an
>>> ever-changing SNAPSHOT lucene.
>>>
>>> - Houston
>>>
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>

Solr Docker Dependencies Question

[Mike Drob]

In our Dockerfile I see the line:

> apt-get -y install acl dirmngr lsof procps wget netcat gosu tini;

I understand how we use some of these, but not all of them.

acl package provides setfacl, and gosu provides gosu, which look to be only
used in tests? How much would we miss them if they were gone?

dirmngr is for network access by gpg, but I don't see where we use gpg in
the docker image at all.

Tini comes baked with docker engine since 1.25, so maybe we don't need it
and can provide instructions to run with --init? This seems convenient for
users though so maybe we can leave it in.

Where do we use netcat?

lsof is used in the startup scripts, wget is used at a minimum to download
jattach, procps is probably used in the startup scripts somewhere too for
top or ps or something similar to get a pid.

Thanks,
Mike

Re: Gradle: Verifying dependencies / version locks

[Mike Drob]

This feature was added to Gradle 6.2, which wasn't available when we first
did the conversion from ant.

This plugin doesn't do any verification of license and notice files like we
do, so that's one thing that we will still need our custom validation for.

We could potentially move the checksum verification to the plugin, but that
seems like a lot of effort for I'm not sure what the payoff is.

I don't trust the state of signatures in open source repositories to know
if going down that path is worthwhile, but I also suspect not.


Mike

On Mon, Feb 22, 2021 at 3:45 PM David Smiley  wrote:

> I noticed that Gradle has a built-in dependency version locking mechanism
> that is different than the one we are using:
> https://docs.gradle.org/current/userguide/dependency_verification.html
> Dawid (or anyone), why are we using something different?  Is our mechanism
> completely defined ad-hoc in Groovy in gradle/validation/jar-checks.gradle
> or is there some related plugin for this?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

Re: GC cost of creating String resource description on IndexInput clone

[Mike Drob]

Late evaluation sounds like it would definitely be nice, but I worry about
holding on to object instances longer than necessary might lead to memory
leaks. Sounds like a good issue to open on JIRA.

On Fri, Feb 19, 2021 at 3:58 PM Viral Gandhi  wrote:

> Hello everyone,
>
> We recently added Java Flight Recorder (JFR) to our internal benchmarking.
> While looking through top heap allocations from JFR output, we noticed the
> following to be in top 5 contributors to garbage creation.
>
> org.apache.lucene.store.IndexInput#getFullSliceDescription()
>   at org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
>at org.apache.lucene.store.ByteBufferIndexInput#slice()
>at
> org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#slice()
>at org.apache.lucene.store.IndexInput#randomAccessSlice()
>
> It seems that we construct a new String resource description for each
> clone/slice of IndexInput instance. Resource description for a clone/slice
> instance is a String concatenation of resource description from original
> instance and current slice description. Also, clone/slice can happen
> multiple times per query per segment.
>
> Can we avoid upfront String allocation by late-evaluating
> IndexInput.toString() on the cloned instance? One approach could be to
> hold a reference of the base IndexInput instance in the cloned instance.
> Then while evaluating IndexInput.toString() on a cloned instance, we can
> construct a resource description by concatenating base instance's
> toString() with current resource description. My understanding is that
> IndexInput.toString() is primarily used for debugging purposes hence we
> can benefit from the late-evaluation.
>
> With this approach, we are seeing sustainable GC time reduction of ~6% and
> gain of ~1% to red-line QPS (throughput). My intention to start this thread
> is to collect feedback on this approach as well as to discuss any other
> ideas.
>
> Thanks,
> Viral Gandhi
>

Re: Solr ClusterEventProducerTest failures

[Mike Drob]

There’s a ton of discussion on
https://issues.apache.org/jira/browse/SOLR-15122, feel free to annotate it
if you feel that isn’t moving fast enough.

On Tue, Feb 2, 2021 at 8:35 PM David Smiley  wrote:

> This test, ClusterEventProducerTest, has been failing over 20% of the
> time.  Is anyone working on fixing it?  Andrzej?  It should be BadApple'd
> ASAP if it won't be fixed soon.
>
>
> http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cluster.events.ClusterEventProducerTest.testEvents
>
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

Re: [VOTE] Release Lucene/Solr 8.8.0 RC1

[Mike Drob]

-1

I have been able to reproduce the regression that Anshum found, and would
like to change my vote accordingly.

On Thu, Jan 21, 2021 at 4:31 PM Anshum Gupta  wrote:

> https://issues.apache.org/jira/browse/SOLR-15097
>
>
>
> On Thu, Jan 21, 2021 at 2:28 PM Anshum Gupta 
> wrote:
>
>> While testing the artifacts some more, I realized that the cluster graph
>> doesn't show up on the admin UI any more.
>>
>> I started the cluster with -e cloud -noprompt, and the getting started
>> collection was created successfully. The clusterstate, as well as the admin
>> UI confirm that.
>>
>> I also tried creating a collection using the admin API, which was
>> successful, but the graph didn't reflect that either.
>>
>> I've tested this using both Safari and Firefox, and it's clearly not a
>> browser issue.
>>
>> Considering that the users rely on the graph, I think we might need to
>> fix and respin this.
>>
>>
>> On Tue, Jan 19, 2021 at 8:51 AM Ishan Chattopadhyaya <
>> ichattopadhy...@gmail.com> wrote:
>>
>>> Please vote for release candidate 1 for Lucene/Solr 8.8.0
>>>
>>> The artifacts can be downloaded from:
>>>
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.0-RC1-rev737cb9c49b08f6e3964c1e8a80132da3c764e027
>>>
>>> You can run the smoke tester directly with this command:
>>>
>>> python3 -u dev-tools/scripts/smokeTestRelease.py \
>>>
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.0-RC1-rev737cb9c49b08f6e3964c1e8a80132da3c764e027
>>>
>>> The vote will be open for at least 72 hours i.e. until 2021-01-22 17:00
>>> UTC.
>>>
>>> [ ] +1  approve
>>> [ ] +0  no opinion
>>> [ ] -1  disapprove (and reason why)
>>>
>>> Here is my +1
>>> 
>>>
>>
>>
>> --
>> Anshum Gupta
>>
>
>
> --
> Anshum Gupta
>

Re: [VOTE] Release Lucene/Solr 8.8.0 RC1

[Mike Drob]

+1 (binding)

Ran performance tests comparing Solr 8.7 to 8.8, saw significant
improvement with collection creation as reported on
https://issues.apache.org/jira/browse/SOLR-15052

On Thu, Jan 21, 2021 at 2:28 PM Julie Tibshirani 
wrote:

> +1
> SUCCESS! [0:52:56.337991]
>
> Julie
>
> On Thu, Jan 21, 2021 at 12:13 PM Anshum Gupta 
> wrote:
>
>> +1
>> SUCCESS! [0:59:24.721238]
>>
>> Smoke tester passed, tried running a sample app with tons of collections
>> created, random data indexed, and some querying and all looked fine w/
>> SolrJ.
>>
>> The changelog looks good too.
>>
>>
>> On Tue, Jan 19, 2021 at 8:51 AM Ishan Chattopadhyaya <
>> ichattopadhy...@gmail.com> wrote:
>>
>>> Please vote for release candidate 1 for Lucene/Solr 8.8.0
>>>
>>> The artifacts can be downloaded from:
>>>
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.0-RC1-rev737cb9c49b08f6e3964c1e8a80132da3c764e027
>>>
>>> You can run the smoke tester directly with this command:
>>>
>>> python3 -u dev-tools/scripts/smokeTestRelease.py \
>>>
>>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.0-RC1-rev737cb9c49b08f6e3964c1e8a80132da3c764e027
>>>
>>> The vote will be open for at least 72 hours i.e. until 2021-01-22 17:00
>>> UTC.
>>>
>>> [ ] +1  approve
>>> [ ] +0  no opinion
>>> [ ] -1  disapprove (and reason why)
>>>
>>> Here is my +1
>>> 
>>>
>>
>>
>> --
>> Anshum Gupta
>>
>

Re: Lucene CHANGES.txt 8.8 adjustment proposals

[Mike Drob]

Never mind, that was correctly backported to branch_8x, not sure why I
thought that it wasn't. Apologies for the confusion.

If you feel that it needs to be under a different section, I have no
objections to it. Alternatively, we can move that SOLR issue to LUCENE jira
and edit the entry if that's what you have concerns about.

On Thu, Jan 14, 2021 at 11:11 AM Mike Drob  wrote:

> Lucene had Jetty versions listed under its own dependencies, I didn't
> investigate further on how it's used. I suspect it is for lucene
> replication, but I didn't look further.
>
> That actually reminds me that I think I never finished the backport to
> 8.8; I think this change is currently only on master. Let me double check.
>
> On Thu, Jan 14, 2021 at 10:43 AM David Smiley  wrote:
>
>> I'm reviewing the 8.8 section of CHANGES.txt to see what's coming.
>> Perhaps my proposals might come off as nit-picking; if you don't care
>> then you don't have to say so :-). I care about CHANGES.txt being well
>> organized because it helps us all understand the changes better, especially
>> for our users.
>>
>> Should "New Features" include new options/toggles on existing features,
>> leaving such for "Improvements"?  It could helps delineate larger release
>> highlights from incremental improvement features but I really have no
>> strong opinion.  It'd be nice to have some consistency on the matter;
>> otherwise the distinct sections here become arbitrary.  For 8.8, it appears
>> all are on New Features; the two Improvements we have actually strike me as
>> Optimizations...
>>
>> This Improvement issues sounds like Optimization issues:
>> * LUCENE-9455: ExitableTermsEnum should sample timeout and interruption
>>   check before calling next(). (Zach Chen via Bruno Roustant)
>> * LUCENE-9023: GlobalOrdinalsWithScore should not compute occurrences
>> when the
>>   provided min is 1. (Jim Ferenczi)
>>
>> Mike Drob, why did you add this Solr item to Lucene under Other?:
>> * SOLR-14995: Update Jetty to 9.4.34 (Mike Drob)
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>

Re: 2021-01 Lucene/Solr Committer meeting

[Mike Drob]

The notes are available on the confluence page for the meeting, and copied
here for posterity:


9.0 Release Planing

   - Will release Lucene/Solr separately
   - Vector work in Lucene, lots of work in Solr as well burning to be
   released.
   - 8.x releases could continue… (8.9 Solr depending on 8.8 Lucene?)
   - Reminder that there are issues with new minors after a major release
   (8.9 after a 9.0) - somebody to research.
   - Action Item: Label as blockers any issues that MUST be resolved before
   a release.
   - Solr packages need an upgrade strategy (minor-minor, minor-major)
   - May consider decoupling organizational change from code change.
   - Release train keeps getting bigger and bigger?
   - Action item: define critical path to releasing split code.


Solr TLP Creation

   - Should happen before 9.0
   - Ton of interest initially, but activity has since seemed to slow down
   - Just more forward with everybody current on the PMC/Committer list?
   - Several people claim that it “feels wrong” but difficult to articulate
   the actual concerns
   - Potential compromise is to mark folks as emeritus on website, while
   still providing commit bit
  - Auto-opt-in for anybody active in the past X years
  - Smiley volunteered to reach out to all committers to ascertain
  their desired relationship with Solr TLP.
  - Important to have communication for restoring status for people who
  may have been missed.
   - …And then send board resolution
   - Expect all of this to take ~month between vote and next board
   meeting/report.


Solr Sandbox

   - Isolation of components that need a separate release cadence
   - Goal is to work and contribute directly to Apache (some companies have
   issues with personal projects)
   - Apache Commons had a “commons sandbox” with lower barrier to entry and
   lower expectations
   - Solr Extras, Solr Contrib, Solr Commons?
   - First party packages should still live in the repo
   - - example LTR?
   - - example Streaming Expressions that could work with multiple versions
   - Best candidates are contribs that “sit there” once they are done


Docker Image

   - Post 9.0, plan to provide Docker images via solr repo instead of
   docker-solr repo
   - How is it going to be released - official image or apache image or ???
   - Need to understand verification steps for “official” images
   - Can we release at the same time (Release and Docker)
   - Are these releasable according to ASF policy (source v binary)
   - Likely need to be released to ASF org on docker hub.




On Tue, Jan 5, 2021 at 1:15 PM Anshum Gupta  wrote:

> Hi committers,
>
> I'd like to organize a virtual Lucene/Solr committer meeting this month
> with an intention to discuss the plan for 9.0 release, and the subsequent
> creation of the Solr TLP. I've started a confluence page to organize the
> agenda for this -
> https://cwiki.apache.org/confluence/display/LUCENE/2021-01+Committer+Meeting
>
>
> I'll share a link to the "Doodle Poll" to figure out a time that suits
> most of us. You'll be able to find a link to the poll on #lucene-dev and
> #solr-dev channels on the ASF Slack. Please email me to ask for the link if
> you are a committer who isn't on Slack and would like to participate.
>
> For this virtual committer meeting and future ones:
>
>- This is in the spirit of committer meetings co-located with
>conferences.  ASF policy says that no "decisions" can be made in such a
>venue.  We make decisions on this dev list and indirectly via JIRA out in
>the open and with the opportunity for anyone to comment.
>- Who:  Committer-only
>- Video chat with option of audio dial-in.  This time I will use
>Google Hangout but open to using something else.
>- I wouldn't be recording this, but would provide detailed meeting
>notes that I can share with everyone who signs up.
>- Published notes:  I (or someone) will take written meeting notes
>that are ultimately published for anyone to see (not restricted to those
>invited). They will be transmitted to the dev list.
>
>
> --
> Anshum Gupta
>

Re: Lucene CHANGES.txt 8.8 adjustment proposals

[Mike Drob]

Lucene had Jetty versions listed under its own dependencies, I didn't
investigate further on how it's used. I suspect it is for lucene
replication, but I didn't look further.

That actually reminds me that I think I never finished the backport to 8.8;
I think this change is currently only on master. Let me double check.

On Thu, Jan 14, 2021 at 10:43 AM David Smiley  wrote:

> I'm reviewing the 8.8 section of CHANGES.txt to see what's coming.
> Perhaps my proposals might come off as nit-picking; if you don't care then
> you don't have to say so :-). I care about CHANGES.txt being well organized
> because it helps us all understand the changes better, especially for our
> users.
>
> Should "New Features" include new options/toggles on existing features,
> leaving such for "Improvements"?  It could helps delineate larger release
> highlights from incremental improvement features but I really have no
> strong opinion.  It'd be nice to have some consistency on the matter;
> otherwise the distinct sections here become arbitrary.  For 8.8, it appears
> all are on New Features; the two Improvements we have actually strike me as
> Optimizations...
>
> This Improvement issues sounds like Optimization issues:
> * LUCENE-9455: ExitableTermsEnum should sample timeout and interruption
>   check before calling next(). (Zach Chen via Bruno Roustant)
> * LUCENE-9023: GlobalOrdinalsWithScore should not compute occurrences when
> the
>   provided min is 1. (Jim Ferenczi)
>
> Mike Drob, why did you add this Solr item to Lucene under Other?:
> * SOLR-14995: Update Jetty to 9.4.34 (Mike Drob)
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

Re: Requesting a new GH repository for CrossDC modules

[Mike Drob]

I'm seeing valid reasons to prefer one solr sandbox repo, or prefer
multiple many repos for future plugins or integrations. In this specific
case, I think the relevant deciding points are 1) we don't have multiple
things yet, so deciding between a "mono-repo" and a "multi-repo" is not
very consequential 2) we can always rename things later 3) in the absence
of a strong reason otherwise i'll defer to the people doing the work (in
this case, Anshum). We considered sandbox and can always create one in the
future. If Anshum feels that solr-cross-dc is better for now than I'm fine
with that too.

On Sun, Jan 10, 2021 at 5:07 PM David Smiley  wrote:

> (palm-to-face) -- LOL okay sorry.  I'm getting my threads crossed.
>
> A repo which holds multiple independent modules that can work with Solr
> need not release them all at once.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sat, Jan 9, 2021 at 4:48 PM Anshum Gupta 
> wrote:
>
>> David, this is about the Cross DC work that was supposed to be done :-)
>>
>> The independent release cadence is primarily the reason why a new repo
>> makes sense to me in this case.
>>
>> On Sat, Jan 9, 2021 at 1:24 PM David Smiley  wrote:
>>
>>> While I like the idea of a single (Apache!) repo for multiple
>>> packages/plugins, that does not apply to the Solr Operator, which isn't
>>> even in Java.  It's too unique.  So I agree with Anshum & others about
>>> creating an Apache repo for the Solr Operator.
>>>
>>> I think the ship has sailed on the Solr Operator being an Apache project
>>> instead of some committer's pet project.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Fri, Jan 8, 2021 at 4:47 AM Ishan Chattopadhyaya <
>>> ichattopadhy...@gmail.com> wrote:
>>>
>>>> Not necessarily. Most people contribute to Apache Lucene/Solr using
>>>> external repositories (forks) and raise pull requests against Apache owned
>>>> repositories. There's no SGA needed on such occasions.
>>>>
>>>> I see two paths forward from here.
>>>>
>>>> a) Lets setup a single repository for all packages/plugins, say
>>>> lucene-solr-extras or lucene-solr-contribs or lucene-solr-sandbox etc., and
>>>> develop it there.
>>>> b) All development for this effort happens in an external repository (
>>>> https://github.com/apple/solr-dc or https://github.com/anshumg/solr-dc)
>>>> and we raise a PR against Apache owned repository (which can be created if
>>>> needed once we are all onboard).
>>>>
>>>> What does everyone else think?
>>>>
>>>> On Fri, Jan 8, 2021 at 10:23 AM Mike Drob  wrote:
>>>>
>>>>> An external repository probably ends up requiring a software grant? I
>>>>> know there is a material difference between code originating externally 
>>>>> and
>>>>> code originating within the umbrella of the ASF in terms of IP, copyright,
>>>>> or other legal status.
>>>>>
>>>>>
>>>>> On Thu, Jan 7, 2021 at 8:11 PM Ishan Chattopadhyaya <
>>>>> ichattopadhy...@gmail.com> wrote:
>>>>>
>>>>>> If all we need now is a place to commit a PoC for now (and something
>>>>>> like sandbox repo or contribs won't suffice), why can't we have a 
>>>>>> separate
>>>>>> repository in GitHub outside Apache and merge into an Apache repository
>>>>>> only once the code takes reasonable shape?
>>>>>>
>>>>>> On Fri, 8 Jan, 2021, 2:31 am Anshum Gupta, 
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the feedback, Mike.
>>>>>>>
>>>>>>> I like the idea of the sandbox, but that might be restricting when
>>>>>>> we want to work on more than one repos.
>>>>>>>
>>>>>>> I'm not sure if that would happen in the near future, but as we can
>>>>>>> always discard the repo and it doesn't really come at a cost, I don't 
>>>>>>> see a
>>>>>>> problem with having a repo created for this specific reason.
>>>>>>>
>>>>>>> On Thu, Jan 7, 2021 at 12:45 PM Mike Drob  wrote:
>>>>>>>
>>>>>>>

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Mike Drob]

Do the shard metadata files list all of the segments that make up the
backup, or only the segments that were uploaded in this incremental update?

On Fri, Jan 8, 2021 at 3:19 PM Jason Gerlowski 
wrote:

> Sure thing.  I put together a writeup on the file layout and formats
> here:
> https://cwiki.apache.org/confluence/display/SOLR/Incremental+Backup+File+Format
> The details get a little verbose, so I made it a subpage that the
> SIP-proper calls out to.
>
> Let me know what you think when you get a chance to read - hopefully
> that's sufficient to fill the gap.
>
> Jason
>
> On Thu, Jan 7, 2021 at 8:34 PM Tomás Fernández Löbbe
>  wrote:
> >
> > Thanks Jason! This is great, and a very much needed feature.
> >
> > > This helps to avoid confusion that would
> > > otherwise arise between identically named files when e.g. a shard
> > > leader changes between two incremental backups.  (I'll try to expand
> > > on this in the SIP, as it's a bit hard to give the full context here.)
> >
> > Thanks, I was wondering the same thing. Maybe it would be good to put an
> example of how the file structure of a backup looks like in the backup? and
> how the manifest file looks like. As you said, a file with the same name
> may refer to different segments created by different cores or the same one
> (even if the leader changed, it may be a file from a previous replication).
> >
> > On Thu, Jan 7, 2021 at 1:20 PM Jason Gerlowski 
> wrote:
> >>
> >> Thanks for the feedback Mike.  I've gotta give any credit to Shalin
> >> though, he wrote most of it before the holiday.  He and Dat wrote much
> >> of the code involved as well.  I haven't done more than steward things
> >> along so far.  As you suggested, I've updated the SIP to mention the
> >> related SOLR-13608 (see the bottom of the "Motivation" section).
> >>
> >> As for your questions, I've tried to answer them below.
> >>
> >> 1. Good catch - it doesn't. The SIP should read that each backup
> >> creates its own manifest files as needed for directories it creates
> >> under the base "location".  This way, additional backups can be added
> >> to the same location without needing to modify existing metadata
> >> files.  I've updated the SIP to reflect this.
> >>
> >> 2. The proposed metadata file is a lot like segments_n (in spirit) in
> >> that it has pointers to each index file that comprise an
> >> index/replica.  But it differs in that it stores additional
> >> information about each file (checksum, size) separate from the file
> >> itself.  It also allows a layer of naming indirection between what
> >> files are named in the storage repository and what name they should be
> >> given upon restoration.  This helps to avoid confusion that would
> >> otherwise arise between identically named files when e.g. a shard
> >> leader changes between two incremental backups.  (I'll try to expand
> >> on this in the SIP, as it's a bit hard to give the full context here.)
> >>
> >> 3. My intention was that the 'maxNumBackups' parameter would only
> >> refer to the incremental backups in a given location.  This was mostly
> >> informed by the fact that traditional backups today are required to be
> >> 1-per-location.  (i.e. a backup in 8.6.3 will error out if the
> >> specified directory has files in it.).  We could fix that aspect of
> >> traditional backups and find semantics for 'maxNumBackups' that might
> >> include traditional ones, but IMO it'd add complexity and work for a
> >> format that the SIP is trying to replace more broadly anyways.
> >>
> >> 4. I definitely intended to update LocalFileSystemRepository.  I have
> >> code to update HdfsBackupRepository as well, but wasn't quite sure
> >> where that stood since it's currently deprecated.  I haven't seen
> >> plans to make it a plugin, but might've just missed those discussions
> >> in other mail.  Anyway, I plan to update it but that assumes it's
> >> sticking around in one form or another.
> >>
> >> 5. Good idea - I didn't realize that was an option.  But it would be
> >> really nice if possible.  I don't have an estimate on resources.  I
> >> expect the need would be relatively small - you could restrict the
> >> tests to running on the nightly runs on ASF's Jenkins unless devs
> >> provide their own (e.g.) s3 creds.  But that's just a guess obviously,
> >> and not even in concrete terms.
> >>
> >> Thanks again for taki

Re: Requesting a new GH repository for CrossDC modules

[Mike Drob]

An external repository probably ends up requiring a software grant? I know
there is a material difference between code originating externally and code
originating within the umbrella of the ASF in terms of IP, copyright, or
other legal status.


On Thu, Jan 7, 2021 at 8:11 PM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> If all we need now is a place to commit a PoC for now (and something like
> sandbox repo or contribs won't suffice), why can't we have a separate
> repository in GitHub outside Apache and merge into an Apache repository
> only once the code takes reasonable shape?
>
> On Fri, 8 Jan, 2021, 2:31 am Anshum Gupta,  wrote:
>
>> Thanks for the feedback, Mike.
>>
>> I like the idea of the sandbox, but that might be restricting when we
>> want to work on more than one repos.
>>
>> I'm not sure if that would happen in the near future, but as we can
>> always discard the repo and it doesn't really come at a cost, I don't see a
>> problem with having a repo created for this specific reason.
>>
>> On Thu, Jan 7, 2021 at 12:45 PM Mike Drob  wrote:
>>
>>> I'm not sure where I sit on this, going to start typing things and then
>>> hopefully I'll reach a conclusion by the end.
>>>
>>> This definitely needs to be outside of the core solr repo so that it can
>>> be versioned and released independently. And I disagree with Ishan about
>>> the consequence of abandoning the repository - if we realize that it's a
>>> bad direction then we can pivot, but we shouldn't let a fear of the unknown
>>> stop us from doing it in the first place.
>>>
>>> However, if all we need right now is a place to commit code that is WIP,
>>> then what we really want is a sandbox to play with, and not necessarily a
>>> strongly directed repo. Lucene has a sandbox in the main code. We could
>>> similarly start this under Solr contrib and move it out before an actual
>>> release of 9x happens. Or maybe we start with a [lucene-]solr-sandbox
>>> repository that we can throw all sorts of stuff into and then when
>>> components are mature enough they get to graduate into their own repo?
>>>
>>> Mike
>>>
>>> On Thu, Jan 7, 2021 at 2:32 PM Anshum Gupta 
>>> wrote:
>>>
>>>> I understand your concern, but this is the placeholder for where the
>>>> code would be, not what the code would look like.
>>>>
>>>> Considering we agreed to do this in a repository outside of the core, I
>>>> believe this is a good place to start. The idea that the release cadence
>>>> for the cross-dc effort should be different from that of core is an
>>>> argument in favor of this approach, but I'm happy to talk more about it.
>>>> I just thought that based on the original email, folks were on-board
>>>> with the idea of this being outside of core Solr artifact/release.
>>>>
>>>> On Thu, Jan 7, 2021 at 11:06 AM Ishan Chattopadhyaya <
>>>> ichattopadhy...@gmail.com> wrote:
>>>>
>>>>> -1 on this. Without finalizing on the shape of how the solution will
>>>>> look like, I don't think we should start a repository: it would be bad if
>>>>> we have to abandon the repository of our approach changes (say we want to
>>>>> keep it tightly integrated inside Solr).
>>>>>
>>>>> On Thu, 7 Jan, 2021, 11:45 pm Anshum Gupta, 
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Inline with my earlier email, I'll be requesting a new repository to
>>>>>> host the cross-dc work. Please let me know if you have any questions or
>>>>>> concerns.
>>>>>>
>>>>>> *Repository name: *solr-crossdc
>>>>>> *Generated name:* lucene-solr-crossdc.git (that's auto-generated, so
>>>>>> can't remove the TLP prefix)
>>>>>> *Commit notification list:* commits-cros...@lucene.apache.org (I
>>>>>> think it makes sense for these commit notifications to go to a new list,
>>>>>> but I'm open to reusing the old one)
>>>>>> *GitHub notification list:* dev@lucene.apache.org
>>>>>>
>>>>>> I'll be submitting a request for the same later in the day today if
>>>>>> there are no concerns.
>>>>>>
>>>>>> --
>>>>>> Anshum Gupta
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Anshum Gupta
>>>>
>>>
>>
>> --
>> Anshum Gupta
>>
>

Re: Requesting a new GH repository for CrossDC modules

[Mike Drob]

I'm not sure where I sit on this, going to start typing things and then
hopefully I'll reach a conclusion by the end.

This definitely needs to be outside of the core solr repo so that it can be
versioned and released independently. And I disagree with Ishan about the
consequence of abandoning the repository - if we realize that it's a bad
direction then we can pivot, but we shouldn't let a fear of the unknown
stop us from doing it in the first place.

However, if all we need right now is a place to commit code that is WIP,
then what we really want is a sandbox to play with, and not necessarily a
strongly directed repo. Lucene has a sandbox in the main code. We could
similarly start this under Solr contrib and move it out before an actual
release of 9x happens. Or maybe we start with a [lucene-]solr-sandbox
repository that we can throw all sorts of stuff into and then when
components are mature enough they get to graduate into their own repo?

Mike

On Thu, Jan 7, 2021 at 2:32 PM Anshum Gupta  wrote:

> I understand your concern, but this is the placeholder for where the code
> would be, not what the code would look like.
>
> Considering we agreed to do this in a repository outside of the core, I
> believe this is a good place to start. The idea that the release cadence
> for the cross-dc effort should be different from that of core is an
> argument in favor of this approach, but I'm happy to talk more about it.
> I just thought that based on the original email, folks were on-board with
> the idea of this being outside of core Solr artifact/release.
>
> On Thu, Jan 7, 2021 at 11:06 AM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
>
>> -1 on this. Without finalizing on the shape of how the solution will look
>> like, I don't think we should start a repository: it would be bad if we
>> have to abandon the repository of our approach changes (say we want to keep
>> it tightly integrated inside Solr).
>>
>> On Thu, 7 Jan, 2021, 11:45 pm Anshum Gupta, 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Inline with my earlier email, I'll be requesting a new repository to
>>> host the cross-dc work. Please let me know if you have any questions or
>>> concerns.
>>>
>>> *Repository name: *solr-crossdc
>>> *Generated name:* lucene-solr-crossdc.git (that's auto-generated, so
>>> can't remove the TLP prefix)
>>> *Commit notification list:* commits-cros...@lucene.apache.org (I think
>>> it makes sense for these commit notifications to go to a new list, but I'm
>>> open to reusing the old one)
>>> *GitHub notification list:* dev@lucene.apache.org
>>>
>>> I'll be submitting a request for the same later in the day today if
>>> there are no concerns.
>>>
>>> --
>>> Anshum Gupta
>>>
>>
>
> --
> Anshum Gupta
>

Re: 2021-01 Lucene/Solr Committer meeting

[Mike Drob]

I would like to explicitly propose that we do not attempt to bring up the
Solr reference impl branch, since I'm reasonably sure that will take up a
full hour by itself. I chatted with Mark about it this morning, and he said
he's working on a wiki page and then we can take it from there (next
meeting, next month?)

On Tue, Jan 5, 2021 at 4:31 PM David Smiley  wrote:

> Thanks so much for organizing this Anshum!  We are much overdue.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Jan 5, 2021 at 4:17 PM Anshum Gupta 
> wrote:
>
>> Hi committers,
>>
>> I'd like to organize a virtual Lucene/Solr committer meeting this month
>> with an intention to discuss the plan for 9.0 release, and the subsequent
>> creation of the Solr TLP. I've started a confluence page to organize the
>> agenda for this -
>> https://cwiki.apache.org/confluence/display/LUCENE/2021-01+Committer+Meeting
>>
>>
>> I'll share a link to the "Doodle Poll" to figure out a time that suits
>> most of us. You'll be able to find a link to the poll on #lucene-dev and
>> #solr-dev channels on the ASF Slack. Please email me to ask for the link if
>> you are a committer who isn't on Slack and would like to participate.
>>
>> For this virtual committer meeting and future ones:
>>
>>- This is in the spirit of committer meetings co-located with
>>conferences.  ASF policy says that no "decisions" can be made in such a
>>venue.  We make decisions on this dev list and indirectly via JIRA out in
>>the open and with the opportunity for anyone to comment.
>>- Who:  Committer-only
>>- Video chat with option of audio dial-in.  This time I will use
>>Google Hangout but open to using something else.
>>- I wouldn't be recording this, but would provide detailed meeting
>>notes that I can share with everyone who signs up.
>>- Published notes:  I (or someone) will take written meeting notes
>>that are ultimately published for anyone to see (not restricted to those
>>invited). They will be transmitted to the dev list.
>>
>>
>> --
>> Anshum Gupta
>>
>

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Mike Drob]

This is a very thorough SIP, thank you for spending the time on it, Jason!

I have a few minor questions about points that are unclear to me.

1) If we assume that we cannot overwrite files, how does the manifest file
stay current for incremental backup operations to the same directory?
2) How is the manifest file functionally different from the segments_n and
segments.gen files?
3) Does the maxNumBackups parameter consider incremental backups or only
full backups? What happens if we have a full backup and then N incremental
ones? Do we delete the full backup and convert the oldest incremental one
into a full? I imagine this might be a metadata operation, but then the
concerns from question 1 apply.
4) Do we plan to retrofit HDFS Backup and Local File Backup to use the new
interfaces? I believe we should, but may be willing to accept this as out
of scope.
5) Regarding cloud provider test resources, we can also approach the ASF
Infra team to ask for cloud credits. Can you give rough estimates on what
kind of resourcing would be needed?

I did not examine the new APIs in detail, but they looked fine at a high
level overview. Will probably look again after questions regarding v1/v2
are figured out.

On Tue, Jan 5, 2021 at 10:11 AM Mike Drob  wrote:

> Can you explicitly call out in the SIP how it relates to the work done in
> SOLR-13608?
>
> On Tue, Jan 5, 2021 at 8:55 AM Jason Gerlowski 
> wrote:
>
>> Hey, Happy New Year everybody.
>>
>> Some SIP updates based on the discussion above:
>>
>> I added v2 examples for each API to the SIP.  Feedback welcome,
>> especially on the v2 APIs that are net-new to this proposal (namely:
>> "list backups" and "delete backup").
>>
>> I've also amended the backcompat/migration section to mention Jan's
>> suggestion that the "incremental" features be exposed in the v2 API
>> only.  Though it's unclear to me whether that's still something people
>> want since it turns out that we'll still have backcompat concerns with
>> the existing v2 backup/restore APIs.  So I've held off from
>> removing/replacing the original plan.
>>
>> Link for convenience:
>>
>> https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore
>>
>> Best,
>>
>> Jason
>>
>>
>> On Thu, Dec 24, 2020 at 8:11 AM Jan Høydahl 
>> wrote:
>> >
>> > Ok, that’s the one I was looking for, it’s not documented in the backup
>> chapter of ref-guide :(
>> >
>> > Jan Høydahl
>> >
>> > > 23. des. 2020 kl. 17:10 skrev Jason Gerlowski > >:
>> > >
>> > > 
>> > >>
>> > >> We have a path alias to the old API ... but we don’t have a true v2
>> API spec for it, do we?
>> > >
>> > > Tbh I'm not yet familiar enough with the v2 APIs to understand the
>> > > distinction you're making.  (Do you have a pointer to something that'd
>> > > fill me in?)
>> > >
>> > > To zoom in on "backup" as an example, the v2 API I'm referring to
>> > > looks like:  /v2/collections" -d '{ "backup-collection":
>> > > {"collection": "books", "name": "asdf3", "location": "/tmp/foo"}}'.
>> > > And it's included in the v2 "introspect" documentation returned by
>> > > this API: /v2/collections/_introspect?command=backup-collection".  To
>> > > me that looked like a v2 API, but maybe path-aliases are also covered
>> > > in the introspect docs.
>> > >
>> > > Jason
>> > >
>> > >> On Wed, Dec 23, 2020 at 10:29 AM Jan Høydahl 
>> wrote:
>> > >>
>> > >> Actually, don’t think we do have a v2 Backup/Restore API. We have a
>> path alias to the old API which takes GET ...=backup... but we don’t
>> have a true v2 API spec for it, do we? Where is that documented?
>> > >>
>> > >> Jan Høydahl
>> > >>
>> > >>>> 22. des. 2020 kl. 18:04 skrev Jason Gerlowski <
>> gerlowsk...@gmail.com>:
>> > >>>
>> > >>> Hey guys,
>> > >>>
>> > >>> Following up to make sure I understand the specifics you're
>> > >>> suggesting.  You're proposing that:
>> > >>>
>> > >>> 1. The brand new backup-related APIs (list-backups and
>> delete-backup)
>> > >>> be added in v2-form only.
>> > >>> 2. Tweaks to existing backup-related APIs (create-backup, restore)
>>

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Mike Drob]

Can you explicitly call out in the SIP how it relates to the work done in
SOLR-13608?

On Tue, Jan 5, 2021 at 8:55 AM Jason Gerlowski 
wrote:

> Hey, Happy New Year everybody.
>
> Some SIP updates based on the discussion above:
>
> I added v2 examples for each API to the SIP.  Feedback welcome,
> especially on the v2 APIs that are net-new to this proposal (namely:
> "list backups" and "delete backup").
>
> I've also amended the backcompat/migration section to mention Jan's
> suggestion that the "incremental" features be exposed in the v2 API
> only.  Though it's unclear to me whether that's still something people
> want since it turns out that we'll still have backcompat concerns with
> the existing v2 backup/restore APIs.  So I've held off from
> removing/replacing the original plan.
>
> Link for convenience:
>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore
>
> Best,
>
> Jason
>
>
> On Thu, Dec 24, 2020 at 8:11 AM Jan Høydahl  wrote:
> >
> > Ok, that’s the one I was looking for, it’s not documented in the backup
> chapter of ref-guide :(
> >
> > Jan Høydahl
> >
> > > 23. des. 2020 kl. 17:10 skrev Jason Gerlowski :
> > >
> > > 
> > >>
> > >> We have a path alias to the old API ... but we don’t have a true v2
> API spec for it, do we?
> > >
> > > Tbh I'm not yet familiar enough with the v2 APIs to understand the
> > > distinction you're making.  (Do you have a pointer to something that'd
> > > fill me in?)
> > >
> > > To zoom in on "backup" as an example, the v2 API I'm referring to
> > > looks like:  /v2/collections" -d '{ "backup-collection":
> > > {"collection": "books", "name": "asdf3", "location": "/tmp/foo"}}'.
> > > And it's included in the v2 "introspect" documentation returned by
> > > this API: /v2/collections/_introspect?command=backup-collection".  To
> > > me that looked like a v2 API, but maybe path-aliases are also covered
> > > in the introspect docs.
> > >
> > > Jason
> > >
> > >> On Wed, Dec 23, 2020 at 10:29 AM Jan Høydahl 
> wrote:
> > >>
> > >> Actually, don’t think we do have a v2 Backup/Restore API. We have a
> path alias to the old API which takes GET ...=backup... but we don’t
> have a true v2 API spec for it, do we? Where is that documented?
> > >>
> > >> Jan Høydahl
> > >>
> >  22. des. 2020 kl. 18:04 skrev Jason Gerlowski <
> gerlowsk...@gmail.com>:
> > >>>
> > >>> Hey guys,
> > >>>
> > >>> Following up to make sure I understand the specifics you're
> > >>> suggesting.  You're proposing that:
> > >>>
> > >>> 1. The brand new backup-related APIs (list-backups and delete-backup)
> > >>> be added in v2-form only.
> > >>> 2. Tweaks to existing backup-related APIs (create-backup, restore) be
> > >>> made in V2-form only.
> > >>> 3. All existing v1 backup-related APIs be deprecated and left
> > >>> unchanged.  Incremental backups will not be possible using the v1
> API.
> > >>>
> > >>> I'm not against going this route if there's consensus around it.  But
> > >>> I'm not 100% clear on how it means we don't need to worry about
> > >>> backcompat.  Backup and Restore currently exist as both a v1 and a v2
> > >>> API - I understand how leaving the v1 APIs untouched (other than
> > >>> deprecation) frees us of some backcompat concerns there, but we would
> > >>> still need to make tweaks to the v2 backup/restore APIs and would
> have
> > >>> to tread just as carefully there in terms of backcompat, afaict.
> > >>> Unless Solr's backcompatibility guarantees only cover the v1 API and
> > >>> leave v2 changes to be made freely?  I looked around to see if the v2
> > >>> APIs had any sort of "experimental" designation, but couldn't find
> > >>> that clearly stated anywhere.  Am I missing something?
> > >>>
> > >>> Best,
> > >>>
> > >>> Jason
> > >>>
> >  On Tue, Dec 22, 2020 at 2:49 AM Noble Paul 
> wrote:
> > 
> > > , and implement the new imporved version as a V2-api only, and
> then deprecate the v1 API?
> > 
> > 
> >  V2 only please
> > 
> > > On Tue, Dec 22, 2020 at 1:34 AM Jason Gerlowski <
> gerlowsk...@gmail.com> wrote:
> > >
> > > Hey Jan, thanks for the review.
> > >
> > > I hadn't thought about the V2 API in connection to this work.
> You're
> > > right though I think - the SIP proposes net-new APIs, so it should
> add
> > > V2 equivalents at the very least.  I'll draft tentative details for
> > > these APIs on the SIP and we can refine things from there.
> > >
> > > I'm more up in the air on your specific suggestion to restrict the
> SIP
> > > changes to these v2 APIs.  It is an elegant approach to the
> > > backcompat, and it provides a carrot for v2 adoption - both of
> which I
> > > like.  But it would let users create snapshot-based backups (and
> keep
> > > us maintaining that code) longer than there's any strict need to.
> And
> > > users are left on the less-efficient format by default.  (By
> contrast,
> > > the current SIP has 

Re: Code reformatting

[Mike Drob]

They both look fine to me, I’d likely go with the default so that there is
less for us to maintain.

On Wed, Dec 23, 2020 at 8:08 AM Erick Erickson 
wrote:

> I took a quick look at lucene/queries just to get my feet wet. Before
> working on it seriously. I did a fast scan through about half of the
> changes and have only one question.
>
> The formatter tightens up non-block comments, i.e.
>
> if (this == obj)
>   return true;
>
> becomes
>
> if (this == obj) return true;
>
> Which one is clearer is always a matter of debate, but my take is that
> braces should be added when the _original_ code was on multiple lines,
> respecting the opinion of the author. So the above would get braces added
> manually:
>
> if (this == obj) {
>   return true;
> }
>
> However, if the original code were:
>
> if( this == obj ) return true;
>
> and it reformatted to:
>
> if (this == obj) return true;
>
> leave it be.
>
> Does this align with other people’s opinion?
>
> Or is the consensus that any changed non-block should have braces added on
> the theory that braces are always clearer and the code was changed anyway?
> In which case the second example would get braces.
>
> I don’t particularly have any strong opinion, and I don’t think we need to
> be draconian about it, just thought that I’d do whatever others are doing.
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Shared Storage -- BlobDirectory, SOLR-15051

[Mike Drob]

Hi David,

Thanks for sharing. I am sure I will have thoughts on this, but won’t be
able to substantively comment until January. Just letting you know that
there is interest and not to be discouraged if you get only silence for a
while.

Hopefully others will look and comment as well.

Mike

On Tue, Dec 22, 2020 at 10:00 AM David Smiley  wrote:

> Hello,
>
> There's lots of exciting work going on in Solr at the moment, judging from
> the SIPs & some JIRA issues.  I want to draw attention to my proposal for
> shared storage in SolrCloud that I call "BlobDirectory" -- SOLR-15051 [1].
> It has a linked proposal document[2] in Google Docs.  If any of you have
> comments / concerns on the design, now is a good time to share them.  I
> expect to share a very early draft WIP PR today, containing an early form
> of only some of the components.  I'll repeat the issue description here:
>
> [1] https://issues.apache.org/jira/browse/SOLR-15051
> [2]
> https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing
> 
>
> This proposal is a way to accomplish shared storage in SolrCloud with a
> few key characteristics: (A) using a Directory implementation, (B)
> delegates to a backing local file Directory as a kind of read/write cache
> (C) replicas have their own "space", (D) , de-duplication across replicas
> via reference counting, (E) uses ZK but separately from SolrCloud stuff.
>
> The Directory abstraction is a good one, and helps isolate shared storage
> from the rest of SolrCloud that doesn't care.  Using a backing normal file
> Directory is faster for reads and is simpler than Solr's HDFSDirectory's
> BlockCache.  Replicas having their own space solves the problem of multiple
> writers (e.g. of the same shard) trying to own and write to the same space,
> and it implies that any of Solr's replica types can be used along with what
> goes along with them like peer-to-peer replication (sometimes
> faster/cheaper than pulling from shared storage).  A de-duplication feature
> solves needless duplication of files across replicas and from parent shards
> (i.e. from shard splitting).  The de-duplication feature requires a place
> to cache directory listings so that they can be shared across replicas and
> atomically updated; this is handled via ZooKeeper.  Finally, some sort of
> Solr daemon / auto-scaling code should be added to implement
> "autoAddReplicas", especially to provide for a scenario where the leader is
> gone and can't be replicated from directly but we can access shared storage.
>
> For more about shared storage concepts, consider looking at the
> description in SOLR-13101
>  and the linked Google
> Doc.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

Re: jira patch "precommit" jenkins jobs? (don't seem to be running lately)

[Mike Drob]

Github actions should be running at least SolrJ tests.
https://issues.apache.org/jira/browse/SOLR-14877

Maybe it would make sense to make a similar action for running lucene tests
when something under lucene/ has changed, or more generally run tests for a
module if something in that module has changed.

On Sat, Dec 12, 2020 at 9:36 AM Mikhail Khludnev  wrote:

> Hello, Christine.
> I don't think it runs tests at all, I suppose it just does
> https://github.com/apache/lucene-solr/blob/ccf3e604537e884e25d33dc9d921cc5e5e1fa284/.github/workflows/gradle-precommit.yml#L36
>
> On Wed, Dec 9, 2020 at 5:15 AM Christine Poerschke (BLOOMBERG/ LONDON) <
> cpoersc...@bloomberg.net> wrote:
>
>> Am not familiar with the yetus setup but from the
>> https://github.com/apache/lucene-solr/tree/master/.github/workflows
>> content and observation of github checks on pull requests my understanding
>> is that github runs some tests but not all tests and what gets run depends
>> on the content of the pull request.
>>
>> Hope that helps.
>>
>> Christine
>>
>> From: dev@lucene.apache.org At: 12/05/20 18:44:40
>> To: dev@lucene.apache.org
>> Cc: hossman_luc...@fucit.org
>> Subject: Re: jira patch "precommit" jenkins jobs? (don't seem to be
>> running lately)
>>
>> Uwe, thank you for your response. I remember that yetus run tests via
>> JIRA's precommit before, but github checks doesn't run tests. Is it
>> correct?
>>
>> --
>> Mikhail
>>
>> On Mon, Oct 12, 2020 at 9:18 AM Uwe Schindler  wrote:
>>
>>> It no longer works since jenkins was moved to new hardware.
>>>
>>> IMHO, you should use pull requests in GitHub. There we have full support
>>> for automatic precommit. We use it every day, much easier than Jira. I'd
>>> not spend much time in reactivating it. It's dead.
>>>
>>> On Jira it's disabled since longer time.
>>>
>>> Uwe
>>>
>>> Am October 12, 2020 3:50:26 PM UTC schrieb Chris Hostetter <
>>> hossman_luc...@fucit.org>:


 Does anyone know / un derstand the current status of the "PreCommit"
 jenkins jobs that are suppose to run against jira issues in the "Path
 Available" status?

 For example: I noticed this AM that even though SOLR-14870 was in the
 "Path Available" status all weekend (I didn't want to commit build changes
 on a friday afternoon) it never got a comment from the jenkins build bot
 regarding the patch -- when I went looking for the "PreCommit" build jobs
 in jenkins i found that they are marked "N/A" for last
 success/failure/durration -- which i believe means they haven't run at all
 since all the jenkins jobs were moved to ci-builds.apache.org?

 https://ci-builds.apache.org/job/Lucene/
 https://ci-builds.apache.org/job/Lucene/job/PreCommit-LUCENE-Build/
 https://ci-builds.apache.org/job/Lucene/job/PreCommit-SOLR-Build/

 ...the descriptions of those jobs say they are run by the
 "PreCommit-Admin" job, but the links is a 404.  searching jenkinds jobs
 turns up a few other "PreCommit" jobs in other projects -- most are
 disabled, except for this "Atlas" one which has run somewhat recently --
 but it looks like people were manually triggering it?

 https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/

 do any of hte jenkins admins/experts know what's the status of the
 infra/jira hooks to get these jobs working again?



 -Hoss
 http://www.lucidworks.com/

 --

 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org


>>> --
>>> Uwe Schindler
>>> Achterdiek 19, 28357 Bremen
>>> https://www.thetaphi.de
>>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>>
>>
>>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: [DISCUSS][SOLR] Multiple Repos For Contributions

[Mike Drob]

> By "hard to link", are you basically saying pasting URLs is hard

Both JIRA and GH give nice visual indicators to the state of an issue when
it is linked from another issue. When you manually create links, you lose
that. That's all I was saying.

I like JIRA because everything is in JIRA and I do not envy the person who
has to move mountains to migrate it to whatever new system. I think Jan is
right here - let each project choose for themselves.

> But there are downsides to managing multiple repositories that make me
> hesitant about the idea more generally.  There's no easy way to
> prevent changes in one repo from unintentionally breaking another.
> There's at least some duplication in the maintenance of things all
> repos need (build systems, etc.).  It may add overhead on release
> volunteers and the PMC if there are more releases.

I tried to address some of these in my second mail to the thread. We'll
need to invest in additional CI, and possibly in some ways to refactor and
DRY the build config.

I don't think adding overhead on releases is a problem - releases are kind
of the reason we are all here anyway. If there's not enough motivation to
make releases happen, then they won't happen, and that will be the end of
it.

> But maybe it'd make sense to use solr-operator as a test case for a few
> releases before putting in the effort to move out our current contribs
> or change our process of adopting new ones

I agree with this, but for a huge caveat. Big things have been deprecated
and removed, and users are going to be very confused on what to do when 9.0
comes around. I don't know to what extent it will impact adoption. But I'd
rather have a plan and a narrative then going into it hoping for the best.
At the same time, I really don't want to have to wait for the solr-operator
experiment to settle before we can get a green light on 9.0 in general. But
that's a different conversation.

Mike

On Wed, Nov 25, 2020 at 10:46 AM Jason Gerlowski 
wrote:

> > then I'll start putting together individual proposals for a few repos to
> exercise the process and get more contributions going
>
> solr-operator is a great example of PMC-maintained code that makes
> sense to have in a separate repository.  It's written primarily in a
> different language, it's an integration with 3rd party software, etc.
>
> But there are downsides to managing multiple repositories that make me
> hesitant about the idea more generally.  There's no easy way to
> prevent changes in one repo from unintentionally breaking another.
> There's at least some duplication in the maintenance of things all
> repos need (build systems, etc.).  It may add overhead on release
> volunteers and the PMC if there are more releases.
>
> I'm not sure how much those'll cause problems in practice.  Hopefully
> they'll be minimal, but it's possible they won't be.  They might end
> up outweighing the benefits.  I'm not saying we should be afraid of
> additional repositories where it makes sense for the domain.  But
> maybe it'd make sense to use solr-operator as a test case for a few
> releases before putting in the effort to move out our current contribs
> or change our process of adopting new ones.  Since this is more about
> long term management, and less about getting in a particular feature
> or value for users, we've got a cool opportunity to let the
> solr-operator experiment play out before we necessarily need to decide
> how to handle similar scenarios.
>
> Just my two cents.
>
> Jason
>
> On Wed, Nov 25, 2020 at 8:23 AM Jan Høydahl  wrote:
> >
> > Let each sub project decide for themselves. PYLUCENE has its own svn
> repo and its own Jira space.
> > Solr-operator should be allowed to continue with GH issues and PRs
> i.m.o. No need to force them into JIRA as long as the ASF allows projects
> to choose.
> >
> > Jan
> >
> > 24. nov. 2020 kl. 20:59 skrev David Smiley :
> >
> >> > Q: Should we create a separate JIRA for these contribs... or ditch
> JIRA entirely for them, relying on GitHub alone?
> >>
> >> I'd start with same JIRA, with a separate component or label. I don't
> think GH issues would be good because it becomes harder to link between
> core and contrib issues in case of compat or tandme feature development.
> >
> >
> > By "hard to link", are you basically saying pasting URLs is hard ;-).
> ?  There was a committer meeting in Montreal where some folks like Jan
> Hoydal and Varun (if I'm not mistaken; I may be) advocated for considering
> more GitHub centric issue tracking.  I was not in favor of that... however
> for contribs/modules that get their own separate repos, it affords an
> opportunity for a break with the past in the interests of simplicity and

Welcome Houston Putman to the PMC

[Mike Drob]

I am pleased to announce that Houston Putman has accepted the PMC's invitation 
to join.

Congratulations and welcome, Houston!

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene » Solr-Artifacts-8.x - Build # 114 - Failure!

[Mike Drob]

Apologies for breaking branch 8.x with my cherry-pick, double checking
things locally and will push a fix shortly.

On Mon, Nov 30, 2020 at 2:53 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://ci-builds.apache.org/job/Lucene/job/Solr-Artifacts-8.x/114/
>
> No tests ran.
>
> Build Log:
> [...truncated 2393 lines...]
> [javac] Compiling 1360 source files to
> /home/jenkins/jenkins-slave/workspace/Lucene/Solr-Artifacts-8.x/solr/build/solr-core/classes/java
> [javac]
> /home/jenkins/jenkins-slave/workspace/Lucene/Solr-Artifacts-8.x/solr/core/src/java/org/apache/solr/core/CachingDirectoryFactory.java:329:
> error: cannot find symbol
> [javac] Path dirPath = Path.of(path);
> [javac]^
> [javac]   symbol:   method of(String)
> [javac]   location: interface Path
> [javac] Note: Some input files use or override a deprecated API.
> [javac] Note: Recompile with -Xlint:deprecation for details.
> [javac] Note: Some input files use unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
> [javac] 1 error
>
> BUILD FAILED
> /home/jenkins/jenkins-slave/workspace/Lucene/Solr-Artifacts-8.x/solr/build.xml:506:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene/Solr-Artifacts-8.x/solr/common-build.xml:406:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene/Solr-Artifacts-8.x/lucene/common-build.xml:581:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene/Solr-Artifacts-8.x/lucene/common-build.xml:2083:
> Compile failed; see the compiler error output for details.
>
> Total time: 54 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Publishing Javadoc
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
> For additional commands, e-mail: builds-h...@lucene.apache.org

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Mike Drob]

Congratulations and welcome, Julie!

On Wed, Nov 18, 2020 at 8:29 AM Christian Moen  wrote:

> Congrats, Julie.
>
> On Thu, Nov 19, 2020 at 0:07 Michael Sokolov  wrote:
>
>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> invitation to become a committer.
>>
>> Julie, the tradition is that new committers introduce themselves with
>> a brief bio.
>>
>> I think we may still be sorting out the details of your Apache account
>> (julie@ may have been taken?), but as soon as that has been sorted out
>>  and karma has been granted, you can use your new powers to add
>> yourself to the committers section of the Who We Are page on the
>> website: 
>>
>> Congratulations and welcome!
>>
>> Mike Sokolov
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: [DISCUSS][SOLR] Multiple Repos For Contributions

[Mike Drob]

uppose so.
>> Q: Would each contrib have it's own release vote?  I suppose so, as it
>> has its own artifact.  I think the ASF requires this.
>> Q: Is it "okay" to release new Solr versions that break any of these
>> external contribs?  Knowingly or unknowingly -- does it matter?
>> Q: What technical work is needed to extricate an internal contrib to an
>> external?
>> * source control history.  (note: i've done this git history in a single
>> folder extraction before, with a popular Stackoverflow answer)
>> * mandatory ASF files, e.g. license, notice
>> * more files that we may want: CHANGES.txt
>> * More build files; copying the rules/setup/standards of the Solr
>> mothership and will become divergent over time no doubt.  Or just KISS
>> principle; no sharing; simple Maven projects.
>> Q: Could & should many contribs live in one repo (no more internal
>> contribs), yet each still have its own release cycle?  This could make
>> sharing build infrastructure easier, and detecting Solr compatibility with
>> them easier.  Although it would mean sharing GitHub project area, thus
>> sharing issues/PRs.
>> Q: Should we create a separate JIRA for these contribs... or ditch JIRA
>> entirely for them, relying on GitHub alone?
>> Q: Would contribs be treated as first class citizens in the Solr
>> Reference Guide (they are still in the ASF after all), or would they be
>> banished like the DIH was?
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Thu, Nov 12, 2020 at 6:40 PM Mike Drob  wrote:
>>
>>> Solr Devs,
>>>
>>> We've slowly been moving into a multi-repository model, and I wanted to
>>> bring some more attention to it and have a more focused discussion. We've
>>> recently embarked upon the acceptance of solr-operator as a distinct
>>> repo[1] under the care of the Lucene (soon to be Solr) PMC. I expect that
>>> there will be more cases of this as we transition additional contribs out
>>> of core, or as more plugins, packages, and integrations mature. Some will
>>> make sense as externally maintained code bases, but I believe other
>>> contributions may benefit our community more as part of the Apache
>>> Foundation.
>>>
>>> I think there was a very insightful comment[2] made by GP regarding
>>> adopting a similar model to Apache Commons governance, bringing attention
>>> to it here because I fear it may have gotten lost deep in the thread. Based
>>> on observations of Commons and a few other Apache projects with multi-repo
>>> setups, there thankfully does not appear to be a limit on how many
>>> repositories a PMC can maintain. The size and scope of each individual
>>> repository can vary greatly. I see potential ideas for anything that could
>>> be standalone and not tied to a release cycle (Admin UI, DIH, etc...), or
>>> anything that bridges integrations between Solr and other systems (k8s,
>>> HDFS, etc...).
>>>
>>> The risks that new repos face are similar to the risks they would have
>>> encountered as contrib modules, but I don't think they should dissuade us.
>>> Each project would need to start with a champion or sponsor and a
>>> discussion on the mailing list. From there, we can vote to accept the code,
>>> or just the idea if there is no code yet, as a community and create the
>>> repo. As part of a natural lifecycle, if there's not enough momentum or
>>> adoption over time, then we can update the README and docs and "retire"
>>> certain projects. The exact mechanisms can be undetermined for now; maybe
>>> it's a repo rename, maybe it's marking the repo read-only, maybe it's
>>> something else.
>>>
>>> The Commons model is that everyone is a committer on everything. There
>>> are other governance models, like Hadoop, with "area committers" who are
>>> limited to the specific repositories they have contributed frequently to.
>>> I'm not sure which model ultimately suits us better, but I think that
>>> leveraging area committers would allow us to recognize and empower
>>> contributors sooner and more frequently. Releases would still need to be
>>> voted on and approved by the singular PMC.
>>>
>>> There's no real action items here, it's more of a discussion prompt. If
>>> it looks like we have general consensus to this approach, then I'll start
>>> putting together individual proposals for a few repos to exercise the
>>> process and get more contributions going. I'll probably put the proposals
>>> together even if there's no replies here, but I'd much rather have some
>>> acknowledgement from the community that I'm headed in a sustainable
>>> direction!
>>>
>>> Mike
>>>
>>> [1]:
>>> https://lists.apache.org/thread.html/rb90f530155dc6edc6f1ccd5f056db1618142fdfcbd32d83f539d984b%40%3Cdev.lucene.apache.org%3E
>>> [2]:
>>> https://lists.apache.org/thread.html/r9965cb693369d927a942f805c134bfeb45c5e80f447ad0fe2f663fae%40%3Cdev.lucene.apache.org%3E
>>>
>>

[DISCUSS][SOLR] Multiple Repos For Contributions

[Mike Drob]

Solr Devs,

We've slowly been moving into a multi-repository model, and I wanted to
bring some more attention to it and have a more focused discussion. We've
recently embarked upon the acceptance of solr-operator as a distinct
repo[1] under the care of the Lucene (soon to be Solr) PMC. I expect that
there will be more cases of this as we transition additional contribs out
of core, or as more plugins, packages, and integrations mature. Some will
make sense as externally maintained code bases, but I believe other
contributions may benefit our community more as part of the Apache
Foundation.

I think there was a very insightful comment[2] made by GP regarding
adopting a similar model to Apache Commons governance, bringing attention
to it here because I fear it may have gotten lost deep in the thread. Based
on observations of Commons and a few other Apache projects with multi-repo
setups, there thankfully does not appear to be a limit on how many
repositories a PMC can maintain. The size and scope of each individual
repository can vary greatly. I see potential ideas for anything that could
be standalone and not tied to a release cycle (Admin UI, DIH, etc...), or
anything that bridges integrations between Solr and other systems (k8s,
HDFS, etc...).

The risks that new repos face are similar to the risks they would have
encountered as contrib modules, but I don't think they should dissuade us.
Each project would need to start with a champion or sponsor and a
discussion on the mailing list. From there, we can vote to accept the code,
or just the idea if there is no code yet, as a community and create the
repo. As part of a natural lifecycle, if there's not enough momentum or
adoption over time, then we can update the README and docs and "retire"
certain projects. The exact mechanisms can be undetermined for now; maybe
it's a repo rename, maybe it's marking the repo read-only, maybe it's
something else.

The Commons model is that everyone is a committer on everything. There are
other governance models, like Hadoop, with "area committers" who are
limited to the specific repositories they have contributed frequently to.
I'm not sure which model ultimately suits us better, but I think that
leveraging area committers would allow us to recognize and empower
contributors sooner and more frequently. Releases would still need to be
voted on and approved by the singular PMC.

There's no real action items here, it's more of a discussion prompt. If it
looks like we have general consensus to this approach, then I'll start
putting together individual proposals for a few repos to exercise the
process and get more contributions going. I'll probably put the proposals
together even if there's no replies here, but I'd much rather have some
acknowledgement from the community that I'm headed in a sustainable
direction!

Mike

[1]:
https://lists.apache.org/thread.html/rb90f530155dc6edc6f1ccd5f056db1618142fdfcbd32d83f539d984b%40%3Cdev.lucene.apache.org%3E
[2]:
https://lists.apache.org/thread.html/r9965cb693369d927a942f805c134bfeb45c5e80f447ad0fe2f663fae%40%3Cdev.lucene.apache.org%3E

Re: Communicating the future of DIH?

[Mike Drob]

I'm somewhat unclear - is the suggestion to submit projects to Apache
Commons repositories or to do a similar process with our own multiple
repositories.

On Tue, Nov 3, 2020 at 2:43 PM Gézapeti  wrote:

> I'm not that close with Apache Commons either, but I think the
> solr-operator is a perfect candidate to figure out how things should work.
> I'd be happy to do some exploration and reach out to other projects about
> their experience in this matter.
> gp
>
>
> On Tue, Nov 3, 2020 at 9:31 PM David Smiley  wrote:
>
>> Thanks Gezapeti for recommending that model.  It makes sense to me
>> conceptually... yet it seems burdensome, at least to set it up (website,
>> lists, repos, misc.) and get us familiar with a new way of working.  I have
>> nothing but questions about how they operate.  As it is, we haven't even
>> done our TLP split yet, which is maybe a little embarrassing as a PMC
>> member.  I wonder if other ASF projects do anything similar for their
>> plugins?  I'd guess Maven or Ant might be similar.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Wed, Oct 28, 2020 at 2:12 PM Gézapeti Cseh 
>> wrote:
>>
>>> Hey everyone,
>>>
>>> Solr could keep some plugins with separate git repositories under the
>>> Apache umbrella if it adapts the Apache Commons model
>>> where plugins can have their own release
>>> schedule and community without the risk of being abandoned completely
>>> without notice. There is a process to get into Apache Commons via the
>>> sandbox  and to end development
>>> in dormant .
>>> I know this solution would still add the load of vulnerabilities and
>>> releases to the Apache Solr project, but components/plugins could have a
>>> level of separation while keeping the trust in ASF behind the plugins.
>>> Also, if a vulnerability surfaces for a particular plugin/component not the
>>> whole Solr release would be marked as affected.
>>> I'm not saying we should do this with DataImportHandler, but I think. it
>>> worth considering for other plugins.
>>>
>>> I'm not sure if this was discussed before, I've tried to search the
>>> archives without any luck.
>>> gp
>>>
>>>
>>>
>>> On Thu, Oct 15, 2020 at 11:35 PM Anshum Gupta 
>>> wrote:
>>>
 I think that the fact that this code is no longer under the Apache
 umbrella will lead to such issues and moving this into a different GitHub
 org wouldn't really help with the problem that's been highlighted. Also,
 the maintainers of a plugin should be people who understand and monitor the
 project and if those rights are opened up, it'd be hard to maintain
 reliability of the plugin.

 Thanks for updating the Ref Guide, Cassandra :) In line with my
 suggestion above, I think we need to be clear with the messaging of the
 fact that the code no longer is owned and maintained by the Apache
 Lucene/Solr community and any vulnerability or bug reported in the 3rd
 party packages wouldn't be the responsibility of the Apache Lucene
 community.


 On Thu, Oct 15, 2020 at 2:16 PM Cassandra Targett <
 casstarg...@gmail.com> wrote:

> I updated the Ref Guide for 8.7 to include a link to the plugin repo
> (for all plugins which have an established repo, not just DIH), hoping 
> that
> would help answer user questions and spur adoption. That’s just a
> super-minor thing, but it’s something.
>
> If Rohit doesn’t have time to be a maintainer now and no one else
> wants to be, would a separate GitHub org help that? I understand the
> motivation for sharing the burden…I guess you’re thinking that single org
> would allow people to be maintainers on multiple plugins?
>
> Draft for 8.7 Guide is here if interested to see what I did:
> https://nightlies.apache.org/Lucene/Solr-reference-guide-8.x/uploading-structured-data-store-data-with-the-data-import-handler.html
> On Oct 15, 2020, 3:52 PM -0500, Jan Høydahl ,
> wrote:
>
> Some of those issues are opened by me, not beause I plan to be a DIH
> maintainer myself, but I was hoping that Rohit had some real interest in
> forming a comunity.
> Turns out that the plugin is as good as dead on arrival, which is
> really disappointing.
>
> We as the donator could perhaps help by sending an email, with a
> reminder that DIH is being deprecated and that the new plugin really needs
> more maintainers.
> That’s why I filed
> https://github.com/rohitbemax/dataimporthandler/issues/12, else
> people arriving to the page would not even know how to contribute or 
> become
> a committer.
> I could whip up a PR for the README inviting contributors, but I’m
> honestly not so sure that newcomers would feel welcome, as their
> contributions would likely attract no 

Re: [SOLR] Closing CDCR issues as Won't Fix

[Mike Drob]

I saw these issues but I didn’t find the blob store deprecation decision to
reference when closing them.

On Thu, Oct 29, 2020 at 6:59 AM Eric Pugh 
wrote:

> Mike,
>
> There are five that come up:
> https://issues.apache.org/jira/browse/SOLR-12501?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20blobstore
>
>
>
> On Oct 28, 2020, at 6:01 PM, Mike Drob  wrote:
>
> This is done for CDCR and Velocity.
>
> I don't really understand which parts of HDFS are deprecated and which
> ones are staying (this is not a request for somebody to explain it to me),
> so I left that one alone.
>
> If you link me to the blobstore deprecation JIRA, then I can close those
> as well. I think I have the mechanics of this figured out by now.
>
> On Wed, Oct 28, 2020 at 3:36 PM Eric Pugh 
> wrote:
>
>> Thank you Mike!
>>
>> While you are at it, is “blobstore” also a candidate for this?
>>
>> Eric
>>
>>
>> On Oct 28, 2020, at 4:30 PM, Mike Drob  wrote:
>>
>> That's a good point, David. I'll make sure to include that.
>>
>> @Eric - I'll handle the Velocity issues for you as well.
>>
>> On Wed, Oct 28, 2020 at 2:59 PM David Smiley  wrote:
>>
>>> In these bulk changes, please add a comment pointing to the deprecation.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Wed, Oct 28, 2020 at 10:34 AM Eric Pugh <
>>> ep...@opensourceconnections.com> wrote:
>>>
>>>> Hopefully not hijacking your thread….  I ran the report for the
>>>> Velocity contrib module, and wondered if we should do the same thing there?
>>>>
>>>>
>>>> https://issues.apache.org/jira/browse/SOLR-6702?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20%22contrib%20-%20Velocity%22
>>>>
>>>>
>>>>
>>>> On Oct 28, 2020, at 10:19 AM, Mike Drob  wrote:
>>>>
>>>> Devs,
>>>>
>>>> Just a heads up that since Solr CDCR is on the slate for
>>>> deprecation/removal, I'm going to clean up some JIRA issues related to it,
>>>> namely
>>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20CDCR
>>>>
>>>> There's a few that I'll leave untouched, like SOLR-10091 and
>>>> SOLR-14611, since they are not related to the current implementation of
>>>> CDCR, but the rest should be closed as Won't Fix. I'll try to do this
>>>> without spamming the mailing list, but apologize pre-emptively if this
>>>> generated a bunch of traffic.
>>>>
>>>> Mike
>>>>
>>>>
>>>> ___
>>>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>>>> | http://www.opensourceconnections.com | My Free/Busy
>>>> <http://tinyurl.com/eric-cal>
>>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>>>> This e-mail and all contents, including attachments, is considered to
>>>> be Company Confidential unless explicitly stated otherwise, regardless
>>>> of whether attachments are marked as such.
>>>>
>>>>
>> ___
>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>> <http://tinyurl.com/eric-cal>
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>>
>>
> ___
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>

Re: [SOLR] Closing CDCR issues as Won't Fix

[Mike Drob]

This is done for CDCR and Velocity.

I don't really understand which parts of HDFS are deprecated and which ones
are staying (this is not a request for somebody to explain it to me), so I
left that one alone.

If you link me to the blobstore deprecation JIRA, then I can close those as
well. I think I have the mechanics of this figured out by now.

On Wed, Oct 28, 2020 at 3:36 PM Eric Pugh 
wrote:

> Thank you Mike!
>
> While you are at it, is “blobstore” also a candidate for this?
>
> Eric
>
>
> On Oct 28, 2020, at 4:30 PM, Mike Drob  wrote:
>
> That's a good point, David. I'll make sure to include that.
>
> @Eric - I'll handle the Velocity issues for you as well.
>
> On Wed, Oct 28, 2020 at 2:59 PM David Smiley  wrote:
>
>> In these bulk changes, please add a comment pointing to the deprecation.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Wed, Oct 28, 2020 at 10:34 AM Eric Pugh <
>> ep...@opensourceconnections.com> wrote:
>>
>>> Hopefully not hijacking your thread….  I ran the report for the Velocity
>>> contrib module, and wondered if we should do the same thing there?
>>>
>>>
>>> https://issues.apache.org/jira/browse/SOLR-6702?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20%22contrib%20-%20Velocity%22
>>>
>>>
>>>
>>> On Oct 28, 2020, at 10:19 AM, Mike Drob  wrote:
>>>
>>> Devs,
>>>
>>> Just a heads up that since Solr CDCR is on the slate for
>>> deprecation/removal, I'm going to clean up some JIRA issues related to it,
>>> namely
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20CDCR
>>>
>>> There's a few that I'll leave untouched, like SOLR-10091 and SOLR-14611,
>>> since they are not related to the current implementation of CDCR, but the
>>> rest should be closed as Won't Fix. I'll try to do this without spamming
>>> the mailing list, but apologize pre-emptively if this generated a bunch of
>>> traffic.
>>>
>>> Mike
>>>
>>>
>>> ___
>>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>>> | http://www.opensourceconnections.com | My Free/Busy
>>> <http://tinyurl.com/eric-cal>
>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>>> This e-mail and all contents, including attachments, is considered to be
>>> Company Confidential unless explicitly stated otherwise, regardless
>>> of whether attachments are marked as such.
>>>
>>>
> ___
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> <http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>

Re: [SOLR] Closing CDCR issues as Won't Fix

[Mike Drob]

That's a good point, David. I'll make sure to include that.

@Eric - I'll handle the Velocity issues for you as well.

On Wed, Oct 28, 2020 at 2:59 PM David Smiley  wrote:

> In these bulk changes, please add a comment pointing to the deprecation.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Wed, Oct 28, 2020 at 10:34 AM Eric Pugh <
> ep...@opensourceconnections.com> wrote:
>
>> Hopefully not hijacking your thread….  I ran the report for the Velocity
>> contrib module, and wondered if we should do the same thing there?
>>
>>
>> https://issues.apache.org/jira/browse/SOLR-6702?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20%22contrib%20-%20Velocity%22
>>
>>
>>
>> On Oct 28, 2020, at 10:19 AM, Mike Drob  wrote:
>>
>> Devs,
>>
>> Just a heads up that since Solr CDCR is on the slate for
>> deprecation/removal, I'm going to clean up some JIRA issues related to it,
>> namely
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20CDCR
>>
>> There's a few that I'll leave untouched, like SOLR-10091 and SOLR-14611,
>> since they are not related to the current implementation of CDCR, but the
>> rest should be closed as Won't Fix. I'll try to do this without spamming
>> the mailing list, but apologize pre-emptively if this generated a bunch of
>> traffic.
>>
>> Mike
>>
>>
>> ___
>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>> <http://tinyurl.com/eric-cal>
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>>
>>

[SOLR] Closing CDCR issues as Won't Fix

[Mike Drob]

Devs,

Just a heads up that since Solr CDCR is on the slate for
deprecation/removal, I'm going to clean up some JIRA issues related to it,
namely
https://issues.apache.org/jira/issues/?jql=project%20%3D%20SOLR%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20CDCR

There's a few that I'll leave untouched, like SOLR-10091 and SOLR-14611,
since they are not related to the current implementation of CDCR, but the
rest should be closed as Won't Fix. I'll try to do this without spamming
the mailing list, but apologize pre-emptively if this generated a bunch of
traffic.

Mike