Re: [VOTE] Release Lucene/Solr 8.11.3 RC1

[Jason Gerlowski]

Here's my +1 (binding)

SUCCESS! [0:56:16.591754]

On Mon, Feb 5, 2024 at 5:23 PM Houston Putman  wrote:

> Please vote for release candidate 1 for Lucene/Solr 8.11.3
>
> The artifacts can be downloaded from:
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.3-RC1-revbaa7c80af4278cc8951a344d8e9320386588d12d
>
> You can run the smoke tester directly with this command:
>
> python3 -u dev-tools/scripts/smokeTestRelease.py \
>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.3-RC1-revbaa7c80af4278cc8951a344d8e9320386588d12d
>
> The vote will be open for at least 72 hours i.e. until 2024-02-08 23:00
> UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>

Re: [VOTE] Release Lucene/Solr 8.11.2 RC2

[Jason Gerlowski]

+1

SUCCESS! [1:03:16.496710]

Also did some manual testing - mostly of backup/restore (GCS and S3)

On Thu, Jun 16, 2022 at 12:58 PM Houston Putman  wrote:
>
> +1 (binding)
>
> SUCCESS! [1:10:19.762602]
>
> - Houston
>
> On Wed, Jun 15, 2022 at 10:04 PM Anshum Gupta  wrote:
>>
>> Thanks for reporting that, Tim.
>>
>> Seems like this one is known to fail. I tried reproducing this w/ same seed 
>> as well as randomizing, but wasn't able to do that over 100 runs.
>>
>>
>> On Wed, Jun 15, 2022 at 11:12 AM Timothy Potter  wrote:
>>>
>>> I guess this is a -0 ? test suite failed and I don't have time to
>>> fight with it :-(
>>>
>>>[junit4] Tests with failures [seed: B73581D35178CCBC]:
>>>[junit4]   - org.apache.solr.cloud.LeaderVoteWaitTimeoutTest.basicTest
>>>
>>> On Tue, Jun 14, 2022 at 4:36 AM Jan Høydahl  wrote:
>>> >
>>> > +1 (binding)
>>> >
>>> > SUCCESS! [1:19:45.192785]
>>> >
>>> > Jan
>>> >
>>> > 13. jun. 2022 kl. 21:05 skrev Mike Drob :
>>> >
>>> > Please vote for release candidate 2 for Lucene/Solr 8.11.2
>>> >
>>> > The artifacts can be downloaded from:
>>> > https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC2-rev17dee71932c683e345508113523e764c3e4c80fa
>>> >
>>> > You can run the smoke tester directly with this command:
>>> >
>>> > python3 -u dev-tools/scripts/smokeTestRelease.py \
>>> > https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.11.2-RC2-rev17dee71932c683e345508113523e764c3e4c80fa
>>> >
>>> > The vote will be open for at least 72 hours i.e. until 2022-06-16 20:00 
>>> > UTC
>>> >
>>> > [ ] +1  approve
>>> > [ ] +0  no opinion
>>> > [ ] -1  disapprove (and reason why)
>>> >
>>> > Here is my +1
>>> > 
>>> >
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>
>>
>> --
>> Anshum Gupta

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Migration to GitHub issue from Jira (LUCENE-10557)

[Jason Gerlowski]

+1 (PMC)

I understand concerns about handing governance over to a 3rd party, but
letting that drive our decision-making here feels like optimizing for a
rare case that might never occur.  I'd m,uch rather optimize for making
things easiest for contributors, and then accommodate any "Github ToS ban,
sanctions, etc." situations if and when they crop up on a case by case
basis.

Best,

Jason

On Tue, May 31, 2022 at 10:09 AM Gus Heck  wrote:

> -1 I think the disruption and bifurcation of where to find history is not
> worth it. I also noticed a comment in the lucene issue for migration with
> summaries by date range, status, affects version,  etc. sub-area, exactly
> the sort of thing I expect to be much more difficult to obtain from github.
> What I would find interesting is a deep integration of the two systems so
> that initiation and basic commenting could be handled on github, but
> transmitted to Jira where full metadata and reporting/tracking could be
> maintained.
>
> On Tue, May 31, 2022 at 12:17 AM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
>
>> -1
>>
>> On Tue, 31 May, 2022, 4:06 am Xi Chen, 
>> wrote:
>>
>>> +1 from me (committer, non-PMC)
>>>
>>> Thanks Tomoko for starting the discussion and organizing / leading this
>>> effort!
>>>
>>> Best,
>>> Zach
>>>
>>> On May 30, 2022, at 2:56 PM, Houston Putman  wrote:
>>>
>>> 
>>> +1 Approve (PMC)
>>>
>>> Thanks so much for doing all of the work for this Tomoko!
>>>
>>> - Houston
>>>
>>> On Mon, May 30, 2022 at 5:38 PM David Smiley  wrote:
>>>
 +1 Approve (PMC)

 ~ David Smiley
 Apache Lucene/Solr Search Developer
 http://www.linkedin.com/in/davidwsmiley


 On Mon, May 30, 2022 at 11:40 AM Tomoko Uchida <
 tomoko.uchida.1...@gmail.com> wrote:

> Hi everyone!
>
> As we had previous discussion thread [1], I propose migration to
> GitHub issue from Jira.
> It'd be technically possible (see [2] for details) and I think it'd be
> good for the project - not only for welcoming new developers who are not
> familiar with Jira, but also for improving the experiences of long-term
> committers/contributors by consolidating the conversation platform.
>
> You can see a short summary of the discussion, some stats on current
> Jira issues, and a draft migration plan in [2].
> Please review [2] if you haven't seen it and vote for this proposal.
>
> The vote will be open until 2022-06-06 16:00 UTC.
>
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
>
> Here is my +1
>
> *IMPORTANT NOTE*
> I set a local protocol for this vote.
> There are 95 committers on this project [3] - the vote will be
> effective if it successfully gains more than 15% of voters (>= 15) from
> committers (including PMC members). This means, that although only PMC
> member votes are counted for the final result, the votes from all
> committers are important to make the vote result effective.
>
> If there are less than 15 votes at 2022-06-06 16:00 UTC, I will expand
> the term to 2022-06-13 16:00 UTC. If this fails to get sufficient voters
> after the expanded time limit, I'll cancel this vote regardless of the
> result.
> But why do I set such an extra bar? My fear is that if such things are
> decided by the opinions of a few members, the result shouldn't yield a 
> good
> outcome for the future. It isn't my goal to just pass the vote [4].
>
> [1] https://lists.apache.org/thread/78wj0vll73sct065m5jjm4z8gqb5yffk
> [2] https://issues.apache.org/jira/browse/LUCENE-10557
> [3] https://projects.apache.org/committee.html?lucene
> [4] I'm sorry for being overly cautious, but I have never met in
> person or virtually any of the committers (with a very few exceptions),
> therefore cannot assess if the vote result is reliable or not unless there
> is certain explicit feedback.
>
> Tomoko
>

>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: 8.11 release candidate

[Jason Gerlowski]

Hey Adrien,

A pretty bad bug in Solr backups came in last week or so: SOLR-15706.
If it's not too late I'd like to try to get it into 8.11.  Would that
be ok?

I'm just digging into things now so I don't have a complete
understanding yet, but I'm optimistic it won't long delay your
timeline for cutting the RC.

Best,

Jason

On Tue, Nov 2, 2021 at 1:31 PM Adrien Grand  wrote:
>
> Hello,
>
> Assuming CI is green and no blockers have been raised until then, I plan to 
> build the first release candidate for Lucene/Solr 8.11 on Thursday November 
> 4th.
>
> Please let me know if you are aware of any blocker that we should address 
> before building the first RC.
>
> --
> Adrien

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - Still Failing!

[Jason Gerlowski]

Looks like the latest changes did the trick - the smoke-release job
passed overnight:
https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-SmokeRelease-8.9/7/

Sorry to block you for a bit Mayya; good luck with the rest of the release!

Jason

On Wed, Jun 9, 2021 at 10:07 AM Jason Gerlowski  wrote:
>
> Hey all,
>
> Quick update: I've removed the javax.annotation dependency declaration
> that the build didn't like.  Precommit and tests pass.  I've pushed
> the fix to branch_8_9.  I'm currently running buildAndPushRelease.py +
> smoke-tester and will give the all-clear once I see that pass.
>
> Sorry to hold things up on the release!
>
> Jason
>
> On Tue, Jun 8, 2021 at 4:17 AM Christine Poerschke (BLOOMBERG/ LONDON)
>  wrote:
> >
> > I've opened https://github.com/apache/lucene-solr/pull/2509 for the v8.10.0 
> > > v8.9.0 comparison issue. Hope that helps.
> >
> > From: dev@lucene.apache.org At: 06/08/21 01:28:25 UTC+1:00
> > To: dev@lucene.apache.org
> > Subject: Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 
> > - Still Failing!
> >
> > Thanks Robert for the explanation.
> >
> > Jason, thank you for your reply and investigations.
> >
> > > Do you happen to have the command that reproduced this for you?
> > You can see the build failures in Jenkins.  The last 3 failures were after 
> > June 2nd, and all contained similar failures about sheisty classes in solr.
> >
> > About specific RC for the smoke-tester, I think you can first build a 
> > release locally with buildAndPushRelease.py script, and then use this local 
> > release for the smoke tester.
> >
> > I've also looked at the failure on 8.x branch: "Future release 8.9.0 is 
> > greater than 8.10.0", and it looks like smokeTestRelease.py fails to 
> > correctly compare v8.10.0 > v8.9.0, as (('8', '10', '0') < ('8', '9', '0')).
> >
> >
> >
> >
> >
> > On Mon, Jun 7, 2021 at 3:24 PM Robert Muir  wrote:
> >>
> >> Sheisty means classes that collide with jdk packages. E.g. this
> >> javax.annotation looks like a problem, as it collides with existing
> >> jdk package in an xml module:
> >> https://docs.oracle.com/javase/8/docs/api/javax/annotation/package-summary.html
> >>
> >> Looks like the whole jar being used here is archived/deprecated [1] in
> >> favor of "jakarta annotations" [2] which uses a new
> >> non-colliding/jar-helling "jakarta.annotation" package/module:
> >>
> >> 1. https://github.com/javaee/javax.annotation/
> >> 2. https://github.com/eclipse-ee4j/common-annotations-api
> >>
> >>
> >>
> >>
> >> M
> >>
> >> On Mon, Jun 7, 2021 at 2:59 PM Jason Gerlowski  
> >> wrote:
> >> >
> >> > Hey Mayya,
> >> >
> >> > My "fix" on branch_8x is already present in 8.9: see
> >> > c461c506ffc02d4f3d16c7a0b0ec4250ba79fb7d from June 2nd.  Evidently
> >> > there's still other problems though.
> >> >
> >> > I'll look into this on my end for sure.  Do you happen to have the
> >> > command that reproduced this for you?  Or understand what "sheisty"
> >> > means here haha?
> >> >
> >> > Jason
> >> >
> >> > On Mon, Jun 7, 2021 at 1:12 PM Mayya Sharipova
> >> >  wrote:
> >> > >
> >> > > Hello Jason,
> >> > > thanks for the update.
> >> > >
> >> > > While trying to build a release candidate and doing  the smoke test, I 
> >> > > am getting the following error:
> >> > > RuntimeError: JAR file 
> >> > > "../.lucene-releases/8.9.0/RC1/smoketest/unpack/solr-8.9.0/contrib/gcs-repository/lib/javax.annotation-api-1.3.2.jar"
> >> > >  contains sheisty class 
> >> > > "javax/annotation/sql/DataSourceDefinitions.class"
> >> > >
> >> > > I guess we would need to backport your changes to branch_8_9 as well. 
> >> > > Would it be possible for you to do this?
> >> > >
> >> > > I will look into how to fix another error about "Future release 8.9.0 
> >> > > is greater than 8.10.0".
> >> > >
> >> > > On Thu, Jun 3, 2021 at 7:54 AM Jason Gerlowski  
> >> > > wrote:
> >> > >>
> >> > >> I pushed a commit to branch_8x yesterday that I expect has fixed this.
> >> > >>

Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - Still Failing!

[Jason Gerlowski]

Hey all,

Quick update: I've removed the javax.annotation dependency declaration
that the build didn't like.  Precommit and tests pass.  I've pushed
the fix to branch_8_9.  I'm currently running buildAndPushRelease.py +
smoke-tester and will give the all-clear once I see that pass.

Sorry to hold things up on the release!

Jason

On Tue, Jun 8, 2021 at 4:17 AM Christine Poerschke (BLOOMBERG/ LONDON)
 wrote:
>
> I've opened https://github.com/apache/lucene-solr/pull/2509 for the v8.10.0 > 
> v8.9.0 comparison issue. Hope that helps.
>
> From: dev@lucene.apache.org At: 06/08/21 01:28:25 UTC+1:00
> To: dev@lucene.apache.org
> Subject: Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - 
> Still Failing!
>
> Thanks Robert for the explanation.
>
> Jason, thank you for your reply and investigations.
>
> > Do you happen to have the command that reproduced this for you?
> You can see the build failures in Jenkins.  The last 3 failures were after 
> June 2nd, and all contained similar failures about sheisty classes in solr.
>
> About specific RC for the smoke-tester, I think you can first build a release 
> locally with buildAndPushRelease.py script, and then use this local release 
> for the smoke tester.
>
> I've also looked at the failure on 8.x branch: "Future release 8.9.0 is 
> greater than 8.10.0", and it looks like smokeTestRelease.py fails to 
> correctly compare v8.10.0 > v8.9.0, as (('8', '10', '0') < ('8', '9', '0')).
>
>
>
>
>
> On Mon, Jun 7, 2021 at 3:24 PM Robert Muir  wrote:
>>
>> Sheisty means classes that collide with jdk packages. E.g. this
>> javax.annotation looks like a problem, as it collides with existing
>> jdk package in an xml module:
>> https://docs.oracle.com/javase/8/docs/api/javax/annotation/package-summary.html
>>
>> Looks like the whole jar being used here is archived/deprecated [1] in
>> favor of "jakarta annotations" [2] which uses a new
>> non-colliding/jar-helling "jakarta.annotation" package/module:
>>
>> 1. https://github.com/javaee/javax.annotation/
>> 2. https://github.com/eclipse-ee4j/common-annotations-api
>>
>>
>>
>>
>> M
>>
>> On Mon, Jun 7, 2021 at 2:59 PM Jason Gerlowski  wrote:
>> >
>> > Hey Mayya,
>> >
>> > My "fix" on branch_8x is already present in 8.9: see
>> > c461c506ffc02d4f3d16c7a0b0ec4250ba79fb7d from June 2nd.  Evidently
>> > there's still other problems though.
>> >
>> > I'll look into this on my end for sure.  Do you happen to have the
>> > command that reproduced this for you?  Or understand what "sheisty"
>> > means here haha?
>> >
>> > Jason
>> >
>> > On Mon, Jun 7, 2021 at 1:12 PM Mayya Sharipova
>> >  wrote:
>> > >
>> > > Hello Jason,
>> > > thanks for the update.
>> > >
>> > > While trying to build a release candidate and doing  the smoke test, I 
>> > > am getting the following error:
>> > > RuntimeError: JAR file 
>> > > "../.lucene-releases/8.9.0/RC1/smoketest/unpack/solr-8.9.0/contrib/gcs-repository/lib/javax.annotation-api-1.3.2.jar"
>> > >  contains sheisty class 
>> > > "javax/annotation/sql/DataSourceDefinitions.class"
>> > >
>> > > I guess we would need to backport your changes to branch_8_9 as well. 
>> > > Would it be possible for you to do this?
>> > >
>> > > I will look into how to fix another error about "Future release 8.9.0 is 
>> > > greater than 8.10.0".
>> > >
>> > > On Thu, Jun 3, 2021 at 7:54 AM Jason Gerlowski  
>> > > wrote:
>> > >>
>> > >> I pushed a commit to branch_8x yesterday that I expect has fixed this.
>> > >> The 'generate-maven-artifacts' task now succeeds for me locally at
>> > >> least.
>> > >>
>> > >> The Jenkins job failed in the overnight run, but it looks to be a
>> > >> consequence of the release process that Mayya has in flight:
>> > >>
>> > >> "RuntimeError: Future release 8.9.0 is greater than 8.10.0 in
>> > >> file:///home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/smokeTestRelease/dist/lucene/changes/Changes.html"
>> > >>
>> > >> If anyone sees the "gcs-repository" related error message crop up
>> > >> anywhere else, please let me know.
>> > >>
>> > >> Jason
&

Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - Still Failing!

[Jason Gerlowski]

> Do you happen to have the command that reproduced this for you?

To clarify, I know how to run the smoke-tester generally, but every
time I've run it in the past I've had a link to a specific RC, and I'm
not sure what to do without that.

Jason

On Mon, Jun 7, 2021 at 2:59 PM Jason Gerlowski  wrote:
>
> Hey Mayya,
>
> My "fix" on branch_8x is already present in 8.9: see
> c461c506ffc02d4f3d16c7a0b0ec4250ba79fb7d from June 2nd.  Evidently
> there's still other problems though.
>
> I'll look into this on my end for sure.  Do you happen to have the
> command that reproduced this for you?  Or understand what "sheisty"
> means here haha?
>
> Jason
>
> On Mon, Jun 7, 2021 at 1:12 PM Mayya Sharipova
>  wrote:
> >
> > Hello Jason,
> > thanks for the update.
> >
> > While trying to build a release candidate and doing  the smoke test, I am 
> > getting the following error:
> > RuntimeError: JAR file 
> > "../.lucene-releases/8.9.0/RC1/smoketest/unpack/solr-8.9.0/contrib/gcs-repository/lib/javax.annotation-api-1.3.2.jar"
> >  contains sheisty class "javax/annotation/sql/DataSourceDefinitions.class"
> >
> > I guess we would need to backport your changes to branch_8_9 as well. Would 
> > it be possible for you to do this?
> >
> > I will look into how to fix another error about "Future release 8.9.0 is 
> > greater than 8.10.0".
> >
> > On Thu, Jun 3, 2021 at 7:54 AM Jason Gerlowski  
> > wrote:
> >>
> >> I pushed a commit to branch_8x yesterday that I expect has fixed this.
> >> The 'generate-maven-artifacts' task now succeeds for me locally at
> >> least.
> >>
> >> The Jenkins job failed in the overnight run, but it looks to be a
> >> consequence of the release process that Mayya has in flight:
> >>
> >> "RuntimeError: Future release 8.9.0 is greater than 8.10.0 in
> >> file:///home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/smokeTestRelease/dist/lucene/changes/Changes.html"
> >>
> >> If anyone sees the "gcs-repository" related error message crop up
> >> anywhere else, please let me know.
> >>
> >> Jason
> >>
> >> On Tue, Jun 1, 2021 at 7:55 AM Jason Gerlowski  
> >> wrote:
> >> >
> >> > Hey all,
> >> >
> >> > This is my fault.  Will look at fixing it this morning.  Sorry for the
> >> > disruption!
> >> >
> >> > If this is blocking anyone, let me know and I'll revert the offending
> >> > commit while I investigate the cause.  Otherwise I'll just leave it
> >> > as-is and push to have a fix as soon as I can.
> >> >
> >> > Jason
> >> >
> >> > On Sat, May 29, 2021 at 8:35 PM Robert Muir  wrote:
> >> > >
> >> > > The latest 8.x failure seems to be related to POM files from solr gcs
> >> > > repository. I don't currently know what is needed to move this along.
> >> > >
> >> > > On Sat, May 29, 2021 at 8:28 PM Apache Jenkins Server
> >> > >  wrote:
> >> > > >
> >> > > > Build: 
> >> > > > https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-SmokeRelease-8.x/220/
> >> > > >
> >> > > > No tests ran.
> >> > > >
> >> > > > Build Log:
> >> > > > [...truncated 28456 lines...]
> >> > > > BUILD FAILED
> >> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/build.xml:431:
> >> > > >  The following error occurred while executing this line:
> >> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:685:
> >> > > >  The following error occurred while executing this line:
> >> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:664:
> >> > > >  The following error occurred while executing this line:
> >> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/common-build.xml:471:
> >> > > >  The following error occurred while executing this line:
> >> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:1767:
> >> > > >  The following error occurred w

Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - Still Failing!

[Jason Gerlowski]

Hey Mayya,

My "fix" on branch_8x is already present in 8.9: see
c461c506ffc02d4f3d16c7a0b0ec4250ba79fb7d from June 2nd.  Evidently
there's still other problems though.

I'll look into this on my end for sure.  Do you happen to have the
command that reproduced this for you?  Or understand what "sheisty"
means here haha?

Jason

On Mon, Jun 7, 2021 at 1:12 PM Mayya Sharipova
 wrote:
>
> Hello Jason,
> thanks for the update.
>
> While trying to build a release candidate and doing  the smoke test, I am 
> getting the following error:
> RuntimeError: JAR file 
> "../.lucene-releases/8.9.0/RC1/smoketest/unpack/solr-8.9.0/contrib/gcs-repository/lib/javax.annotation-api-1.3.2.jar"
>  contains sheisty class "javax/annotation/sql/DataSourceDefinitions.class"
>
> I guess we would need to backport your changes to branch_8_9 as well. Would 
> it be possible for you to do this?
>
> I will look into how to fix another error about "Future release 8.9.0 is 
> greater than 8.10.0".
>
> On Thu, Jun 3, 2021 at 7:54 AM Jason Gerlowski  wrote:
>>
>> I pushed a commit to branch_8x yesterday that I expect has fixed this.
>> The 'generate-maven-artifacts' task now succeeds for me locally at
>> least.
>>
>> The Jenkins job failed in the overnight run, but it looks to be a
>> consequence of the release process that Mayya has in flight:
>>
>> "RuntimeError: Future release 8.9.0 is greater than 8.10.0 in
>> file:///home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/smokeTestRelease/dist/lucene/changes/Changes.html"
>>
>> If anyone sees the "gcs-repository" related error message crop up
>> anywhere else, please let me know.
>>
>> Jason
>>
>> On Tue, Jun 1, 2021 at 7:55 AM Jason Gerlowski  wrote:
>> >
>> > Hey all,
>> >
>> > This is my fault.  Will look at fixing it this morning.  Sorry for the
>> > disruption!
>> >
>> > If this is blocking anyone, let me know and I'll revert the offending
>> > commit while I investigate the cause.  Otherwise I'll just leave it
>> > as-is and push to have a fix as soon as I can.
>> >
>> > Jason
>> >
>> > On Sat, May 29, 2021 at 8:35 PM Robert Muir  wrote:
>> > >
>> > > The latest 8.x failure seems to be related to POM files from solr gcs
>> > > repository. I don't currently know what is needed to move this along.
>> > >
>> > > On Sat, May 29, 2021 at 8:28 PM Apache Jenkins Server
>> > >  wrote:
>> > > >
>> > > > Build: 
>> > > > https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-SmokeRelease-8.x/220/
>> > > >
>> > > > No tests ran.
>> > > >
>> > > > Build Log:
>> > > > [...truncated 28456 lines...]
>> > > > BUILD FAILED
>> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/build.xml:431:
>> > > >  The following error occurred while executing this line:
>> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:685:
>> > > >  The following error occurred while executing this line:
>> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:664:
>> > > >  The following error occurred while executing this line:
>> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/common-build.xml:471:
>> > > >  The following error occurred while executing this line:
>> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:1767:
>> > > >  The following error occurred while executing this line:
>> > > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:682:
>> > > >  Unable to initialize POM pom.xml: Could not find the model file 
>> > > > '/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/poms/solr/contrib/gcs-repository/pom.xml'.
>> > > >  for project unknown
>> > > >
>> > > > Total time: 7 minutes 13 seconds
>> > > > Build step 'Invoke Ant' marked build as failure
>> > > > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
>> > > > Email was triggered for: Failure - Any
>> > > > Sending email for trigger: Failure - Any
>> > > > Setting JD

Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - Still Failing!

[Jason Gerlowski]

I pushed a commit to branch_8x yesterday that I expect has fixed this.
The 'generate-maven-artifacts' task now succeeds for me locally at
least.

The Jenkins job failed in the overnight run, but it looks to be a
consequence of the release process that Mayya has in flight:

"RuntimeError: Future release 8.9.0 is greater than 8.10.0 in
file:///home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/smokeTestRelease/dist/lucene/changes/Changes.html"

If anyone sees the "gcs-repository" related error message crop up
anywhere else, please let me know.

Jason

On Tue, Jun 1, 2021 at 7:55 AM Jason Gerlowski  wrote:
>
> Hey all,
>
> This is my fault.  Will look at fixing it this morning.  Sorry for the
> disruption!
>
> If this is blocking anyone, let me know and I'll revert the offending
> commit while I investigate the cause.  Otherwise I'll just leave it
> as-is and push to have a fix as soon as I can.
>
> Jason
>
> On Sat, May 29, 2021 at 8:35 PM Robert Muir  wrote:
> >
> > The latest 8.x failure seems to be related to POM files from solr gcs
> > repository. I don't currently know what is needed to move this along.
> >
> > On Sat, May 29, 2021 at 8:28 PM Apache Jenkins Server
> >  wrote:
> > >
> > > Build: 
> > > https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-SmokeRelease-8.x/220/
> > >
> > > No tests ran.
> > >
> > > Build Log:
> > > [...truncated 28456 lines...]
> > > BUILD FAILED
> > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/build.xml:431:
> > >  The following error occurred while executing this line:
> > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:685:
> > >  The following error occurred while executing this line:
> > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:664:
> > >  The following error occurred while executing this line:
> > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/common-build.xml:471:
> > >  The following error occurred while executing this line:
> > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:1767:
> > >  The following error occurred while executing this line:
> > > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:682:
> > >  Unable to initialize POM pom.xml: Could not find the model file 
> > > '/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/poms/solr/contrib/gcs-repository/pom.xml'.
> > >  for project unknown
> > >
> > > Total time: 7 minutes 13 seconds
> > > Build step 'Invoke Ant' marked build as failure
> > > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > > Email was triggered for: Failure - Any
> > > Sending email for trigger: Failure - Any
> > > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene » Lucene-Solr-SmokeRelease-8.x - Build # 220 - Still Failing!

[Jason Gerlowski]

Hey all,

This is my fault.  Will look at fixing it this morning.  Sorry for the
disruption!

If this is blocking anyone, let me know and I'll revert the offending
commit while I investigate the cause.  Otherwise I'll just leave it
as-is and push to have a fix as soon as I can.

Jason

On Sat, May 29, 2021 at 8:35 PM Robert Muir  wrote:
>
> The latest 8.x failure seems to be related to POM files from solr gcs
> repository. I don't currently know what is needed to move this along.
>
> On Sat, May 29, 2021 at 8:28 PM Apache Jenkins Server
>  wrote:
> >
> > Build: 
> > https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-SmokeRelease-8.x/220/
> >
> > No tests ran.
> >
> > Build Log:
> > [...truncated 28456 lines...]
> > BUILD FAILED
> > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/build.xml:431:
> >  The following error occurred while executing this line:
> > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:685:
> >  The following error occurred while executing this line:
> > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/build.xml:664:
> >  The following error occurred while executing this line:
> > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/solr/common-build.xml:471:
> >  The following error occurred while executing this line:
> > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:1767:
> >  The following error occurred while executing this line:
> > /home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/common-build.xml:682:
> >  Unable to initialize POM pom.xml: Could not find the model file 
> > '/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-Solr-SmokeRelease-8.x/lucene/build/poms/solr/contrib/gcs-repository/pom.xml'.
> >  for project unknown
> >
> > Total time: 7 minutes 13 seconds
> > Build step 'Invoke Ant' marked build as failure
> > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > Email was triggered for: Failure - Any
> > Sending email for trigger: Failure - Any
> > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
> > Setting JDK_1_9_LATEST_HOME=/home/jenkins/tools/java/latest1.9
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Review request - New Solr website

[Jason Gerlowski]

The Solr website looks great, thanks for doing this Jan. +1 from me.

Jason

On Wed, Mar 3, 2021 at 7:23 AM Jan Høydahl  wrote:
>
> Privacy policy on the web sites (I copied Lucene's to the new Solr site)
>
> https://lucene.apache.org/privacy.html
> https://lucene-solrtlp.staged.apache.org/privacy.html
>
> Jan
>
>
> 3. mar. 2021 kl. 09:49 skrev Jan Høydahl :
>
> ASF dont want us to use any user tracking - they tell us to get stats from 
> INFRA based on httpd logs. But they currently don't track downloads (i.e 
> redirects to mirrors).
> So I don't expect any ASF help.
>
> I'm setting up a GA account for Solr and will invite any PMC member to manage 
> that account.
> I have sent an email to check whether anyone known anything about the 
> existing Lucene account.
>
> Jan
>
> 3. mar. 2021 kl. 09:33 skrev Anshum Gupta :
>
> Is there an ASF GA account we could use?
>
> On Wed, Mar 3, 2021 at 12:06 AM Jan Høydahl  wrote:
>>
>> You are right, we still use the Lucene GA code in there, which noone knows 
>> anything about.
>> I should probably switch GA code on Solr side to a new GA account?
>>
>> Jan
>>
>> > 2. mar. 2021 kl. 23:14 skrev Alexandre Rafalovitch :
>> >
>> > Did we get to any consensus on Google Analytics (or other) tracking?
>> > Would have been nice to track the moment of split.
>> >
>> > Regards,
>> >   Alex.
>> >
>> > On Tue, 2 Mar 2021 at 16:44, Jan Høydahl  wrote:
>> >>
>> >> We fixed JavaDoc and RefGuide on Solr side 
>> >> (https://issues.apache.org/jira/browse/SOLR-15177) and redirects on 
>> >> Lucene side (https://issues.apache.org/jira/browse/SOLR-15171) -- 
>> >> hopefully.
>> >>
>> >> The Lucene redirects can be tested right now, e.g. 
>> >> https://lucene-new.staged.apache.org/solr/guide/8_8/ redirects to 
>> >> https://solr.apache.org/guide/8_8/ which currently does not exist but 
>> >> will once we publish.
>> >>
>> >> I plan to depoy the new Solr site tomorrow, Wednesday. And then 
>> >> afterwards the new Lucene site.
>> >> After the deploy we can do rapid commits to fix whatever comes up.
>> >>
>> >> Feel free to continue to comment here, or simply commit fixes to the 
>> >> main/solr and main/lucene branches at 
>> >> https://github.com/apache/lucene-site
>> >>
>> >> Jan
>> >>
>> >> 2. mar. 2021 kl. 20:16 skrev Uwe Schindler :
>> >>
>> >> Hi,
>> >>
>> >> Javadocs and refguide do not work in staging sites. They can only be seen 
>> >> on production servers. We can’t test those, but I am confident, Jan’s new 
>> >> htaccess rules will work (they point to the special subversion production 
>> >> repo, still shared by lucene/solr for the time being).
>> >>
>> >> Uwe
>> >>
>> >> -
>> >> Uwe Schindler
>> >> Achterdiek 19, D-28357 Bremen
>> >> https://www.thetaphi.de
>> >> eMail: u...@thetaphi.de
>> >>
>> >> From: Jan Høydahl 
>> >> Sent: Tuesday, March 2, 2021 7:25 PM
>> >> To: dev@lucene.apache.org
>> >> Subject: Re: Review request - New Solr website
>> >>
>> >> Thanks for reviewing. Yes, the javadocs and refguide are not part of the 
>> >> site git repo, but will materialize through .htaccess rules once the site 
>> >> is published to the final location.
>> >>
>> >> Jan
>> >>
>> >>
>> >> 2. mar. 2021 kl. 17:17 skrev Michael McCandless 
>> >> :
>> >>
>> >> Thank you Jan!  I clicked around a bit and it looks awesome, but I hit a 
>> >> broken link for the Solr javadocs "Latest Release" link on this page: 
>> >> https://lucene-solrtlp.staged.apache.org/resources.html -- likely this is 
>> >> not an issue because
>> >> once this is moved to the right location, that link should "just work" 
>> >> maybe?
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >>
>> >> On Tue, Mar 2, 2021 at 10:32 AM Jan Høydahl  wrote:
>> >>
>> >> On the new "Project" page 
>> >> (https://lucene-solrtlp.staged.apache.org/whoweare.html) I have attempted 
>> >> to make some kind of TLP landing page, framing "what" and "who we are".
>> >>
>> >> The path I have taken so far is, instead of duplicating a list of names, 
>> >> which will get out of date, instead link to official roster at 
>> >> https://projects.apache.org/committee.html?solr for the authoritative 
>> >> list of PMC and committers. It will steal away the opportunity for new 
>> >> committers to have something to commit at day 1, but I can live with that.
>> >>
>> >> The missing piece is the "Emeritus" list. So let me throw out some 
>> >> questions:
>> >> * The ASF does not operate with emeritus PMC or committers. Do we need to?
>> >> * If we want to stick to a notion of "emeritus" committers, what should 
>> >> the initial list of emeritus Solr committers be?
>> >>
>> >> Please also proof-read that entire page, it is brand new and English is 
>> >> not my 1st language.
>> >>
>> >> Jan
>> >>
>> >>> 1. mar. 2021 kl. 09:56 skrev Jan Høydahl :
>> >>>
>> >>> Hi,
>> >>>
>> >>> I have been working on https://issues.apache.org/jira/browse/SOLR-14499 
>> >>> to prepare the separate website for Solr.
>> >>> I believe the work 

Re: Revisiting Standardized Test Names in Solr

[Jason Gerlowski]

> I hope that doesn’t sound too negative

Not to me.  But I'm a little confused what your ultimate stand is on
these renames Marcus is proposing.  I'm hearing different messages in
different sections of your email.

> There are already so many conflicts, you will cry and then realize there are 
> more.

Sounds very much like you're saying that the test renames will cause
really painful merge-conflicts, and that renames should wait because
of the pain involved in reconciling ref_impl.

But...

> You can’t let a specter freeze the tireless day to day shifting and shuffling 
> of names and rules and locations.

Sounds like you're saying that we shouldn't let fear of ref_impl
complications stop us from doing renames, file-moves, etc.

Sorry if I'm just being daft, but can you clarify please?  Are you
saying that we should avoid big changes because of the ugly merges
with ref_impl?  Or that we shouldn't let fear of ref_impl
complications stop us from anything on master?  Or something else
altogether?

Best,

Jason

On Fri, Feb 26, 2021 at 7:50 AM Mark Miller  wrote:
>
> I hope that doesn’t sound too negative, “clinging” never sounds as positive 
> as I’d like and I do negative plenty well without doing it by accident. Not a 
> pessimistic statement though, I made it even better than I was planning or 
> remembering I could or however that works. Resistance is built into the 
> equation - this isn’t rock and roll, I’m a science bachelor. Though only a 
> small few liberal arts classes made me go, so I wouldn’t trust the cert 
> myself. Anyway, I learned from multiple Star Wars movies what to do here, you 
> have to setup an ambush on the trench run and then just make the thing look 
> like a huge black star.
>
> On Fri, Feb 26, 2021 at 4:38 AM Mark Miller  wrote:
>>
>> There are already so many conflicts, you will cry and then realize there are 
>> more. Even worse, some things have been changed due to their cost/benefit 
>> failings, things that someone, somewhere, will cling to like a life vest.
>>
>> The ref branch waits for no man, and expects the same.
>>
>> It lives on ridiculous speed and stability and throws mergability to the 
>> crows.
>>
>> It could not be merged into anything and survive, but it can absorb 
>> anything, as long as it behaves like a boss or can be jostled into doing so. 
>> So fear not for the fearless. You can’t let a specter freeze the tireless 
>> day to day shifting and shuffling of names and rules and locations. I swear, 
>> enough lucky shifts and this thing can rise to meet the living. I’ve seen it 
>> see dead people.
>>
>> End of the day, if the ref branch can’t survive even a large and lengthy 
>> divergence, if that is the freeze in its tracks, it’s not at all what I’ve 
>> said ive been working on and so does it even matter?
>>
>>
>> On Mon, Feb 22, 2021 at 9:39 AM Jason Gerlowski  
>> wrote:
>>>
>>> I'm fine with standardization, whichever convention we choose.  I have
>>> a slight preference for FooTest, for the same reason Gus mentioned,
>>> but any standard is better than none here IMO.
>>>
>>> > prefer that we not make a sweeping change like this until after Mark's 
>>> > "ref branch" is reconciled
>>>
>>> Personally I disagree about the need to wait.  It'd be one thing if
>>> there was an agreed-upon plan or a timeframe for merging "ref-branch".
>>> But since that's not the case today, I don't think it makes sense to
>>> ignore concrete/mergeable improvements.  It seems like a "bird in the
>>> hand vs two in the bush" situation.  Especially when there are
>>> strategies for handling the conflicts that might arise with Mark's
>>> "ref-branch" (e.g. do the test renames on both master and ref_impl).
>>>
>>> Jason
>>>
>>> On Sun, Feb 21, 2021 at 12:44 PM David Smiley  wrote:
>>> >
>>> > I look forward to a standardization on *something* but would prefer that 
>>> > we not make a sweeping change like this until after Mark's "ref branch" 
>>> > is reconciled.  I don't want that to hang over the project indefinitely, 
>>> > but we can wait; we've not had this standardization yet for many years, 
>>> > after all.
>>> >
>>> > That said, it would be good to choose the standard name now so that there 
>>> > is less to change later.  Can someone dig up the statistics on Solr's 
>>> > name choice to see if there is a clear winner (e.g. >60%)?  I don't have 
>>> > a strong opinion on whatever the standard should be so long as there 

Re: Revisiting Standardized Test Names in Solr

[Jason Gerlowski]

I'm fine with standardization, whichever convention we choose.  I have
a slight preference for FooTest, for the same reason Gus mentioned,
but any standard is better than none here IMO.

> prefer that we not make a sweeping change like this until after Mark's "ref 
> branch" is reconciled

Personally I disagree about the need to wait.  It'd be one thing if
there was an agreed-upon plan or a timeframe for merging "ref-branch".
But since that's not the case today, I don't think it makes sense to
ignore concrete/mergeable improvements.  It seems like a "bird in the
hand vs two in the bush" situation.  Especially when there are
strategies for handling the conflicts that might arise with Mark's
"ref-branch" (e.g. do the test renames on both master and ref_impl).

Jason

On Sun, Feb 21, 2021 at 12:44 PM David Smiley  wrote:
>
> I look forward to a standardization on *something* but would prefer that we 
> not make a sweeping change like this until after Mark's "ref branch" is 
> reconciled.  I don't want that to hang over the project indefinitely, but we 
> can wait; we've not had this standardization yet for many years, after all.
>
> That said, it would be good to choose the standard name now so that there is 
> less to change later.  Can someone dig up the statistics on Solr's name 
> choice to see if there is a clear winner (e.g. >60%)?  I don't have a strong 
> opinion on whatever the standard should be so long as there is a standard :-)
>
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Sun, Feb 21, 2021 at 12:18 PM Gus Heck  wrote:
>>
>> FWIW, I'm not really in favor of the convention Lucene adopted. I probably 
>> lost track of the debate and failed to object which is on me, but I guess it 
>> was because that was the lower number of changes there? It's certainly much 
>> less legible in the IDE to have a wall of classes all starting with T. Maybe 
>> given that the projects are splitting Solr can Stick with FooTest not 
>> TestFoo? I think *Test suffix is more common in Solr... (though I haven't 
>> attempted to quantify it)
>>
>> On Sun, Feb 21, 2021 at 12:05 PM Eric Pugh  
>> wrote:
>>>
>>> Makes sense to me.
>>>
>>>
>>> On Feb 20, 2021, at 2:42 PM, Marcus Eagan  wrote:
>>>
>>> Hi all,
>>>
>>> Now that Lucene’s standardization is complete and I believe enforced, 
>>> should we discuss if we could bring the same consistency to Solr?
>>>
>>> Best,
>>>
>>> Marcus
>>> --
>>> Marcus Eagan
>>>
>>>
>>> ___
>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
>>> http://www.opensourceconnections.com | My Free/Busy
>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> This e-mail and all contents, including attachments, is considered to be 
>>> Company Confidential unless explicitly stated otherwise, regardless of 
>>> whether attachments are marked as such.
>>>
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

[Jason Gerlowski]

Congrats!

On Fri, Feb 19, 2021 at 10:06 AM Divye  wrote:
>
> Congratulations Jan!
>
> Regards,
> Divye
>
> On Fri, 19 Feb, 2021, 00:26 Anshum Gupta,  wrote:
>
> > Hi everyone,
> >
> > I’d like to inform everyone that the newly formed Apache Solr PMC nominated
> > and elected Jan Høydahl for the position of the Solr PMC Chair and Vice
> > President. This decision was approved by the board in its February 2021
> > meeting.
> >
> > Congratulations Jan!
> >
> > --
> > Anshum Gupta
> >

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Separate git repo(s) for Solr modules

[Jason Gerlowski]

> Shoving such a component into lucene-solr repo makes no sense, given its 
> branching strategy is based on master / branch_8x

I get how this could cause issues - I def hadn't thought much about
multi-version support and branching.  But I don't think moving plugins
to a separate repo solves that problem for us.  If our first class
plugins are set up to have different release "lines" that don't line
up with major Solr versions, it's only a matter of time before two of
those plugins have branch requirements that conflict.  Unless I'm
missing something here?

> I'd prefer that a module only leave our "contribs" area when the 
> concerns/limitations become real.  Doing it prematurely could lead to atrophy 
> of the module

+1 to David's comment.   I def hadn't considered the branching-scheme
issues that Ishan brought up in his last reply, and they seem like
valid concerns to me.  But the risk and the downsides of "atrophy" are
serious enough that I'd vote to not risk them until this starts to
cause us issues in practice.  Even if, for now, that means we won't be
able to build a single plugin jar that supports (e.g.) 3 major Solr
versions.  IMO that's a much smaller loss.

On Tue, Feb 16, 2021 at 9:40 AM David Smiley  wrote:
>
> On Tue, Feb 16, 2021 at 8:38 AM Eric Pugh  
> wrote:
>>
>> Testing across multiple versions is always very difficult ;-).  I recently 
>> saw this very interesting approach to using our Dockerized Solr’s to test a 
>> component against a number of previous versions of Solr.  
>> https://github.com/querqy/querqy/pull/147. I’m hopeful it could be a model 
>> for other packages to adopt.
>
>
> Thanks for the link to that Querqy PR.  That is *very* similar to what I do 
> at work (minus multi-version testing), and also similar to how I test 
> multiple versions in one of my pet projects by using a CI build matrix of a 
> configurable dependency.  I didn't know Testcontainer.org has its own Solr 
> module -- https://www.testcontainers.org/modules/solr/ but we implemented 
> that ourselves; not hard.
>
>>
>> Trying to maintain across multiple versions is kind of a Sisyphean task, and 
>> I don’t think plays to open source communities strengths.   It’s hard enough 
>> to keep all components of Solr up to date with the latest Lucene and the 
>> latest Solr….  (Try out wt=xlsx Response Writer, it’s currently broken on 
>> master) .  Reminds me of the Apache Gump project ;-)
>>
>> If something is really going to be backcompatible across certain versions, 
>> then maybe having it’s own repo makes sense,
>
>
> I'd prefer that a module only leave our "contribs" area when the 
> concerns/limitations become real.  Doing it prematurely could lead to atrophy 
> of the module
>
>>
>> but I suspect it means those components may go stale….   Look at DIH and 
>> Velocity components that are moved over to their own repos, both are getting 
>> stale, and I’d argue it’s because they don’t live in the main project where 
>> all of us have oversight and easy access.
>
>
> Agreed :-(
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Separate git repo(s) for Solr modules

[Jason Gerlowski]

> separate repo can be used to test against many Solr versions. Imagine a 
> component that works across Solr versions 6x through 9x from day one. I can't 
> imagine such a component being part of the lucene-solr repo itself.

It'll be great to test contrib's on multiple Solr versions, but I
can't see any blockers for this that are same-repo specific.  Can you
elaborate on what makes this tricky in a same-repo setup?  AFAICT you
can set up multiple gradle test tasks that rely on different
solr-test-framework versions and run tests that way - regardless of
where the code lives.   (Or alternatively, docker images for various
Solr versions as David suggested.)

I think Jan's got a good point about how do-able it'll be in practice
to have plugins support multiple major release versions.  That said
though, a lot of these questions (docker tests vs otherwise, cross
major-version plugins?) will likely sort themselves out when it comes
time to implement.

On Tue, Feb 16, 2021 at 12:21 AM David Smiley  wrote:
>
> I think embracing Docker is how we can identify if a plugin works in multiple 
> versions of Solr.  The setup of such a test would start Solr via Docker and 
> install the plugin into a Solr server using the package manager.  The test 
> code itself would interact with Solr purely via SolrClient.  This would be an 
> integration / smoke level test for the plugin.  I test my plugins at work 
> similar to this way at a "smoke" level, which follows "integration".  What's 
> cool is that we can take the very same test and either have it run via an 
> embedded Solr (which we call an integration test) or a remote'ed Solr (run 
> via Docker, which we call our "smoke" test).  Naturally, if a test makes 
> assumptions that only work in embedded, then it won't run in smoke mode, so 
> we have annotations to categorize the tests.  I'd like to work on some JUnit 
> "Rule" for Solr, similar to what I have at work (and have done in the past) 
> -- https://issues.apache.org/jira/browse/SOLR-11872
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Feb 15, 2021 at 9:17 AM Ishan Chattopadhyaya 
>  wrote:
>>
>> Hoss,
>> One important reason why separate repo for solr-extras is a good idea, as 
>> opposed to conrib modules, is that separate repo can be used to test against 
>> many Solr versions. Imagine a component that works across Solr versions 6x 
>> through 9x from day one. I can't imagine such a component being part of the 
>> lucene-solr repo itself.
>>
>> Also, today, contrib modules are shipped with Solr, which we don't want for 
>> new things we might want to build.
>>
>> On Mon, Jan 25, 2021 at 4:25 PM Ishan Chattopadhyaya 
>>  wrote:
>>>
>>> I haven't been able to follow up on creation of the extras repo, but more 
>>> importantly I wanted to respond to Hoss. I'm out on an emergency for a week 
>>> or so, shall resume then. If there's a decision on this until then, I shall 
>>> accept it.
>>>
>>> On Mon, 25 Jan, 2021, 9:04 am Jason Gerlowski,  
>>> wrote:
>>>>
>>>> Tentative +1 to Hoss' questions.  I agree with his summary of the
>>>> potential risks here, and I share his ignorance of the perceived
>>>> benefits.
>>>>
>>>> (I thought for a time that this was driven by interest in having
>>>> release cadences independent from Solr-core releases.  I'm all for
>>>> that goal, but if that's the motivation I'm not sure what the obstacle
>>>> is to doing that with a single repo - all build systems these days
>>>> support versioning and releasing modules independent of one another.
>>>> But maybe that was never a driving factor here.)
>>>>
>>>> I know there have been a lot of discussions about this, and I know the
>>>> repo has already been created.  So maybe it's too late to object even
>>>> if I wanted to, which I'm not sure I do.  But can someone that
>>>> understands the motivation please summarize what multiple-repos gets
>>>> us over a single repo that outweighs the "cons" that Hoss mentioned?
>>>>
>>>> Jason
>>>>
>>>> On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
>>>>  wrote:
>>>> >
>>>> >
>>>> > : As we discussed over the last few months, there seems a need to move
>>>> > : non-core pieces away from the Solr core module. The contribs are 
>>>> > presently
>>>> > : a good place, but it makes sense to have a separate git repositor

Re: [VOTE] Release Lucene/Solr 8.8.0 RC2

[Jason Gerlowski]

I agree with Noble that we shouldn't let those flakies hold up the
release. It really sucks, but Solr wouldn't've gotten out a release in
years if that was our bar.

FWIW though I disagree with his finer point that we shouldn't worry
about autoscaling in particular because it's slated for removal in
9.0.  We're not releasing 9.0, we're releasing 8.8.  And there's
likely to be an 8.8.1 or 8.9 at some point later.  So there's
definitely a "point in trying to fix" those failures, generally
speaking.

Jason

On Thu, Jan 28, 2021 at 6:01 PM Noble Paul  wrote:
>
> I'm not sure why the tests fail. But, there is no point in trying to
> fix them if that code is removed from master and we do not plan to
> support them anyway
>
> On Fri, Jan 29, 2021 at 9:43 AM David Smiley  wrote:
> >
> > Ah; I see they appear to be 8x only.  Any idea what the story is with them? 
> >  Do you think it is just test flakiness or flakiness in Solr's related 
> > functionality?
> >
> > ~ David
> >
> >
> > On Thu, Jan 28, 2021 at 5:36 PM Noble Paul  wrote:
> >>
> >> Both of those are gone for good @David Smiley  I think we can safely
> >> make them BadApples
> >>
> >> On Fri, Jan 29, 2021 at 9:07 AM David Smiley  wrote:
> >> >
> >> > I ran it twice on my old-ish MacBook, and it failed each time on Solr 
> >> > tests that have a history of flakiness (as shown by Hossman's awesome 
> >> > fucit.org site) -- AutoscalingHistoryHandlerTest & TestWithCollection.  
> >> > Neither seed repeated.  I'll pass on voting.
> >> >
> >> > ~ David Smiley
> >> > Apache Lucene/Solr Search Developer
> >> > http://www.linkedin.com/in/davidwsmiley
> >> >
> >> >
> >> > On Thu, Jan 28, 2021 at 11:35 AM Cassandra Targett 
> >> >  wrote:
> >> >>
> >> >> I’ve updated the Ref Guide to remove the DRAFT status on 8.8 pages, so 
> >> >> that’s done. I’ll also take care of the next Ref Guide step to update 
> >> >> the website to show 8.8 as the latest.
> >> >> On Jan 28, 2021, 9:59 AM -0600, Tomás Fernández Löbbe 
> >> >> , wrote:
> >> >>
> >> >> It’s 9 binding: Noble, Tim, me, Ignacio, Mike M, Namgyu, Atri, Anshum 
> >> >> and Mike S.
> >> >>
> >> >> 1 non-binding: Haoyu
> >> >>
> >> >>
> >> >> On Thu, Jan 28, 2021 at 5:34 AM Michael Sokolov  
> >> >> wrote:
> >> >>>
> >> >>> SUCCESS! [0:58:25.213071]
> >> >>>
> >> >>> +1 better late than never?
> >> >>>
> >> >>> On Thu, Jan 28, 2021 at 8:04 AM Ishan Chattopadhyaya
> >> >>>  wrote:
> >> >>> >
> >> >>> > Thanks Noble!
> >> >>> >
> >> >>> > On Thu, 28 Jan, 2021, 4:24 pm Noble Paul,  
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> [+1]  9  (4 binding)
> >> >>> >>
> >> >>> >>  [0]  0
> >> >>> >>
> >> >>> >> [-1]  0
> >> >>> >>
> >> >>> >>
> >> >>> >> This vote has PASSED.
> >> >>> >>
> >> >>> >> I shall proceed with the rest of the release process
> >> >>> >>
> >> >>> >> On Thu, Jan 28, 2021 at 6:24 PM Anshum Gupta 
> >> >>> >>  wrote:
> >> >>> >> >
> >> >>> >> > Thanks for handing the release, Noble!
> >> >>> >> >
> >> >>> >> > +1 (binding)
> >> >>> >> >
> >> >>> >> > SUCCESS! [0:56:12.016387]
> >> >>> >> >
> >> >>> >> > Ran the smoke tester, a demo app, and checked the change log. All 
> >> >>> >> > of that looks good.
> >> >>> >> >
> >> >>> >> > On Mon, Jan 25, 2021 at 2:22 AM Noble Paul  
> >> >>> >> > wrote:
> >> >>> >> >>
> >> >>> >> >> Please vote for release candidate 2 for Lucene/Solr 8.8.0
> >> >>> >> >>
> >> >>> >> >> The artifacts can be downloaded from:
> >> >>> >> >>
> >> >>> >> >> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.0-RC2-revb10659f0fc18b58b90929cfdadde94544d202c4a/
> >> >>> >> >>
> >> >>> >> >> python3 -u dev-tools/scripts/smokeTestRelease.py \
> >> >>> >> >> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.0-RC2-revb10659f0fc18b58b90929cfdadde94544d202c4a/
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> The vote will be open for at least 72 hours
> >> >>> >> >>
> >> >>> >> >> [ ] +1  approve
> >> >>> >> >> [ ] +0  no opinion
> >> >>> >> >> [ ] -1  disapprove (and reason why)
> >> >>> >> >>
> >> >>> >> >> Here is my +1
> >> >>> >> >> --
> >> >>> >> >> -
> >> >>> >> >> Noble Paul
> >> >>> >> >>
> >> >>> >> >> -
> >> >>> >> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> >>> >> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >> >>> >> >>
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > --
> >> >>> >> > Anshum Gupta
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> -
> >> >>> >> Noble Paul
> >> >>> >>
> >> >>> >> -
> >> >>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> >>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >> >>> >>
> >> >>>
> >> >>> -
> >> >>> To unsubscribe, e-mail: 

Re: Separate git repo(s) for Solr modules

[Jason Gerlowski]

Tentative +1 to Hoss' questions.  I agree with his summary of the
potential risks here, and I share his ignorance of the perceived
benefits.

(I thought for a time that this was driven by interest in having
release cadences independent from Solr-core releases.  I'm all for
that goal, but if that's the motivation I'm not sure what the obstacle
is to doing that with a single repo - all build systems these days
support versioning and releasing modules independent of one another.
But maybe that was never a driving factor here.)

I know there have been a lot of discussions about this, and I know the
repo has already been created.  So maybe it's too late to object even
if I wanted to, which I'm not sure I do.  But can someone that
understands the motivation please summarize what multiple-repos gets
us over a single repo that outweighs the "cons" that Hoss mentioned?

Jason

On Thu, Jan 14, 2021 at 12:34 PM Chris Hostetter
 wrote:
>
>
> : As we discussed over the last few months, there seems a need to move
> : non-core pieces away from the Solr core module. The contribs are presently
> : a good place, but it makes sense to have a separate git repository hosting
> : such modules. Some candidates that come to mind are the present day contrib
>
> can you explain why it makes sense to have a separate git repo for these
> things?
>
> I can think of lots of reasons why it makes sense to have a single
> repo for all things solr (unified CI that quickly identifies if core
> changes break "first order" plugins, shared feature branches & monotomic
> commits of code that affects APIs and impls of those APIs, etc...) but
> I've yet to see any concrete specifics of why multiple git repos are
> "better" then just having distinct sub-projects (with distinct artifacts)
> in the same repo other then "it makes sense"
>
> why does it make sense?
>
> why can't the ideas of "solr-sandbox" and "solr-extras" just be
> directories in the "solr repo" ? ... what value is gained by making them
> new repos?
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Is it Time to Deprecate the Legacy Facets API

[Jason Gerlowski]

Personally I'd love to see us stop maintaining the duplicated code of
the underlying implementations.  I wouldn't mind losing the legacy
syntax as well - I'll take a clear, verbose API over a less-clear,
concise one any day.  But I'm probably a minority there.

Either way I agree with Michael when he said above that the first step
would have to be a parity investigation for features and performance.

Best,

Jason

On Fri, Jan 22, 2021 at 10:05 AM Michael Gibney
 wrote:
>
> I agree it would make long-term sense to consolidate the backend 
> implementation. I think leaving the "classic" user-facing facet API (with 
> JSON Facet module as a backend) would be a good idea. Either way, I think a 
> first step would be checking for parity between existing backend 
> implementations -- possibly in terms of features [1], but certainly in terms 
> of performance for common use cases [2].
>
> I think removal of the "classic" user-facing API would cause a lot of 
> consternation in the user community. I can even see a 
> non-backward-compatibility argument for preserving the "classic" user-facing 
> API: it's simpler for simple use cases. _If_ the ultimate goal is removal of 
> the "classic" user-facing API (not presuming that it is), that approach could 
> be facilitated in the short term by enticing users towards "JSON Facet" API 
> ... basically with a "feature freeze" on the legacy implementation. No new 
> features [3], no new optimizations [4] for "classic"; concentrate such 
> efforts on JSON Facet. This seems to already be the de facto case, but it 
> could be a more intentional decision -- e.g. in [3] it's straightforward to 
> extend the the proposed "facet cache" to the "classic" impl ... but I could 
> see an argument for intentionally not doing so.
>
> Robert, I think your concerns about UninvertedField could be addressed by the 
> `uninvertible="false"` property (currently defaults to "true" for backward 
> compatibility iiuc; but could default to "false", or at least provide the 
> ability to set the default for all fields to "false" at node level solr.xml? 
> -- I know I've wished for the latter!). Also fwiw I'm not aware of any JSON 
> Facet processors that work with string values in RAM ... I do think all JSON 
> Facet processors use OrdinalMap now, where relevant.
>
> [1] https://issues.apache.org/jira/browse/SOLR-14921
> [2] https://issues.apache.org/jira/browse/SOLR-14764
> [3] https://issues.apache.org/jira/browse/SOLR-13807
> [4] https://issues.apache.org/jira/browse/SOLR-10732
>
> On Fri, Jan 22, 2021 at 12:46 AM Robert Muir  wrote:
>>
>> Do these two options conflate concerns of input format vs. actual
>> algorithm? That was always my disappointment.
>>
>> I feel like the java apis are off here at the lower level, and it
>> hurts the user.
>> I don't talk about the input format from the user, instead I mean the
>> execution of the faceting query.
>>
>> IMO: building top-level caches (e.g. uninvertedfield) or
>> on-the-fly-caches (e.g. fieldcache) is totally trappy already.
>> But with the uninvertedfield of json facets it does its own thing,
>> even if you went thru the trouble to enable docvalues at index time:
>> that's sad.
>>
>> the code by default should not give the user jvm
>> heap/garbage-collector hell. If you want to do that to yourself, for a
>> totally static index, IMO that should be opt-in.
>>
>> But for the record, it is no longer just two shitty choices like
>> "top-level vs per-segment". There are different field types, e.g.
>> numeric types where the per-segment approach works efficiently.
>> Then you have the strings, but there is a newish middle ground for
>> Strings: OrdinalMap (lucene Multi* interfaces do it) which builds
>> top-level integers structures to speed up string-faceting, but doesnt
>> need *string values* in ram.
>> It is just integers and mostly compresses as deltas. Adrien compresses
>> the shit out of it.
>>
>> So I'd hate for the user to lose the option here of using docvalues to
>> keep faceting out of heap memory, which should not be hassling them
>> already in 2021.
>> Maybe better to refactor the code such that all these concerns aren't
>> unexpectedly tied together.
>>
>> On Thu, Jan 21, 2021 at 10:08 PM David Smiley  wrote:
>> >
>> > There's a JIRA issue about this from 5 years ago: 
>> > https://issues.apache.org/jira/browse/SOLR-7296
>> > I don't recall seeing any resistance to the idea of having the JSON 
>> > Faceting module act as a back-end to the front-end (API surface) of Solr's 
>> > common/classic/original/whatever faceting API.  I don't think that simple 
>> > API should go away; it's strength is simple/common cases that are 
>> > comparatively verbose in the JSON one.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>> >
>> > On Thu, Jan 21, 2021 at 9:57 PM Marcus Eagan  wrote:
>> >>
>> >> Hi all,
>> >>
>> >> Sorry to spam the list. I am querying the list in such quick 

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

> Announce your intention to proceed with lazy consensus in two business days.

I intend to proceed then. : )

I'm going to file JIRAs for the SIP today and start pushing up PRs.
Based on how this discussion went I also plan on updating the "SIP"
process page to say that lazy consensus can be used over a strict VOTE
if no one objects.

Thanks everyone for the attention and feedback.

Jason

On Fri, Jan 15, 2021 at 10:43 AM David Smiley  wrote:
>
> I think embrace lazy consensus -- no formal vote.  Announce your intention to 
> proceed with lazy consensus in two business days.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jan 15, 2021 at 9:32 AM Jason Gerlowski  wrote:
>>
>> Jan - I've modified the file-structure page to include support for
>> storing multiple collections within the same "location" per your
>> suggestion.
>>
>> I've also clarified the file-structure page to state that _all_ files
>> required to restore a shard are recorded in the shard-metadata file
>> (regardless of whether the file was uploaded by the current backup or
>> a past one).
>>
>> This DISCUSS thread has been open for a full month now, and it seems
>> like the feedback is winding down.  Pending any objections, I'd like
>> to move out of the DISCUSS phase and start implementing.  The "SIP
>> Process" page in Confluence mentions holding a VOTE thread for the
>> final proposal, but I can't find any examples of that actually being
>> done.  Is that a required part of the process, or is the process page
>> out of date?  IMO a VOTE seems slightly redundant, unless success on a
>> VOTE means that individual PRs can't be -1'd on design grounds?
>>
>> Either way I'm happy to do whatever the process requires here.  I'll
>> plan on starting the next step on Monday, whatever that needs to be.
>>
>> Best,
>>
>> Jason
>>
>> On Thu, Jan 14, 2021 at 1:10 PM Jason Gerlowski  
>> wrote:
>> >
>> > Yeah, that's a good point.  I think the only change we'd need to make
>> > to the file-structure to support multi-collection down the road would
>> > be to introduce a top-level directories for each collection.  I'll
>> > experiment with that and tweak the described file structure to handle
>> > that (assuming the testing pans out).
>> >
>> > Best,
>> >
>> > Jason
>> >
>> > On Thu, Jan 14, 2021 at 5:10 AM Jan Høydahl  wrote:
>> > >
>> > > > It's tough to decide between /v2/cluster/backups and
>> > > > /v2/collections//backups as alternatives until you
>> > > > figure out whether we currently support multi-collection backup, or
>> > > > want to in the near future.
>> > >
>> > > I suppose multi-collection / TRA support cold be expanded on later
>> > > by supporting e.g. collection= or alias=my-tra.
>> > >
>> > > However, the file layout chosen here dictates whether one named backup
>> > > will be capable of storing more than one collection in the future, so 
>> > > that's
>> > > perhaps something to consider. But if it gets too complicated we should
>> > > just delay it and redesign the storage structure once again when we cross
>> > > that bridge. I'll not veto the current suggestion.
>> > >
>> > > Jan
>> > >
>> > > > 12. jan. 2021 kl. 17:53 skrev Jason Gerlowski :
>> > > >
>> > > > Hey all,
>> > > >
>> > > > Two follow ups on recent discussion.
>> > > >
>> > > > I reviewed the gc/ref-counting part of the BlobDirectory proposal on
>> > > > SOLR-15051 that David mentioned.  We talked about it a bit offline and
>> > > > agreed that while an automatic gc mechanism is really needed for what
>> > > > he's trying to do, the requirements of the backup usecase are
>> > > > different enough that SIP-12 can get by with manually-triggered
>> > > > 'purging'.  Mostly because infrequent static backups produce much less
>> > > > garbage than continually tracking all files for a (possibly
>> > > > ever-changing) index.
>> > > >
>> > > >> I'd be open to creating a new v2 backup endpoint (without adding TRA, 
>> > > >> etc. compatibility) if there was consensus on that approach to 
>> > > >> handling backcompat and on the specific appearance of the API
>> >

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Jan - I've modified the file-structure page to include support for
storing multiple collections within the same "location" per your
suggestion.

I've also clarified the file-structure page to state that _all_ files
required to restore a shard are recorded in the shard-metadata file
(regardless of whether the file was uploaded by the current backup or
a past one).

This DISCUSS thread has been open for a full month now, and it seems
like the feedback is winding down.  Pending any objections, I'd like
to move out of the DISCUSS phase and start implementing.  The "SIP
Process" page in Confluence mentions holding a VOTE thread for the
final proposal, but I can't find any examples of that actually being
done.  Is that a required part of the process, or is the process page
out of date?  IMO a VOTE seems slightly redundant, unless success on a
VOTE means that individual PRs can't be -1'd on design grounds?

Either way I'm happy to do whatever the process requires here.  I'll
plan on starting the next step on Monday, whatever that needs to be.

Best,

Jason

On Thu, Jan 14, 2021 at 1:10 PM Jason Gerlowski  wrote:
>
> Yeah, that's a good point.  I think the only change we'd need to make
> to the file-structure to support multi-collection down the road would
> be to introduce a top-level directories for each collection.  I'll
> experiment with that and tweak the described file structure to handle
> that (assuming the testing pans out).
>
> Best,
>
> Jason
>
> On Thu, Jan 14, 2021 at 5:10 AM Jan Høydahl  wrote:
> >
> > > It's tough to decide between /v2/cluster/backups and
> > > /v2/collections//backups as alternatives until you
> > > figure out whether we currently support multi-collection backup, or
> > > want to in the near future.
> >
> > I suppose multi-collection / TRA support cold be expanded on later
> > by supporting e.g. collection= or alias=my-tra.
> >
> > However, the file layout chosen here dictates whether one named backup
> > will be capable of storing more than one collection in the future, so that's
> > perhaps something to consider. But if it gets too complicated we should
> > just delay it and redesign the storage structure once again when we cross
> > that bridge. I'll not veto the current suggestion.
> >
> > Jan
> >
> > > 12. jan. 2021 kl. 17:53 skrev Jason Gerlowski :
> > >
> > > Hey all,
> > >
> > > Two follow ups on recent discussion.
> > >
> > > I reviewed the gc/ref-counting part of the BlobDirectory proposal on
> > > SOLR-15051 that David mentioned.  We talked about it a bit offline and
> > > agreed that while an automatic gc mechanism is really needed for what
> > > he's trying to do, the requirements of the backup usecase are
> > > different enough that SIP-12 can get by with manually-triggered
> > > 'purging'.  Mostly because infrequent static backups produce much less
> > > garbage than continually tracking all files for a (possibly
> > > ever-changing) index.
> > >
> > >> I'd be open to creating a new v2 backup endpoint (without adding TRA, 
> > >> etc. compatibility) if there was consensus on that approach to handling 
> > >> backcompat and on the specific appearance of the API
> > >
> > > On second thought, I'm going to flip-flop on this.  Coming up with a
> > > better v2 API for backup/restore will be easier *after* some of the
> > > questions Jan raised (multi-collection? alias support? etc.) have been
> > > dealt with.  i.e. It's tough to decide between /v2/cluster/backups and
> > > /v2/collections//backups as alternatives until you
> > > figure out whether we currently support multi-collection backup, or
> > > want to in the near future.  If people feel strongly or would veto the
> > > vote otherwise, then I'll try my best.  But otherwise I think we're
> > > best served waiting until other stuff settles out to revisit larger v2
> > > backup API changes.
> > >
> > > Best,
> > >
> > > Jason
> > >
> > > On Mon, Jan 11, 2021 at 10:41 AM Jason Gerlowski  
> > > wrote:
> > >>
> > >> Hey all,
> > >>
> > >> I've put replies to everyone's questions below.  Hope they help!
> > >>
> > >>> Do the shard metadata files list all of the segments that make up the 
> > >>> backup, or only the segments that were uploaded in this incremental 
> > >>> update?
> > >>
> > >> Mike: The former - they're intended to hold metadata about all of the
> > >> segments that

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Yeah, that's a good point.  I think the only change we'd need to make
to the file-structure to support multi-collection down the road would
be to introduce a top-level directories for each collection.  I'll
experiment with that and tweak the described file structure to handle
that (assuming the testing pans out).

Best,

Jason

On Thu, Jan 14, 2021 at 5:10 AM Jan Høydahl  wrote:
>
> > It's tough to decide between /v2/cluster/backups and
> > /v2/collections//backups as alternatives until you
> > figure out whether we currently support multi-collection backup, or
> > want to in the near future.
>
> I suppose multi-collection / TRA support cold be expanded on later
> by supporting e.g. collection= or alias=my-tra.
>
> However, the file layout chosen here dictates whether one named backup
> will be capable of storing more than one collection in the future, so that's
> perhaps something to consider. But if it gets too complicated we should
> just delay it and redesign the storage structure once again when we cross
> that bridge. I'll not veto the current suggestion.
>
> Jan
>
> > 12. jan. 2021 kl. 17:53 skrev Jason Gerlowski :
> >
> > Hey all,
> >
> > Two follow ups on recent discussion.
> >
> > I reviewed the gc/ref-counting part of the BlobDirectory proposal on
> > SOLR-15051 that David mentioned.  We talked about it a bit offline and
> > agreed that while an automatic gc mechanism is really needed for what
> > he's trying to do, the requirements of the backup usecase are
> > different enough that SIP-12 can get by with manually-triggered
> > 'purging'.  Mostly because infrequent static backups produce much less
> > garbage than continually tracking all files for a (possibly
> > ever-changing) index.
> >
> >> I'd be open to creating a new v2 backup endpoint (without adding TRA, etc. 
> >> compatibility) if there was consensus on that approach to handling 
> >> backcompat and on the specific appearance of the API
> >
> > On second thought, I'm going to flip-flop on this.  Coming up with a
> > better v2 API for backup/restore will be easier *after* some of the
> > questions Jan raised (multi-collection? alias support? etc.) have been
> > dealt with.  i.e. It's tough to decide between /v2/cluster/backups and
> > /v2/collections//backups as alternatives until you
> > figure out whether we currently support multi-collection backup, or
> > want to in the near future.  If people feel strongly or would veto the
> > vote otherwise, then I'll try my best.  But otherwise I think we're
> > best served waiting until other stuff settles out to revisit larger v2
> > backup API changes.
> >
> > Best,
> >
> > Jason
> >
> > On Mon, Jan 11, 2021 at 10:41 AM Jason Gerlowski  
> > wrote:
> >>
> >> Hey all,
> >>
> >> I've put replies to everyone's questions below.  Hope they help!
> >>
> >>> Do the shard metadata files list all of the segments that make up the 
> >>> backup, or only the segments that were uploaded in this incremental 
> >>> update?
> >>
> >> Mike: The former - they're intended to hold metadata about all of the
> >> segments that are needed to restore to the given
> >> snapshot/commit-point.  So it's likely to hold metadata about files
> >> just uploaded, as well as ones that were added to the blob by previous
> >> backups.  I'll see if I can make that clearer in the file
> >> descriptions.
> >>
> >>> leave the old Backup/Restore API as-is, deprecated, and add a new one on 
> >>> /v2/cluster/backup
> >>
> >> Jan: Ultimately I agree with your concerns about scope, so I'd vote
> >> against trying to cover TRAs, multiple collection backups, etc. in
> >> this effort here.
> >>
> >> That aside though, I agree that the existing v2 backup API is a bit of
> >> a headscratcher.  Why is it /v2/collections instead of
> >> /v2/collections/ or a subpath of /v2/cluster?  Does it
> >> have something to do with aliases?  Or did it end up there mostly by
> >> default?  I'd be open to creating a new v2 backup endpoint (without
> >> adding TRA, etc. compatibility) if there was consensus on that
> >> approach to handling backcompat and on the specific appearance of the
> >> API.  It would help with backcompat after all.  Though if finding
> >> consensus bogs down it may not be worth the addition.
> >>
> >>> I know you've seen SOLR-15051 (Shared storage -- BlobDirectory) ... We 
> >>> both want to store checksums and f

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Hey all,

Two follow ups on recent discussion.

I reviewed the gc/ref-counting part of the BlobDirectory proposal on
SOLR-15051 that David mentioned.  We talked about it a bit offline and
agreed that while an automatic gc mechanism is really needed for what
he's trying to do, the requirements of the backup usecase are
different enough that SIP-12 can get by with manually-triggered
'purging'.  Mostly because infrequent static backups produce much less
garbage than continually tracking all files for a (possibly
ever-changing) index.

> I'd be open to creating a new v2 backup endpoint (without adding TRA, etc. 
> compatibility) if there was consensus on that approach to handling backcompat 
> and on the specific appearance of the API

On second thought, I'm going to flip-flop on this.  Coming up with a
better v2 API for backup/restore will be easier *after* some of the
questions Jan raised (multi-collection? alias support? etc.) have been
dealt with.  i.e. It's tough to decide between /v2/cluster/backups and
/v2/collections//backups as alternatives until you
figure out whether we currently support multi-collection backup, or
want to in the near future.  If people feel strongly or would veto the
vote otherwise, then I'll try my best.  But otherwise I think we're
best served waiting until other stuff settles out to revisit larger v2
backup API changes.

Best,

Jason

On Mon, Jan 11, 2021 at 10:41 AM Jason Gerlowski  wrote:
>
> Hey all,
>
> I've put replies to everyone's questions below.  Hope they help!
>
> > Do the shard metadata files list all of the segments that make up the 
> > backup, or only the segments that were uploaded in this incremental update?
>
> Mike: The former - they're intended to hold metadata about all of the
> segments that are needed to restore to the given
> snapshot/commit-point.  So it's likely to hold metadata about files
> just uploaded, as well as ones that were added to the blob by previous
> backups.  I'll see if I can make that clearer in the file
> descriptions.
>
> > leave the old Backup/Restore API as-is, deprecated, and add a new one on 
> > /v2/cluster/backup
>
> Jan: Ultimately I agree with your concerns about scope, so I'd vote
> against trying to cover TRAs, multiple collection backups, etc. in
> this effort here.
>
> That aside though, I agree that the existing v2 backup API is a bit of
> a headscratcher.  Why is it /v2/collections instead of
> /v2/collections/ or a subpath of /v2/cluster?  Does it
> have something to do with aliases?  Or did it end up there mostly by
> default?  I'd be open to creating a new v2 backup endpoint (without
> adding TRA, etc. compatibility) if there was consensus on that
> approach to handling backcompat and on the specific appearance of the
> API.  It would help with backcompat after all.  Though if finding
> consensus bogs down it may not be worth the addition.
>
> > I know you've seen SOLR-15051 (Shared storage -- BlobDirectory) ... We both 
> > want to store checksums and file lengths. ... Your proposal did not discuss 
> > how these files are GC'ed
>
> David: SIP-12 does address this, though maybe the writeup needs
> clarifying.  The Delete Backup API includes a "purge" parameter which
> triggers GC activity.  This probably works about the way you'd expect
> - Solr gets the list of UUID-named index files from the blob store,
> and then it compares that list to the set of UUID's referenced by any
> shard-metadata file (which requires reading all the shard-metadata
> files).  This avoids adding to Solr's ZK state, but does so at the
> cost of requiring users to trigger sporadic cleanup manually instead
> of detecting orphans automatically like BlobDirectory does (assuming I
> understand that correctly).
>
> I'm def not saying this is the best approach necessarily.  I like it,
> though it has downsides for sure.  Just that there is a proposed
> approach that's easy to miss buried in the SIP.
>
> More broadly though - I share your sense that we should consider
> alignment.  It may end up that Backup/Restore is different enough from
> the BlobDirectory usecase that it doesn't make sense, but it's at
> least worth figuring out.  That's about as far as my understanding
> goes right now though.  I'll read up on BlobDirectory while you absorb
> SIP-12 and maybe we can circle back to this shortly.
>
> Best,
>
> Jason
>
> On Sun, Jan 10, 2021 at 7:20 AM Jan Høydahl  wrote:
> >
> > Jason, Shalin and Dat, thanks for the thorough work. This is an example for 
> > other SIPs to follow!
> >
> > > I've also amended the backcompat/migration section to mention Jan's
> > > suggestion that the "incremental" features be exposed in the v2 API
> > > only.  Though it's uncle

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Hey all,

I've put replies to everyone's questions below.  Hope they help!

> Do the shard metadata files list all of the segments that make up the backup, 
> or only the segments that were uploaded in this incremental update?

Mike: The former - they're intended to hold metadata about all of the
segments that are needed to restore to the given
snapshot/commit-point.  So it's likely to hold metadata about files
just uploaded, as well as ones that were added to the blob by previous
backups.  I'll see if I can make that clearer in the file
descriptions.

> leave the old Backup/Restore API as-is, deprecated, and add a new one on 
> /v2/cluster/backup

Jan: Ultimately I agree with your concerns about scope, so I'd vote
against trying to cover TRAs, multiple collection backups, etc. in
this effort here.

That aside though, I agree that the existing v2 backup API is a bit of
a headscratcher.  Why is it /v2/collections instead of
/v2/collections/ or a subpath of /v2/cluster?  Does it
have something to do with aliases?  Or did it end up there mostly by
default?  I'd be open to creating a new v2 backup endpoint (without
adding TRA, etc. compatibility) if there was consensus on that
approach to handling backcompat and on the specific appearance of the
API.  It would help with backcompat after all.  Though if finding
consensus bogs down it may not be worth the addition.

> I know you've seen SOLR-15051 (Shared storage -- BlobDirectory) ... We both 
> want to store checksums and file lengths. ... Your proposal did not discuss 
> how these files are GC'ed

David: SIP-12 does address this, though maybe the writeup needs
clarifying.  The Delete Backup API includes a "purge" parameter which
triggers GC activity.  This probably works about the way you'd expect
- Solr gets the list of UUID-named index files from the blob store,
and then it compares that list to the set of UUID's referenced by any
shard-metadata file (which requires reading all the shard-metadata
files).  This avoids adding to Solr's ZK state, but does so at the
cost of requiring users to trigger sporadic cleanup manually instead
of detecting orphans automatically like BlobDirectory does (assuming I
understand that correctly).

I'm def not saying this is the best approach necessarily.  I like it,
though it has downsides for sure.  Just that there is a proposed
approach that's easy to miss buried in the SIP.

More broadly though - I share your sense that we should consider
alignment.  It may end up that Backup/Restore is different enough from
the BlobDirectory usecase that it doesn't make sense, but it's at
least worth figuring out.  That's about as far as my understanding
goes right now though.  I'll read up on BlobDirectory while you absorb
SIP-12 and maybe we can circle back to this shortly.

Best,

Jason

On Sun, Jan 10, 2021 at 7:20 AM Jan Høydahl  wrote:
>
> Jason, Shalin and Dat, thanks for the thorough work. This is an example for 
> other SIPs to follow!
>
> > I've also amended the backcompat/migration section to mention Jan's
> > suggestion that the "incremental" features be exposed in the v2 API
> > only.  Though it's unclear to me whether that's still something people
> > want since it turns out that we'll still have backcompat concerns with
> > the existing v2 backup/restore APIs.  So I've held off from
> > removing/replacing the original plan.
>
> Since we already have v2 for the existing backup API, I guess my suggestion 
> is not that 'clean' after all.
>
> Another approach would be to leave the old Backup/Restore API as-is, 
> deprecated, and add a new one on /v2/cluster/backup, with support for backing 
> up multiple collections in one go, or backup a TRA alias with hundreds of 
> concrete "sub" collections. But as I write these words I imagine it probably 
> is way outside the scope for this SIP which is large enough. Anyone even 
> tried to backup a TRA with today's API?
>
> Jan
>
> > 5. jan. 2021 kl. 15:55 skrev Jason Gerlowski :
> >
> > Hey, Happy New Year everybody.
> >
> > Some SIP updates based on the discussion above:
> >
> > I added v2 examples for each API to the SIP.  Feedback welcome,
> > especially on the v2 APIs that are net-new to this proposal (namely:
> > "list backups" and "delete backup").
> >
> > I've also amended the backcompat/migration section to mention Jan's
> > suggestion that the "incremental" features be exposed in the v2 API
> > only.  Though it's unclear to me whether that's still something people
> > want since it turns out that we'll still have backcompat concerns with
> > the existing v2 backup/restore APIs.  So I've held off from
> > removing/replacing the original plan.
> >
> > Link for convenience:
> > https://cwiki.apache.org/

Re: SolrJ ResponseParser Expectations/Compatibility

[Jason Gerlowski]

Ok, that makes sense.  I was aware of "json.nl" and figured it would
mean that lining up the NamedList produced by JSON and other formats
would be "best effort" at most.  This would be good to document in the
"Using SolrJ" ref-guide page.  I'll put that on my todo list.

But it's good to get confirmation that javabin and XML _are_ expected
to 'round-trip' a NamedList.  That implies that it's definitely a
design issue then that our XML ser/de code doesn't round-trip the
"Map" type the way javabin does.

Jason

On Sat, Jan 9, 2021 at 4:40 PM David Smiley  wrote:
>
> Your understanding is, I think, not fully correct.  Only JavaBin & XML ought 
> to round-trip a NamedList and some other structures, but not JSON which maps 
> them in different ways (see the json.nl param).  Since you discovered this on 
> wt=xml, you found a bug.  Undoubtedly, we need better testing here and 
> consequently hardening to make javabin & XML more consistent.  We can only do 
> so much for JSON.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Jan 7, 2021 at 11:01 AM Jason Gerlowski  wrote:
>>
>> Hi all,
>>
>> I recently spent some time on SOLR-15070 - a SolrJ ClassCastException
>> that cropped up for a customer hitting /suggest with wt=xml.
>> SOLR-15070 itself was a relatively straightforward fix, but the more I
>> think about the underlying cause the more I wonder whether we don't
>> have a larger, fundamental problem in how our ResponseParser's/codec's
>> work in SolrJ.  I wanted to run it by everyone here.
>>
>> My understanding of SolrJ's response serialization/deserialization was
>> that responses from Solr are supposed to deserialize to the same
>> NamedList on the client side, regardless of the wire format Solr used
>> to send the response. This guarantee is seemingly important for other
>> classes in SolrJ as well as for users who inspect NamedList's directly
>> - it would be a big burden if (e.g.) SolrJ's QueryResponse had to
>> handle the slight divergences of responses in 3 or 4 different
>> formats.
>>
>> But in working on SOLR-15070 I noticed that ResponseParser's don't
>> even support the same "collection" types in their deserialization.
>> BinaryResponseParser (the default) creates structures containing
>> ArrayLists, SimpleOrderedMaps, NamedLists, and HashMaps.
>> XMLResponseParser only supports a subset of those: ArrayLists and
>> SimpleOrderedMaps.  Any API that triggers any of javabin's unique
>> mappings then will by necessity result in different NamedList
>> structures on the client side.  The biggest problem in practice here
>> is probably "Map".
>>
>> Is my understanding of how NamedList and ResponseParsers are supposed
>> to work correct?  If so, has anyone else noticed the divergence
>> between the supported output-types before?  Does anyone have opinions
>> on a fix?  Presumably this would require bringing our ser/de code for
>> other formats up to par with javabin, which seems like a big breaking
>> change for those formats.  Just fishing for general thoughts and
>> advice I guess.
>>
>> Best,
>>
>> Jason
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Sure thing.  I put together a writeup on the file layout and formats
here: 
https://cwiki.apache.org/confluence/display/SOLR/Incremental+Backup+File+Format
The details get a little verbose, so I made it a subpage that the
SIP-proper calls out to.

Let me know what you think when you get a chance to read - hopefully
that's sufficient to fill the gap.

Jason

On Thu, Jan 7, 2021 at 8:34 PM Tomás Fernández Löbbe
 wrote:
>
> Thanks Jason! This is great, and a very much needed feature.
>
> > This helps to avoid confusion that would
> > otherwise arise between identically named files when e.g. a shard
> > leader changes between two incremental backups.  (I'll try to expand
> > on this in the SIP, as it's a bit hard to give the full context here.)
>
> Thanks, I was wondering the same thing. Maybe it would be good to put an 
> example of how the file structure of a backup looks like in the backup? and 
> how the manifest file looks like. As you said, a file with the same name may 
> refer to different segments created by different cores or the same one (even 
> if the leader changed, it may be a file from a previous replication).
>
> On Thu, Jan 7, 2021 at 1:20 PM Jason Gerlowski  wrote:
>>
>> Thanks for the feedback Mike.  I've gotta give any credit to Shalin
>> though, he wrote most of it before the holiday.  He and Dat wrote much
>> of the code involved as well.  I haven't done more than steward things
>> along so far.  As you suggested, I've updated the SIP to mention the
>> related SOLR-13608 (see the bottom of the "Motivation" section).
>>
>> As for your questions, I've tried to answer them below.
>>
>> 1. Good catch - it doesn't. The SIP should read that each backup
>> creates its own manifest files as needed for directories it creates
>> under the base "location".  This way, additional backups can be added
>> to the same location without needing to modify existing metadata
>> files.  I've updated the SIP to reflect this.
>>
>> 2. The proposed metadata file is a lot like segments_n (in spirit) in
>> that it has pointers to each index file that comprise an
>> index/replica.  But it differs in that it stores additional
>> information about each file (checksum, size) separate from the file
>> itself.  It also allows a layer of naming indirection between what
>> files are named in the storage repository and what name they should be
>> given upon restoration.  This helps to avoid confusion that would
>> otherwise arise between identically named files when e.g. a shard
>> leader changes between two incremental backups.  (I'll try to expand
>> on this in the SIP, as it's a bit hard to give the full context here.)
>>
>> 3. My intention was that the 'maxNumBackups' parameter would only
>> refer to the incremental backups in a given location.  This was mostly
>> informed by the fact that traditional backups today are required to be
>> 1-per-location.  (i.e. a backup in 8.6.3 will error out if the
>> specified directory has files in it.).  We could fix that aspect of
>> traditional backups and find semantics for 'maxNumBackups' that might
>> include traditional ones, but IMO it'd add complexity and work for a
>> format that the SIP is trying to replace more broadly anyways.
>>
>> 4. I definitely intended to update LocalFileSystemRepository.  I have
>> code to update HdfsBackupRepository as well, but wasn't quite sure
>> where that stood since it's currently deprecated.  I haven't seen
>> plans to make it a plugin, but might've just missed those discussions
>> in other mail.  Anyway, I plan to update it but that assumes it's
>> sticking around in one form or another.
>>
>> 5. Good idea - I didn't realize that was an option.  But it would be
>> really nice if possible.  I don't have an estimate on resources.  I
>> expect the need would be relatively small - you could restrict the
>> tests to running on the nightly runs on ASF's Jenkins unless devs
>> provide their own (e.g.) s3 creds.  But that's just a guess obviously,
>> and not even in concrete terms.
>>
>> Thanks again for taking the time to wade through the SIP - really
>> appreciate the feedback.  Hope the answers help!
>>
>> Best,
>>
>> Jason
>>
>> On Tue, Jan 5, 2021 at 11:52 AM Mike Drob  wrote:
>> >
>> > This is a very thorough SIP, thank you for spending the time on it, Jason!
>> >
>> > I have a few minor questions about points that are unclear to me.
>> >
>> > 1) If we assume that we cannot overwrite files, how does the manifest file 
>> > stay current for incremental backup opera

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Thanks for the feedback Mike.  I've gotta give any credit to Shalin
though, he wrote most of it before the holiday.  He and Dat wrote much
of the code involved as well.  I haven't done more than steward things
along so far.  As you suggested, I've updated the SIP to mention the
related SOLR-13608 (see the bottom of the "Motivation" section).

As for your questions, I've tried to answer them below.

1. Good catch - it doesn't. The SIP should read that each backup
creates its own manifest files as needed for directories it creates
under the base "location".  This way, additional backups can be added
to the same location without needing to modify existing metadata
files.  I've updated the SIP to reflect this.

2. The proposed metadata file is a lot like segments_n (in spirit) in
that it has pointers to each index file that comprise an
index/replica.  But it differs in that it stores additional
information about each file (checksum, size) separate from the file
itself.  It also allows a layer of naming indirection between what
files are named in the storage repository and what name they should be
given upon restoration.  This helps to avoid confusion that would
otherwise arise between identically named files when e.g. a shard
leader changes between two incremental backups.  (I'll try to expand
on this in the SIP, as it's a bit hard to give the full context here.)

3. My intention was that the 'maxNumBackups' parameter would only
refer to the incremental backups in a given location.  This was mostly
informed by the fact that traditional backups today are required to be
1-per-location.  (i.e. a backup in 8.6.3 will error out if the
specified directory has files in it.).  We could fix that aspect of
traditional backups and find semantics for 'maxNumBackups' that might
include traditional ones, but IMO it'd add complexity and work for a
format that the SIP is trying to replace more broadly anyways.

4. I definitely intended to update LocalFileSystemRepository.  I have
code to update HdfsBackupRepository as well, but wasn't quite sure
where that stood since it's currently deprecated.  I haven't seen
plans to make it a plugin, but might've just missed those discussions
in other mail.  Anyway, I plan to update it but that assumes it's
sticking around in one form or another.

5. Good idea - I didn't realize that was an option.  But it would be
really nice if possible.  I don't have an estimate on resources.  I
expect the need would be relatively small - you could restrict the
tests to running on the nightly runs on ASF's Jenkins unless devs
provide their own (e.g.) s3 creds.  But that's just a guess obviously,
and not even in concrete terms.

Thanks again for taking the time to wade through the SIP - really
appreciate the feedback.  Hope the answers help!

Best,

Jason

On Tue, Jan 5, 2021 at 11:52 AM Mike Drob  wrote:
>
> This is a very thorough SIP, thank you for spending the time on it, Jason!
>
> I have a few minor questions about points that are unclear to me.
>
> 1) If we assume that we cannot overwrite files, how does the manifest file 
> stay current for incremental backup operations to the same directory?
> 2) How is the manifest file functionally different from the segments_n and 
> segments.gen files?
> 3) Does the maxNumBackups parameter consider incremental backups or only full 
> backups? What happens if we have a full backup and then N incremental ones? 
> Do we delete the full backup and convert the oldest incremental one into a 
> full? I imagine this might be a metadata operation, but then the concerns 
> from question 1 apply.
> 4) Do we plan to retrofit HDFS Backup and Local File Backup to use the new 
> interfaces? I believe we should, but may be willing to accept this as out of 
> scope.
> 5) Regarding cloud provider test resources, we can also approach the ASF 
> Infra team to ask for cloud credits. Can you give rough estimates on what 
> kind of resourcing would be needed?
>
> I did not examine the new APIs in detail, but they looked fine at a high 
> level overview. Will probably look again after questions regarding v1/v2 are 
> figured out.
>
> On Tue, Jan 5, 2021 at 10:11 AM Mike Drob  wrote:
>>
>> Can you explicitly call out in the SIP how it relates to the work done in 
>> SOLR-13608?
>>
>> On Tue, Jan 5, 2021 at 8:55 AM Jason Gerlowski  wrote:
>>>
>>> Hey, Happy New Year everybody.
>>>
>>> Some SIP updates based on the discussion above:
>>>
>>> I added v2 examples for each API to the SIP.  Feedback welcome,
>>> especially on the v2 APIs that are net-new to this proposal (namely:
>>> "list backups" and "delete backup").
>>>
>>> I've also amended the backcompat/migration section to mention Jan's
>>> suggestion that the "incremental" fea

SolrJ ResponseParser Expectations/Compatibility

[Jason Gerlowski]

Hi all,

I recently spent some time on SOLR-15070 - a SolrJ ClassCastException
that cropped up for a customer hitting /suggest with wt=xml.
SOLR-15070 itself was a relatively straightforward fix, but the more I
think about the underlying cause the more I wonder whether we don't
have a larger, fundamental problem in how our ResponseParser's/codec's
work in SolrJ.  I wanted to run it by everyone here.

My understanding of SolrJ's response serialization/deserialization was
that responses from Solr are supposed to deserialize to the same
NamedList on the client side, regardless of the wire format Solr used
to send the response. This guarantee is seemingly important for other
classes in SolrJ as well as for users who inspect NamedList's directly
- it would be a big burden if (e.g.) SolrJ's QueryResponse had to
handle the slight divergences of responses in 3 or 4 different
formats.

But in working on SOLR-15070 I noticed that ResponseParser's don't
even support the same "collection" types in their deserialization.
BinaryResponseParser (the default) creates structures containing
ArrayLists, SimpleOrderedMaps, NamedLists, and HashMaps.
XMLResponseParser only supports a subset of those: ArrayLists and
SimpleOrderedMaps.  Any API that triggers any of javabin's unique
mappings then will by necessity result in different NamedList
structures on the client side.  The biggest problem in practice here
is probably "Map".

Is my understanding of how NamedList and ResponseParsers are supposed
to work correct?  If so, has anyone else noticed the divergence
between the supported output-types before?  Does anyone have opinions
on a fix?  Presumably this would require bringing our ser/de code for
other formats up to par with javabin, which seems like a big breaking
change for those formats.  Just fishing for general thoughts and
advice I guess.

Best,

Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Hey, Happy New Year everybody.

Some SIP updates based on the discussion above:

I added v2 examples for each API to the SIP.  Feedback welcome,
especially on the v2 APIs that are net-new to this proposal (namely:
"list backups" and "delete backup").

I've also amended the backcompat/migration section to mention Jan's
suggestion that the "incremental" features be exposed in the v2 API
only.  Though it's unclear to me whether that's still something people
want since it turns out that we'll still have backcompat concerns with
the existing v2 backup/restore APIs.  So I've held off from
removing/replacing the original plan.

Link for convenience:
https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore

Best,

Jason


On Thu, Dec 24, 2020 at 8:11 AM Jan Høydahl  wrote:
>
> Ok, that’s the one I was looking for, it’s not documented in the backup 
> chapter of ref-guide :(
>
> Jan Høydahl
>
> > 23. des. 2020 kl. 17:10 skrev Jason Gerlowski :
> >
> > 
> >>
> >> We have a path alias to the old API ... but we don’t have a true v2 API 
> >> spec for it, do we?
> >
> > Tbh I'm not yet familiar enough with the v2 APIs to understand the
> > distinction you're making.  (Do you have a pointer to something that'd
> > fill me in?)
> >
> > To zoom in on "backup" as an example, the v2 API I'm referring to
> > looks like:  /v2/collections" -d '{ "backup-collection":
> > {"collection": "books", "name": "asdf3", "location": "/tmp/foo"}}'.
> > And it's included in the v2 "introspect" documentation returned by
> > this API: /v2/collections/_introspect?command=backup-collection".  To
> > me that looked like a v2 API, but maybe path-aliases are also covered
> > in the introspect docs.
> >
> > Jason
> >
> >> On Wed, Dec 23, 2020 at 10:29 AM Jan Høydahl  wrote:
> >>
> >> Actually, don’t think we do have a v2 Backup/Restore API. We have a path 
> >> alias to the old API which takes GET ...=backup... but we don’t 
> >> have a true v2 API spec for it, do we? Where is that documented?
> >>
> >> Jan Høydahl
> >>
> >>>> 22. des. 2020 kl. 18:04 skrev Jason Gerlowski :
> >>>
> >>> Hey guys,
> >>>
> >>> Following up to make sure I understand the specifics you're
> >>> suggesting.  You're proposing that:
> >>>
> >>> 1. The brand new backup-related APIs (list-backups and delete-backup)
> >>> be added in v2-form only.
> >>> 2. Tweaks to existing backup-related APIs (create-backup, restore) be
> >>> made in V2-form only.
> >>> 3. All existing v1 backup-related APIs be deprecated and left
> >>> unchanged.  Incremental backups will not be possible using the v1 API.
> >>>
> >>> I'm not against going this route if there's consensus around it.  But
> >>> I'm not 100% clear on how it means we don't need to worry about
> >>> backcompat.  Backup and Restore currently exist as both a v1 and a v2
> >>> API - I understand how leaving the v1 APIs untouched (other than
> >>> deprecation) frees us of some backcompat concerns there, but we would
> >>> still need to make tweaks to the v2 backup/restore APIs and would have
> >>> to tread just as carefully there in terms of backcompat, afaict.
> >>> Unless Solr's backcompatibility guarantees only cover the v1 API and
> >>> leave v2 changes to be made freely?  I looked around to see if the v2
> >>> APIs had any sort of "experimental" designation, but couldn't find
> >>> that clearly stated anywhere.  Am I missing something?
> >>>
> >>> Best,
> >>>
> >>> Jason
> >>>
> >>>> On Tue, Dec 22, 2020 at 2:49 AM Noble Paul  wrote:
> >>>>
> >>>>> , and implement the new imporved version as a V2-api only, and then 
> >>>>> deprecate the v1 API?
> >>>>
> >>>>
> >>>> V2 only please
> >>>>
> >>>>> On Tue, Dec 22, 2020 at 1:34 AM Jason Gerlowski  
> >>>>> wrote:
> >>>>>
> >>>>> Hey Jan, thanks for the review.
> >>>>>
> >>>>> I hadn't thought about the V2 API in connection to this work.  You're
> >>>>> right though I think - the SIP proposes net-new APIs, so it should add
> >>>>> V2 equivalents at the very leas

Re: Old programmers do fade away

[Jason Gerlowski]

Hey Erick,

I'm sorry for our users and for the project overall to hear you'll be
"hanging up the spurs", but excited that you've found new, fun things
to expand into in your retirement!  I hope they're awesome!

Best of luck man, it's been great working with you!

Best,

Jason

On Mon, Jan 4, 2021 at 1:05 PM Uwe Schindler  wrote:
>
> Hi Erick,
>
> too bad that you want to leave us. I really hope to see you in person after
> COVID19 allows us to meet at conferences again. It was always nice
> discussions with you on the mailing list, although I am not subscribed to
> solr-user@lao, so the whole work you were doing is not so obvious to me.
>
> About the squirrels: As an European, I have no idea why those grey American
> squirrels are so aggressive and eat all your tomatoes!
> https://www.dropbox.com/s/g8w441njq8xvaxd/20210104_185757.jpg?dl=0 (sitting
> in my garden)
>
> Here in Bremen, I have no problem at all with tomatoes and squirrels. The
> European red squirrels only collect and eat nuts (or to be more correct:
> they collect nuts and then forget about them, making hazelnut trees grow
> everywhere in my garden, if you do not destroy them).
> https://www.dropbox.com/s/e9twup0850vxpdz/20210104_185720.jpg?dl=0
> (leftovers of the nuts...)
>
> My tomato plants all survived and I was able to harvest this year.
>
> Uwe
>
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> > -Original Message-
> > From: Erick Erickson 
> > Sent: Wednesday, December 30, 2020 3:09 PM
> > To: dev@lucene.apache.org
> > Subject: Old programmers do fade away
> >
> > 40 years is enough. OK, it's only been 39 1/2 years. Dear Lord, has it
> really been
> > that long? Programming's been fun, I've gotten to solve puzzles every day.
> The
> > art and science of programming has changed over that time. Let me tell you
> > about the joys of debugging with a Z80 stack emulator that required that
> you
> > to look on the stack for variables and trace function calls by knowing how
> to
> > follow frame pointers. Oh the tedium! Oh the (lack of) speed! Not to
> mention
> > that 64K of memory was all you had to work with. I had a co-worker who
> could
> > predict the number of bytes by which the program would shrink based on
> > extracting common code to functions. The "good old days"...weren't...
> >
> > I'd been thinking that I'd treat Lucene/Solr as a hobby, doing occasional
> work
> > on it when I was bored over long winter nights. I've discovered, though,
> that
> > I've been increasingly reluctant to crack open the code. I guess that
> after this
> > much time, I'm ready to hang up my spurs. One major factor is the
> realization
> > that there's so much going on with Lucene/Solr that simply being aware of
> the
> > changes, much less trying to really understand them, isn't something I can
> do
> > casually.
> >
> > I bought a welder and find myself more interested in playing with that
> than
> > programming. Wait until you see the squirrel-proof garden enclosure I'm
> > building with it. If my initial plan doesn't work, next up is an electric
> fence
> > along the top. The laser-sighted automatic machine gun emplacement will
> take
> > more planning...Ahhh, probably won't be able to get a permit from the
> > township for that though. Do you think the police would notice? Perhaps I
> > should add that the local police station is two blocks away and in the
> line of
> > fire. But an infrared laser powerful enough to "pre-cook" them wouldn't be
> as
> > obvious would it?
> >
> > Why am I so fixated on squirrels? One of the joys of gardening is fresh
> > tomatoes rather than those red things they sell in the store. The
> squirrels ATE
> > EVERY ONE OF MY TOMATOES WHILE THEY WERE STILL GREEN LAST YEAR! And
> > the melons. In the words of B. Bunny: "Of course you realize this means
> war"
> > (https://www.youtube.com/watch?v=4XNr-BQgpd0)...
> >
> > Then there's working in the garden and landscaping, the desk I want to
> build
> > for my wife, travel as soon as I can, maybe seeing if some sailboats need
> > crew...you get the idea.
> >
> > It's been a privilege to work with this group, you're some of the best and
> > brightest. Many thanks to all who've generously given me their time and
> > guidance. It's been a constant source of amazement to me how willing
> people
> > are to take time out of their own life and work to help me when I've had
> > questions. I owe a lot of people beers ;)
> >
> > I'll be stopping my list subscriptions, Slack channels (dm me if you need
> > something), un-assigning any JIRAs and that kind of thing over the next
> while. If
> > anyone's interested in taking over the BadApple report, let me know and I
> can
> > put the code up somewhere. It takes about 10 minutes to do each week. I
> won't
> > disappear entirely, things like the code-reformatting effort are nicely
> self-
> > contained for instance and something I can to casually.
> >
> > My e-mail 

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

> We have a path alias to the old API ... but we don’t have a true v2 API spec 
> for it, do we?

Tbh I'm not yet familiar enough with the v2 APIs to understand the
distinction you're making.  (Do you have a pointer to something that'd
fill me in?)

To zoom in on "backup" as an example, the v2 API I'm referring to
looks like:  /v2/collections" -d '{ "backup-collection":
{"collection": "books", "name": "asdf3", "location": "/tmp/foo"}}'.
And it's included in the v2 "introspect" documentation returned by
this API: /v2/collections/_introspect?command=backup-collection".  To
me that looked like a v2 API, but maybe path-aliases are also covered
in the introspect docs.

Jason

On Wed, Dec 23, 2020 at 10:29 AM Jan Høydahl  wrote:
>
> Actually, don’t think we do have a v2 Backup/Restore API. We have a path 
> alias to the old API which takes GET ...=backup... but we don’t have a 
> true v2 API spec for it, do we? Where is that documented?
>
> Jan Høydahl
>
> > 22. des. 2020 kl. 18:04 skrev Jason Gerlowski :
> >
> > Hey guys,
> >
> > Following up to make sure I understand the specifics you're
> > suggesting.  You're proposing that:
> >
> > 1. The brand new backup-related APIs (list-backups and delete-backup)
> > be added in v2-form only.
> > 2. Tweaks to existing backup-related APIs (create-backup, restore) be
> > made in V2-form only.
> > 3. All existing v1 backup-related APIs be deprecated and left
> > unchanged.  Incremental backups will not be possible using the v1 API.
> >
> > I'm not against going this route if there's consensus around it.  But
> > I'm not 100% clear on how it means we don't need to worry about
> > backcompat.  Backup and Restore currently exist as both a v1 and a v2
> > API - I understand how leaving the v1 APIs untouched (other than
> > deprecation) frees us of some backcompat concerns there, but we would
> > still need to make tweaks to the v2 backup/restore APIs and would have
> > to tread just as carefully there in terms of backcompat, afaict.
> > Unless Solr's backcompatibility guarantees only cover the v1 API and
> > leave v2 changes to be made freely?  I looked around to see if the v2
> > APIs had any sort of "experimental" designation, but couldn't find
> > that clearly stated anywhere.  Am I missing something?
> >
> > Best,
> >
> > Jason
> >
> >> On Tue, Dec 22, 2020 at 2:49 AM Noble Paul  wrote:
> >>
> >>> , and implement the new imporved version as a V2-api only, and then 
> >>> deprecate the v1 API?
> >>
> >>
> >> V2 only please
> >>
> >>> On Tue, Dec 22, 2020 at 1:34 AM Jason Gerlowski  
> >>> wrote:
> >>>
> >>> Hey Jan, thanks for the review.
> >>>
> >>> I hadn't thought about the V2 API in connection to this work.  You're
> >>> right though I think - the SIP proposes net-new APIs, so it should add
> >>> V2 equivalents at the very least.  I'll draft tentative details for
> >>> these APIs on the SIP and we can refine things from there.
> >>>
> >>> I'm more up in the air on your specific suggestion to restrict the SIP
> >>> changes to these v2 APIs.  It is an elegant approach to the
> >>> backcompat, and it provides a carrot for v2 adoption - both of which I
> >>> like.  But it would let users create snapshot-based backups (and keep
> >>> us maintaining that code) longer than there's any strict need to.  And
> >>> users are left on the less-efficient format by default.  (By contrast,
> >>> the current SIP has snapshot-backup creation being replaced by
> >>> incremental-backup creation as soon as the latter is available.).  Did
> >>> you have a particular lifespan in mind for snapshot-based creation if
> >>> we go with this approach?
> >>>
> >>> Jason
> >>>
> >>> On Thu, Dec 17, 2020 at 3:54 PM Jan Høydahl  wrote:
> >>>>
> >>>> Much needed! Thanks for initiating this Jason!
> >>>>
> >>>> As we want to move away from v1 APIs where a HTTP GET is used for 
> >>>> creation and deletion, would it be an idea to leave the old 
> >>>> backup/resotre APIs as-is, and implement the new imporved version as a 
> >>>> V2-api only, and then deprecate the v1 API? Then we don't need to worry 
> >>>> about back-compat, and we get a head-start on converting the COLLECTION 
> >>>> API to v2 styl

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Hey guys,

Following up to make sure I understand the specifics you're
suggesting.  You're proposing that:

1. The brand new backup-related APIs (list-backups and delete-backup)
be added in v2-form only.
2. Tweaks to existing backup-related APIs (create-backup, restore) be
made in V2-form only.
3. All existing v1 backup-related APIs be deprecated and left
unchanged.  Incremental backups will not be possible using the v1 API.

I'm not against going this route if there's consensus around it.  But
I'm not 100% clear on how it means we don't need to worry about
backcompat.  Backup and Restore currently exist as both a v1 and a v2
API - I understand how leaving the v1 APIs untouched (other than
deprecation) frees us of some backcompat concerns there, but we would
still need to make tweaks to the v2 backup/restore APIs and would have
to tread just as carefully there in terms of backcompat, afaict.
Unless Solr's backcompatibility guarantees only cover the v1 API and
leave v2 changes to be made freely?  I looked around to see if the v2
APIs had any sort of "experimental" designation, but couldn't find
that clearly stated anywhere.  Am I missing something?

Best,

Jason

On Tue, Dec 22, 2020 at 2:49 AM Noble Paul  wrote:
>
> >, and implement the new imporved version as a V2-api only, and then 
> >deprecate the v1 API?
>
>
> V2 only please
>
> On Tue, Dec 22, 2020 at 1:34 AM Jason Gerlowski  wrote:
> >
> > Hey Jan, thanks for the review.
> >
> > I hadn't thought about the V2 API in connection to this work.  You're
> > right though I think - the SIP proposes net-new APIs, so it should add
> > V2 equivalents at the very least.  I'll draft tentative details for
> > these APIs on the SIP and we can refine things from there.
> >
> > I'm more up in the air on your specific suggestion to restrict the SIP
> > changes to these v2 APIs.  It is an elegant approach to the
> > backcompat, and it provides a carrot for v2 adoption - both of which I
> > like.  But it would let users create snapshot-based backups (and keep
> > us maintaining that code) longer than there's any strict need to.  And
> > users are left on the less-efficient format by default.  (By contrast,
> > the current SIP has snapshot-backup creation being replaced by
> > incremental-backup creation as soon as the latter is available.).  Did
> > you have a particular lifespan in mind for snapshot-based creation if
> > we go with this approach?
> >
> > Jason
> >
> > On Thu, Dec 17, 2020 at 3:54 PM Jan Høydahl  wrote:
> > >
> > > Much needed! Thanks for initiating this Jason!
> > >
> > > As we want to move away from v1 APIs where a HTTP GET is used for 
> > > creation and deletion, would it be an idea to leave the old 
> > > backup/resotre APIs as-is, and implement the new imporved version as a 
> > > V2-api only, and then deprecate the v1 API? Then we don't need to worry 
> > > about back-compat, and we get a head-start on converting the COLLECTION 
> > > API to v2 style.
> > >
> > > Jan
> > >
> > > > 15. des. 2020 kl. 15:48 skrev Jason Gerlowski :
> > > >
> > > > Hey all,
> > > >
> > > > This morning I published SIP-12, which proposes an overhaul of Solr's
> > > > backup and restore functionality.  While the "headline" improvement in
> > > > this SIP is a change to do backups incrementally, it bundles in a
> > > > number of other improvements as well, including the addition of
> > > > corruption checks, APIs to list and delete backups, and stronger
> > > > integration points with popular object storage APIs.
> > > >
> > > > The SIP can be found here:
> > > > https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore
> > > >
> > > > Please read the SIP description and come back here for discussion.  As
> > > > the discussion progresses we will update the SIP page with any
> > > > outcomes and eventually move things to a VOTE.
> > > >
> > > > Looking forward to hearing your feedback.
> > > >
> > > > Best,
> > > >
> > > > Jason
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > > > For additional commands, e-mail: dev-h...@lucene.apache.org
> > > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Hey Jan, thanks for the review.

I hadn't thought about the V2 API in connection to this work.  You're
right though I think - the SIP proposes net-new APIs, so it should add
V2 equivalents at the very least.  I'll draft tentative details for
these APIs on the SIP and we can refine things from there.

I'm more up in the air on your specific suggestion to restrict the SIP
changes to these v2 APIs.  It is an elegant approach to the
backcompat, and it provides a carrot for v2 adoption - both of which I
like.  But it would let users create snapshot-based backups (and keep
us maintaining that code) longer than there's any strict need to.  And
users are left on the less-efficient format by default.  (By contrast,
the current SIP has snapshot-backup creation being replaced by
incremental-backup creation as soon as the latter is available.).  Did
you have a particular lifespan in mind for snapshot-based creation if
we go with this approach?

Jason

On Thu, Dec 17, 2020 at 3:54 PM Jan Høydahl  wrote:
>
> Much needed! Thanks for initiating this Jason!
>
> As we want to move away from v1 APIs where a HTTP GET is used for creation 
> and deletion, would it be an idea to leave the old backup/resotre APIs as-is, 
> and implement the new imporved version as a V2-api only, and then deprecate 
> the v1 API? Then we don't need to worry about back-compat, and we get a 
> head-start on converting the COLLECTION API to v2 style.
>
> Jan
>
> > 15. des. 2020 kl. 15:48 skrev Jason Gerlowski :
> >
> > Hey all,
> >
> > This morning I published SIP-12, which proposes an overhaul of Solr's
> > backup and restore functionality.  While the "headline" improvement in
> > this SIP is a change to do backups incrementally, it bundles in a
> > number of other improvements as well, including the addition of
> > corruption checks, APIs to list and delete backups, and stronger
> > integration points with popular object storage APIs.
> >
> > The SIP can be found here:
> > https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore
> >
> > Please read the SIP description and come back here for discussion.  As
> > the discussion progresses we will update the SIP page with any
> > outcomes and eventually move things to a VOTE.
> >
> > Looking forward to hearing your feedback.
> >
> > Best,
> >
> > Jason
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[DISCUSS] SIP-12: Incremental Backup and Restore

[Jason Gerlowski]

Hey all,

This morning I published SIP-12, which proposes an overhaul of Solr's
backup and restore functionality.  While the "headline" improvement in
this SIP is a change to do backups incrementally, it bundles in a
number of other improvements as well, including the addition of
corruption checks, APIs to list and delete backups, and stronger
integration points with popular object storage APIs.

The SIP can be found here:
https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore

Please read the SIP description and come back here for discussion.  As
the discussion progresses we will update the SIP page with any
outcomes and eventually move things to a VOTE.

Looking forward to hearing your feedback.

Best,

Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS][SOLR] Multiple Repos For Contributions

[Jason Gerlowski]

> then I'll start putting together individual proposals for a few repos to 
> exercise the process and get more contributions going

solr-operator is a great example of PMC-maintained code that makes
sense to have in a separate repository.  It's written primarily in a
different language, it's an integration with 3rd party software, etc.

But there are downsides to managing multiple repositories that make me
hesitant about the idea more generally.  There's no easy way to
prevent changes in one repo from unintentionally breaking another.
There's at least some duplication in the maintenance of things all
repos need (build systems, etc.).  It may add overhead on release
volunteers and the PMC if there are more releases.

I'm not sure how much those'll cause problems in practice.  Hopefully
they'll be minimal, but it's possible they won't be.  They might end
up outweighing the benefits.  I'm not saying we should be afraid of
additional repositories where it makes sense for the domain.  But
maybe it'd make sense to use solr-operator as a test case for a few
releases before putting in the effort to move out our current contribs
or change our process of adopting new ones.  Since this is more about
long term management, and less about getting in a particular feature
or value for users, we've got a cool opportunity to let the
solr-operator experiment play out before we necessarily need to decide
how to handle similar scenarios.

Just my two cents.

Jason

On Wed, Nov 25, 2020 at 8:23 AM Jan Høydahl  wrote:
>
> Let each sub project decide for themselves. PYLUCENE has its own svn repo and 
> its own Jira space.
> Solr-operator should be allowed to continue with GH issues and PRs i.m.o. No 
> need to force them into JIRA as long as the ASF allows projects to choose.
>
> Jan
>
> 24. nov. 2020 kl. 20:59 skrev David Smiley :
>
>> > Q: Should we create a separate JIRA for these contribs... or ditch JIRA 
>> > entirely for them, relying on GitHub alone?
>>
>> I'd start with same JIRA, with a separate component or label. I don't think 
>> GH issues would be good because it becomes harder to link between core and 
>> contrib issues in case of compat or tandme feature development.
>
>
> By "hard to link", are you basically saying pasting URLs is hard ;-).  ?  
> There was a committer meeting in Montreal where some folks like Jan Hoydal 
> and Varun (if I'm not mistaken; I may be) advocated for considering more 
> GitHub centric issue tracking.  I was not in favor of that... however for 
> contribs/modules that get their own separate repos, it affords an opportunity 
> for a break with the past in the interests of simplicity and familiarity for 
> what contributors are already familiar with.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Nov 17, 2020 at 7:42 PM Mike Drob  wrote:
>>
>> Thank you for the replies so far.
>>
>> I think that each contrib would necessarily have to have their own release 
>> schedule and release vote. I suspect that there might be frequent releases 
>> at first, and then these will smooth out into basically once per major 
>> release. I also think that contribs having releases could reduce the number 
>> of minor releases that we need to do, if a certain feature is well contained.
>>
>> Compatibility breaks will happen, but I feel like we should try to avoid 
>> them. Sometimes they're inevitable though, and we'll need to clearly mark 
>> that version X of the contrib is only compatible with version X of Solr, and 
>> for newer versions of Solr you have to use Y. Maybe we'll be able to release 
>> contrib Y first, and have it bridge the Solr releases. I think we'll need to 
>> invest in CI tooling to catch these kinds of situations sooner.
>>
>> > * More build files; copying the rules/setup/standards of the Solr 
>> > mothership and will become divergent over time no doubt.  Or just KISS 
>> > principle; no sharing; simple Maven projects.
>>
>> I wonder what the Gradle equivalent would be here. In maven-land, we can 
>> define a parent pom and attach a bunch of configuration and rules and 
>> plugins to it, and reuse across repositories and projects. Maybe the gradle 
>> build rules turn into an externally referenced project as well. I don't know 
>> what we'll need, but being able to apply all of our validation and precommit 
>> rules consistently to the contribs seems important.
>>
>> > Q: Could & should many contribs live in one repo (no more internal 
>> > contribs), yet each still have its own release cycle?  This could make 
>> > sharing build infrastructure easier, and detecting Solr compatibility with 
>> > them easier.  Although it would mean sharing GitHub project area, thus 
>> > sharing issues/PRs.
>>
>> I don't know. It would make source releases more complicated, which are what 
>> the ASF releases provide. I think it would make testing a contrib against 
>> multiple versions of Solr more difficult as well.
>>
>> > Q: 

Re: 8.6.3 Release

[Jason Gerlowski]

On the topic - is there a particular reason behind the convention of
having the page titles be named without spaces or periods?  As is they
look more like Java classnames than page titles. e.g.
"ReleaseNote852" vs "Release Notes: 8.5.2" vs. "8.5.2 Release Notes"

On Thu, Oct 22, 2020 at 9:50 AM Jason Gerlowski  wrote:
>
> I added that "DRAFT-" prefix mostly for the reason Atri mentioned: to
> indicate that the page was a WIP.  I thought it might also have the
> side benefit of catching the eye of any committers who had Confluence
> open and a few minutes to review the content.  The page still has
> "DRAFT-" because, as Cassandra suspected, I forgot to update it once
> the release was finished.  Fixing now.
>
> On Thu, Oct 22, 2020 at 9:47 AM Atri Sharma  wrote:
> >
> > Fair point -- I have changed accordingly.
> >
> > On Thu, Oct 22, 2020 at 7:09 PM Cassandra Targett  
> > wrote:
> > >
> > > That’s a fair thing to consider, but does it really matter if someone 
> > > looks at the draft notes pre-release? What would be the harm if users 
> > > happen across them before they’re done?
> > >
> > > (And to David’s point, it’s not in the RM steps to fix the page name 
> > > after release, so it’s pretty easy to forget to do it.)
> > > On Oct 22, 2020, 8:31 AM -0500, Atri Sharma , wrote:
> > >
> > > I added it since it looked a safe way to indicate that the draft notes
> > > are in progress and should not be referred to in case somebody is
> > > surfing the release notes.
> > >
> > > Atri
> > >
> > > On Thu, Oct 22, 2020 at 6:23 PM David Smiley  wrote:
> > >
> > >
> > > Jason: This is the first time I recall seeing release note pages in 
> > > Confluence with a "DRAFT-" prefix. And I also see that Atri has follow-ed 
> > > suit (following your lead, no doubt) for 8.6. Why? Looking at the page 
> > > navigation, it's clearly an oddity -- a change. And it's still DRAFT 
> > > despite it being released.
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> > >
> > >
> > > On Thu, Oct 1, 2020 at 10:28 AM Jason Gerlowski  
> > > wrote:
> > >
> > >
> > > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can
> > > someone please sanity check the summaries there when they get a
> > > chance? Would appreciate the review.
> > >
> > > 8.6.3 is a bit interesting in that Lucene has no changes in this
> > > bugfix release. As a result I had to omit the standard phrase in the
> > > Solr release notes about there being additional changes at the Lucene
> > > level, and change some of the wording in the Lucene announcement to
> > > indicate the lack of changes. So that's something to pay particular
> > > attention to, if someone can check my wording there.
> > >
> > > [1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863
> > > [2] 
> > > https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863
> > >
> > > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski  
> > > wrote:
> > >
> > >
> > > The only one that was previously mentioned as a blocker was
> > > SOLR-14835, but from the comments on the ticket it looks like it ended
> > > up being purely a cosmetic issue. Andrzej left a comment there
> > > suggesting that we "address" this with documentation for 8.6.3 but
> > > otherwise leave it as-is.
> > >
> > > So it looks like we're unblocked on starting the release process.
> > > Will begin the preliminary steps this afternoon.
> > >
> > > On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett  
> > > wrote:
> > >
> > >
> > > It looks to me like everything for 8.6.3 is resolved now 
> > > (https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it 
> > > seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a 
> > > Jetty upgrade less compelling to try.
> > >
> > > Are there any other issues not currently marked for 8.6.3 we’re waiting 
> > > for before starting the RC?
> > > On Sep 29, 2020, 12:04 PM -0500, Jason Gerlowski , 
> > > wrote:
> > >
> > > That said, if someone can use 8.6.3, what’s stopping them from going to 
> > > 8.7 when it’e released?
> > >
> > >
> &g

Re: 8.6.3 Release

[Jason Gerlowski]

I added that "DRAFT-" prefix mostly for the reason Atri mentioned: to
indicate that the page was a WIP.  I thought it might also have the
side benefit of catching the eye of any committers who had Confluence
open and a few minutes to review the content.  The page still has
"DRAFT-" because, as Cassandra suspected, I forgot to update it once
the release was finished.  Fixing now.

On Thu, Oct 22, 2020 at 9:47 AM Atri Sharma  wrote:
>
> Fair point -- I have changed accordingly.
>
> On Thu, Oct 22, 2020 at 7:09 PM Cassandra Targett  
> wrote:
> >
> > That’s a fair thing to consider, but does it really matter if someone looks 
> > at the draft notes pre-release? What would be the harm if users happen 
> > across them before they’re done?
> >
> > (And to David’s point, it’s not in the RM steps to fix the page name after 
> > release, so it’s pretty easy to forget to do it.)
> > On Oct 22, 2020, 8:31 AM -0500, Atri Sharma , wrote:
> >
> > I added it since it looked a safe way to indicate that the draft notes
> > are in progress and should not be referred to in case somebody is
> > surfing the release notes.
> >
> > Atri
> >
> > On Thu, Oct 22, 2020 at 6:23 PM David Smiley  wrote:
> >
> >
> > Jason: This is the first time I recall seeing release note pages in 
> > Confluence with a "DRAFT-" prefix. And I also see that Atri has follow-ed 
> > suit (following your lead, no doubt) for 8.6. Why? Looking at the page 
> > navigation, it's clearly an oddity -- a change. And it's still DRAFT 
> > despite it being released.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Thu, Oct 1, 2020 at 10:28 AM Jason Gerlowski  
> > wrote:
> >
> >
> > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can
> > someone please sanity check the summaries there when they get a
> > chance? Would appreciate the review.
> >
> > 8.6.3 is a bit interesting in that Lucene has no changes in this
> > bugfix release. As a result I had to omit the standard phrase in the
> > Solr release notes about there being additional changes at the Lucene
> > level, and change some of the wording in the Lucene announcement to
> > indicate the lack of changes. So that's something to pay particular
> > attention to, if someone can check my wording there.
> >
> > [1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863
> > [2] https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863
> >
> > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski  
> > wrote:
> >
> >
> > The only one that was previously mentioned as a blocker was
> > SOLR-14835, but from the comments on the ticket it looks like it ended
> > up being purely a cosmetic issue. Andrzej left a comment there
> > suggesting that we "address" this with documentation for 8.6.3 but
> > otherwise leave it as-is.
> >
> > So it looks like we're unblocked on starting the release process.
> > Will begin the preliminary steps this afternoon.
> >
> > On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett  
> > wrote:
> >
> >
> > It looks to me like everything for 8.6.3 is resolved now 
> > (https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it 
> > seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a 
> > Jetty upgrade less compelling to try.
> >
> > Are there any other issues not currently marked for 8.6.3 we’re waiting for 
> > before starting the RC?
> > On Sep 29, 2020, 12:04 PM -0500, Jason Gerlowski , 
> > wrote:
> >
> > That said, if someone can use 8.6.3, what’s stopping them from going to 8.7 
> > when it’e released?
> >
> >
> > The same things that always stop users from going directly to the
> > latest-and-greatest: fear of instability from new minor-release
> > features, reliance on behavior changed across minor versions, breaking
> > changes on Lucene elements that don't guarantee backcompat (e.g.
> > SOLR-14254), security issues in later versions (new libraries pulled
> > in with vulns), etc. There's lots of reasons a given user might want
> > to stick on 8.6.x rather than 8.7 (in the short/medium term).
> >
> > I'm ambivalent to whether we upgrade Jetty in 8.6.3 - as I said above
> > the worst of the Jetty issue should be mitigated by work on our end -
> > but I think there's a lot of reasons users might not upgrade as far as
> > we'd expect/like.
> >
> >

Re: 8.6.3 Release

[Jason Gerlowski]

Thanks for the pointer Cassandra and clarification Jan.  I'll tidy the
patch and look to commit if there's no additional complications.

On Fri, Oct 9, 2020 at 1:01 PM Jan Høydahl  wrote:
>
> It was not my intention to hold up LUCENE-9434, I was worried that it could 
> affect formatting.
> That should be easy to verify though, so feel free to continue with the patch!
>
> Jan
>
> 9. okt. 2020 kl. 18:52 skrev Cassandra Targett :
>
> We discussed this before and agreed that could be removed. I made a patch to 
> remove it but my editor always removes trailing whitespace and Jan doesn’t 
> want that for some reason and I haven’t had time to go back to it. See 
> https://issues.apache.org/jira/browse/LUCENE-9434.
> On Oct 9, 2020, 11:09 AM -0500, Jason Gerlowski , 
> wrote:
>
> Small correction: I see now some pages for 8.4 and 8.6 in a different
> section of the wiki tree. But the overall point still stands I think
> - this hasn't been done consistently and it doesn't seem like that's
> caused any problems (as the pages are all stubs anyways).
>
> On Fri, Oct 9, 2020 at 12:05 PM Jason Gerlowski  wrote:
>
>
> The traditional (non-docker) part of the release should now be wrapped
> up. Thanks everyone for the help and answering my questions here and
> in Slack. One final question:
>
> The final releaseWizard.py step instructs:
>
> "The Solr WIKI has a page for every version which is often linked to
> from WIKI pages to indicate differences between versions, example:
> http://wiki.apache.org/solr/Solr4.3. Do the following: Update the page
> for the released version with release date and link to release
> statement. Create a new placeholder page for the "next" version, if it
> does not exist"
>
> But looking at our wiki, the latest of these pages is 8.2
> (https://cwiki.apache.org/confluence/display/SOLR/Solr8.2). I've
> created the pages as instructed for now. But if we're not following
> this step regularly and it hasn't caused any issues maybe we should
> remove it from the release process altogether?
>
> Jason
>
> On Thu, Oct 8, 2020 at 5:16 PM David Smiley  wrote:
>
>
> The way GitHub works for contributors is that you are expected to fork a repo 
> and then push to your fork. At that point when you go to the PR area, you'll 
> see a convenient yellow dialog to create a PR based on your pushed branch.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Oct 8, 2020 at 10:20 AM Chris Hostetter  
> wrote:
>
>
>
> FWIW: I followed the docs to update the Dockerfiles + TAGS for 8.6.3, and
> run tests; but since it's in a distinct github repo I don't think i can
> push to it?
>
> so i creaed a GH issue w/patch...
>
> https://github.com/docker-solr/docker-solr/issues/349
>
>
>
> : Date: Tue, 6 Oct 2020 11:33:15 -0400
> : From: Houston Putman 
> : Reply-To: dev@lucene.apache.org
> : To: Solr/Lucene Dev 
> : Subject: Re: 8.6.3 Release
> :
> : That is correct. 8.x docker builds have not been affected in any way.
> :
> : On Tue, Oct 6, 2020 at 11:30 AM Cassandra Targett 
> : wrote:
> :
> : > I wanted to ask now that the 8.6.3 vote is underway - for the docker-solr
> : > image, are the update instructions in the docker-solr repo still the same
> : > for 8.x even though the build process has been moved to the main project
> : > for 9.0? Meaning, to release the 8.6.3 image there’s no change from 
> before,
> : > right?
> : >
> : > I’m asking specifically about these instructions:
> : >
> : > https://github.com/docker-solr/docker-solr/blob/master/update.md
> : > On Oct 1, 2020, 9:28 AM -0500, Jason Gerlowski ,
> : > wrote:
> : >
> : > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can
> : > someone please sanity check the summaries there when they get a
> : > chance? Would appreciate the review.
> : >
> : > 8.6.3 is a bit interesting in that Lucene has no changes in this
> : > bugfix release. As a result I had to omit the standard phrase in the
> : > Solr release notes about there being additional changes at the Lucene
> : > level, and change some of the wording in the Lucene announcement to
> : > indicate the lack of changes. So that's something to pay particular
> : > attention to, if someone can check my wording there.
> : >
> : > [1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863
> : > [2]
> : > https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863
> : >
> : > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski 
> : > wrote:
> : >
> : >
> : > The on

Re: 8.6.3 Release

[Jason Gerlowski]

Small correction: I see now some pages for 8.4 and 8.6 in a different
section of the wiki tree.  But the overall point still stands I think
- this hasn't been done consistently and it doesn't seem like that's
caused any problems (as the pages are all stubs anyways).

On Fri, Oct 9, 2020 at 12:05 PM Jason Gerlowski  wrote:
>
> The traditional (non-docker) part of the release should now be wrapped
> up.  Thanks everyone for the help and answering my questions here and
> in Slack.  One final question:
>
> The final releaseWizard.py step instructs:
>
> "The Solr WIKI has a page for every version which is often linked to
> from WIKI pages to indicate differences between versions, example:
> http://wiki.apache.org/solr/Solr4.3. Do the following: Update the page
> for the released version with release date and link to release
> statement. Create a new placeholder page for the "next" version, if it
> does not exist"
>
> But looking at our wiki, the latest of these pages is 8.2
> (https://cwiki.apache.org/confluence/display/SOLR/Solr8.2).  I've
> created the pages as instructed for now.  But if we're not following
> this step regularly and it hasn't caused any issues maybe we should
> remove it from the release process altogether?
>
> Jason
>
> On Thu, Oct 8, 2020 at 5:16 PM David Smiley  wrote:
> >
> > The way GitHub works for contributors is that you are expected to fork a 
> > repo and then push to your fork.  At that point when you go to the PR area, 
> > you'll see a convenient yellow dialog to create a PR based on your pushed 
> > branch.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Thu, Oct 8, 2020 at 10:20 AM Chris Hostetter  
> > wrote:
> >>
> >>
> >> FWIW: I followed the docs to update the Dockerfiles + TAGS for 8.6.3, and
> >> run tests; but since it's in a distinct github repo I don't think i can
> >> push to it?
> >>
> >> so i creaed a GH issue w/patch...
> >>
> >> https://github.com/docker-solr/docker-solr/issues/349
> >>
> >>
> >>
> >> : Date: Tue, 6 Oct 2020 11:33:15 -0400
> >> : From: Houston Putman 
> >> : Reply-To: dev@lucene.apache.org
> >> : To: Solr/Lucene Dev 
> >> : Subject: Re: 8.6.3 Release
> >> :
> >> : That is correct. 8.x docker builds have not been affected in any way.
> >> :
> >> : On Tue, Oct 6, 2020 at 11:30 AM Cassandra Targett 
> >> : wrote:
> >> :
> >> : > I wanted to ask now that the 8.6.3 vote is underway - for the 
> >> docker-solr
> >> : > image, are the update instructions in the docker-solr repo still the 
> >> same
> >> : > for 8.x even though the build process has been moved to the main 
> >> project
> >> : > for 9.0? Meaning, to release the 8.6.3 image there’s no change from 
> >> before,
> >> : > right?
> >> : >
> >> : > I’m asking specifically about these instructions:
> >> : >
> >> : > https://github.com/docker-solr/docker-solr/blob/master/update.md
> >> : > On Oct 1, 2020, 9:28 AM -0500, Jason Gerlowski ,
> >> : > wrote:
> >> : >
> >> : > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can
> >> : > someone please sanity check the summaries there when they get a
> >> : > chance? Would appreciate the review.
> >> : >
> >> : > 8.6.3 is a bit interesting in that Lucene has no changes in this
> >> : > bugfix release. As a result I had to omit the standard phrase in the
> >> : > Solr release notes about there being additional changes at the Lucene
> >> : > level, and change some of the wording in the Lucene announcement to
> >> : > indicate the lack of changes. So that's something to pay particular
> >> : > attention to, if someone can check my wording there.
> >> : >
> >> : > [1] 
> >> https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863
> >> : > [2]
> >> : > https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863
> >> : >
> >> : > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski 
> >> 
> >> : > wrote:
> >> : >
> >> : >
> >> : > The only one that was previously mentioned as a blocker was
> >> : > SOLR-14835, but from the comments on the ticket it looks like it ended
> >> : > up being purely a cosmetic issue. Andrze

Re: 8.6.3 Release

[Jason Gerlowski]

The traditional (non-docker) part of the release should now be wrapped
up.  Thanks everyone for the help and answering my questions here and
in Slack.  One final question:

The final releaseWizard.py step instructs:

"The Solr WIKI has a page for every version which is often linked to
from WIKI pages to indicate differences between versions, example:
http://wiki.apache.org/solr/Solr4.3. Do the following: Update the page
for the released version with release date and link to release
statement. Create a new placeholder page for the "next" version, if it
does not exist"

But looking at our wiki, the latest of these pages is 8.2
(https://cwiki.apache.org/confluence/display/SOLR/Solr8.2).  I've
created the pages as instructed for now.  But if we're not following
this step regularly and it hasn't caused any issues maybe we should
remove it from the release process altogether?

Jason

On Thu, Oct 8, 2020 at 5:16 PM David Smiley  wrote:
>
> The way GitHub works for contributors is that you are expected to fork a repo 
> and then push to your fork.  At that point when you go to the PR area, you'll 
> see a convenient yellow dialog to create a PR based on your pushed branch.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Oct 8, 2020 at 10:20 AM Chris Hostetter  
> wrote:
>>
>>
>> FWIW: I followed the docs to update the Dockerfiles + TAGS for 8.6.3, and
>> run tests; but since it's in a distinct github repo I don't think i can
>> push to it?
>>
>> so i creaed a GH issue w/patch...
>>
>> https://github.com/docker-solr/docker-solr/issues/349
>>
>>
>>
>> : Date: Tue, 6 Oct 2020 11:33:15 -0400
>> : From: Houston Putman 
>> : Reply-To: dev@lucene.apache.org
>> : To: Solr/Lucene Dev 
>> : Subject: Re: 8.6.3 Release
>> :
>> : That is correct. 8.x docker builds have not been affected in any way.
>> :
>> : On Tue, Oct 6, 2020 at 11:30 AM Cassandra Targett 
>> : wrote:
>> :
>> : > I wanted to ask now that the 8.6.3 vote is underway - for the docker-solr
>> : > image, are the update instructions in the docker-solr repo still the same
>> : > for 8.x even though the build process has been moved to the main project
>> : > for 9.0? Meaning, to release the 8.6.3 image there’s no change from 
>> before,
>> : > right?
>> : >
>> : > I’m asking specifically about these instructions:
>> : >
>> : > https://github.com/docker-solr/docker-solr/blob/master/update.md
>> : > On Oct 1, 2020, 9:28 AM -0500, Jason Gerlowski ,
>> : > wrote:
>> : >
>> : > I've put together draft Release Notes for 8.6.3 here. [1] [2]. Can
>> : > someone please sanity check the summaries there when they get a
>> : > chance? Would appreciate the review.
>> : >
>> : > 8.6.3 is a bit interesting in that Lucene has no changes in this
>> : > bugfix release. As a result I had to omit the standard phrase in the
>> : > Solr release notes about there being additional changes at the Lucene
>> : > level, and change some of the wording in the Lucene announcement to
>> : > indicate the lack of changes. So that's something to pay particular
>> : > attention to, if someone can check my wording there.
>> : >
>> : > [1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863
>> : > [2]
>> : > https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863
>> : >
>> : > On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski 
>> : > wrote:
>> : >
>> : >
>> : > The only one that was previously mentioned as a blocker was
>> : > SOLR-14835, but from the comments on the ticket it looks like it ended
>> : > up being purely a cosmetic issue. Andrzej left a comment there
>> : > suggesting that we "address" this with documentation for 8.6.3 but
>> : > otherwise leave it as-is.
>> : >
>> : > So it looks like we're unblocked on starting the release process.
>> : > Will begin the preliminary steps this afternoon.
>> : >
>> : > On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett 
>> : > wrote:
>> : >
>> : >
>> : > It looks to me like everything for 8.6.3 is resolved now (
>> : > https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it
>> : > seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a
>> : > Jetty upgrade less compelling to try.
>> : >
>> : > Are there any other issues not currently marked for 8.6.3 we’re waiting
>>

[ANNOUNCE] Apache Solr 8.6.3 released

[Jason Gerlowski]

The Lucene PMC is pleased to announce the release of Apache Solr 8.6.3.

Solr is the popular, blazing fast, open source NoSQL search platform
from the Apache Lucene project. Its major features include powerful
full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, rich document handling, and
geospatial search. Solr is highly scalable, providing fault tolerant
distributed search and indexing, and powers the search and navigation
features of many of the world's largest internet sites.

Solr 8.6.3 is available for immediate download at:
  

### Solr 8.6.3 Release Highlights:

 * SOLR-14898: Prevent duplicate header accumulation on internally
forwarded requests
 * SOLR-14768: Fix HTTP multipart POST requests to Solr (8.6.0 regression)
 * SOLR-14859: PrefixTree-based fields now reject invalid schema
properties instead of quietly failing certain queries
 * SOLR-14663: CREATE ConfigSet action now copies base node content

Please refer to the Upgrade Notes in the Solr Ref Guide for
information on upgrading from previous Solr versions:
  

Please read CHANGES.txt for a full list of bugfixes:
  

Solr 8.6.3 also includes bugfixes in the corresponding Apache Lucene release:
  

Note: The Apache Software Foundation uses an extensive mirroring network for
distributing releases. It is possible that the mirror you are using may not have
replicated the release yet. If that is the case, please try another mirror.
This also applies to Maven access.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[ANNOUNCE] Apache Lucene 8.6.3 released

[Jason Gerlowski]

The Lucene PMC is pleased to announce the release of Apache Lucene 8.6.3.

Apache Lucene is a high-performance, full-featured text search engine
library written entirely in Java. It is a technology suitable for
nearly any application that requires full-text search, especially
cross-platform.

This release contains no additional bug fixes over the previous
version 8.6.2. The release is available for immediate download at:

  

Note: The Apache Software Foundation uses an extensive mirroring network for
distributing releases. It is possible that the mirror you are using may not have
replicated the release yet. If that is the case, please try another mirror.

This also applies to Maven access.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release Lucene/Solr 8.6.3 RC1

[Jason Gerlowski]

It's been >72h since the vote was initiated (I left some extra time since
the vote started over the weekend) and the result is:


+1  11  (10 binding)

 0  0

-1  0


This vote has PASSED

Thanks everyone, I'll start working to finalize the release.

On Wed, Oct 7, 2020 at 10:22 AM Michael Sokolov  wrote:

> +1 (binding)
>
> SUCCESS! [0:56:16.661736]
>
> On Tue, Oct 6, 2020 at 2:18 AM Tomás Fernández Löbbe
>  wrote:
> >
> > +1 (binding)
> >
> > SUCCESS! [1:05:14.591357]
> >
> > On Mon, Oct 5, 2020 at 1:13 PM Anshum Gupta 
> wrote:
> >>
> >> +1 (binding)
> >>
> >> SUCCESS! [1:00:37.423566]
> >>
> >> Tried basic indexing and search and ran the smoke tester.
> >>
> >> On Sat, Oct 3, 2020 at 6:53 PM Jason Gerlowski 
> wrote:
> >>>
> >>> Please vote for release candidate 1 for Lucene/Solr 8.6.3
> >>>
> >>>
> >>> The artifacts can be downloaded from:
> >>>
> >>>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.3-RC1-reve001c2221812a0ba9e9378855040ce72f93eced4
> >>>
> >>>
> >>> You can run the smoke tester directly with this command:
> >>>
> >>>
> >>> python3 -u dev-tools/scripts/smokeTestRelease.py \
> >>>
> >>>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.3-RC1-reve001c2221812a0ba9e9378855040ce72f93eced4
> >>>
> >>>
> >>> The vote will be open for at least 72 hours i.e. until 2020-10-07
> 02:00 UTC.
> >>>
> >>>
> >>> [ ] +1  approve
> >>>
> >>> [ ] +0  no opinion
> >>>
> >>> [ ] -1  disapprove (and reason why)
> >>>
> >>>
> >>> Here is my +1
> >>
> >>
> >>
> >> --
> >> Anshum Gupta
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[VOTE] Release Lucene/Solr 8.6.3 RC1

[Jason Gerlowski]

Please vote for release candidate 1 for Lucene/Solr 8.6.3


The artifacts can be downloaded from:

https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.3-RC1-reve001c2221812a0ba9e9378855040ce72f93eced4


You can run the smoke tester directly with this command:


python3 -u dev-tools/scripts/smokeTestRelease.py \

https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.3-RC1-reve001c2221812a0ba9e9378855040ce72f93eced4


The vote will be open for at least 72 hours i.e. until 2020-10-07 02:00 UTC.


[ ] +1  approve

[ ] +0  no opinion

[ ] -1  disapprove (and reason why)


Here is my +1

Re: 8.6.3 Release

[Jason Gerlowski]

I've put together draft Release Notes for 8.6.3 here. [1] [2].  Can
someone please sanity check the summaries there when they get a
chance?  Would appreciate the review.

8.6.3 is a bit interesting in that Lucene has no changes in this
bugfix release.  As a result I had to omit the standard phrase in the
Solr release notes about there being additional changes at the Lucene
level, and change some of the wording in the Lucene announcement to
indicate the lack of changes.  So that's something to pay particular
attention to, if someone can check my wording there.

[1] https://cwiki.apache.org/confluence/display/SOLR/DRAFT-ReleaseNote863
[2] https://cwiki.apache.org/confluence/display/LUCENE/DRAFT-ReleaseNote863

On Wed, Sep 30, 2020 at 10:57 AM Jason Gerlowski  wrote:
>
> The only one that was previously mentioned as a blocker was
> SOLR-14835, but from the comments on the ticket it looks like it ended
> up being purely a cosmetic issue.  Andrzej left a comment there
> suggesting that we "address" this with documentation for 8.6.3 but
> otherwise leave it as-is.
>
> So it looks like we're unblocked on starting the release process.
> Will begin the preliminary steps this afternoon.
>
> On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett  
> wrote:
> >
> > It looks to me like everything for 8.6.3 is resolved now 
> > (https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it 
> > seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a 
> > Jetty upgrade less compelling to try.
> >
> > Are there any other issues not currently marked for 8.6.3 we’re waiting for 
> > before starting the RC?
> > On Sep 29, 2020, 12:04 PM -0500, Jason Gerlowski , 
> > wrote:
> >
> > That said, if someone can use 8.6.3, what’s stopping them from going to 8.7 
> > when it’e released?
> >
> >
> > The same things that always stop users from going directly to the
> > latest-and-greatest: fear of instability from new minor-release
> > features, reliance on behavior changed across minor versions, breaking
> > changes on Lucene elements that don't guarantee backcompat (e.g.
> > SOLR-14254), security issues in later versions (new libraries pulled
> > in with vulns), etc. There's lots of reasons a given user might want
> > to stick on 8.6.x rather than 8.7 (in the short/medium term).
> >
> > I'm ambivalent to whether we upgrade Jetty in 8.6.3 - as I said above
> > the worst of the Jetty issue should be mitigated by work on our end -
> > but I think there's a lot of reasons users might not upgrade as far as
> > we'd expect/like.
> >
> >
> > On Mon, Sep 28, 2020 at 2:05 PM Erick Erickson  
> > wrote:
> >
> >
> > For me, there’s a sharp distinction between changing a dependency in a 
> > point release just because there’s a new version, and changing the 
> > dependency because there’s a bug in it. That said, if someone can use 
> > 8.6.3, what’s stopping them from going to 8.7 when it’e released? Would it 
> > make more sense to do the upgrades for 8.7 and get that out the door rather 
> > than backport?
> >
> > FWIW,
> > Erick
> >
> > On Sep 28, 2020, at 1:45 PM, Jason Gerlowski  wrote:
> >
> > Hey all,
> >
> > I wanted to add 2 more blocker tickets to the list: SOLR-14897 and
> > SOLR-14898. These tickets (while bad bugs in their own right) are
> > especially necessary because they work around a Jetty buffer-reuse bug
> > (see SOLR-14896) that causes sporadic request failures once triggered.
> >
> > So that brings the list of 8.6.3 blockers up to: SOLR-14850,
> > SOLR-14835, SOLR-14897, and SOLR-14898. (Thanks David for the quick
> > work on SOLR-14768!)
> >
> > Additionally, should we also consider a Jetty upgrade for 8.6.3 in
> > light of the issue mentioned above? I know it's atypical for bug-fix
> > releases to change deps, but here the bug is serious and tied directly
> > to the dep. SOLR-14897 and SOLR-14898 help greatly here, but the
> > Jetty bug is likely still a problem for users making requests that
> > match a specific (albeit rare) profile. Anyone have thoughts?
> >
> > Best,
> >
> > Jason
> >
> > On Fri, Sep 25, 2020 at 12:28 AM Houston Putman  
> > wrote:
> >
> >
> > If I recall correctly, thats a step in the release wizard.
> >
> > After checking, I think this fits the bill:
> > https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/releaseWizard.yaml#L1435
> >
> > - Houston
> >
> > On Fri, Sep 25, 2020 at 12:06 AM David Smiley  wrote:
> >
> >
> > When movi

Re: 8.6.3 Release

[Jason Gerlowski]

The only one that was previously mentioned as a blocker was
SOLR-14835, but from the comments on the ticket it looks like it ended
up being purely a cosmetic issue.  Andrzej left a comment there
suggesting that we "address" this with documentation for 8.6.3 but
otherwise leave it as-is.

So it looks like we're unblocked on starting the release process.
Will begin the preliminary steps this afternoon.

On Tue, Sep 29, 2020 at 3:40 PM Cassandra Targett  wrote:
>
> It looks to me like everything for 8.6.3 is resolved now 
> (https://issues.apache.org/jira/projects/SOLR/versions/12348713), and it 
> seems from comments in SOLR-14897 and SOLR-14898 that those fixes make a 
> Jetty upgrade less compelling to try.
>
> Are there any other issues not currently marked for 8.6.3 we’re waiting for 
> before starting the RC?
> On Sep 29, 2020, 12:04 PM -0500, Jason Gerlowski , 
> wrote:
>
> That said, if someone can use 8.6.3, what’s stopping them from going to 8.7 
> when it’e released?
>
>
> The same things that always stop users from going directly to the
> latest-and-greatest: fear of instability from new minor-release
> features, reliance on behavior changed across minor versions, breaking
> changes on Lucene elements that don't guarantee backcompat (e.g.
> SOLR-14254), security issues in later versions (new libraries pulled
> in with vulns), etc. There's lots of reasons a given user might want
> to stick on 8.6.x rather than 8.7 (in the short/medium term).
>
> I'm ambivalent to whether we upgrade Jetty in 8.6.3 - as I said above
> the worst of the Jetty issue should be mitigated by work on our end -
> but I think there's a lot of reasons users might not upgrade as far as
> we'd expect/like.
>
>
> On Mon, Sep 28, 2020 at 2:05 PM Erick Erickson  
> wrote:
>
>
> For me, there’s a sharp distinction between changing a dependency in a point 
> release just because there’s a new version, and changing the dependency 
> because there’s a bug in it. That said, if someone can use 8.6.3, what’s 
> stopping them from going to 8.7 when it’e released? Would it make more sense 
> to do the upgrades for 8.7 and get that out the door rather than backport?
>
> FWIW,
> Erick
>
> On Sep 28, 2020, at 1:45 PM, Jason Gerlowski  wrote:
>
> Hey all,
>
> I wanted to add 2 more blocker tickets to the list: SOLR-14897 and
> SOLR-14898. These tickets (while bad bugs in their own right) are
> especially necessary because they work around a Jetty buffer-reuse bug
> (see SOLR-14896) that causes sporadic request failures once triggered.
>
> So that brings the list of 8.6.3 blockers up to: SOLR-14850,
> SOLR-14835, SOLR-14897, and SOLR-14898. (Thanks David for the quick
> work on SOLR-14768!)
>
> Additionally, should we also consider a Jetty upgrade for 8.6.3 in
> light of the issue mentioned above? I know it's atypical for bug-fix
> releases to change deps, but here the bug is serious and tied directly
> to the dep. SOLR-14897 and SOLR-14898 help greatly here, but the
> Jetty bug is likely still a problem for users making requests that
> match a specific (albeit rare) profile. Anyone have thoughts?
>
> Best,
>
> Jason
>
> On Fri, Sep 25, 2020 at 12:28 AM Houston Putman  
> wrote:
>
>
> If I recall correctly, thats a step in the release wizard.
>
> After checking, I think this fits the bill:
> https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/releaseWizard.yaml#L1435
>
> - Houston
>
> On Fri, Sep 25, 2020 at 12:06 AM David Smiley  wrote:
>
>
> When moving changes from 8.7 to 8.6.3, must we (the mover of an individual 
> change) move the CHANGES.txt entry on all branches -- master, branch_8x, 
> branch_8_6? I expect the release branch but am unsure of the other two. In 
> the past I have but it's annoying. Does the RM sync CHANGES.txt on the other 
> branches in one go? If not, I think it'd make sense for that to happen.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Sep 24, 2020 at 6:22 AM Atri Sharma  wrote:
>
>
> I will push the 8.7 release by a week to give Jason enough headroom to
>
>
> do the 8.6.3 release.
>
>
>
>
>
> Jason, let me know if you need me to assist on the 8.6.3 release.
>
>
>
>
>
> On Thu, Sep 24, 2020 at 3:23 PM Jason Gerlowski  wrote:
>
>
>
>
>
> OK, in that case I'll try my best to keep the 8.6.3 process moving
>
>
>
> then, so Atri can stick as close to his proposed schedule as possible.
>
>
>
> My apologies - I didn't realize I'd be putting the brakes on 8.7 by
>
>
>
> proposing a bug-fix release. But the reasons make sense given what
>
>

Re: 8.6.3 Release

[Jason Gerlowski]

> That said, if someone can use 8.6.3, what’s stopping them from going to 8.7 
> when it’e released?

The same things that always stop users from going directly to the
latest-and-greatest: fear of instability from new minor-release
features, reliance on behavior changed across minor versions, breaking
changes on Lucene elements that don't guarantee backcompat (e.g.
SOLR-14254), security issues in later versions (new libraries pulled
in with vulns), etc.  There's lots of reasons a given user might want
to stick on 8.6.x rather than 8.7 (in the short/medium term).

I'm ambivalent to whether we upgrade Jetty in 8.6.3 - as I said above
the worst of the Jetty issue should be mitigated by work on our end -
but I think there's a lot of reasons users might not upgrade as far as
we'd expect/like.


On Mon, Sep 28, 2020 at 2:05 PM Erick Erickson  wrote:
>
> For me, there’s a sharp distinction between changing a dependency in a point 
> release just because there’s a new version, and changing the dependency 
> because there’s a bug in it. That said, if someone can use 8.6.3, what’s 
> stopping them from going to 8.7 when it’e released? Would it make more sense 
> to do the upgrades for 8.7 and get that out the door rather than backport?
>
> FWIW,
> Erick
>
> > On Sep 28, 2020, at 1:45 PM, Jason Gerlowski  wrote:
> >
> > Hey all,
> >
> > I wanted to add 2 more blocker tickets to the list: SOLR-14897 and
> > SOLR-14898.  These tickets (while bad bugs in their own right) are
> > especially necessary because they work around a Jetty buffer-reuse bug
> > (see SOLR-14896) that causes sporadic request failures once triggered.
> >
> > So that brings the list of 8.6.3 blockers up to: SOLR-14850,
> > SOLR-14835, SOLR-14897, and SOLR-14898.  (Thanks David for the quick
> > work on SOLR-14768!)
> >
> > Additionally, should we also consider a Jetty upgrade for 8.6.3 in
> > light of the issue mentioned above?  I know it's atypical for bug-fix
> > releases to change deps, but here the bug is serious and tied directly
> > to the dep.  SOLR-14897 and SOLR-14898 help greatly here, but the
> > Jetty bug is likely still a problem for users making requests that
> > match a specific (albeit rare) profile.  Anyone have thoughts?
> >
> > Best,
> >
> > Jason
> >
> > On Fri, Sep 25, 2020 at 12:28 AM Houston Putman  
> > wrote:
> >>
> >> If I recall correctly, thats a step in the release wizard.
> >>
> >> After checking, I think this fits the bill:
> >> https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/releaseWizard.yaml#L1435
> >>
> >> - Houston
> >>
> >> On Fri, Sep 25, 2020 at 12:06 AM David Smiley  wrote:
> >>>
> >>> When moving changes from 8.7 to 8.6.3, must we (the mover of an 
> >>> individual change) move the CHANGES.txt entry on all branches -- master, 
> >>> branch_8x, branch_8_6?  I expect the release branch but am unsure of the 
> >>> other two.  In the past I have but it's annoying.  Does the RM sync 
> >>> CHANGES.txt on the other branches in one go?  If not, I think it'd make 
> >>> sense for that to happen.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>>
> >>>
> >>> On Thu, Sep 24, 2020 at 6:22 AM Atri Sharma  wrote:
> >>>>
> >>>> I will push the 8.7 release by a week to give Jason enough headroom to
> >>>>
> >>>>
> >>>> do the 8.6.3 release.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Jason, let me know if you need me to assist on the 8.6.3 release.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Sep 24, 2020 at 3:23 PM Jason Gerlowski  
> >>>> wrote:
> >>>>
> >>>>
> >>>>>
> >>>>
> >>>>
> >>>>> OK, in that case I'll try my best to keep the 8.6.3 process moving
> >>>>
> >>>>
> >>>>> then, so Atri can stick as close to his proposed schedule as possible.
> >>>>
> >>>>
> >>>>> My apologies - I didn't realize I'd be putting the brakes on 8.7 by
> >>>>
> >>>>
> >>>>> proposing a bug-fix release.  But the reasons make sense given what
> >>>>
> >>>>
> >>>>> others

Re: 8.6.3 Release

[Jason Gerlowski]

Hey all,

I wanted to add 2 more blocker tickets to the list: SOLR-14897 and
SOLR-14898.  These tickets (while bad bugs in their own right) are
especially necessary because they work around a Jetty buffer-reuse bug
(see SOLR-14896) that causes sporadic request failures once triggered.

So that brings the list of 8.6.3 blockers up to: SOLR-14850,
SOLR-14835, SOLR-14897, and SOLR-14898.  (Thanks David for the quick
work on SOLR-14768!)

Additionally, should we also consider a Jetty upgrade for 8.6.3 in
light of the issue mentioned above?  I know it's atypical for bug-fix
releases to change deps, but here the bug is serious and tied directly
to the dep.  SOLR-14897 and SOLR-14898 help greatly here, but the
Jetty bug is likely still a problem for users making requests that
match a specific (albeit rare) profile.  Anyone have thoughts?

Best,

Jason

On Fri, Sep 25, 2020 at 12:28 AM Houston Putman  wrote:
>
> If I recall correctly, thats a step in the release wizard.
>
> After checking, I think this fits the bill:
> https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/releaseWizard.yaml#L1435
>
> - Houston
>
> On Fri, Sep 25, 2020 at 12:06 AM David Smiley  wrote:
>>
>> When moving changes from 8.7 to 8.6.3, must we (the mover of an individual 
>> change) move the CHANGES.txt entry on all branches -- master, branch_8x, 
>> branch_8_6?  I expect the release branch but am unsure of the other two.  In 
>> the past I have but it's annoying.  Does the RM sync CHANGES.txt on the 
>> other branches in one go?  If not, I think it'd make sense for that to 
>> happen.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Thu, Sep 24, 2020 at 6:22 AM Atri Sharma  wrote:
>>>
>>> I will push the 8.7 release by a week to give Jason enough headroom to
>>>
>>>
>>> do the 8.6.3 release.
>>>
>>>
>>>
>>>
>>>
>>> Jason, let me know if you need me to assist on the 8.6.3 release.
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Sep 24, 2020 at 3:23 PM Jason Gerlowski  
>>> wrote:
>>>
>>>
>>> >
>>>
>>>
>>> > OK, in that case I'll try my best to keep the 8.6.3 process moving
>>>
>>>
>>> > then, so Atri can stick as close to his proposed schedule as possible.
>>>
>>>
>>> > My apologies - I didn't realize I'd be putting the brakes on 8.7 by
>>>
>>>
>>> > proposing a bug-fix release.  But the reasons make sense given what
>>>
>>>
>>> > others mentioned above.
>>>
>>>
>>> >
>>>
>>>
>>> > > As branch_8_6 should be pretty stable by now I wonder if we really need 
>>> > > to wait one week?
>>>
>>>
>>> >
>>>
>>>
>>> > There's no special reason on my end.  I suggested a week to give
>>>
>>>
>>> > others time to backport anything they wanted included, but I'm happy
>>>
>>>
>>> > to start the process as soon as all the expected changes land.
>>>
>>>
>>> >
>>>
>>>
>>> > Best,
>>>
>>>
>>> >
>>>
>>>
>>> > Jason
>>>
>>>
>>> >
>>>
>>>
>>> > On Thu, Sep 24, 2020 at 1:48 AM Anshum Gupta  
>>> > wrote:
>>>
>>>
>>> > >
>>>
>>>
>>> > > Simultaneous releases are also confusing for users, in addition to the 
>>> > > back-compat tests as our website chronologically lists our releases and 
>>> > > it gets complicated for someone reading the 'News' page.
>>>
>>>
>>> > >
>>>
>>>
>>> > > As 8.7 isn't a release that needs to be rushed, waiting until 8.6.3 is 
>>> > > released and back-compat indexes are pushed will make things easier for 
>>> > > the RMs and community.
>>>
>>>
>>> > >
>>>
>>>
>>> > > On Wed, Sep 23, 2020 at 1:43 PM David Smiley  wrote:
>>>
>>>
>>> > >>
>>>
>>>
>>> > >> Jason: Thanks for volunteering to do an 8.6.3!  I recently fixed 
>>> > >> SOLR-14768, multipart HTTP POST was broken in 8.6 (a regression I 
>>> > >> introduced).  If you can't do the rele

Re: Filestore, subsume userfiles requirement

[Jason Gerlowski]

I think some expressions are SolrCloud specific, but there are
expressions that work in standalone

e.g. https://paste.apache.org/zm3sw and https://paste.apache.org/vs35m

On Fri, Sep 25, 2020 at 11:53 AM Eric Pugh
 wrote:
>
> Can you do streaming expression in standalone solr?   I was under the 
> impression that it required SolrCloud.
>
>
> On Sep 25, 2020, at 11:45 AM, Alexandre Rafalovitch  
> wrote:
>
> It is not just a directory. There is a security feature that only
> allows to read from several file locations (configurable in solr.xml).
> Userfiles is part of default list, so whatever replacement is there
> will need to be explicitly-named as well.
>
> Regards,
>   Alex.
>
> On Fri, 25 Sep 2020 at 11:43, Jason Gerlowski  wrote:
>
>
> You're left with the /filestore/ dir to put there what you want in 
> standalone mode.
>
> That's fine by me as that's effectively what "userfiles" provides now.
> The main difference I imagine is that the 'filestore' directory won't
> exist at all in standalone? In that case standalone users will have to
> know where to create the dir if they want to reference 'filestore'
> files in their streaming expressions.  But that's not all that much
> more to ask I guess.
>
> I'm curious - is the filestore unimplemented in standalone because no
> one prioritized it, or is there a specific technical blocker?
>
> On Fri, Sep 25, 2020 at 11:24 AM David Smiley  wrote:
>
>
> The "userfiles" thing is merely a directory.  There's no CRUD API on it.  
> Even if the CRUD API of the file store isn't active in standalone -- I think 
> that doesn't matter.  You're left with the /filestore/ dir to put 
> there what you want in standalone mode.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Sep 25, 2020 at 11:00 AM Ishan Chattopadhyaya 
>  wrote:
>
>
> Standalone isn't supported.
>
> We need to transition away from the standalone mode and get rid of it 
> completely.
>
> On Fri, 25 Sep, 2020, 7:50 pm Jason Gerlowski,  wrote:
>
>
> I don't know much about the new package/file-store, but it does sound
> like a good replacement for 'userfiles' (which Eric is right- is
> pretty awkward to work with).
>
> My only concerns with using the filestore would be around its
> availability.  Is it enabled by default (or will it be in an upcoming
> release)?  Does it work in standalone as well as SolrCloud?  If the
> answer to those questions are both "yes", then it makes sense to me as
> a replacement on the face of things.
>
> Jason
>
> On Fri, Sep 25, 2020 at 9:39 AM Eric Pugh
>  wrote:
>
>
> +1.  I’ve found the user files really useful when doing things with 
> streaming, but it’s also awkward to reach to put files into.
>
>
> On Sep 25, 2020, at 8:47 AM, David Smiley  wrote:
>
> Yes.  And I think it's high time that coreRootDirectory default to 
> /something (e.g. "cores")
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Sep 25, 2020 at 8:42 AM Alexandre Rafalovitch  
> wrote:
>
>
> Would that file store also be under solr.home? Because if it is and
> the user can upload core.property into it as well as other things that
> core discovery will then load bypassing security
>
> Regards,
>  Alex.
>
> On Fri, 25 Sep 2020 at 08:32, David Smiley  wrote:
>
>
> I'm looking to see that we can deprecate "userfiles" and remove for 9.0
>
> Solr has a "userfiles" directory under Solr home that Jason added in some 
> issue relating to streaming expressions accessing a local file.  I bet only a 
> few of you have even heard of it.  I think the "File Store" that came along 
> with the package manager obsoletes "userfiles".  If you have not heard of the 
> file store either, I wouldn't be surprised -- it's new and it's name was 
> changed from "package store" last minute, since it's general purpose, with 
> "packages" being a directory at the root of the file store for packages.  
> It's documented as the "package store" (should be renamed) on the package 
> manager internals doc: 
> https://lucene.apache.org/solr/guide/8_6/package-manager-internals.html#package-store
> However IMO it's worthy of its own doc page as it's a very useful new 
> component of the Solr platform.  It can store "user files" (hence obsoleting 
> the userfiles dir), ML models, or basically any file that's too large to put 
> in ZK.  I'd be nice if SolrResourceLoader could resolve resources from it -- 
> an issue for another day.  That wo

Re: Filestore, subsume userfiles requirement

[Jason Gerlowski]

> You're left with the /filestore/ dir to put there what you want in 
> standalone mode.
That's fine by me as that's effectively what "userfiles" provides now.
The main difference I imagine is that the 'filestore' directory won't
exist at all in standalone? In that case standalone users will have to
know where to create the dir if they want to reference 'filestore'
files in their streaming expressions.  But that's not all that much
more to ask I guess.

I'm curious - is the filestore unimplemented in standalone because no
one prioritized it, or is there a specific technical blocker?

On Fri, Sep 25, 2020 at 11:24 AM David Smiley  wrote:
>
> The "userfiles" thing is merely a directory.  There's no CRUD API on it.  
> Even if the CRUD API of the file store isn't active in standalone -- I think 
> that doesn't matter.  You're left with the /filestore/ dir to put 
> there what you want in standalone mode.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Sep 25, 2020 at 11:00 AM Ishan Chattopadhyaya 
>  wrote:
>>
>> Standalone isn't supported.
>>
>> We need to transition away from the standalone mode and get rid of it 
>> completely.
>>
>> On Fri, 25 Sep, 2020, 7:50 pm Jason Gerlowski,  wrote:
>>>
>>> I don't know much about the new package/file-store, but it does sound
>>> like a good replacement for 'userfiles' (which Eric is right- is
>>> pretty awkward to work with).
>>>
>>> My only concerns with using the filestore would be around its
>>> availability.  Is it enabled by default (or will it be in an upcoming
>>> release)?  Does it work in standalone as well as SolrCloud?  If the
>>> answer to those questions are both "yes", then it makes sense to me as
>>> a replacement on the face of things.
>>>
>>> Jason
>>>
>>> On Fri, Sep 25, 2020 at 9:39 AM Eric Pugh
>>>  wrote:
>>> >
>>> > +1.  I’ve found the user files really useful when doing things with 
>>> > streaming, but it’s also awkward to reach to put files into.
>>> >
>>> >
>>> > On Sep 25, 2020, at 8:47 AM, David Smiley  wrote:
>>> >
>>> > Yes.  And I think it's high time that coreRootDirectory default to 
>>> > /something (e.g. "cores")
>>> >
>>> > ~ David Smiley
>>> > Apache Lucene/Solr Search Developer
>>> > http://www.linkedin.com/in/davidwsmiley
>>> >
>>> >
>>> > On Fri, Sep 25, 2020 at 8:42 AM Alexandre Rafalovitch 
>>> >  wrote:
>>> >>
>>> >> Would that file store also be under solr.home? Because if it is and
>>> >> the user can upload core.property into it as well as other things that
>>> >> core discovery will then load bypassing security
>>> >>
>>> >> Regards,
>>> >>   Alex.
>>> >>
>>> >> On Fri, 25 Sep 2020 at 08:32, David Smiley  wrote:
>>> >> >
>>> >> > I'm looking to see that we can deprecate "userfiles" and remove for 9.0
>>> >> >
>>> >> > Solr has a "userfiles" directory under Solr home that Jason added in 
>>> >> > some issue relating to streaming expressions accessing a local file.  
>>> >> > I bet only a few of you have even heard of it.  I think the "File 
>>> >> > Store" that came along with the package manager obsoletes "userfiles". 
>>> >> >  If you have not heard of the file store either, I wouldn't be 
>>> >> > surprised -- it's new and it's name was changed from "package store" 
>>> >> > last minute, since it's general purpose, with "packages" being a 
>>> >> > directory at the root of the file store for packages.  It's documented 
>>> >> > as the "package store" (should be renamed) on the package manager 
>>> >> > internals doc: 
>>> >> > https://lucene.apache.org/solr/guide/8_6/package-manager-internals.html#package-store
>>> >> > However IMO it's worthy of its own doc page as it's a very useful new 
>>> >> > component of the Solr platform.  It can store "user files" (hence 
>>> >> > obsoleting the userfiles dir), ML models, or basically any file that's 
>>> >> > too large to put in ZK.  I'd be nice if SolrResour

Re: Filestore, subsume userfiles requirement

[Jason Gerlowski]

I don't know much about the new package/file-store, but it does sound
like a good replacement for 'userfiles' (which Eric is right- is
pretty awkward to work with).

My only concerns with using the filestore would be around its
availability.  Is it enabled by default (or will it be in an upcoming
release)?  Does it work in standalone as well as SolrCloud?  If the
answer to those questions are both "yes", then it makes sense to me as
a replacement on the face of things.

Jason

On Fri, Sep 25, 2020 at 9:39 AM Eric Pugh
 wrote:
>
> +1.  I’ve found the user files really useful when doing things with 
> streaming, but it’s also awkward to reach to put files into.
>
>
> On Sep 25, 2020, at 8:47 AM, David Smiley  wrote:
>
> Yes.  And I think it's high time that coreRootDirectory default to 
> /something (e.g. "cores")
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Sep 25, 2020 at 8:42 AM Alexandre Rafalovitch  
> wrote:
>>
>> Would that file store also be under solr.home? Because if it is and
>> the user can upload core.property into it as well as other things that
>> core discovery will then load bypassing security
>>
>> Regards,
>>   Alex.
>>
>> On Fri, 25 Sep 2020 at 08:32, David Smiley  wrote:
>> >
>> > I'm looking to see that we can deprecate "userfiles" and remove for 9.0
>> >
>> > Solr has a "userfiles" directory under Solr home that Jason added in some 
>> > issue relating to streaming expressions accessing a local file.  I bet 
>> > only a few of you have even heard of it.  I think the "File Store" that 
>> > came along with the package manager obsoletes "userfiles".  If you have 
>> > not heard of the file store either, I wouldn't be surprised -- it's new 
>> > and it's name was changed from "package store" last minute, since it's 
>> > general purpose, with "packages" being a directory at the root of the file 
>> > store for packages.  It's documented as the "package store" (should be 
>> > renamed) on the package manager internals doc: 
>> > https://lucene.apache.org/solr/guide/8_6/package-manager-internals.html#package-store
>> > However IMO it's worthy of its own doc page as it's a very useful new 
>> > component of the Solr platform.  It can store "user files" (hence 
>> > obsoleting the userfiles dir), ML models, or basically any file that's too 
>> > large to put in ZK.  I'd be nice if SolrResourceLoader could resolve 
>> > resources from it -- an issue for another day.  That would be another 
>> > avenue of use separate from the configSet.  You can already upload single 
>> > files to the file store :-)
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> http://www.opensourceconnections.com | My Free/Busy
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> This e-mail and all contents, including attachments, is considered to be 
> Company Confidential unless explicitly stated otherwise, regardless of 
> whether attachments are marked as such.
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: bin/solr testing surprise with techproducts example

[Jason Gerlowski]

I couldn't reproduce your error on running techproducts.  Though
whatever is causing it locally for you sounds a bit related to
SOLR-13690 maybe?

Jason

On Wed, Sep 23, 2020 at 11:28 AM Munendra S N  wrote:
>
> The wiki has steps to build solr with gradle
> https://cwiki.apache.org/confluence/display/SOLR/Building+Solr+with+Gradle
>
> ./gradlew assemble or ./gradlew dev will create runnable solr instance.
>
>
> On Wed, Sep 23, 2020, 8:01 PM Christine Poerschke (BLOOMBERG/ LONDON) 
>  wrote:
>>
>> Hello everyone.
>>
>> So I was trying to locally test the small 
>> https://issues.apache.org/jira/browse/SOLR-11167 change on master branch and 
>> encountered two things:
>>
>> Question: What is the replacement for "cd solr ; ant dist server" usage?
>>
>> If there is an equivalent -- "./gradlew -p solr assembleDist" perhaps? -- 
>> then I'd be happy to update 
>> https://github.com/apache/lucene-solr/blob/master/help/ant.txt with the info.
>>
>> Observation: "cd solr ; bin/solr start -e techproducts" on master branch 
>> (but not branch_8x) gives me an error. Is this a known issue already or if 
>> not could someone try to reproduce the issue before a JIRA ticket is opened?
>>
>> ERROR: Error CREATEing SolrCore 'techproducts': Unable to create core 
>> [techproducts] Caused by: [schema.xml] analyzer/tokenizer: missing mandatory 
>> attribute 'class'
>>
>> Thanks,
>>
>> Christine

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 8.6.3 Release

[Jason Gerlowski]

OK, in that case I'll try my best to keep the 8.6.3 process moving
then, so Atri can stick as close to his proposed schedule as possible.
My apologies - I didn't realize I'd be putting the brakes on 8.7 by
proposing a bug-fix release.  But the reasons make sense given what
others mentioned above.

> As branch_8_6 should be pretty stable by now I wonder if we really need to 
> wait one week?

There's no special reason on my end.  I suggested a week to give
others time to backport anything they wanted included, but I'm happy
to start the process as soon as all the expected changes land.

Best,

Jason

On Thu, Sep 24, 2020 at 1:48 AM Anshum Gupta  wrote:
>
> Simultaneous releases are also confusing for users, in addition to the 
> back-compat tests as our website chronologically lists our releases and it 
> gets complicated for someone reading the 'News' page.
>
> As 8.7 isn't a release that needs to be rushed, waiting until 8.6.3 is 
> released and back-compat indexes are pushed will make things easier for the 
> RMs and community.
>
> On Wed, Sep 23, 2020 at 1:43 PM David Smiley  wrote:
>>
>> Jason: Thanks for volunteering to do an 8.6.3!  I recently fixed SOLR-14768, 
>> multipart HTTP POST was broken in 8.6 (a regression I introduced).  If you 
>> can't do the release or need help, I will take over.  It's the least I can 
>> offer in repentance for the regression.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>>
>> On Wed, Sep 23, 2020 at 10:07 AM Jason Gerlowski  
>> wrote:
>>>
>>> Hi all,
>>>
>>> I ran into a query-parsing bug recently in SOLR-14859 that caused
>>> problems for some of my usecases.  I wanted to volunteer as RM for an
>>> 8.6.3 to get a bugfix release out for users that aren't ready for some
>>> of the bigger changes in 8.7
>>>
>>> I was thinking of cutting the branch in a week's time to give others a
>>> chance to backport any bug-fixes they might want included, with an RC
>>> to follow shortly.  Does anyone have any concerns with that plan, or
>>> have anything they'd like to fix or backport before an 8.6.3 goes out?
>>>
>>> Best,
>>>
>>> Jason
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>
>
> --
> Anshum Gupta

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 8.6.3 Release

[Jason Gerlowski]

> do we want to delay that?

It wasn't my intention to delay 8.7.  The releases will end up being
close together, but they're on different release lines so there's
nothing wrong with that necessarily.  Unless there's something
inherent to the release process that makes it difficult to have two
in-flight simultaneously? (I haven't RM'd a release yet, so there may
well be.  Though now that I think back I'm pretty sure other
committers have done this sort of thing before...)

Was there a particular reason you brought up delaying 8.7?

Jason


On Wed, Sep 23, 2020 at 11:11 AM Atri Sharma  wrote:
>
> I was planning on a branch cut on 30th September for 8.7 — do we want to 
> delay that?
>
> On Wed, 23 Sep 2020 at 19:37, Jason Gerlowski  wrote:
>>
>> Hi all,
>>
>>
>>
>> I ran into a query-parsing bug recently in SOLR-14859 that caused
>>
>> problems for some of my usecases.  I wanted to volunteer as RM for an
>>
>> 8.6.3 to get a bugfix release out for users that aren't ready for some
>>
>> of the bigger changes in 8.7
>>
>>
>>
>> I was thinking of cutting the branch in a week's time to give others a
>>
>> chance to backport any bug-fixes they might want included, with an RC
>>
>> to follow shortly.  Does anyone have any concerns with that plan, or
>>
>> have anything they'd like to fix or backport before an 8.6.3 goes out?
>>
>>
>>
>> Best,
>>
>>
>>
>> Jason
>>
>>
>>
>> -
>>
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>>
> --
> Regards,
>
> Atri
> Apache Concerted

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

8.6.3 Release

[Jason Gerlowski]

Hi all,

I ran into a query-parsing bug recently in SOLR-14859 that caused
problems for some of my usecases.  I wanted to volunteer as RM for an
8.6.3 to get a bugfix release out for users that aren't ready for some
of the bigger changes in 8.7

I was thinking of cutting the branch in a week's time to give others a
chance to backport any bug-fixes they might want included, with an RC
to follow shortly.  Does anyone have any concerns with that plan, or
have anything they'd like to fix or backport before an 8.6.3 goes out?

Best,

Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Our test failure rate is unacceptable.

[Jason Gerlowski]

> I don't think it is along the Apache way to revert somebody's commit without 
> an explicit permission to do so
Interesting, I made the Devil's Advocate argument above with the
Apache Way specifically in mind.

"Community over Code" comes up most often in terms of navigating
interpersonal conflict and fostering contributors; that's valid and
important.  But broken builds cause concrete "Community harm" as well.
100%-fails slow down every developer working on Solr for whatever
length of time the project is in that state.  Established committers,
first-PR contributors, Github forks, everyone.  Leaving 100%-fails out
there while waiting for a committer to respond or fix an issue
prolongs that period: slowing down development and turning off new
potential contributors all the while.  So I think there's a concrete
Apache Way argument for reverting early.

Obviously the revert has to be done diplomatically or it risks
alienating committers and undermining the other Apache Way benefits.
But that's a question of execution not of approach.  There are
tactful, inoffensive ways to roll back a change without implying
negligence, impugning skill-sets, etc.   It's also not a concern
that's specific to reverts - any JIRA comment or dev-list discussion
pointing out issues runs into that.

All that said, this is a Devil's Advocate argument I'm making here.  I
have no plans to go around reverting other's commits; I was just
curious where others were on this in case it came up again later.

Best,

Jason

On Fri, Sep 18, 2020 at 3:59 PM Tomás Fernández Löbbe
 wrote:
>
> I thought we were talking about reverting your own commits, not someone 
> else’s...
>
> On Fri, Sep 18, 2020 at 12:31 PM Dawid Weiss  wrote:
>>
>> I don't think it is along the Apache way to revert somebody's commit
>>
>> without an explicit permission to do so... Not that I would personally
>>
>> mind if somebody did it for me.
>>
>>
>>
>> On Fri, Sep 18, 2020 at 9:06 PM Tomás Fernández Löbbe
>>
>>  wrote:
>>
>> >
>>
>> > Sometimes Jenkins may take hours to take your commit, may fail in the 
>> > middle of your night, may not fail consistently, etc. That's why I don't 
>> > think giving specific timeframes makes sense, but yes, as soon as you 
>> > notice it's failing, it's either fix immediately or revert IMO.
>>
>> >
>>
>> > On Fri, Sep 18, 2020 at 12:03 PM Jason Gerlowski  
>> > wrote:
>>
>> >>
>>
>> >> > If it’s inadvertently added, we either fix it within an hour or so or 
>> >> > revert the offending commit
>>
>> >>
>>
>> >> > I don't want to set specific time frames,
>>
>> >>
>>
>> >> To play Devil's Advocate here: why wait even an hour to revert a 100%
>>
>> >> test failure?  Reverts are usually trivial to do, unblock others
>>
>> >> immediately, and don't interfere with the fix process at all.
>>
>> >> Remembering the times I've broken the build myself, reverts even seem
>>
>> >> preferable from that position - reverting up front takes all the
>>
>> >> time-pressure off of getting out a fix.  Why work under the gun when
>>
>> >> you don't have to?
>>
>> >>
>>
>> >> On Fri, Sep 18, 2020 at 1:14 PM Tomás Fernández Löbbe
>>
>> >>  wrote:
>>
>> >> >
>>
>> >> > I believe these failures are associated to 
>> >> > https://issues.apache.org/jira/browse/SOLR-14151
>>
>> >> >
>>
>> >> > • FAILED:  org.apache.solr.pkg.TestPackages.classMethod
>>
>> >> > • FAILED:  
>> >> > org.apache.solr.schema.PreAnalyzedFieldManagedSchemaCloudTest.testAdd2Fields
>>
>> >> > • FAILED:  
>> >> > org.apache.solr.schema.ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin
>>
>> >> >
>>
>> >> > > IMO if a temporary instability is to be introduced deliberately, it 
>> >> > > should be published on the list. If it’s inadvertently added, we 
>> >> > > either fix it within an hour or so or revert the offending commit
>>
>> >> > I don't want to set specific time frames, but sometimes it's obviously 
>> >> > too much time.
>>
>> >> >
>>
>> >> > On Fri, Sep 18, 2020 at 8:48 AM Atri Sharma  wrote:
>>
>> >> >>
>>
>> >> >> When I said temporary, I meant 3-4 hours. Definitely not more than 
&g

Re: Our test failure rate is unacceptable.

[Jason Gerlowski]

> If it’s inadvertently added, we either fix it within an hour or so or revert 
> the offending commit

> I don't want to set specific time frames,

To play Devil's Advocate here: why wait even an hour to revert a 100%
test failure?  Reverts are usually trivial to do, unblock others
immediately, and don't interfere with the fix process at all.
Remembering the times I've broken the build myself, reverts even seem
preferable from that position - reverting up front takes all the
time-pressure off of getting out a fix.  Why work under the gun when
you don't have to?

On Fri, Sep 18, 2020 at 1:14 PM Tomás Fernández Löbbe
 wrote:
>
> I believe these failures are associated to 
> https://issues.apache.org/jira/browse/SOLR-14151
>
> • FAILED:  org.apache.solr.pkg.TestPackages.classMethod
> • FAILED:  
> org.apache.solr.schema.PreAnalyzedFieldManagedSchemaCloudTest.testAdd2Fields
> • FAILED:  
> org.apache.solr.schema.ManagedSchemaRoundRobinCloudTest.testAddFieldsRoundRobin
>
> > IMO if a temporary instability is to be introduced deliberately, it should 
> > be published on the list. If it’s inadvertently added, we either fix it 
> > within an hour or so or revert the offending commit
> I don't want to set specific time frames, but sometimes it's obviously too 
> much time.
>
> On Fri, Sep 18, 2020 at 8:48 AM Atri Sharma  wrote:
>>
>> When I said temporary, I meant 3-4 hours. Definitely not more than that.
>>
>> IMO we should just roll back offending commits if they are easily 
>> identifiable. I agree with you — we all have been guilty of breaking builds 
>> (mea culpa as well). The bad part here is the longevity of the failures.
>>
>>
>> On Fri, 18 Sep 2020 at 21:05, Erick Erickson  wrote:
>>>
>>> bq. IMO if a temporary instability is to be introduced deliberately, it 
>>> should be published on the list
>>>
>>>
>>>
>>> Actually, I disagree. Having anything in the tests that fail 100% of the 
>>> time is just unacceptable since it becomes a barrier for everyone else. 
>>> AFAIK, if the problem can be identified to a particular push, I have no 
>>> problems with that push being unilaterally rolled back.
>>>
>>>
>>>
>>> The exception for me is when the problem is addressed immediately, I’ve 
>>> certainly been the source of that kind of problem, as have others.
>>>
>>>
>>>
>>> What I take great exception to is the fact that some of these tests have 
>>> been failing 100% of the time for the last seven days! If it’s the case 
>>> that the full test suite was never run before the push that’s another 
>>> discussion. Yeah, it takes a long time but…
>>>
>>>
>>>
>>> Erick
>>>
>>>
>>>
>>> > On Sep 18, 2020, at 11:28 AM, Atri Sharma  wrote:
>>>
>>> >
>>>
>>> > IMO if a temporary instability is to be introduced deliberately, it 
>>> > should be published on the list. If it’s inadvertently added, we either 
>>> > fix it within an hour or so or revert the offending commit.
>>>
>>> >
>>>
>>> > On Fri, 18 Sep 2020 at 20:26, Erick Erickson  
>>> > wrote:
>>>
>>> > http://fucit.org/solr-jenkins-reports/failure-report.html
>>>
>>> >
>>>
>>> >
>>>
>>> >
>>>
>>> > HdfsAutoAddReplicasTest failing 100% of the time.
>>>
>>> >
>>>
>>> > TestPackages.classMethod failing 100% of the time
>>>
>>> >
>>>
>>> > 3-4 AutoAddReplicas tests failing 98% of the time.
>>>
>>> >
>>>
>>> >
>>>
>>> >
>>>
>>> > Is anyone looking at these? I realize the code base is changing a lot, 
>>> > and some temporary instability is to be expected. What I’d like is for 
>>> > some indication that people are actively addressing these.
>>>
>>> >
>>>
>>> >
>>>
>>> >
>>>
>>> > Erick
>>>
>>> >
>>>
>>> > -
>>>
>>> >
>>>
>>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>>
>>> >
>>>
>>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>> >
>>>
>>> >
>>>
>>> >
>>>
>>> > --
>>>
>>> > Regards,
>>>
>>> >
>>>
>>> > Atri
>>>
>>> > Apache Concerted
>>>
>>>
>>>
>>>
>>>
>>> -
>>>
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>>
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>
>>>
>> --
>> Regards,
>>
>> Atri
>> Apache Concerted

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Tests that use bin/solr?

[Jason Gerlowski]

I created a few JIRAs to spike "bin/solr" tests several years back
(SOLR-11749).  I didn't have quite enough time to get it across the finish
line previously but I think it's a great idea to have tests of this sort.
At the time I was writing tests in bash - but realized that wasn't a
particularly scalable approach.

Would be happy to help where I can if the effort gets restarted.

Best,

Jason

On Mon, Sep 7, 2020 at 3:14 AM Uwe Schindler  wrote:

> It should be part of an integration test suite, only on module "package"
> after assembly. That's something to setup. Please make sure that it works
> with any operating system, because we are leaving Java world here (and
> because of this, don't mix that into default unit tests).
>
> Currently we have a test for bin/solr: Smoketester 
>
> Uwe
>
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> > -Original Message-
> > From: Dawid Weiss 
> > Sent: Monday, September 7, 2020 9:00 AM
> > To: Lucene Dev 
> > Subject: Re: Tests that use bin/solr?
> >
> > Just a note - such integration tests should depend on (and consume)
> > the output of solr/packaging (a ZIP file with fully assembled
> > package). Then you're really sure you're testing the final artifact.
> >
> > Dawid
> >
> > On Mon, Sep 7, 2020 at 7:51 AM David Smiley  wrote:
> > >
> > > Do we have any tests that operate on a "real" Solr instance running
> from
> > "bin/solr"?  Such tests could find problems with bin/solr and any
> classpath
> > matters in how Jetty operates.  Solr does have JettySolrRunner which is
> great
> > but doesn't cover the aforementioned matters.
> > >
> > > We've got some really nice tests in SolrExampleTests which is a base
> class and
> > many implementations that create SolrJ clients in different ways.  I
> could
> > imagine modifying this such that if a magic system property is specified
> to a
> > URL of an existing Solr instance, then the test would not create a
> > JettySolrRunner but instead use the configured one.  This would then be
> > executed by the smoke tester and maybe a future Docker release process.
> I
> > have test infrastructure I wrote where I work that does this sort of
> thing for our
> > Solr plugins, and it works great.
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: [VOTE] Lucene logo contest, third time's a charm

[Jason Gerlowski]

A1, A2, D (binding)

On Wed, Sep 2, 2020 at 10:47 AM Michael McCandless
 wrote:
>
> A2, A1, C5, D (binding)
>
> Thank you to everyone for working so hard to make such cool looking possible 
> future Lucene logos!  And to Ryan for the challenging job of calling this 
> VOTE :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Sep 1, 2020 at 4:21 PM Ryan Ernst  wrote:
>>
>> Dear Lucene and Solr developers!
>>
>> Sorry for the multiple threads. This should be the last one.
>>
>> In February a contest was started to design a new logo for Lucene 
>> [jira-issue]. The initial attempt [first-vote] to call a vote resulted in 
>> some confusion on the rules, as well the request for one additional 
>> submission. The second attempt [second-vote] yesterday had incorrect links 
>> for one of the submissions. I would like to call a new vote, now with more 
>> explicit instructions on how to vote, and corrected links.
>>
>> Please read the following rules carefully before submitting your vote.
>>
>> Who can vote?
>>
>> Anyone is welcome to cast a vote in support of their favorite submission(s). 
>> Note that only PMC member's votes are binding. If you are a PMC member, 
>> please indicate with your vote that the vote is binding, to ease collection 
>> of votes. In tallying the votes, I will attempt to verify only those marked 
>> as binding.
>>
>> How do I vote?
>>
>> Votes can be cast simply by replying to this email. It is a ranked-choice 
>> vote [rank-choice-voting]. Multiple selections may be made, where the order 
>> of preference must be specified. If an entry gets more than half the votes, 
>> it is the winner. Otherwise, the entry with the lowest number of votes is 
>> removed, and the votes are retallied, taking into account the next preferred 
>> entry for those whose first entry was removed. This process repeats until 
>> there is a winner.
>>
>> The entries are broken up by variants, since some entries have multiple 
>> color or style variations. The entry identifiers are first a capital letter, 
>> followed by a variation id (described with each entry below), if applicable. 
>> As an example, if you prefer variant 1 of entry A, followed by variant 2 of 
>> entry A, variant 3 of entry C, entry D, and lastly variant 4e of entry B, 
>> the following should be in your reply:
>>
>> (binding)
>> vote: A1, A2, C3, D, B4e
>>
>> Entries
>>
>> The entries are as follows:
>>
>> A. Submitted by Dustin Haver. This entry has two variants, A1 and A2.
>>
>> [A1] 
>> https://issues.apache.org/jira/secure/attachment/12999548/Screen%20Shot%202020-04-10%20at%208.29.32%20AM.png
>> [A2] https://issues.apache.org/jira/secure/attachment/12997172/LuceneLogo.png
>>
>> B. Submitted by Stamatis Zampetakis. This has several variants. Within the 
>> linked entry there are 7 patterns and 7 color palettes. Any vote for B 
>> should contain the pattern number followed by the lowercase letter of the 
>> color palette. For example, B3e or B1a.
>>
>> [B] https://issues.apache.org/jira/secure/attachment/12997768/zabetak-1-7.pdf
>>
>> C. Submitted by Baris Kazar. This entry has 8 variants.
>>
>> [C1] 
>> https://issues.apache.org/jira/secure/attachment/13006392/lucene_logo1_full.pdf
>> [C2] 
>> https://issues.apache.org/jira/secure/attachment/13006393/lucene_logo2_full.pdf
>> [C3] 
>> https://issues.apache.org/jira/secure/attachment/13006394/lucene_logo3_full.pdf
>> [C4] 
>> https://issues.apache.org/jira/secure/attachment/13006395/lucene_logo4_full.pdf
>> [C5] 
>> https://issues.apache.org/jira/secure/attachment/13006396/lucene_logo5_full.pdf
>> [C6] 
>> https://issues.apache.org/jira/secure/attachment/13006397/lucene_logo6_full.pdf
>> [C7] 
>> https://issues.apache.org/jira/secure/attachment/13006398/lucene_logo7_full.pdf
>> [C8] 
>> https://issues.apache.org/jira/secure/attachment/13006399/lucene_logo8_full.pdf
>>
>> D. The current Lucene logo.
>>
>> [D] https://lucene.apache.org/theme/images/lucene/lucene_logo_green_300.png
>>
>> Please vote for one of the above choices. This vote will close about one 
>> week from today, Mon, Sept 7, 2020 at 11:59PM.
>>
>> Thanks!
>>
>> [jira-issue] https://issues.apache.org/jira/browse/LUCENE-9221
>> [first-vote] 
>> http://mail-archives.apache.org/mod_mbox/lucene-dev/202006.mbox/%3cCA+DiXd74Mz4H6o9SmUNLUuHQc6Q1-9mzUR7xfxR03ntGwo=d...@mail.gmail.com%3e
>> [second-vote] 
>> http://mail-archives.apache.org/mod_mbox/lucene-dev/202009.mbox/%3cCA+DiXd7eBrQu5+aJQ3jKaUtUTJUqaG2U6o+kUZfNe-m=smn...@mail.gmail.com%3e
>> [rank-choice-voting] https://en.wikipedia.org/wiki/Instant-runoff_voting

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Analytics Module; maybe should be 3rd party?

[Jason Gerlowski]

+1 to having some sort of vote before a big change like moving a
contrib.  Also +1 to getting feedback on the user list as well; people
there might be able to give details on some of the functionality gaps
that Houston alluded to.

Personally I haven't done anything with the Analytics Component beyond
some Hello World usage.  It does seem like a large source of
duplication that would be nice to alleviate in some way.

On Mon, Aug 31, 2020 at 5:24 PM Houston Putman  wrote:
>
> Also I think that discussion around the use of a feature should take place on 
> the user list, instead of the dev list. Users are the intended audience for 
> that kind of question.
>
> - Houston
>
> On Mon, Aug 31, 2020 at 5:21 PM Houston Putman  
> wrote:
>>
>> Thanks for bringing this up David.
>>
>> To start, I definitely agree that removal of contribs should have a 
>> discussion, then a vote before any action is taken.
>>
>> There have been a few bug fixes/improvements made this year, but there has 
>> not been a lot of development on it.
>> If there is little use of the component, then I agree taking it outside of 
>> lucene-solr is probably a good idea. If it does have use in the community 
>> and stays as a contrib, I think it is still a good idea to turn it into a 
>> "package". That way it is easily loadable by users.
>> But I guess that is a part of a larger discussion about how we should treat 
>> contrib modules, and whether they should be "first party" packages.
>>
>> As per your question around compatibility with JSON Facets, there is 
>> definitely overlap but differences in the options provided by each. I'm not 
>> an expert on JSON Facets, but there are certainly things that each module 
>> can do that the other can't.
>> Ideally we could take the functionality unique to analytics and add it to 
>> JSON facets, that way there is little reason for the analytics component to 
>> remain. But that is a larger effort.
>>
>> - Houston
>>
>> On Mon, Aug 31, 2020 at 4:43 PM David Smiley  wrote:
>>>
>>> I'd prefer discussion about removal of contribs (or major parts of Solr) 
>>> occur here first with a vote, and _then_ file a JIRA if the conclusion is 
>>> to remove it.  It's more visible to each other and the community.
>>>
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>> On Mon, Aug 31, 2020 at 4:14 PM Ishan Chattopadhyaya 
>>>  wrote:

 +1, please open a jira and I'll take care of it.

 Fyi, Marcus is helping me with migrating contrib modules to first party 
 package, and this was on our radar. I am totally occupied in other things 
 lately and hence slowed down on that effort. Will pick up pace end of this 
 week.

 On Tue, 1 Sep, 2020, 12:32 am David Smiley,  wrote:
>
> Our new package system allows for a viable/healthy ecosystem of 3rd party 
> packages that are easy to find & install.  I wonder if the Analytics 
> contrib module should be 3rd party?  I suspect it is very rarely used, 
> especially with the relative strength and growth of Solr's built-in JSON 
> facet module.  I've never used it.  I raised these points in this comment 
> 2 years ago:
> https://issues.apache.org/jira/browse/SOLR-12045?focusedCommentId=16437754=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16437754
> My biggest point there is the redundancy Solr has with competing ways of 
> doing facets.
>
> I ran "cloc" on a clean checkout of this dir:
>
> github.com/AlDanial/cloc v 1.82  T=0.72 s (416.9 files/s, 69593.9 lines/s)
> ---
> Language files  blankcomment  
>  code
> ---
> Java   290   5950   8886  
> 35035
> XML  6 76123  
>   186
> HTML 4  8 59  
>38
> Ant  1  7 14  
> 7
> Gradle   1  3 16  
> 6
> ---
> SUM:   302   6044   9098  
> 35272
> ---
>
> 35k lines of code is a lot of code to maintain.  AFAIK there haven't been 
> bugs/features reported against it in a long time.  Maybe it would be 
> maintained better if it was a 3rd party package?  There is some 
> maintenance cost as refactors touch it.  Maybe not "a lot" but it's 
> something.
>
> As a counterpoint to all this, 

Re: RoadMap?

[Jason Gerlowski]

I agree with the approach Jan voiced - that at least some of these
features should appear in 9.0 as deprecated to give users more time.

That said, maybe discussing what to do around these features should be
its own thread or should be taken back to some of the specific jiras
where there's already been some discussion (e.g. SOLR-14616).  It just
seems likely to hijack this thread away from other discussions (how to
manage/handle our new Roadmap page).

On Wed, Aug 12, 2020 at 9:35 AM Ishan Chattopadhyaya
 wrote:
>
> > It has been proposed on the list to NOT rip out all deprecations in 9.0, 
> > but allow users to
> > upgrade to 9.0 with e.g. SolrCell still available, and then have yet some 
> > time to change their
> > processes to adapt to the new way of doing stuff. I like that proposal. 
> > Sure, 9.0 will remove lots
> > of deprecated code, but I think it is a mistake to do all of the proposed 
> > removals at once. We
> > can spread removals out in 9.x releases, after users have had a few 
> > releases with a choice between
> > old and new and the new alternative is solid.
>
> I support the DIH, autoscaling and CDCR going away in 9.0, rest of the things 
> can just move into first party packages and continue to be part of the 
> distribution. Does that make sense, Jan?
>
> On Wed, Aug 12, 2020 at 1:36 PM Jan Høydahl  wrote:
>>
>> I edited the page to introduce the (super important) Solr TLP split into the 
>> roadmap.
>> Also added a rough timeframe and a «major theme» for each release above the 
>> issue table.
>> I added 8.8 and 9.1 as I think it is important to track what gets done just 
>> before 9.0 and what can be deferred to after 9.0.
>>
>> It has been proposed on the list to NOT rip out all deprecations in 9.0, but 
>> allow users to upgrade to 9.0 with e.g. SolrCell still available, and then 
>> have yet some time to change their processes to adapt to the new way of 
>> doing stuff. I like that proposal. Sure, 9.0 will remove lots of deprecated 
>> code, but I think it is a mistake to do all of the proposed removals at 
>> once. We can spread removals out in 9.x releases, after users have had a few 
>> releases with a choice between old and new and the new alternative is solid.
>>
>> Thanks Gus for taking ownership and suggesting a process! Feel free to 
>> rework what I edited into a structure you see more fit.
>>
>> Jan
>>
>> 11. aug. 2020 kl. 18:51 skrev Gus Heck :
>>
>> I was thinking that level of detail is in the Jira... I don't see any reason 
>> for things to disappear (in fact rejected should go in a rejected list for 
>> future reference.)
>>
>> On Tue, Aug 11, 2020 at 12:04 PM Ilan Ginzburg  wrote:
>>>
>>> Maybe also add “in progress”? So items do not disappear suddenly from the 
>>> page when work really starts on them?
>>>
>>> On Tue 11 Aug 2020 at 17:15, Gus Heck  wrote:

 Cool, since I brought it up, I can volunteer to help manage the page. We 
 should get jira issue links in there wherever possible. Do we want to 
 build an initial list and have some sort of Proposed/Planned workflow so 
 readers can have confidence (or appropriate lack of confidence) in what 
 they see there? voting on things seems like too much but maybe folks who 
 care watch the page, and if something is on there for a week without 
 objection it can be called accepted? If a discussion starts here it can be 
 marked "Considering" so... something like this:

 4 states: Proposed, Considering, Planned, Rejected

 Workflow like this:
 Proposed ---(no objection 1 wk) --> Planned
 Proposed ---(discussion)--> Considering
 Considering (agreement) --> Planned
 Considering (deferred) ---> Proposed (later release)
 Considering (unsuitable) -> Rejected
 Considering (promoted) ---> Proposed (earlier release)
 Planned (difficulty found) ---> Considering

 Anything in "Considering" should have an active dev list thread, and if it 
 didn't happen on the list it didn't happen :). Any of that (or differences 
 of opinion during Considering) can be overridden by a formal vote of course

 -Gus




 On Tue, Aug 11, 2020 at 10:29 AM Ishan Chattopadhyaya 
  wrote:
>
> I've created a placeholder document here: 
> https://cwiki.apache.org/confluence/display/SOLR/Roadmap
> Let us put in all our items there.
>
> On Tue, Aug 11, 2020 at 4:45 PM Jan Høydahl  wrote:
>>
>> Let’s revive this email thread about Roadmap.
>>
>>
>>
>>
>>
>> With so many large initiatives going on, and the TLP split also, I think 
>> it makes perfect sense with a Roadmap.
>>
>>
>> I know we’re not used to that kind of thing - we tend to just let things 
>> play out as it happens to land in various releases, but this time is 
>> special, and I think we’d benefit from more 

Re: Survey on ManagedResources feature

[Jason Gerlowski]

Hey Noble,

Can you explain what you mean when you say it's not secured?  Just for
those of us who haven't been following the discussion so far?  On the
surface of things users taking advantage of our RuleBasedAuth plugin
can secure this API like they can any other HTTP API.  Or are you
talking about some other security aspect here?

Jason

On Tue, Aug 11, 2020 at 9:55 AM Noble Paul  wrote:
>
> Hi all,
> The end-point for Managed resources is not secured. So it needs to be
> fixed/eliminated.
>
> I would like to know what is the level of adoption for that feature
> and if it is a critical feature for users.
>
> Another possibility is to offer a replacement for the feature using a
> different API
>
> Your feedback will help us decide on what a potential solution should be
>
> --
> -
> Noble Paul

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Standardize Leading Test or Trailing Test

[Jason Gerlowski]

+1

On Fri, Aug 7, 2020 at 8:37 PM Marcus Eagan  wrote:
>
> Any -1's? I just learned more about them from David Smiley earlier today and 
> I recognize they are for the rare case, but I want to ensure consensus before 
> I spend time on this tomorrow evening. I hadn't remembered seeing them in 
> real life except for on the Lucene/Solr separation vote.
>
> Of course, if someone else does this before I get to it today or tomorrow 
> morning, great. :)
>
> Marcus
>
> On Thu, Aug 6, 2020 at 8:05 AM Gus Heck  wrote:
>>
>> Yeah  +1 for standardization +1.01 if it lands on *Test :) but that's just 
>> my personal preference.
>>
>> On Thu, Aug 6, 2020 at 9:17 AM Adrien Grand  wrote:
>>>
>>> +1
>>>
>>> On Thu, Aug 6, 2020 at 1:54 PM Erick Erickson  
>>> wrote:

 This has amused/annoyed me for a long time. But did I ever have the 
 energy to tackle it? N.

 +1

 > On Aug 6, 2020, at 1:50 AM, Tomás Fernández Löbbe 
 >  wrote:
 >
 > +1
 >
 > On Wed, Aug 5, 2020 at 10:37 PM David Smiley  
 > wrote:
 > +1 to standardize on something.
 > This has been brought up before: LUCENE-8626 -- credit to Christine who 
 > started the work.  I recommend resuming the discussion there.
 >
 > ~ David
 >
 >
 > On Thu, Aug 6, 2020 at 12:08 AM Anshum Gupta  
 > wrote:
 > +1
 >
 > Thanks for bringing this up, Marcus. Standardizing this is really great.
 >
 > On Wed, Aug 5, 2020 at 8:01 PM Marcus Eagan  
 > wrote:
 > Hi community, what do you think a small effort to standardize on leading 
 > with the word "Test" or trailing with the word "Test" in the repo. Most 
 > projects do one or the other and it has an impact on developer 
 > productivity. I'll explain my use case:
 >
 > I'm working on a class and I want to modify the test to evaluate my 
 > changes. If the class is named in a standard way, I can find it easily. 
 > If it is not, it's fine. There are typically two options. I consider it 
 > distracting and sloppy. Distraction is expensive for developers. I have 
 > some more important efforts that I'm working on, but if the community 
 > agrees on this one, I can open a ticket and submit a PR. Let me know 
 > what you think.
 >
 > Hoping to make the project more developer friendly.
 >
 > --
 > Marcus Eagan
 >
 >
 >
 > --
 > Anshum Gupta


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

>>>
>>>
>>> --
>>> Adrien
>>
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>
>
>
> --
> Marcus Eagan
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Deprecate Schemaless Mode?

[Jason Gerlowski]

> Is anyone on this list using schemaless mode in production or have you tried 
> to?

Schemaless mode is one of a group of Solr features present for
convenience but not intended for production usage.  It's in the same
boat as "bin/post", and SolrCell, and others.  These features do cause
headaches when users ignore the documented restrictions and use them
for more than prototyping.  But at the same time they're super
valuable for these sort of demo-ing or getting-started use cases.  An
easy getting-started experience is important, and schemaless et al
serve a mostly positive role in that.

I think we'd better serve our users if we left schemaless
in/undeprecated, and instead focused on making it harder to
(unknowingly) use them in ways contrary to community recommendations.
Add louder warnings in the documentation (where not already present).
Add warnings to the Solr logs the first time these features are used.
Disable them by default (where that makes sense).  Taken to the
extreme, we could even add a section into Solr's response that lists
non-production features used in serving a given request.

There are lots of ways to address the "feature X is trappy" problem
without removing X together.

On Mon, Aug 3, 2020 at 11:33 AM Marcus Eagan  wrote:
>
> Community,
>
> There are many of us that have had to deal with the pain of managing the 
> schemaless mode of operation in Solr. I'm curious to get others thoughts 
> about how well it is working for them and if they would like to continue to 
> use it.
>
> I for one don't think Schemaless works as intended and favor deprecating it 
> and replacing it with some more usable but I am sure others have thoughts 
> here.
>
> Is anyone on this list using schemaless mode in production or have you tried 
> to?
>
> A preliminary discussion has occurred in this Jira ticket: 
> https://issues.apache.org/jira/browse/SOLR-14701
>
> Thank you all,
>
> Marcus Eagan
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Newbie task + request

[Jason Gerlowski]

Hi Pramod,

I'm pretty sure Erick answered some of your questions on the
SOLR-13205 jira already, but I wanted to restate those answers here
for other beginners that may be reading this thread or stumble on it
later.

The way our project is set up, only "committers" can assign tickets in
JIRA.  In practice, what you can do instead is post a comment
indicating that you've started looking into the ticket - that's enough
to prevent two people from accidentally doing the same work.

In terms of mini-mentoring- I'd suggest getting set up in our IRC
channels or Slack chatroom.  Of course, anything you ask there can be
asked on the mailing list too.  But you might get feedback more
quickly in Slack/IRC than on the mailing list.  I've sent you an
invite to Slack, in case you're interested.  Feel free to ping me
("@gerlowskija") on any questions you have there, or in JIRA, and I'll
try to help you out.

Good luck on the ticket and on getting started in open source!

Jason

On Sun, Jun 28, 2020 at 10:10 PM Pramod Kumar
 wrote:
>
> Hi all,
>
> I am a newbie and was looking to pick up a quick task as my first 
> (SOLR-13205).
> However, I don’t seem to be able to assign it to myself. Can someone please 
> assign it to me? If I am missing something obvious, let me know how I can do 
> it myself.
>
> There is some additional cleanup that I am planning on doing as a part of 
> that task
> - There are a couple of places that do a charAt. I am planning on doing a 
> null or empty check for the field and use the default if so.
> - Replace charAt with startsWith anyway since default can be null or empty.
> - Planning on also throwing an exception during init() if the arguments are 
> not good.
>
> Let me know if someone disagrees.
>
>
> I also had another request. I am a fairly experienced engineer, but very new 
> to open source and Apache. I am super interested in this space and that’s why 
> I am looking to contribute substantially I hope. Is anyone willing to be a 
> mini-mentor for me? I was just hoping for someone to answer some n00b 
> questions in the beginning, that may cut down my spinning time in the 
> beginning. Nothing super formal, and I promise to not take more than an hour 
> a week at max. That time I would expect to dwindle down to nothing after a 
> month or so.
>
> Thanks.
>
> Pramod

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Lucene logo contest

[Jason Gerlowski]

Option "A"

On Tue, Jun 16, 2020 at 8:37 PM Man with No Name
 wrote:
>
> A, clean and modern.
>
> On Mon, Jun 15, 2020 at 6:08 PM Ryan Ernst  wrote:
>>
>> Dear Lucene and Solr developers!
>>
>> In February a contest was started to design a new logo for Lucene [1]. That 
>> contest concluded, and I am now (admittedly a little late!) calling a vote.
>>
>> The entries are labeled as follows:
>>
>> A. Submitted by Dustin Haver [2]
>>
>> B. Submitted by Stamatis Zampetakis [3] Note that this has several variants. 
>> Within the linked entry there are 7 patterns and 7 color palettes. Any vote 
>> for B should contain the pattern number, like B1 or B3. If a B variant wins, 
>> we will have a followup vote on the color palette.
>>
>> C. The current Lucene logo [4]
>>
>> Please vote for one of the three (or nine depending on your perspective!) 
>> above choices. Note that anyone in the Lucene+Solr community is invited to 
>> express their opinion, though only Lucene+Solr PMC cast binding votes 
>> (indicate non-binding votes in your reply, please). This vote will close one 
>> week from today, Mon, June 22, 2020.
>>
>> Thanks!
>>
>> [1] https://issues.apache.org/jira/browse/LUCENE-9221
>> [2] 
>> https://issues.apache.org/jira/secure/attachment/12999548/Screen%20Shot%202020-04-10%20at%208.29.32%20AM.png
>> [3] https://issues.apache.org/jira/secure/attachment/12997768/zabetak-1-7.pdf
>> [4] https://lucene.apache.org/theme/images/lucene/lucene_logo_green_300.png
>
> --
> Sent from Gmail for IPhone

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Mayya Sharipova as Lucene/Solr committer

[Jason Gerlowski]

Welcome Mayya!

On Wed, Jun 10, 2020 at 4:44 AM Tommaso Teofili
 wrote:
>
> congrats and welcome onboard Mayya!
>
> Regards,
> Tommaso
>
> On Tue, 9 Jun 2020 at 19:27, Trey Grainger  wrote:
>>
>> Woohoo - awesome news! Congrats, Mayya!
>>
>> Trey Grainger
>> Founder, Searchkernel
>>
>> On Mon, Jun 8, 2020 at 12:58 PM jim ferenczi  wrote:
>>>
>>> Hi all,
>>>
>>> Please join me in welcoming Mayya Sharipova as the latest Lucene/Solr 
>>> committer.
>>> Mayya, it's tradition for you to introduce yourself with a brief bio.
>>>
>>> Congratulations and Welcome!
>>>
>>> Jim

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: BadApple report

[Jason Gerlowski]

> Hoss’s rollups are here: 
> http://fucit.org/solr-jenkins-reports/failure-report.html which show the 
> rates, but not where they came from.

If I click on a particular test entry on "failure-report.html", I'm
presented with dialog with links for each failure.  Clicking that link
takes me to a file listing page (e.g.
http://fucit.org/solr-jenkins-reports/job-data/apache/Lucene-Solr-Tests-8.x/1569/),
with Jenkins logs, etc. for that particular failure.  Notably, it also
has a file called "url.txt" with a link to the actual failure in
Jenkins (e.g. 
http://fucit.org/solr-jenkins-reports/job-data/apache/Lucene-Solr-Tests-8.x/1569/url.txt).

Just mentioning what I've seen with a few I've clicked on.  The
rollups might not have that for all failures, or for all different
source-Jenkins.  Just wanted to mention that you can get back to the
Jenkins job in at least _some_ cases with a bit of clicking.

On Mon, May 25, 2020 at 1:27 PM Ilan Ginzburg  wrote:
>
> Thanks that helps. I'll try to have a look at some of the failures related to 
> areas I know.
>
> Ilan
>
> On Mon, May 25, 2020 at 7:07 PM Erick Erickson  
> wrote:
>>
>> Ilan:
>>
>> That’s, unfortunately, not an easy question. Hoss’s rollups are here: 
>> http://fucit.org/solr-jenkins-reports/failure-report.html which show the 
>> rates, but not where they came from.
>>
>> Here’s an example of a failure from Jenkins, if you follow the link you can 
>> see the full output, (click “console output”, then “full log”): 
>> https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/3181/. I usually see 
>> the individual ones go by by subscribing to “bui...@lucene.apache.org”.
>>
>> Otherwise, what I often do is use Mark Miller’s “beasting” script to see if 
>> I can get it to reproduce locally and go from there:
>>
>> https://gist.github.com/markrmiller/dbdb792216dc98b018ad
>>
>> It’s all complicated by the fact that the failures are intermittent.
>>
>> Best,
>> Erick
>>
>> > On May 25, 2020, at 11:22 AM, Ilan Ginzburg  wrote:
>> >
>> > Where are the test failure details?
>> >
>> > On Mon, May 25, 2020 at 4:47 PM Erick Erickson  
>> > wrote:
>> > Here’s the summary:
>> >
>> > Raw fail count by week totals, most recent week first (corresponds to 
>> > bits):
>> > Week: 0  had  113 failures
>> > Week: 1  had  103 failures
>> > Week: 2  had  102 failures
>> > Week: 3  had  343 failures
>> >
>> >
>> > Failures in Hoss' reports for the last 4 rollups.
>> >
>> > There were 511 unannotated tests that failed in Hoss' rollups. Ordered by 
>> > the date I downloaded the rollup file, newest->oldest. See above for the 
>> > dates the files were collected
>> > These tests were NOT BadApple'd or AwaitsFix'd
>> >
>> > Failures in the last 4 reports..
>> >Report   Pct runsfails   test
>> >  0123   0.7 1593 40  BasicDistributedZkTest.test
>> >  0123   2.1 1518 28  MultiThreadedOCPTest.test
>> >  0123   0.7 1613 14  RollingRestartTest.test
>> >  0123   7.1 1635 44  
>> > ScheduledTriggerIntegrationTest.testScheduledTrigger
>> >  0123   2.4 1614 17  
>> > SearchRateTriggerTest.testWaitForElapsed
>> >  0123   0.2 1614  6  
>> > ShardSplitTest.testSplitShardWithRuleLink
>> >  0123   0.5 1577  5  
>> > SolrCloudReportersTest.testExplicitConfiguration
>> >  0123   0.7 1560 19  TestInPlaceUpdatesDistrib.test
>> >  0123   1.0 1566 17  TestPackages.testPluginLoading
>> >  0123   0.8 1598  7  
>> > TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
>> >  0123   0.7 1598  8  TestSimScenario.testAutoAddReplicas
>> > 
>> >
>> >
>> > Full report:
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Lucene-Solr split (Solr promoted to TLP)

[Jason Gerlowski]

> Would this not be eased to some extent if the initial committer base of both 
> the projects was the same?

"Who has commit karma to a project" is a separate question from "Who
will make commits in practice".  Having Lucene committers retain their
status as Solr committers only helps if they're willing and interested
in keeping Solr up to date.  From discussion on this thread so far,
I'm not sure how much of that interest exists.  After all, avoiding
this solr-update-burden was one of the arguments cited in favor of a
split.

> Contrary to Jason I don't think keeping Solr and Lucene code together helps 
> anybody in tackling those issues now or in the future.
That's not the point I was making.  I wasn't saying that the split (or
lack thereof) affects our ability to address test-flakiness (etc.).  I
was citing test-flakiness as an example of how bad us Solr folks have
been historically at prioritizing this sort of work that's crucial for
project health but not tied to a specific feature.  I brought this
historical example up as a parallel or a prediction to how we might do
with a similar task: managing to stay up to date on Lucene.  My whole
point was: "We historically don't do well at getting this sort of work
done; therefore I expect we're going to have some level of lag behind
Lucene"

On Wed, May 13, 2020 at 2:11 PM Dawid Weiss  wrote:
>
> > This might sound a bit harsh, but maybe Lucene devs helping with Solr has 
> > let Solr off the hook a bit too much? I actually like the fact that the 
> > split causes Solr to figure out it's own situation and focus on its 
> > problems.
>
> Well said.
>
> > Take our ongoing test flakiness woes and SolrCloud instability issues as 
> > examples: both are serious threats to the project, both have been around 
> > for years, and both are here to stay for the foreseeable future.
>
> Contrary to Jason I don't think keeping Solr and Lucene code together
> helps anybody in tackling those issues now or in the future. The first
> thing Mark (Miller) did when he started cleaning up the codebase for
> gradle was to *disable* nearly all randomizations and fix certain
> parameters to bring back stability and speed up Solr tests. I bet it would be
> a tad easier if he only had Solr (or Lucene) side to take care of (rather than
> both Lucene AND Solr).
>
> What is good for Lucene may not be as good for Solr. Maybe removing
> randomizations that
> currently happen in LuceneTestCase will calm down tests? Who knows. 
> Sincerely, I
> think a split project may bring a clean slate for more drastic
> refactorings and cleanups. A combined
> codebase keeps the status quo we've been in for years.
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Lucene-Solr split (Solr promoted to TLP)

[Jason Gerlowski]

There's nothing wrong with a harsh "sink or swim" approach if the
risks are bearable.  If the worst case risk here is that we have a few
rough releases as we smooth out the process, I'm all on board with
"sink or swim".  But by the same token - "sink or swim" gets less
appealing as the risks increase.   No sane person would toss their PFD
after a shipwreck because they always meant to learn to backstroke.
So maybe we just disagree on what the worst case harm to Solr looks
like.  I see the harm being pretty serious: if Solr stagnates its
Lucene version relative to other offerings users could go elsewhere
and the project would lose out on adoption and community.  A Very Bad
Thing.  But if you don't see this as even a remote possibility, well
then "sink or swim" makes sense.

> I'd be OK with a stable, robust Solr that got 1-2 major versions behind 
> Lucene, but was rock-solid with a lower barrier to entry...

If that's an option, I might be too.  But I'm not sure how a
Lucene-Solr split (or an older Lucene version) does anything to make
Solr more solid, lower its barrier to entry, etc.  Anecdotally, Solr
bugs rooted in Lucene seem the minority by far.  And Solr committers
can put effort into stability/barrier-to-entry as easily now as they
can in a post-split world.  Is there some connection between the split
and the those -ilities that I'm missing?

> I choose to be more optimistic wrt «Solr committers» ability to integrate new 
> and changed Lucene APIs in Solr
I agree that Solr committers _can_ do this work, and that there are
some awesome committers who straddle the fence and know Lucene very
well.  I wasn't trying to impugn anyone's efforts, interest or
expertise.  My point was just that at the end of the day a split
leaves fewer people around Solr with knowledge of the Lucene APIs and
their perf implications.  And a split is going to burden those
remaining people heavily until the roster of Lucene-literate Solr
committers re-populates.

On Wed, May 13, 2020 at 10:29 AM Jan Høydahl  wrote:
>
> I choose to be more optimistic wrt «Solr committers» ability to integrate new 
> and changed Lucene APIs in Solr. You do not need to be a Lucene committer in 
> order to learn how to USE the Lucene APIs, and I believe there are several 
> «Solr committers» who already posess those skills and are pretty deep in 
> Lucene already. Hopefully they are interested in doing lucene upgrades for 
> Solr, even if that some times includes implementing support for a new 
> fieldType (points vs trie), getting rid of index-time-boost features etc. I 
> may even attempt some of those tasks myself for the areas of Lucene API I am 
> comfortable with.
>
> Jan
>
> 13. mai 2020 kl. 16:24 skrev Doug Turnbull 
> :
>
> Jason, I hear your arguments and think of them FOR a split
>
> This might sound a bit harsh, but maybe Lucene devs helping with Solr has let 
> Solr off the hook a bit too much? I actually like the fact that the split 
> causes Solr to figure out it's own situation and focus on its problems.
>
> Regardless of the split or not, Solr is going to sink or swim based on the 
> efforts of Solr committers, not Lucene committers. I don't think Lucene 
> committers are going to be the ones to really address the systemic issues 
> with Solr. If anything, I imagine they are "let me fix this so the code 
> compiles" level of maintenance.
>
> "Falling behind Lucene" is counterbalanced to me with "Should Solr be on 
> cutting-edge Lucene?"
>
> I'd be OK with a stable, robust Solr that got 1-2 major versions behind 
> Lucene, but was rock-solid with a lower barrier to entry...
>
> On Wed, May 13, 2020 at 10:07 AM Jason Gerlowski  
> wrote:
>>
>> Wanted to add my two cents to the mix, though I'm a little late as the
>> vote has already progressed pretty far.
>>
>> I'm against a split.  From the points raised, I agree that Lucene has
>> much to gain.  But Solr has a lot to lose.
>>
>> Lucene devs would be freed from keeping Solr usage up to date.  That's
>> a great improvement for Lucene itself.  But that burden doesn't
>> disappear - it's just being moved to a different (smaller) group of
>> committers - who by definition don't know Lucene as well, and are less
>> suited to the task.  (Lucene devs still might help post-split, but
>> given that avoiding this burden is one of the arguments made above for
>> a split, it seems unwise to assume how much this generosity will
>> continue.)
>>
>> One likely result is that Solr will fall behind Lucene. Possibly
>> permanently behind.  Lucene folks are doing great work to improve
>> perf, add features etc. so falling behind is a Very Bad Thing.  To
>> Solr, Lucene is not the same as Jett

Re: [DISCUSS] Lucene-Solr split (Solr promoted to TLP)

[Jason Gerlowski]

Wanted to add my two cents to the mix, though I'm a little late as the
vote has already progressed pretty far.

I'm against a split.  From the points raised, I agree that Lucene has
much to gain.  But Solr has a lot to lose.

Lucene devs would be freed from keeping Solr usage up to date.  That's
a great improvement for Lucene itself.  But that burden doesn't
disappear - it's just being moved to a different (smaller) group of
committers - who by definition don't know Lucene as well, and are less
suited to the task.  (Lucene devs still might help post-split, but
given that avoiding this burden is one of the arguments made above for
a split, it seems unwise to assume how much this generosity will
continue.)

One likely result is that Solr will fall behind Lucene. Possibly
permanently behind.  Lucene folks are doing great work to improve
perf, add features etc. so falling behind is a Very Bad Thing.  To
Solr, Lucene is not the same as Jetty or Jackson which Solr can fall
behind on without significant detriment.  Lucene and the core search
functionality it offers is what brings people to Solr (or Elastic).
Putting ourselves in a position to fall behind on Lucene does a huge
disservice to our users, and loses Solr one of its greatest
advantages.

I hope that in the case of a split, the Solr community would rise to
the occasion and prevent this.  But my personal judgement is that it's
unlikely.  I hate to be negative, and I hope to be proven wrong, but
that's how things look to me.  We (Solr folks) have a bad track record
of addressing things with less-tangible, less-sellable benefits.  Take
our ongoing test flakiness woes and SolrCloud instability issues as
examples: both are serious threats to the project, both have been
around for years, and both are here to stay for the foreseeable
future.

If conditions were different in a way that made "falling behind" less
likely, I'd be all for a split.  But given (1) our recent track record
of addressing these sort of issues, (2) our test flakiness which will
make identifying "Lucene snapshot upgrade" bugs exceedingly difficult,
and (3) the current economic conditions which may make it harder for
committers to negotiate time from their employers to work on Lucene
updates...now seems like a bad time to attempt a split.  It will harm
Solr more than it helps Lucene.

On Tue, May 12, 2020 at 3:37 PM Namgyu Kim  wrote:
>
> It's hard to make a decision because it seems to have pros and cons.
> Basically, I agree to separate but there are some questions.
> So I don't not vote right now.
>
> 1) Release version
> Currently, versions of Lucene and Solr are aligned, how will they be managed 
> in the future?
> Other people took Elasticsearch as an example... But it was an independent 
> project from the beginning.
> So there is no problem with the Lucene version. (Elasticsearch 7.7 and Lucene 
> 8.5.1)
> I'm sure if we make solr as an independent project, it will make cracks about 
> the version structure. (like Lucene 8.6.2 and Solr 8.9.1)
> But it's also strange to suddenly start a new version of the Solr. (Solr 1.0)
> Of course it's a matter of adaption, but it's likely to cause some confusion 
> for existing users.
>
> 2) Complementary relationship
> When Lucene and Solr are built together, Solr can always maintain the latest 
> Lucene.
> In my personal opinion, it's a great advantage of Solr.
> Because Solr doesn't have to suffer from Lucene API changes and has latest 
> library.
> But it will be difficult if Solr becomes independent.
> If Solr tracks the master branch of Lucene on separate repository(project), 
> can it always check and reflect Lucene's API changes?
>
> On Tue, May 12, 2020 at 10:12 PM Doug Turnbull 
>  wrote:
>>
>> I'll give a perspective that comes more from the user's / "market" point of 
>> view as at OSC we onboard lots of new organizations into Solr.
>>
>> - Most new users incorrectly think of Solr as an independent Apache project, 
>> and many will have little knowledge or awareness of Lucene itself until 
>> given the full history of Lucene, Solr, Elasticsearch... or they have to 
>> dive into the code/write a plugin
>>
>> - Most orgs / managers think in terms of "Solr" (as in "Solr" vs 
>> "Elasticsearch" vs "Vespa, etc). So the starting point for new devs / folks 
>> is from the Solr angle
>>
>> - Lucene, when discussed, is understood more colloquially as a Solr 
>> dependency
>>
>> - If someone brings down the code to do some kind of work or investigation, 
>> there's typically surprise that Lucene and Solr are bundled together.
>>
>> - There's further surprise as the projects are indeed so different: Lucene 
>> and Solr tests, for example look little alike. They seem to have different 
>> coding syles / practices. One has more server-like and distributed system 
>> concerns; the other is clearly a low-level library for doing search work...
>>
>> I personally have a hard time explaining to new users the rationale for 
>> keeping these together, 

Re: [VOTE] Solr to become a top-level Apache project (TLP)

[Jason Gerlowski]

-1 (binding)

On Tue, May 12, 2020 at 7:31 AM Alan Woodward  wrote:
>
> +1 (binding)
>
> Alan Woodward
>
> > On 12 May 2020, at 12:06, Jan Høydahl  wrote:
> >
> > +1 (binding)
> >
> > Jan Høydahl
> >
> >> 12. mai 2020 kl. 09:36 skrev Dawid Weiss :
> >>
> >> Dear Lucene and Solr developers!
> >>
> >> According to an earlier [DISCUSS] thread on the dev list [2], I am
> >> calling for a vote on the proposal to make Solr a top-level Apache
> >> project (TLP) and separate Lucene and Solr development into two
> >> independent entities.
> >>
> >> To quickly recap the reasons and consequences of such a move: it seems
> >> like the reasons for the initial merge of Lucene and Solr, around 10
> >> years ago, have been achieved. Both projects are in good shape and
> >> exhibit signs of independence already (mailing lists, committers,
> >> patch flow). There are many technical considerations that would make
> >> development much easier if we move Solr out into its own TLP.
> >>
> >> We discussed this issue [2] and both PMC members and committers had a
> >> chance to review all the pros and cons and express their views. The
> >> discussion showed that there are clearly different opinions on the
> >> matter - some people are in favor, some are neutral, others are
> >> against or not seeing the point of additional labor. Realistically, I
> >> don't think reaching 100% level consensus is going to be possible --
> >> we are a diverse bunch with different opinions and personalities. I
> >> firmly believe this is the right direction hence the decision to put
> >> it under the voting process. Should something take a wrong turn in the
> >> future (as some folks worry it may), all blame is on me.
> >>
> >> Therefore, the proposal is to separate Solr from under Lucene TLP, and
> >> make it a TLP on its own. The initial structure of the new PMC,
> >> committer base, git repositories and other managerial aspects can be
> >> worked out during the process if the decision passes.
> >>
> >> Please indicate one of the following (see [1] for guidelines):
> >>
> >> [ ] +1 - yes, I vote for the proposal
> >> [ ] -1 - no, I vote against the proposal
> >>
> >> Please note that anyone in the Lucene+Solr community is invited to
> >> express their opinion, though only Lucene+Solr committers cast binding
> >> votes (indicate non-binding votes in your reply, please).
> >>
> >> The vote will be active for a week to give everyone a chance to read
> >> and cast a vote.
> >>
> >> Dawid
> >>
> >> [1] https://www.apache.org/foundation/voting.html
> >> [2] 
> >> https://lists.apache.org/thread.html/rfae2440264f6f874e91545b2030c98e7b7e3854ddf090f7747d338df%40%3Cdev.lucene.apache.org%3E
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Code overviews

[Jason Gerlowski]

Hi Martin,

If you're looking for an overview of how the code is laid out and
structured, there's not a ton there that I'm aware of. In terms of
particular features, the best resource that comes to mind would be
some of the videos of conference talks from the last several years.
I've gotten some mileage personally out of recordings of from the
"Lucene/Solr Revolution" or "Activate" conferences.  The conference is
hosted by Lucidworks, and their youtube channel has playlists for each
year: https://www.youtube.com/channel/UCPItOdfUk_tjlvqggkY-JsA.
(Disclaimer: I work for Lucidworks.). Again, these are mostly useful
if you're interested in a particular feature that happens to be
covered.  If you're looking for a broader overview, you might be out
of luck.

If you're looking for an overview from more of a user perspective
though, there's plenty of resources available.  The Reference Guide is
available online (https://lucene.apache.org/solr/guide) , and there
are a few different books out there on Solr that, while older, are
still pretty good references for the basics. (Solr in Action being
maybe the most popular).

Hope that helps, and good luck getting started.

Best,

Jason

On Mon, Apr 20, 2020 at 10:15 AM Martin Entwistle
 wrote:
>
> Hello,
>
> I am new to the lucene-solr code base and I am attempting to learn how things 
> work.
>
> Are there any guides/diagrams/documents anywhere to help people get 
> orientated?
>
> Many thanks
>
> Martin Entwistle
> Software Developer
> IBM Security
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
> - To 
> unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional 
> commands, e-mail: dev-h...@lucene.apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Alessandro Benedetti as a Lucene/Solr committer

[Jason Gerlowski]

Congrats and welcome Alessandro.  Well deserved.

On Wed, Mar 18, 2020 at 2:34 PM Tomás Fernández Löbbe
 wrote:
>
> Congrats and welcome Alessandro!
>
> On Wed, Mar 18, 2020 at 9:34 AM Erick Erickson  
> wrote:
>>
>> Welcome Alessandro!
>>
>> On Wed, Mar 18, 2020, 10:32 Houston Putman  wrote:
>>>
>>> Congrats Alessandro!
>>>
>>> On Wed, Mar 18, 2020 at 10:07 AM Tommaso Teofili 
>>>  wrote:

 welcome on board Alessandro, well deserved!
 I still remember when we were sitting together in the same room having fun 
 with Lucene/Solr a few years ago, keep up the good job !

 Regards,
 Tommaso

 On Wed, 18 Mar 2020 at 14:01, David Smiley  
 wrote:
>
> Hi all,
>
> Please join me in welcoming Alessandro Benedetti as the latest 
> Lucene/Solr committer!
>
> Alessandro has been contributing to Lucene and Solr in areas such as More 
> Like This, Synonym boosting, and Suggesters, and other areas for years.  
> Furthermore he's been a help to many users on the solr-user mailing list 
> and has helped others through his blog posts and presentations about 
> search.  We look forward to his future contributions.
>
> Congratulations and welcome!  It is a tradition to introduce yourself 
> with a brief bio, Alessandro.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: CHANGES.txt and issue categorization

[Jason Gerlowski]

> Robert Muir)
>>>>> * LUCENE-9237: Faster UniformSplit intersect TermsEnum. (Bruno Roustant)
>>>>>
>>>>> These "Improvements" I think are better categorized as "Other":
>>>>> * LUCENE-9109: Backport some changes from master (except StackWalker) to 
>>>>> improve
>>>>>   TestSecurityManager (Uwe Schindler)
>>>>> * LUCENE-9110: Backport refactored stack analysis in tests to use 
>>>>> generalized
>>>>>   LuceneTestCase methods (Uwe Schindler)
>>>>> * LUCENE-9141: Simplify LatLonShapeXQuery API by adding a new abstract 
>>>>> class called LatLonGeometry. Queries are
>>>>>   executed with input objects that extend such interface. (Ignacio Vera)
>>>>> * LUCENE-9194: Simplify XYShapeXQuery API by adding a new abstract class 
>>>>> called XYGeometry. Queries are
>>>>>   executed with input objects that extend such interface. (Ignacio Vera)
>>>>>
>>>>> Maybe this "Other" item should be  "Optimization"? (not sure):
>>>>> * LUCENE-9068: FuzzyQuery builds its Automaton up-front (Alan Woodward, 
>>>>> Mike Drob)
>>>>>
>>>>> Solr:
>>>>>
>>>>> "New Features" that maybe should be "Improvements":
>>>>>  * SOLR-13892: New "top-level" docValues join implementation (Jason 
>>>>> Gerlowski, Joel Bernstein)
>>>>>  * SOLR-14242: HdfsDirectory now supports indexing geo-points, ranges or 
>>>>> shapes. (Adrien Grand)
>>>>>
>>>>> "Improvements" that maybe should be "Optimizations":
>>>>> * SOLR-13808: filter in BoolQParser and {"bool":{"filter":..}} in Query 
>>>>> DSL are cached by default (Mikhail Khludnev)
>>>>>
>>>>> "Improvements" that maybe should be "Other":
>>>>> * SOLR-14114: Add WARN to Solr log that embedded ZK is not supported in 
>>>>> production (janhoy)
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> ~ David Smiley
>>>>> Apache Lucene/Solr Search Developer
>>>>> http://www.linkedin.com/in/davidwsmiley
>>>
>>>
>>>
>>> --
>>> Adrien

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Propose JIRA "Fix Version" remove from create-issue screen

[Jason Gerlowski]

+1 on Option A.

[-0] on Option B.  Even though it might not be everyday, I don't think
we should put roadblocks in front of users who want to clean up after
themselves.  We do occasionally see users create jira issues and then
close them themselves when they realize user error or something else
was at play.

On Mon, Feb 3, 2020 at 12:03 AM David Smiley  wrote:
>
> I think too often a "Fix Version" is set prematurely, especially by 
> contributors who don't know better and seem to choose arbitrary values, thus 
> making this JIRA field less meaningful.  Ideally it is set on resolution.  
> We've also used it to assign issues to releases in advance to avoid 
> forgetting about them.[1] The permissions on this field in JIRA appears to be 
> a bit unique[2]; it's tied to the ability to "Resolve" issues.  Reporters 
> (who could be anybody) can resolve issues (e.g. to close) can thus set the 
> fix version.  I can see a couple options to improve the situation *and we 
> could do both*.  I propose we do both in fact.
>
> Option A:  Remove "Fix Version" from the "create issue" screen.
> If usually this shouldn't be set on issue creation, then let's remove the 
> temptation to set it here.  Many contributors, I think, only ever see the 
> create-issue screen and won't edit the issue, which we'll leave open for the 
> ability to set this field.  Implementing this appears easy as we've got our 
> own project-specific screen to manipulate.
>
> Option B:  Revoke the "Resolve" permission for the Reporter.
> It seems unusual for simple contributors to "Resolve" the issue.  They might 
> note it's a duplicate after-the-fact but it seems quite rare to me; it's 
> usually us committers (who retain the right to Resolve any issue) who point 
> out a duplicate or perhaps decide the issue is a "Won't-Fix" or whatever.  
> Implementing this proposal would require moving to a project-specific 
> permission scheme instead of using the default one that's in use by 349 
> projects.
>
> [1]: We might stop the practice of using fix-version as a TODO for unresolved 
> issues, and thus fix-version would simply only ever get set for resolved 
> issues and thus be editable on a resolution screen.  But what other approach? 
>  Maybe Priority of Blocker, though it wouldn't differentiate the next-major 
> release from the next-minor one.  Shrug; the status quo is fine I guess.
>
> [2]: 
> https://confluence.atlassian.com/adminjiraserver083/managing-project-permissions-976767811.html
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: analytics.adoc validation does not pass

[Jason Gerlowski]

> I also think its confusing to have gitattributes at the root and then also 
> under solr/

Oh, that's new.  There wasn't a root gitattributes when I wrote the
solr/ one.  Must have been added around the same time through the
gradle work?

Anyway, there's no technical reason afaik to keep the files separate.
I put my recent addition under solr/ because I guessed the Lucene code
wasn't nearly as affected by the OS-specific script issue we've hit a
few times in Solr.  I didn't want to stick my nose in and affect the
Lucene folks without a concrete benefit to point to.  But if that's
something Lucene guys would want Robert, feel free to pursue a
combination.

Though it probably makes sense to first see where we land on Dawid's
point about narrowing the files affected.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: analytics.adoc validation does not pass

[Jason Gerlowski]

> It created an issue for me in that I spent two hours looking for a bug in the 
> wrong place...

I regret that it made it harder for you to find the issue.  But at the
end of the day the issue wasn't with gitattributes, it was a slip-up
in custom precommit code.

I agree with Jan - "auto" prevents CRLF from being injected into the
files that don't need it, it's a Good Thing.  Now, if gitattributes
breaks your normal workflow in some way - if your diffs start looking
noisy and weird locally, if Eclipse or IntelliJ behave weirdly, if
builds fail, etc. - then absolutely we need to scale back what
gitattributes affects.  But with our custom precommit code fixed, is
that the case?

If you feel strongly enough, I'll scale back what gitattributes
affects.  But as Jan pointed out, this is a concrete benefit to us,
and it doesn't feel right to walk that back just because there might
be some workflow issue out there that we don't know about.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: analytics.adoc validation does not pass

[Jason Gerlowski]

> It's because of the recent change to solr/.gitattributes
> (SOLR-10713) [sic - SOLR-14186] End of lines differ for adoc files. Jason -- 
> please
> change this one from auto to "don't touch line endings on any of other
> files", whatever the magic is.

Since the issue ended up being a bad regex in the build task, do you
still want me to exclude adoc files specifically from gitattributes
Dawid?  I'll make the change if there's some particular reason why
adoc files should be exempted here.  But at a glance it seems like
gitattributes wasn't the core problem here.

Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Change solr/lucene Readme file format

[Jason Gerlowski]

> Perhaps a weekly email with a list of PRs from first timers?

+1.  Would be a helpful nudge for committers to pay attention to other
contributors.

On Thu, Jan 23, 2020 at 4:46 AM Jan Høydahl  wrote:
>
> I added some review comments on the PR: 
> https://github.com/apache/lucene-solr/pull/908 as some of the converted files 
> were not really markdown compliant or did not look good in GitHub. Let’s hope 
> the contributor is still around to address this.
>
> Agree that we should pay special attention to first time contributors! 
> Perhaps a weekly email with a list of PRs from first timers?
>
> Jan Høydahl
>
> 22. jan. 2020 kl. 16:11 skrev Houston Putman :
>
> 
> Markdown is very standard at this point and I think the barrier to entry is 
> very small, at least much smaller than asciidoc, which is used for the ref 
> guide.
>
> +1 on this for me.
>
> On Wed, Jan 22, 2020 at 9:43 AM Doug Turnbull 
>  wrote:
>>
>> I think this got lost in the holidays. I wanted to bump this contribution, 
>> as I feel markdown is pretty standard format for readmes that devs are 
>> expecting these days. (And the files were close to markdown anyway.)
>>
>> Or if the project doesn't want this contribution, I feel we should at least 
>> let Pinkesh (with his 1st contribution) that this isn't something the 
>> project wants, and close the PR
>>
>> Best!
>> -Doug
>>
>> On Thu, Nov 14, 2019 at 12:51 AM Man with No Name 
>>  wrote:
>>>
>>> Hey guys,
>>> I have created a PR on this, please have a look to see if that's helpful.
>>>
>>> Thanks:
>>> Pinkesh Sharma
>>>
>>> On Sun, Nov 10, 2019 at 11:29 AM Uwe Schindler  wrote:

 Hi,

 When building the documentation (ant documentation), all readme files 
 included in the documentation are parsed as markdown (see flexmark task in 
 ant) and converted to html. This works well, although not everything is 
 markdown. If you have a plain readme file it would still parse as valid 
 markdown and HTML output looks fine, so Erik's problem with markdown isn't 
 one.

 Uwe

 Am November 10, 2019 4:00:21 PM UTC schrieb Marcus Eagan 
 :
>
> Most README files in contemporary open source projects are Markdown 
> because of the formatting features. I personally favor convention over 
> ease of use in this case.
>
> Marcus Eagan
>
> On Sun, Nov 10, 2019, 8:58 AM Erick Erickson  
> wrote:
>>
>> Personally I’d make them text files. The last thing I want to do is make 
>> reading/updating these have a barrier to entry. We should save 
>> formatting for the ref guide and/or Wiki.
>>
>> Best,
>> Erick
>>
>> > On Nov 10, 2019, at 1:01 AM, Man with No Name 
>> >  wrote:
>> >
>> > Hey folks,
>> > I have been looking into the solr/lucene source code, and the first 
>> > thing caught my eye was the different Readme files. All the files had 
>> > different file and text format. What do you guys think about making 
>> > all the readmes to markdown file rather than text files, and a 
>> > standard template?
>> >
>> >
>> > --
>> > Regards:
>> > Pinkesh Sharma
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

 --
 Uwe Schindler
 Achterdiek 19, 28357 Bremen
 https://www.thetaphi.de
>>>
>>>
>>>
>>> --
>>> Regards:
>>> Pinkesh Sharma
>>
>>
>>
>> --
>> Doug Turnbull | CTO | OpenSource Connections, LLC | 240.476.9983
>> Author: Relevant Search
>> This e-mail and all contents, including attachments, is considered to be 
>> Company Confidential unless explicitly stated otherwise, regardless of 
>> whether attachments are marked as such.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: gitattributes and line-endings

[Jason Gerlowski]

Thanks for the heads up Erick.

I poked around a little bit there and agree that this shouldn't affect
the gitattributes change.  I'm also curious why it was needed.  Maybe
Yonik or someone else involved with putting it in could clarify for
curiosity's sake.

Jason

On Tue, Jan 14, 2020 at 11:22 AM Erick Erickson  wrote:
>
> Jason:
>
> I’ve been poking around some of the regenerate code. I should say
> up-front that I have no idea whether this is relevant or not, but…
>
> Here’s a command to scare you:
>
> "find . -name build.xml | xargs grep -i fixcrlf”
>
> To be frank, I’m not totally sure what the intent for those is. From
> what I can tell they operate on java files that are the output from
> things like jflex and just modify line endings for the Java output,
> presumably the output may have windows-style line endings, but
> that’s a guess. Originally I had a vague fear that it operated on
> binary output, but this made me look more deeply and I don’t see that.
>
> So since they appear to be working on generated Java code, and
> I assume this is happening before the git operations you
> outlined, I expect it to be totally invisible.
>
> This is mostly FYI, I happened to see the “fixcrlf” in the build files
> and thought I’d pass it along. I don’t think this should hold up the
> commit at all.
>
> Best,
> Erick
>
> > On Jan 14, 2020, at 10:18 AM, Jason Gerlowski  wrote:
> >
> > Hi all,
> >
> > Wanted to sent out a notice about a change I plan on making this week
> > regarding how line endings are maintained in some of our OS-specific
> > files (bin/solr, zkcli, etc.).
> >
> > Previously it's been on committers/contributors individually to put
> > the correct line endings in their files.  This has bitten us a few
> > times recently.  solr.cmd often ends up missing the CRLF EOLs that it
> > needs, which breaks the code surrounding those edits. See SOLR-13977
> > for a recent example.
> >
> > To remedy this, I plan on adding a .gitattributes file to normalize
> > EOL's across Solr code.  File editing is unchanged.  But on certain
> > git commands (add, commit, checkout), git will rewrite the line
> > endings of changed files to match the desired EOL for that file.  For
> > most files, this is LF.  For Windows specific files, this is CRLF.
> > See 
> > https://github.com/apache/lucene-solr/pull/1163/files#diff-c781eab6689a2956b8e2d082fdecbe38
> > for the specific rules.
> >
> > This change should be entirely invisible to developers, with two caveats:
> >
> > 1. Of necessity, the gitattributes PR also corrects a few files that
> > have the wrong line endings currently.  Developers applying old
> > patches that touch one of these files will encounter merge conflicts
> > due to the EOL changes.
> >
> > 2. If developers have incorrect line endings in files unique to a
> > feature branch, and they merge master into that feature branch, they
> > may see unexpected changes in git-status as git normalizes the EOL's
> > in those files.
> >
> > Given how rare incorrect EOLs are (outside of solr.cmd), I expect this
> > change will be invisible to 99% of developers.  If anyone has any
> > questions or concerns, let me know or comment on the PR (#1163)
> > directly.  I plan on merging the change tomorrow or the day after.
> >
> > Best,
> >
> > Jason
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

gitattributes and line-endings

[Jason Gerlowski]

Hi all,

Wanted to sent out a notice about a change I plan on making this week
regarding how line endings are maintained in some of our OS-specific
files (bin/solr, zkcli, etc.).

Previously it's been on committers/contributors individually to put
the correct line endings in their files.  This has bitten us a few
times recently.  solr.cmd often ends up missing the CRLF EOLs that it
needs, which breaks the code surrounding those edits. See SOLR-13977
for a recent example.

To remedy this, I plan on adding a .gitattributes file to normalize
EOL's across Solr code.  File editing is unchanged.  But on certain
git commands (add, commit, checkout), git will rewrite the line
endings of changed files to match the desired EOL for that file.  For
most files, this is LF.  For Windows specific files, this is CRLF.
See 
https://github.com/apache/lucene-solr/pull/1163/files#diff-c781eab6689a2956b8e2d082fdecbe38
for the specific rules.

This change should be entirely invisible to developers, with two caveats:

1. Of necessity, the gitattributes PR also corrects a few files that
have the wrong line endings currently.  Developers applying old
patches that touch one of these files will encounter merge conflicts
due to the EOL changes.

2. If developers have incorrect line endings in files unique to a
feature branch, and they merge master into that feature branch, they
may see unexpected changes in git-status as git normalizes the EOL's
in those files.

Given how rare incorrect EOLs are (outside of solr.cmd), I expect this
change will be invisible to 99% of developers.  If anyone has any
questions or concerns, let me know or comment on the PR (#1163)
directly.  I plan on merging the change tomorrow or the day after.

Best,

Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Commit / Code Review Policy

[Jason Gerlowski]

> and now its turned around as if its consensus everywhere

David didn't say anything about consensus everywhere.  He mentioned
pretty clearly that the agreement was only "in attendance", and that
this thread was a precursor for putting together a proposal to test
the waters further.  That all seems pretty in line with the Apache
process for feeling out/testing/etc. consensus, though maybe there's a
nuance I'm missing?

Jason

On Sat, Nov 30, 2019 at 3:27 PM Shawn Heisey  wrote:
>
> On 11/30/2019 7:39 AM, Robert Muir wrote:
> > -1 ... you even went so far as to discourage lucene committers from
> > attending that meeting, and now its turned around as if its consensus
> > everywhere and should be applied to lucene too?
> >
> > I don't think changing things to review-then-commit will help.
>
> I did attend the meeting, and I think committers who concentrate on
> Lucene should not be discouraged from attending any similar future
> meetings.  Discussions in a meeting about Solr are not very likely to
> dive down that far into the inner workings, but even if they don't, some
> Lucene people might find the the talk about Solr to be interesting, and
> having their perspective available would not be a bad thing.
>
> I'm very wary of making an official change to review-then-commit.  I
> fully support the ideas that went into the proposal, but I think making
> it mandatory for all commits is going to really slow things down and
> cause some problems.  It's not the way most Apache projects work, and it
> makes it a LOT harder to do quick sanity edits like fixing spelling errors.
>
> As much as I really like the idea of more frequent reviews, I think that
> review requirements should be informal.
>
> Most of the time a committer will know whether the changes they are
> working on fall into a "major" or "minor" category.  When it's leaning
> more towards major, I think most of us will agree that a few extra
> eyeballs looking for gotchas is a really good idea.  TLDR note:  The
> number of lines in a change is sometimes NOT an indicator of how major
> the change is.
>
> I certainly want to seek a "looks good to me" on changes I make which
> have any significant impact.  For some of the ideas I have, if those
> ever reach the implementation phase, I think I'd want even more
> assurance that I'm not doing it wrong.  I can pledge that I will seek
> review for non-trivial changes.  I wonder if there is wording we can add
> to any official project rulebook to make such a thing mandatory, without
> a full switch.
>
> I think that a full switch to review-then-commit (RTC) would be the
> wrong thing to do.  I think it would lead to a lot of dissent within the
> project, encouraging rivalries and cliques within the project.
>
> If RTC is preferred by a significant majority, I will work within that
> paradigm ... but I think that it would be a short-lived experiment and
> we would be back to CTR pretty quickly.
>
> Is it possible to vote -0.5 instead of -1?  I don't think it is.
>
> Thanks,
> Shawn
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 8.3.1 release

[Jason Gerlowski]

+1

On Mon, Nov 25, 2019 at 4:19 AM Shalin Shekhar Mangar
 wrote:
>
> +1
>
> On Mon, Nov 25, 2019 at 11:20 AM Noble Paul  wrote:
>>
>> +1
>>
>> On Mon, Nov 25, 2019 at 4:49 PM Ishan Chattopadhyaya
>>  wrote:
>> >
>> > Hi,
>> > Recently, Colvin has reported
>> > https://issues.apache.org/jira/browse/SOLR-13963. Effect of this is
>> > that Solr 8.3 must not be used out of the box due to the reported data
>> > loss. I propose we fix this as soon as possible and release 8.3.1. I
>> > am volunteering as RM for this.
>> > Any thoughts?
>> > Regards,
>> > Ishan
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>>
>> --
>> -
>> Noble Paul
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Bruno Roustant as Lucene/Solr committer

[Jason Gerlowski]

Welcome Bruno!

On Sat, Nov 23, 2019 at 9:19 PM Tomoko Uchida
 wrote:
>
> Congratulations and welcome!
>
> Tomoko
>
> 2019年11月24日(日) 10:22 Otis Gospodnetić :
> >
> > Hi,
> >
> > Welcome and great work!
> >
> > Otis
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> >
> > On Sat, Nov 23, 2019 at 3:29 AM Adrien Grand  wrote:
> >>
> >> Hi all,
> >>
> >> Please join me in welcoming Bruno Roustant as the latest Lucene/Solr 
> >> committer!
> >>
> >> It didn't take many JIRA issues for Bruno to demonstrate good
> >> understanding of the lower level bits of Lucene by writing a new
> >> postings format and more recently exploring ideas that ended up
> >> speeding up FSTs while decreasing their memory usage at the same time.
> >> We are very happy that Bruno accepted the PMC's invitation to join.
> >>
> >> Congratulations and welcome, Bruno! It's a tradition to introduce
> >> yourself with a brief bio.
> >>
> >> --
> >> Adrien
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Houston Putman as Lucene/Solr committer

[Jason Gerlowski]

Congratulations!

On Thu, Nov 14, 2019 at 11:58 AM Gus Heck  wrote:
>
> Congratulations and welcome :)
>
> On Thu, Nov 14, 2019 at 11:52 AM Namgyu Kim  wrote:
>>
>> Congratulations and welcome, Houston! :D
>>
>> On Fri, Nov 15, 2019 at 1:18 AM Ken LaPorte  wrote:
>>>
>>> Congratulations Houston! Well deserved honor.
>>>
>>>
>>>
>>> --
>>> Sent from: 
>>> https://lucene.472066.n3.nabble.com/Lucene-Java-Developer-f564358.html
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Hi I'm Raphael

[Jason Gerlowski]

Hi Raphael,

Welcome and thanks for taking an interest in helping out!  Shawn's
pointers above are all good for getting started.  If there's any other
information you need, let us know.

You know best what you want to get involved with.  But if you're
interested in QA and are interested in the Solr side of things, you
might be interested in a test-failure aggregator that Hoss, a member
of the community here, has set up:
http://fucit.org/solr-jenkins-reports/failure-report.html.  Some tests
fail regularly but others have low failure rates and are challenging
to reproduce.  We always need help with tests and appreciate help
tracking down issues.

Looking forward to working with you as you get ramped up.

Best,

Jason

On Fri, Nov 8, 2019 at 12:42 AM Shawn Heisey  wrote:
>
> On 11/7/2019 6:14 PM, Raphael Bircher wrote:
> > Wich source tree I should build for testing purpose? I heard rumors, that 
> > installing Lucene is a bit rocket science.
>
> I'm here for Solr, and my interest in Lucene is only incidental, because
> Solr derives most of its functionality from Lucene.
>
> Almost all development work here begins in the master branch, and is
> backported to other branches as required.  We release from the stable
> branch, which is currently branch_8x.  At some point, likely next year,
> we will create branch_9x and master will be renumbered to major version 10.
>
> Lucene is not really geared for "installation" ... it is a development
> library.  It is used by developers to write their own software, and it
> will be up to that developer to work out how their software is
> installed.  Lucene is a development dependency, much like this one:
>
> https://dev.mysql.com/downloads/connector/j/
>
> If you want to work on a complete application, you can opt to work on
> Solr instead of Lucene.  Or you can go to project outside the Apache
> umbrella that happens to use Lucene as a dependency.
>
> Thanks,
> Shawn
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Rethinking how we publish the Solr Ref Guide

[Jason Gerlowski]

+1.  That all sounds good to me.  Excited to see some streamlining here.

On Fri, Sep 20, 2019 at 3:46 PM Cassandra Targett  wrote:
>
> Thanks everyone, by the way, for the encouragement and feedback here.
>
> For next steps, how do folks feel about making the change to stop voting on 
> the PDF *now*? Or, I guess, retroactively for 8.2 since that’s not out yet. I 
> could push the HTML and make a PDF but announce to the list that from 8.2 
> forward we consider the HTML the main Ref Guide and the PDF is “for 
> convenience” (and explain the thinking behind it).
>
> If we want a VOTE on this policy change, I can do that - I feel like we have 
> consensus without it, but we could be more formal about it if folks prefer.
>
> For 8.3 we'll see what we can get automated there, but if it’s not ready I’ll 
> just do it manually once the RC is out.
>
> I’ll file a Jira for some of the changes I’ll make to the docs for the 
> process, etc., and another one for automation ideas.
>
> Cassandra
> On Sep 19, 2019, 2:53 PM -0500, Noble Paul , wrote:
>
> First of all a big thanks to Cassandra to help coordinate and build
> our ref guide to make it professional. It really used to be pathetic
> before you took over
>
> . Yes we need to avoid "creating work" . There should be no need for a
> ref guide release.
>
> +1 for your plan
>
> On Thu, Sep 19, 2019 at 11:57 PM Cassandra Targett
>  wrote:
>
>
> The pages do already have a “Site last generated” date on them at the bottom. 
> It’s specifically worded that way for a reason.
>
> We actually wanted the date the .adoc file was last updated to be in the 
> footer too, but the problem has always been that a static site generator 
> always regenerates all pages every time - so every single page, edited or 
> not, always has the same exact date on it.
>
> And, our build works by copying everything under `solr/solr-ref-guide/src` to 
> `solr/build`, so every file really has a create date and last updated date 
> that are always the date you do the build. Basically, the files you see and 
> work with are not actually the same files that get built - we build from 
> copies that are made at build-time.
>
> (That’s all why it says “Site last generated” instead of “Last updated”.)
>
> I’m not saying there’s no way to add a last updated date for the underlying 
> file, it’s just not obvious how to do it so we skipped it.
>
> No problem, though, adding a link to Github - that’s actually kind of a 
> different thing and I’m pretty sure we could do that right now.
>
> Cassandra
> On Sep 19, 2019, 7:07 AM -0500, Anshum Gupta , wrote:
>
> I agree that we should be able to fix mistakes, my only suggestion was that 
> those mistakes not be non-trivial. But the more I think about it, the more I 
> feel convinced about just publishing the updates - however, having a time 
> stamp on when the guide was last updated would be nice to have. Anything else 
> would require much more effort and I'm not sure it's worth it.
>
> On Wed, Sep 18, 2019 at 2:48 PM Chris Hostetter  
> wrote:
>
>
>
> : > However Anshum does make a good point that users wouldn't know when
> : the pages have changed. I think it would be great to have a link on each
> : ref-guide page that shows the last modified date and links to the
> : history of that page in github
>
> : Perhaps we could instead provide a single HTML page or HTML table as
> : part of or alongside each guide, showing all commits touching the guide
> : on that branch after the release. Could probably be baked in as part of
> : the release script. Using the release date or git hash for the release,
>
> Yeah, there are a lot of options we could pursue for generating a
> "changes" list as part of an automated build process -- but i would
> consider this idea a "nice to have" feature that shouldn't block moving
> forward.
>
> Given 2 options, I would much rather:
> a) have the ability to quickly/easily "fix" mistakes/ommisions in
> "official X.y ref-guide" on our site and have it automatically republish,
> w/o it being immediately obvious to users that a page may have changed
> between yesterday and today.
> ... over ...
> b) *NOT* being able to re-publish at all just for the sake of users
> knowing that the (incorrect) content they are reading is consistent
> between yesterday and today.
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
> --
> Anshum Gupta
>
>
>
>
> --
> -
> Noble Paul
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional 

Re: Github PR Collaboration Question

[Jason Gerlowski]

Thanks for chiming in guys.  I created a JIRA for this (SOLR-13775).
Haven't really thought about wording or anything yet, if either of you
have any suggestions.  Otherwise I'll hope to get to this over the
weekend.

Jason

On Tue, Sep 17, 2019 at 3:39 PM David Smiley  wrote:
>
> RE the idea of shash-merge from a PR:  that would be cool were it not for 
> CHANGES.txt; ah well.
>
> +1 to "put it in the PR template checklist"
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Sep 17, 2019 at 2:37 PM Jan Høydahl  wrote:
>>
>> I like the idea to put it in the PR template checklist. Or you can ask the 
>> contributor to check that box. I once made a PR against another PR branch 
>> and it worked but was too many steps.
>>
>> I like the idea of squash-merging from PR branch since (I believe) the 
>> commit will then have the credit of the contributor which then shows up in 
>> his/her stats on GH which is nice.
>>
>> Jan Høydahl
>>
>> > 17. sep. 2019 kl. 16:17 skrev Jason Gerlowski :
>> >
>> > Github does have an option that fork-owners can click when they create
>> > a PR that will give those with karma on the upstream repo the same
>> > karma on their PR branch.  [1] That would solve this problem somewhat.
>> > But it's still up to users to choose that themselves.  Maybe it makes
>> > sense to mention this as an optional checklist item in the PR template
>> > that was recently added.
>> >
>> > Still curious about other approaches though if anyone has suggestions.
>> >
>> > [1] 
>> > https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork
>> >
>> >> On Tue, Sep 17, 2019 at 10:12 AM Jason Gerlowski  
>> >> wrote:
>> >>
>> >> Hey all,
>> >>
>> >> I've hit a small snag reviewing a few PRs on github recently.  Wanted
>> >> to see if anyone has any suggestions for my workflow:
>> >>
>> >> I’ve found myself in the position a few times where I want to add a
>> >> few small changes to a contributor’s PR…either to help them with a
>> >> piece they haven’t gotten to yet, or to show what I’m suggesting with
>> >> a particular review comment, or to clean up little things like
>> >> whitespace formatting, etc.  In the patch world, this is easy to do
>> >> without requiring review/acking by other contributors…you apply the
>> >> latest patch, make your additional changes, and re-upload to jira.
>> >> But in Github, you have to juggle branch/PR ownership.  PR’s are often
>> >> from personal forks, where others don't have write access there.  So I
>> >> can't add to the "main" PR without either (a) asking the contributor
>> >> to give me the right karma, or (b) opening my own "secondary" PR
>> >> against their "main" PR branch, and asking them to review/merge my
>> >> "secondary" PR.
>> >>
>> >> Is there some simpler approach I'm missing?  I love the in-line
>> >> comments that Github supports for code-review, and it seems like more
>> >> committers are starting to use it.  I'd love to figure out how to make
>> >> it work for me.  But two developers collaborating on the same PR seems
>> >> like a pretty fundamental use case to be so heavyweight.  This must be
>> >> a problem that's solved, right?
>> >>
>> >> Best,
>> >>
>> >> Jason
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Atri Sharma as Lucene/Solr committer

[Jason Gerlowski]

Congratulations and welcome Atri!

On Wed, Sep 18, 2019 at 6:24 AM Shalin Shekhar Mangar
 wrote:
>
> Congratulations and welcome Atri!
>
> On Wed, Sep 18, 2019 at 3:12 AM Adrien Grand  wrote:
>>
>> Hi all,
>>
>> Please join me in welcoming Atri Sharma as Lucene/ Solr committer!
>>
>> If you are following activity on Lucene, this name will likely sound
>> familiar to you: Atri has been very busy trying to improve Lucene over
>> the past months. In particular, Atri recently started improving our
>> top-hits optimizations like early termination on sorted indexes and
>> WAND, when indexes are searched using multiple threads.
>>
>> Congratulations and welcome! It is a tradition to introduce yourself
>> with a brief bio.
>>
>> --
>> Adrien
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Github PR Collaboration Question

[Jason Gerlowski]

Github does have an option that fork-owners can click when they create
a PR that will give those with karma on the upstream repo the same
karma on their PR branch.  [1] That would solve this problem somewhat.
But it's still up to users to choose that themselves.  Maybe it makes
sense to mention this as an optional checklist item in the PR template
that was recently added.

Still curious about other approaches though if anyone has suggestions.

[1] 
https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork

On Tue, Sep 17, 2019 at 10:12 AM Jason Gerlowski  wrote:
>
> Hey all,
>
> I've hit a small snag reviewing a few PRs on github recently.  Wanted
> to see if anyone has any suggestions for my workflow:
>
> I’ve found myself in the position a few times where I want to add a
> few small changes to a contributor’s PR…either to help them with a
> piece they haven’t gotten to yet, or to show what I’m suggesting with
> a particular review comment, or to clean up little things like
> whitespace formatting, etc.  In the patch world, this is easy to do
> without requiring review/acking by other contributors…you apply the
> latest patch, make your additional changes, and re-upload to jira.
> But in Github, you have to juggle branch/PR ownership.  PR’s are often
> from personal forks, where others don't have write access there.  So I
> can't add to the "main" PR without either (a) asking the contributor
> to give me the right karma, or (b) opening my own "secondary" PR
> against their "main" PR branch, and asking them to review/merge my
> "secondary" PR.
>
> Is there some simpler approach I'm missing?  I love the in-line
> comments that Github supports for code-review, and it seems like more
> committers are starting to use it.  I'd love to figure out how to make
> it work for me.  But two developers collaborating on the same PR seems
> like a pretty fundamental use case to be so heavyweight.  This must be
> a problem that's solved, right?
>
> Best,
>
> Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Github PR Collaboration Question

[Jason Gerlowski]

Hey all,

I've hit a small snag reviewing a few PRs on github recently.  Wanted
to see if anyone has any suggestions for my workflow:

I’ve found myself in the position a few times where I want to add a
few small changes to a contributor’s PR…either to help them with a
piece they haven’t gotten to yet, or to show what I’m suggesting with
a particular review comment, or to clean up little things like
whitespace formatting, etc.  In the patch world, this is easy to do
without requiring review/acking by other contributors…you apply the
latest patch, make your additional changes, and re-upload to jira.
But in Github, you have to juggle branch/PR ownership.  PR’s are often
from personal forks, where others don't have write access there.  So I
can't add to the "main" PR without either (a) asking the contributor
to give me the right karma, or (b) opening my own "secondary" PR
against their "main" PR branch, and asking them to review/merge my
"secondary" PR.

Is there some simpler approach I'm missing?  I love the in-line
comments that Github supports for code-review, and it seems like more
committers are starting to use it.  I'd love to figure out how to make
it work for me.  But two developers collaborating on the same PR seems
like a pretty fundamental use case to be so heavyweight.  This must be
a problem that's solved, right?

Best,

Jason

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Notes from the committers' meeting at Activate 2019

[Jason Gerlowski]

I missed the part of the meeting/lunch when our use of Github came up.
Can anyone that was present summarize the discussion in a little more
detail?

re: issues on github.  There are challenges like avoiding
fragmentation and keeping multiple issue sources up to date, but those
are problems that probably shrink or go away with appropriate
automation.  IMO at least, issue reporting is largely the same whether
you're on Github, JIRA, trello, etc.  You fill out a form, set the
appropriate tags and fields, etc.  I'm fine w/ changes there, but it's
hard for me to imagine how that would have much/any impact on barrier
to entry.

I see our code-contribution process as a much bigger driver of the
barrier-to-entry. First time contributors must learn how to generate
patches.  They receive code-review on JIRA, so they get one chunk of
text in a comment.  And contributions have very weak version control
(identically named SOLR-.patch files piling up on the same issue).
Github has concrete benefits here.  If the goal is to make it easier
for contributors to get involved, making Github first-class for code
contributions might be where the real gains are.  (We allow Github
PR's, but could steer people towards them more clearly: rewrite
HowToContribute docs to assume Github, hide the patch process for new
contributors, set up workflows in Github to run precommit/tests on
PRs, etc.)

On Tue, Sep 17, 2019 at 8:56 AM Eric Pugh
 wrote:
>
> Ah..   The mythical committer just sitting around waiting to be “interested 
> in the issue” that you have created!   Having said that, I think thats a 
> separate challenge!
>
>
> On Sep 17, 2019, at 12:38 AM, David Smiley  wrote:
>
> +1 to all that Gus said.  On a fresh project then indeed perhaps we would use 
> GitHub issues but it's not nearly so compelling at this point with so much 
> rich history in JIRA, not to mention those issues being referenced in commit 
> messages.
>
> Is the point to lower barriers for contributors that are not in our community 
> yet (thus don't have an ASF JIRA account)?  Well... they don't *have* to 
> create the JIRA issue if a committer is sufficiently interested in the issue 
> at hand to do it.  And as mentioned this could be automated.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Sep 16, 2019 at 6:27 PM Gus Heck  wrote:
>>
>> FWIW, One thing that needs to be figured out is how github would accommodate 
>> security issues (or how the process for those issues would change).  Does 
>> github have the ability to assign roles and visibility (could be I haven't 
>> really worked with organizations on GitHub, all my clients have been on jira 
>> ... except the one that has trello, and another with gitlab... neither of 
>> which have impressed me very much )?
>>
>> Additionally, I'm slightly leery of the move since I don't (yet) see the 
>> real benefit that pays for the splitting of the records into two systems. 
>> Can issues be migrated to github? I've not really been on a large project 
>> that used only GitHub, can someone explain what we *gain* by moving to 
>> GitHub issues. At least two things are lost: continuity and familiarity. My 
>> impression is that there are a lot fewer features, but maybe I've just not 
>> been exposed to them.
>>
>> Part of me worries that this is the usual cycle of "this is simpler" (mass 
>> migration ensues) "but it doesn't x, y and z" (x, y and z are added) "wow 
>> this is complex, but THAT is simpler" (mass migration ensues...) "hmm p, 
>> q an z are missing" (p q and z are added  and cycle repeats). So I'm 
>> skeptical of any "gain" hanging it's hat solely on "simplicity". Jira used 
>> to be the simpler, snazzier, sexier alternative to bugzilla...
>>
>> Sell me, I'm all ears, but not seeing it yet :)
>>
>> -Gus
>>
>> On Mon, Sep 16, 2019 at 3:57 PM Jan Høydahl  wrote:
>>>
>>> Is there any reason at all that we need to hold on to JIRA? ASF allows us 
>>> to move all issue handling over to GH, I'd like us to consider such a move.
>>>
>>> Until then, I made a script that "diffs" GH and JIRA and outputs a report, 
>>> see 
>>> https://github.com/apache/lucene-solr/tree/master/dev-tools/scripts#githubprspy
>>>
>>> Running the tool shows that we're not very successful in keeping the two in 
>>> sync, we also forget to close PRs after JIRAs are resolved:
>>>
>>> $ ./githubPRs.py
>>> Lucene/Solr Github PR report
>>> 
>>> Number of open Pull Requests: 208
>>>
>>> PRs lacking JIRA reference in title
>>>   #882: Add versions.lock entry for "org.apache.yetus:audience-annotations"
>>>   #880: Tweak header format.
>>>   [… many more ]
>>>
>>> Open PRs with a resolved JIRA
>>>   #723: SOLR-13545 status=Closed, resolution=Fixed, 
>>> resolutiondate=2019-06-22T13:04:47.000+ (SOLR-13545: AutoClose stream 
>>> in ContentStreamUpdateRequest)
>>>   #643: SOLR-13391 status=Resolved, resolution=Resolved, 
>>> 

Re: Intervals in Solr json request

[Jason Gerlowski]

Hi Mikhail,

I'm having trouble understanding the exact syntax you're proposing.
Is there a jira where the syntax is described in a little more detail?
 If not, would you care to put together a writeup on a jira somewhere?
 It's hard (for me at least) to weigh in as things are currently.

Best,

Jason

On Sun, Sep 8, 2019 at 3:25 PM Mikhail Khludnev  wrote:
>
> Ok. It might be a parser referring to a json object under some new property
>
> {
>"query": {
>"jinterval":"just a name"  // introducing new QPPlugin
>   },
>"jparams": {   // introducing new top-level entry
> "just a name": {
>   "or":["foo",
>   "bar",
>{ "unordered":
> ["bag",
>   "baz",
>   "ban",
>{ "phrase": ["moo","foo"]}
>  ]
> }
>   ],
>"field":"text_content"
>  }
>  }
> }
>
> Can we consider it as a spec for the new feature?
>
> On Sun, Sep 8, 2019 at 12:16 AM Mikhail Khludnev  wrote:
>>
>> Thanks for your warm responses. I encounter Intervals, and considering 
>> introducing them in Solr JSON Request API.
>> Following Query DSL approach gives me something like
>> "interval":{  "or":["foo",
>>   "bar",
>>{"interval": { "unordered":
>> ["bag",
>>   "baz",
>>   "ban",
>>{ "interval":{ "phrase": 
>> ["moo","foo"]} }
>>  ]}
>>   }
>> ],
>>"field":"text_content"}
>> So, it implies creating {!inteval} query parser, which handles local param 
>> in a certain way, eg  it shouldn't support "or" and "phrase" an the same 
>> node.
>> Not sure how to propagate "filed" to term nodes.
>>
>> I'd rather want to have more control over syntax and JsonQueryConverter.
>>  "interval":{  "or":["foo",
>>   "bar",
>>{ "unordered":
>> ["bag",
>>   "baz",
>>   "ban",
>>{ "phrase": ["moo","foo"]}
>>  ]}
>>   }
>>   ],
>>"field":"text_content"}
>>
>> Any ideas, preferences?
>>
>> On Sat, Sep 7, 2019 at 12:03 AM Mikhail Khludnev  wrote:
>>>
>>> Hello,
>>>
>>> Finally we let users to send span queries via XML (yeah) query parser. But 
>>> I feel awkward to invoke XML under Json. Straightforward approach lead us 
>>> to bunch of span[Or|And|Not|Etc] QParser plugins. Are there any more 
>>> elegant ideas?
>>>
>>> --
>>> Sincerely yours
>>> Mikhail Khludnev
>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13724) Reject v1 API updates for (non-routed) multi-collection aliases

[Jason Gerlowski (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918122#comment-16918122
 ] 

Jason Gerlowski commented on SOLR-13724:


Here's a quick way to reproduce the behavior:

{code}
➜  solr git:(fcbe46c28ce) ✗ bin/solr start -c
Waiting up to 180 seconds to see Solr running on port 8983 [\]  
Started Solr server on port 8983 (pid=20921). Happy searching!

➜  solr git:(fcbe46c28ce) ✗ bin/solr create -c foo
Created collection 'foo' with 1 shard(s), 1 replica(s) with config-set 'foo'
➜  solr git:(fcbe46c28ce) ✗ bin/solr create -c bar
Created collection 'bar' with 1 shard(s), 1 replica(s) with config-set 'bar'
➜  solr git:(fcbe46c28ce) ✗ curl -ilk -X GET 
"http://localhost:8983/solr/admin/collections?action=CREATEALIAS=my_alias=foo,bar;
HTTP/1.1 200 OK
Content-Type: application/json;charset=utf-8
Content-Length: 57

{
  "responseHeader":{
"status":0,
"QTime":128}}
➜  solr git:(fcbe46c28ce) ✗ curl -ilk -X POST -H "Content-type: 
application/json" "http://localhost:8983/solr/my_alias/update?commit=true; -d 
'[{"id": "asdf", "val_i": 4}]'
HTTP/1.1 200 OK
Content-Type: text/plain;charset=utf-8
Content-Length: 69

{
  "responseHeader":{
"rf":1,
"status":0,
"QTime":373}}
{code}

> Reject v1 API updates for (non-routed) multi-collection aliases
> ---
>
> Key: SOLR-13724
> URL: https://issues.apache.org/jira/browse/SOLR-13724
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 8.1, master (9.0)
>Reporter: Jason Gerlowski
>Priority: Major
>
> The intent of SOLR-13407 was to block all updates on aliases that represent 
> multiple collections.  Currently, this works for deletes, commits, and 
> optimize requests.  Users get a message like: {{Update request to non-routed 
> multi-collection alias not supported}}.
> But currently we still allow document adds/updates to go through.
> We should either close this last hole, so _no_ update requests are allowed, 
> or document it so the behavior clearer.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13724) Reject v1 API updates for (non-routed) multi-collection aliases

[Jason Gerlowski (Jira)]

Jason Gerlowski created SOLR-13724:
--

 Summary: Reject v1 API updates for (non-routed) multi-collection 
aliases
 Key: SOLR-13724
 URL: https://issues.apache.org/jira/browse/SOLR-13724
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 8.1, master (9.0)
Reporter: Jason Gerlowski


The intent of SOLR-13407 was to block all updates on aliases that represent 
multiple collections.  Currently, this works for deletes, commits, and optimize 
requests.  Users get a message like: {{Update request to non-routed 
multi-collection alias not supported}}.

But currently we still allow document adds/updates to go through.

We should either close this last hole, so _no_ update requests are allowed, or 
document it so the behavior clearer.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13539) Atomic Update Multivalue remove does not work for field types UUID, Enums, Bool and Binary

[Jason Gerlowski (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-13539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911848#comment-16911848
 ] 

Jason Gerlowski commented on SOLR-13539:


bq. When you say all the cast issues are fixed on master and 8x, does that 
include the small patch I added to this ticket, for the removeregex multi-value 
situation?

It doesn't.  But I have bundled that in with Thomas' changes in my local 
testing and will make sure it makes it to master and branch_8x.

Speaking of which, test "beast" runs look pretty good, so I'll go forward with 
committing this later this week, or this weekend.

> Atomic Update Multivalue remove does not work for field types UUID, Enums, 
> Bool  and Binary
> ---
>
> Key: SOLR-13539
> URL: https://issues.apache.org/jira/browse/SOLR-13539
> Project: Solr
>  Issue Type: Bug
>  Components: UpdateRequestProcessors
>Affects Versions: 7.7.2, 8.1, 8.1.1
>Reporter: Thomas Wöckinger
>Priority: Critical
> Attachments: SOLR-13539.patch
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> When using JavaBinCodec the values of collections are of type 
> ByteArrayUtf8CharSequence, existing field values are Strings so the remove 
> Operation does not have any effect.
>  This is related to following field types: UUID, Enums, Bool and Binary



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13687) Enable the bin/solr script to accept a solr url to run commands

[Jason Gerlowski (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910929#comment-16910929
 ] 

Jason Gerlowski commented on SOLR-13687:


Sure.  My main point was less about url vs zk-conn-string (I think either is 
fine...though there is prior art around the ZK_HOST setting).  My point was 
just that a solr.in.sh property fits well with how other similar things are 
done today.  Whether you add an explicit flag or not, users would probably 
expect this to be set-able in solr.in.sh.

bq. Ideally, we should minimize access to ZK from hosts outside of of solr 
nodes. It's a security hole.
Two things about this:
1. ZK has security mechanisms.  If you don't use those, then yes, it's a 
security hole.  But you can say the same thing about a Solr URL if Solr hasn't 
been secured.
2. I'd love for ZK to be a background detail we can hide from people, but right 
now I think it's a little unrealistic to try abstracting from clients, given 
how fundamental it is to Solr. ZK-conn-strings are stickier and more useful to 
use in cloud environments, where individual Solr nodes might join and leave a 
cluster frequently. etc

That said, I don't really have a dog in the conn-string vs url fight.  Just 
thinking aloud about pros/cons.  I think adding a flag/solr.in.sh option for 
either would be an improvement.

> Enable the bin/solr script to accept a solr url to  run commands
> 
>
> Key: SOLR-13687
> URL: https://issues.apache.org/jira/browse/SOLR-13687
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Priority: Major
>
> The problem we have today with our {{bin/solr}} script is that we have to run 
> it from one of the nodes where Solr is running. This is a security issue b/c 
> only admins are usaully be allowed to login to a machine where solr is 
> running.If you have multiple cluster running in that host we don't know which 
> one it's going to use. It is much easier to write a simple script that works 
> over a url and the user has no ambiguity as to how it works. You can just 
> unpack a solr distribution to your local machine and start using the script 
> without bothering to install solr .
> The following commands can easily be executed remotely. These commands can 
> accept the base url of any solr node in the cluster and perform the opertaion
>  * healthcheck
>  * create
>  * create_core
>  * create_collection
>  * delete, version,
>  * config
>  * autoscaling



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13687) Enable the bin/solr script to accept a solr url to run commands

[Jason Gerlowski (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910728#comment-16910728
 ] 

Jason Gerlowski commented on SOLR-13687:


This is already partially possible: several of the commands above have an 
explicit "zk-host" option that allows you to point them at remote clusters.  
You can also point at a remote cluster by setting ZK_HOST in solr.in.sh.

If {{create_collection}}, {{delete}}, etc don't accept ZK_HOST currently, 
adding ZK_HOST support is probably the easiest and most uniform path forward.

> Enable the bin/solr script to accept a solr url to  run commands
> 
>
> Key: SOLR-13687
> URL: https://issues.apache.org/jira/browse/SOLR-13687
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Priority: Major
>
> The problem we have today with our {{bin/solr}} script is that we have to run 
> it from one of the nodes where Solr is running. This is a security issue b/c 
> only admins are usaully be allowed to login to a machine where solr is 
> running.If you have multiple cluster running in that host we don't know which 
> one it's going to use. It is much easier to write a simple script that works 
> over a url and the user has no ambiguity as to how it works. You can just 
> unpack a solr distribution to your local machine and start using the script 
> without bothering to install solr .
> The following commands can easily be executed remotely. These commands can 
> accept the base url of any solr node in the cluster and perform the opertaion
>  * healthcheck
>  * create
>  * create_core
>  * create_collection
>  * delete, version,
>  * config
>  * autoscaling



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13407) Reject updates sent to non-routed multi collection aliases

[Jason Gerlowski (Jira)]

[ 
https://issues.apache.org/jira/browse/SOLR-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910567#comment-16910567
 ] 

Jason Gerlowski commented on SOLR-13407:


Hey [~ab], I noticed recently that I can seemingly still use (standard) aliases 
to index documents.

Am I missing something here?  I ran {{git checkout 
releases/lucene-solr/8.1.1}}, rebuilt, and can still see the behavior below:

{code}
➜  solr git:(fcbe46c28ce) ✗ bin/solr start -c
Waiting up to 180 seconds to see Solr running on port 8983 [\]  
Started Solr server on port 8983 (pid=20921). Happy searching!

➜  solr git:(fcbe46c28ce) ✗ bin/solr create -c foo
Created collection 'foo' with 1 shard(s), 1 replica(s) with config-set 'foo'
➜  solr git:(fcbe46c28ce) ✗ bin/solr create -c bar
Created collection 'bar' with 1 shard(s), 1 replica(s) with config-set 'bar'
➜  solr git:(fcbe46c28ce) ✗ curl -ilk -X GET 
"http://localhost:8983/solr/admin/collections?action=CREATEALIAS=my_alias=foo,bar;
HTTP/1.1 200 OK
Content-Type: application/json;charset=utf-8
Content-Length: 57

{
  "responseHeader":{
"status":0,
"QTime":128}}
➜  solr git:(fcbe46c28ce) ✗ curl -ilk -X POST -H "Content-type: 
application/json" "http://localhost:8983/solr/my_alias/update?commit=true; -d 
'[{"id": "asdf", "val_i": 4}]'
HTTP/1.1 200 OK
Content-Type: text/plain;charset=utf-8
Content-Length: 69

{
  "responseHeader":{
"rf":1,
"status":0,
"QTime":373}}
{code}

> Reject updates sent to non-routed multi collection aliases
> --
>
> Key: SOLR-13407
> URL: https://issues.apache.org/jira/browse/SOLR-13407
> Project: Solr
>  Issue Type: Improvement
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13407.patch
>
>
> Spin-off from SOLR-13262.
> Currently Solr uses a convention that updates sent to multi-collection 
> aliases are applied only to the first collection on the list, which is 
> nonintuitive and may hide bugs or accidental configuration changes made 
> either in Solr or in client applications.
> This issue proposes to reject all such updates with an error.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org