Re: Lucene/Solr 7

[Ramkumar R. Aiyengar]

Should SOLR-8396 be a prerequisite?

On 25 Jan 2017 10:38, "Christine Poerschke (BLOOMBERG/ LONDON)" <
cpoersc...@bloomberg.net> wrote:

> +1 for May.
>
> I'd like to see https://issues.apache.org/jira/browse/SOLR-8668 in the
> 7.0 release (and have tagged/updated the ticket to indicate so).
>
> Christine
>
> From: dev@lucene.apache.org At: 01/24/17 17:17:27
> To: dev@lucene.apache.org
> Subject: Re: Lucene/Solr 7
>
> I would love to see SOLR-5944, SOLR-8029, SOLR-9835 in a 7.0 release. I
> think all of these are very close to landing up on master.
>
> On Tue, Jan 24, 2017 at 10:22 PM, Adrien Grand  wrote:
>
>> Hi all,
>>
>> We have accumulated some good changes in master, like point support in
>> Solr or sparse norms/doc-values in Lucene. I think it would be nice to
>> expose these new features to our users, so what would you think about
>> starting to work on making master ready to be released?
>>
>> Since the question about the timeframe will be asked, I think we could
>> target something like early May 2017, which is a bit more than 3 months
>> away from now. What do you think?
>>
>> Adrien
>>
>
>

Re: Welcome Mikhail Khludnev to the PMC

[Ramkumar R. Aiyengar]

Congratulations Mikhail!

On 30 Dec 2016 15:16, "Adrien Grand"  wrote:

> I am pleased to announce that Mikhail Khludnev has accepted the PMC's
> invitation to join.
>
> Welcome Mikhail!
>
> Adrien
>

Re: Welcome Jim Ferenczi as a Lucene/Solr committer

[Ramkumar R. Aiyengar]

Welcome Jim!

On 1 Jan 2017 10:05, "Michael McCandless"  wrote:

> I'm pleased to announce that Jim Ferenczi has accepted the Lucene
> PMC's invitation to become a committer.
>
> Jim, it's tradition that you introduce yourself with a brief bio.
>
> Your handle "jimczi" has been added to the “lucene" LDAP group, so you
> now have commit privileges. Please test this by adding yourself to the
> committers section of the Who We Are page on the website:
>  (instructions here
> ).
>
> The ASF dev page also has lots of useful links: <
> http://www.apache.org/dev/>.
>
> Congratulations and welcome and Happy New Year,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Christine Poerschke to the PMC

[Ramkumar R. Aiyengar]

Congratulations Christine!

On 30 Dec 2016 12:47, "Adrien Grand"  wrote:

> I am pleased to announce that Christine Poerschke has accepted the PMC's
> invitation to join.
>
> Welcome Christine!
>
> Adrien
>

Re: Welcome Ishan Chattopadhyaya as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Welcome Ishan :)

On Tue, Nov 29, 2016 at 6:17 PM, Mark Miller  wrote:

> I'm pleased to announce that Ishan Chattopadhyaya has accepted the PMC's
> invitation to become a committer.
>
> Ishan, it's tradition that you introduce yourself with a brief bio /
> origin story, explaining how you arrived here.
>
> Your handle "ishan" has already added to the “lucene" LDAP group, so
> you now have commit privileges.
>
> Please celebrate this rite of passage, and confirm that the right
> karma has in fact enabled, by embarking on the challenge of adding
> yourself to the committers section of the Who We Are page on the
> website: http://lucene.apache.org/whoweare.html (use the ASF CMS
> bookmarklet
> at the bottom of the page here: https://cms.apache.org/#bookmark -
> more info here http://www.apache.org/dev/cms.html).
>
> Congratulations and welcome!
> --
> - Mark
> about.me/markrmiller
>



-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: GIT migration complete

[Ramkumar R. Aiyengar]

What's the error? I don't think you can still merge into Github, it remains
read-only..
On 24 Jan 2016 03:59, "Joel Bernstein" <joels...@gmail.com> wrote:

> I had no problem pushing my first commit. But I didn't seem to have
> permission to merge a pull-request from Github. Is there another layer of
> permissions that need to be setup for this?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Sat, Jan 23, 2016 at 1:23 PM, Yonik Seeley <ysee...@gmail.com> wrote:
>
>> Here's a useful thing you can add to your .profile , it conditionally
>> displays the branch you are on in your shell prompt:
>> https://gist.github.com/yonik/04b2e759eecf5ef96383
>>
>> Example:
>>
>> /opt/code$ cd lusolr
>> [master] /opt/code/lusolr$
>>
>> -Yonik
>>
>>
>> On Sat, Jan 23, 2016 at 12:55 PM, Ramkumar R. Aiyengar
>> <andyetitmo...@gmail.com> wrote:
>> > Not a big deal, but guys, do check if your mail is set properly in
>> > ~/.gitconfig, otherwise commits would show your personal mail as the
>> author.
>> > Committer still shows up as your apache mail address, but many UIs hide
>> it
>> > by default, and might not be apparent to those new to the project/git..
>> >
>> > Example .gitconfig snippet allowing you to keep your existing name/mail
>> but
>> > override for apache.org..
>> >
>> > [user "https://git-wip-us.apache.org;]
>> > name = Ramkumar Aiyengar
>> > mail = andyetitmo...@apache.org
>> >
>> >
>> > On Sat, Jan 23, 2016 at 3:41 PM, Uwe Schindler <u...@thetaphi.de> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I migrated all Policeman and ASF Jenkins Jobs to Git. Everything
>> should be
>> >> set up now.
>> >> The Jobs for 5.3 and 5.4 were deleted.
>> >>
>> >> Uwe
>> >>
>> >> -
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: u...@thetaphi.de
>> >>
>> >>
>> >> > -Original Message-
>> >> > From: Uwe Schindler [mailto:u...@thetaphi.de]
>> >> > Sent: Saturday, January 23, 2016 11:05 AM
>> >> > To: dev@lucene.apache.org
>> >> > Subject: RE: GIT migration complete
>> >> >
>> >> > Hi,
>> >> >
>> >> > Thanks! I am talking care of Jenkins at the moment.
>> >> >
>> >> > First Policeman one is working:
>> >> > http://jenkins.thetaphi.de/job/Lucene-Solr-
>> >> > trunk-Linux/15619/console
>> >> >
>> >> > The Changes I do for jenkins is:
>> >> > - Deleted *all* workspaces to have clean start
>> >> > - Change all Jobs (is a bit of work as manually, especially for ASF)
>> to
>> >> > use Git
>> >> > checkouts of corresponding branch
>> >> > - Add the following "additional behaviour" config: Clean before
>> >> > checkout:
>> >> > Clean up the workspace before every checkout by deleting all
>> untracked
>> >> > files
>> >> > and directories, including those which are specified in .gitignore.
>> It
>> >> > also resets
>> >> > all tracked files to their versioned state. This ensures that the
>> >> > workspace is in
>> >> > the same state as if you cloned and checked out in a brand-new empty
>> >> > directory, and ensures that your build is not affected by the files
>> >> > generated
>> >> > by the previous build.
>> >> >
>> >> > Uwe
>> >> >
>> >> > -
>> >> > Uwe Schindler
>> >> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> > http://www.thetaphi.de
>> >> > eMail: u...@thetaphi.de
>> >> >
>> >> > > From: Dawid Weiss [mailto:dawid.we...@gmail.com]
>> >> > > Sent: Saturday, January 23, 2016 10:13 AM
>> >> > > To: dev@lucene.apache.org
>> >> > > Subject: GIT migration complete
>> >> > >
>> >> > > Ok, folks. Seems like code migration is complete, with both gh
>> mirrors
>> >> > > and Apache repos up.
>> >> > >
>> >> > > 1. Please read the docs at: https://git-wip-us.apache.org/
>> >> > >
>> >

Re: GIT migration complete

[Ramkumar R. Aiyengar]

https://git-scm.com/docs/git-credential-cache

Should help with the constant password typing..
On 23 Jan 2016 09:13, "Dawid Weiss"  wrote:

> Ok, folks. Seems like code migration is complete, with both gh mirrors
> and Apache repos up.
>
> 1. Please read the docs at: https://git-wip-us.apache.org/
>
> 2. I didn't realize Apache's repositories are over https, which makes
> authorization rather painful for Linux users (requires password typing
> on every push). I think I'll stick with gh for this reason and only
> push my changes when they're fully baked.
>
> 3. The master branch (former trunk) passes ant precommit for me, but I
> didn't apply build patches to branch_5x yet.
>
> If there are any problems or concerns, let me know (perhaps I'll be
> able to help).
>
> Dawid
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: GIT migration complete

[Ramkumar R. Aiyengar]

You're welcome! :)
On 23 Jan 2016 09:31, "Dawid Weiss" <dawid.we...@gmail.com> wrote:

> Thanks Ramkumar. Also thank you for saving that list of PRs, that was
> very thoughtful of you.
>
> Dawid
>
> On Sat, Jan 23, 2016 at 10:26 AM, Ramkumar R. Aiyengar
> <andyetitmo...@gmail.com> wrote:
> > https://git-scm.com/docs/git-credential-cache
> >
> > Should help with the constant password typing..
> >
> > On 23 Jan 2016 09:13, "Dawid Weiss" <dawid.we...@gmail.com> wrote:
> >>
> >> Ok, folks. Seems like code migration is complete, with both gh mirrors
> >> and Apache repos up.
> >>
> >> 1. Please read the docs at: https://git-wip-us.apache.org/
> >>
> >> 2. I didn't realize Apache's repositories are over https, which makes
> >> authorization rather painful for Linux users (requires password typing
> >> on every push). I think I'll stick with gh for this reason and only
> >> push my changes when they're fully baked.
> >>
> >> 3. The master branch (former trunk) passes ant precommit for me, but I
> >> didn't apply build patches to branch_5x yet.
> >>
> >> If there are any problems or concerns, let me know (perhaps I'll be
> >> able to help).
> >>
> >> Dawid
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: dev-h...@lucene.apache.org
> >>
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: GIT migration complete

[Ramkumar R. Aiyengar]

Not a big deal, but guys, do check if your mail is set properly in
~/.gitconfig, otherwise commits would show your personal mail as the
author. Committer still shows up as your apache mail address, but many UIs
hide it by default, and might not be apparent to those new to the
project/git..

Example .gitconfig snippet allowing you to keep your existing name/mail but
override for apache.org..

[user "https://git-wip-us.apache.org;]
name = Ramkumar Aiyengar
mail = andyetitmo...@apache.org


On Sat, Jan 23, 2016 at 3:41 PM, Uwe Schindler  wrote:

> Hi,
>
> I migrated all Policeman and ASF Jenkins Jobs to Git. Everything should be
> set up now.
> The Jobs for 5.3 and 5.4 were deleted.
>
> Uwe
>
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
> > -Original Message-
> > From: Uwe Schindler [mailto:u...@thetaphi.de]
> > Sent: Saturday, January 23, 2016 11:05 AM
> > To: dev@lucene.apache.org
> > Subject: RE: GIT migration complete
> >
> > Hi,
> >
> > Thanks! I am talking care of Jenkins at the moment.
> >
> > First Policeman one is working:
> http://jenkins.thetaphi.de/job/Lucene-Solr-
> > trunk-Linux/15619/console
> >
> > The Changes I do for jenkins is:
> > - Deleted *all* workspaces to have clean start
> > - Change all Jobs (is a bit of work as manually, especially for ASF) to
> use Git
> > checkouts of corresponding branch
> > - Add the following "additional behaviour" config: Clean before checkout:
> > Clean up the workspace before every checkout by deleting all untracked
> files
> > and directories, including those which are specified in .gitignore. It
> also resets
> > all tracked files to their versioned state. This ensures that the
> workspace is in
> > the same state as if you cloned and checked out in a brand-new empty
> > directory, and ensures that your build is not affected by the files
> generated
> > by the previous build.
> >
> > Uwe
> >
> > -
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> > > From: Dawid Weiss [mailto:dawid.we...@gmail.com]
> > > Sent: Saturday, January 23, 2016 10:13 AM
> > > To: dev@lucene.apache.org
> > > Subject: GIT migration complete
> > >
> > > Ok, folks. Seems like code migration is complete, with both gh mirrors
> > > and Apache repos up.
> > >
> > > 1. Please read the docs at: https://git-wip-us.apache.org/
> > >
> > > 2. I didn't realize Apache's repositories are over https, which makes
> > > authorization rather painful for Linux users (requires password typing
> > > on every push). I think I'll stick with gh for this reason and only
> > > push my changes when they're fully baked.
> > >
> > > 3. The master branch (former trunk) passes ant precommit for me, but I
> > > didn't apply build patches to branch_5x yet.
> > >
> > > If there are any problems or concerns, let me know (perhaps I'll be
> > > able to help).
> > >
> > > Dawid
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: GIT migration date (SVN freeze)

[Ramkumar R. Aiyengar]

More to Mark's point, the focus of this effort is to move the repo and not
the development model. Sure, we can debate a lot more on whether rebase or
merge is preferred, bit given that the amount of git experience varies
across committers, it would help to start off as close to SVN as possible,
and linear history fits the bill..
On 20 Jan 2016 06:30, "Mark Miller"  wrote:

> On Tue, Jan 19, 2016 at 11:51 PM Noble Paul  wrote:
>
>>
>> > Yes, patches in JIRA will still be the primary force, with our secondary
>> > GitHub integration hooks. Agreed, 3rd party submissions will get
>> effectively
>> > squashed anyway.
>>
>> In the Git world users are happy give a pull request instead of a
>> patch. Why do you think patches in Jira will be the primary force?
>>
>>
> Because we are not changing anything about the current process right now.
> Just moving from SVN to Git.
>
> First class support is still patches in JIRA. We have a second class
> integration with Github as well. As before, contributors are free to use
> either. Nothing changes in that regard.
>
> - Mark
> --
> - Mark
> about.me/markrmiller
>

Re: GIT migration date (SVN freeze)

[Ramkumar R. Aiyengar]

+1. There might be ways to enforce it on the remote side..

http://stackoverflow.com/questions/2039773/have-remote-git-repository-refuse-merge-commits-on-push

But I am not a git expert enough to tell you exactly what the solution
there is trying to do..
On 19 Jan 2016 19:00, "Mark Miller"  wrote:

> I think for everyone's sanity we want to mostly ban merge commits and
> insist people use rebase and a linear history. I think we want to keep the
> history nice and clean just like it is now.
>
> You can change the pull command so that it does rebase rather than merge
> via git config --global pull.rebase true
>
> Even if everyone does not agree with that, we need some discussion and
> guidelines. Everyone going crazy in Git with merge commits makes an ungodly
> mess.
>
> - Mark
>
> On Tue, Jan 19, 2016 at 1:49 PM Dawid Weiss  wrote:
>
>> > So I'm clear, this also means that from Saturday morning (call it 6:00
>> UTC)
>> > until you give the OK (assuming the original schedule) I shouldn't count
>> > on having access to any of the source code, right?
>>
>> You will have access to the source code -- the SVN remains functional,
>> but it'll be an empty folder on the branches I mentioned.
>>
>> > And when you do give the OK, I should plan on rebasing whatever I'm
>> > working on to the new Git repo. There, did I use at least one Git term
>> > correctly?
>>
>> Well, the workflow is up to you. One possible way to work is like this
>> (I assume command line, because that's what I use).
>>
>> 1) git clone 
>> cd lucene_solr
>>
>> 2) git checkout master -b mybranch
>>
>> 3) while (not done) {
>>   // work on mybranch's files.
>>   git add -A .   # add any changes (and removals) to the
>> commit, recursively.
>>   git diff --cached # see what would be committed.
>>   git commit -m "interim commit"
>> }
>>
>> 4) When done, fetch any new commits from Apache. Merge them with your
>> feature branch. If there are conflicts, git will guide you. Note there
>> are no rebases here, you just merge stuff with master much like you
>> did with SVN.
>>
>> git fetch origin
>> git merge origin/master
>>
>> 5) Create a patch against master.
>>
>> git diff origin/master > myfeature.patch
>>
>> Done. Place the patch in Jira.
>>
>> If you wish to commit your changes to master, I'd "squash" all your
>> interim changes into one single commit (easier on the eyes and simpler
>> to revert).
>>
>> git checkout master
>> git pull
>> git merge --squash mybranch --nocommit
>> # review what would be changed again, etc.
>> git stat
>> git diff --cached
>> # finally, commit
>> git commit -m "My changes."
>> # and push the commit from your local repository and current branch to
>> remote (Apache) repository.
>> git push origin HEAD
>>
>> I guarantee you these steps are conceptually very simple. I'd run gitk
>> --all after every single one (having read the document I pointed to
>> previously) -- you'd see how the graph of patches and merges unfolds.
>>
>> Dawid
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>> --
> - Mark
> about.me/markrmiller
>

Re: [VOTE] Release Lucene/Solr 5.4.1 RC2

[Ramkumar R. Aiyengar]

+1

SUCCESS! [0:56:50.593184]

On Mon, Jan 18, 2016 at 7:19 PM, Erick Erickson 
wrote:

> +1
>
> SUCCESS! [0:42:55.587563]
>
> So I see I'm between Mike and Shalin in terms of how fast my machine
> goes
>
> On Mon, Jan 18, 2016 at 8:58 AM, Shalin Shekhar Mangar
>  wrote:
> > +1
> >
> > SUCCESS! [1:11:14.331645]
> >
> > On Mon, Jan 18, 2016 at 8:08 PM, Adrien Grand  wrote:
> >> Please vote for the RC2 release candidate for Lucene/Solr 5.4.1
> >>
> >> This release candidate contains 3 additional changes compared to the
> RC1:
> >>  - SOLR-8496: multi-select faceting and getDocSet(List) can match
> >> deleted docs
> >>  - SOLR-8418: Adapt to changes in LUCENE-6590 for use of boosts with
> >> MLTHandler and Simple/CloudMLTQParser
> >>  - SOLR-8561: Add fallback to ZkController.getLeaderProps for a mixed
> >> 5.4-pre-5.4 deployments
> >>
> >> The artifacts can be downloaded from:
> >>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.4.1-RC2-rev1725212
> >>
> >> You can run the smoke tester directly with this command:
> >> python3 -u dev-tools/scripts/smokeTestRelease.py
> >>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.4.1-RC2-rev1725212
> >>
> >> The smoke tester already passed for me both with the local and remote
> >> artifacts, so here is my +1.
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: [VOTE] Release Lucene/Solr 5.4.1 RC1

[Ramkumar R. Aiyengar]

Thanks for waiting Adrien. I have now backported SOLR-8418 to 5.4.

On Sun, Jan 17, 2016 at 10:01 PM, Adrien Grand  wrote:

> Le dim. 17 janv. 2016 à 04:00, Mark Miller  a
> écrit :
>
>> Always nice to be patient and accommodating for reroll 1. You know, since
>> we are all friends here. Best to get picky later on.
>
>
> I don't think it is fair to qualify the fact that we should release the
> corruption fix as soon as possible as picky.
>
> I can't make everybody happy here as there are two valid opposite
> arguments that we should release the corruption fix as soon as possible on
> the one hand and that it is ridiculous to release Solr with such a major
> bug on the other hand.
>
> I won't release the current artifacts and will respin tomorrow morning EU
> time (about 12 hours from now). If someone could backport SOLR-8418 to the
> 5.4 branch until then, that would be great, otherwise I will do it myself
> before building the RC.
>



-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: [VOTE] Release Lucene/Solr 5.4.1 RC1

[Ramkumar R. Aiyengar]

I missed the initial RC, but if we are going to re-spin (and looks like
Yonik has a patch up now), I would like to back-port SOLR-8418. Should be
fairly benign, let me know if there are any concerns..

On Fri, Jan 15, 2016 at 5:59 PM, Adrien Grand  wrote:

> Thanks Yonik, let's see what this bug boils down to and how quickly we can
> get it fixed.
>
> Le ven. 15 janv. 2016 à 17:15, Yonik Seeley  a écrit :
>
>> We've discovered a very serious bug, currently unknown how deep it
>> runs, but may make sense to respin if we can get it ironed out quickly
>> enough:
>> https://issues.apache.org/jira/browse/SOLR-8496
>>
>> -Yonik
>>
>>
>> On Thu, Jan 14, 2016 at 5:41 AM, Adrien Grand  wrote:
>> > Please vote for the RC1 release candidate for Lucene/Solr 5.4.1
>> >
>> > The artifacts can be downloaded from:
>> >
>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.4.1-RC1-rev1724447/
>> >
>> > You can run the smoke tester directly with this command:
>> > python3 -u dev-tools/scripts/smokeTestRelease.py
>> >
>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.4.1-RC1-rev1724447/
>> >
>> > The smoke tester already passed for me both with the local and remote
>> > artifacts, so here is my +1.
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>


-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: [JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.7.0_80) - Build # 15104 - Still Failing!

[Ramkumar R. Aiyengar]

Looks like my bad on merge to branch 5x. I should be near a computer in a
few hours, and will fix it then if no one else gets to it by then..


On Fri, 1 Jan 2016 13:01 Policeman Jenkins Server 
wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/15104/
> Java: 64bit/jdk1.7.0_80 -XX:-UseCompressedOops -XX:+UseG1GC
>
> All tests passed
>
> Build Log:
> [...truncated 9026 lines...]
> [javac] Compiling 868 source files to
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/classes/java
> [javac]
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java:147:
> error: incompatible types
> [javac] ? null : new
> ArrayList<>(mlt.mlt.getMaxQueryTerms());
> [javac] ^
> [javac]   required: List
> [javac]   found:ArrayList
> [javac] Note: Some input files use or override a deprecated API.
> [javac] Note: Recompile with -Xlint:deprecation for details.
> [javac] Note: Some input files use unchecked or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked for details.
> [javac] 1 error
>
> BUILD FAILED
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:794: The following
> error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:738: The following
> error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:59: The following
> error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build.xml:233: The
> following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:526:
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:476:
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:389:
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:520:
> The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1956:
> Compile failed; see the compiler error output for details.
>
> Total time: 24 minutes 48 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> [WARNINGS] Skipping publisher since build result is FAILURE
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Tomás Fernández Löbbe to the PMC

[Ramkumar R. Aiyengar]

Congratulations Tomas :)
On 6 Nov 2015 09:53, "Anshum Gupta"  wrote:

> I'm pleased to announce that Tomás Fernández Löbbe has accepted the PMC’s
> invitation to join.
>
> Welcome Tomás!
>
> --
> Anshum Gupta
>

Re: Welcome Dennis Gove as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Congratulations and welcome :)
On 6 Nov 2015 15:19, "Joel Bernstein"  wrote:

> I'm pleased to announce that Dennis Gove has accepted the PMC's
> invitation to become a committer.
>
> Dennis, it's tradition that you introduce yourself with a brief bio.
>
> Your account is not entirely ready yet. We will let you know when it is
> created
> and karma has been granted so that you can add yourself to the committers
> section of the Who We Are page on the website:
> .
>
> Congratulations and welcome!
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>

Re: Solr replication making use of timestamps?

[Ramkumar R. Aiyengar]

Crickets..

I have raised https://issues.apache.org/jira/browse/SOLR-7932 anyway for
this issue..

On Sat, Aug 8, 2015 at 8:50 PM, Ramkumar R. Aiyengar 
andyetitmo...@gmail.com wrote:

 Noticed while looking at https://issues.apache.org/jira/browse/SOLR-7859
 that wall time recorded as commit data on a commit to check if replication
 needs to be done. In IndexFetcher, there is this code:

   if (!forceReplication  
 IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
 //master and slave are already in sync just return
 LOG.info(Slave in sync with master.);
 successfulInstall = true;
 return true;
   }

 We are checking wall times across machines to check if we are in sync?
 That sounds wrong.. Or I am mistaken here? Why can't we just check
 generations? Another place below checks both generations and timestamps to
 see if a full copy is needed..

   // if the generation of master is older than that of the slave , it 
 means they are not compatible to be copied  // then a new index directory 
 to be created and all the files need to be copied  boolean 
 isFullCopyNeeded = IndexDeletionPolicyWrapper
   .getCommitTimestamp(commit) = latestVersion
   || commit.getGeneration() = latestGeneration || forceReplication;





-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: Better DocSetCollector

[Ramkumar R. Aiyengar]

I wonder if there might be value in BitDocIdSet.Builder which Lucene uses.
It had perf issues of its soon, but LUCENE-6645 seems to have fixed them,
and it does a similar approach as above (int array and then fixedbitset).
On 3 Aug 2015 12:35, Toke Eskildsen t...@statsbiblioteket.dk wrote:

 On Sat, 2015-08-01 at 15:09 -0700, Yonik Seeley wrote:
  I also investigated going the other way and tracking a Listint[] and
  allocating in smaller chunks (and even having a memory pool to pull
  the fixed size chunks from) but it was slower on my first attempt and
  I haven't returned to try more variants yet.  It *feels* like we
  should be able to get overall speedups by allocating in 8K chunks or
  so when the effects of memory bandwidth (the cost of zeroing) and GC
  are considered.

 Chunked allocations of int[] would still have the problem of having the
 copy-to-bitmap step if the result set gets too big.

 Chunks might work better with the garbage collector, compared to the
 current solution, but I greatly prefer the idea of re-using structures.

 That being said, I realize that it is not simple to choose the proper
 strategy:

 http://stackoverflow.com/questions/1955322/at-what-point-is-it-worth-reusing-arrays-in-java

 In the case of an update-tracked structure, the cost of zeroing is
 linear to the amount of changed values. This makes it even harder to
 determine the best strategy as it will be tied to concrete index size
 and query pattern.

 - Toke Eskildsen, State and University Library, Denmark


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Solr replication making use of timestamps?

[Ramkumar R. Aiyengar]

Noticed while looking at https://issues.apache.org/jira/browse/SOLR-7859
that wall time recorded as commit data on a commit to check if replication
needs to be done. In IndexFetcher, there is this code:

  if (!forceReplication 
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) ==
latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
  }

We are checking wall times across machines to check if we are in sync? That
sounds wrong.. Or I am mistaken here? Why can't we just check generations?
Another place below checks both generations and timestamps to see if a full
copy is needed..

  // if the generation of master is older than that of the slave ,
it means they are not compatible to be copied  // then a new index
directory to be created and all the files need to be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
  .getCommitTimestamp(commit) = latestVersion
  || commit.getGeneration() = latestGeneration || forceReplication;

RE: Strange comment in CdcrReplicationHandlerTest.java

[Ramkumar R. Aiyengar]

Ah okay, my bad then..
On 29 Jul 2015 21:28, Uwe Schindler u...@thetaphi.de wrote:

 RAT would only fail if the license header is missing completely. I don't
 think it checks for copyright notices.

 If there is no license header, we should check our RAT config! What does
 it list as license for that file?

 Uwe

 -
 Uwe Schindler
 H.-H.-Meier-Allee 63, D-28213 Bremen
 http://www.thetaphi.de
 eMail: u...@thetaphi.de


  -Original Message-
  From: Erick Erickson [mailto:erickerick...@gmail.com]
  Sent: Wednesday, July 29, 2015 7:36 PM
  To: dev@lucene.apache.org
  Subject: Re: Strange comment in CdcrReplicationHandlerTest.java
 
  Yeah, I wondered that myself.
 
  On Wed, Jul 29, 2015 at 1:35 PM, Ramkumar R. Aiyengar
  andyetitmo...@gmail.com wrote:
   Hmm.. I would have expected rat to fail this in precommit actually..
  
   On 29 Jul 2015 18:01, Timothy Potter thelabd...@gmail.com wrote:
  
   Why is this in the code?
  
   /**
* Copyright (c) 2015 Renaud Delbru. All Rights Reserved.
*/
   package org.apache.solr.cloud;
  
   -
   To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For
   additional commands, e-mail: dev-h...@lucene.apache.org
  
  
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional
  commands, e-mail: dev-h...@lucene.apache.org


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Strange comment in CdcrReplicationHandlerTest.java

[Ramkumar R. Aiyengar]

Hmm.. I would have expected rat to fail this in precommit actually..
On 29 Jul 2015 18:01, Timothy Potter thelabd...@gmail.com wrote:

 Why is this in the code?

 /**
  * Copyright (c) 2015 Renaud Delbru. All Rights Reserved.
  */
 package org.apache.solr.cloud;

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Christine Poerschke as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Welcome Christine. Glad to have company a couple of desks away :)
On 24 Jul 2015 08:28, Adrien Grand jpou...@gmail.com wrote:

 I'm pleased to announce that Christine Poerschke has accepted the PMC's
 invitation to become a committer.

 Christine, it's tradition that you introduce yourself with a brief bio.

 Your account is not entirely ready yet. We will let you know when it is
 created
 and karma has been granted so that you can add yourself to the committers
 section of the Who We Are page on the website:
 http://lucene.apache.org/whoweare.html.

 Congratulations and welcome!

 --
 Adrien

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Mikhail Khludnev as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Welcome Mikhail!

On Tue, Jul 21, 2015 at 8:21 AM, Adrien Grand jpou...@gmail.com wrote:

 I'm pleased to announce that Mikhail Khludnev has accepted the PMC's
 invitation to become a committer.

 Mikhail, it's tradition that you introduce yourself with a brief bio.

 Your handle mkhl has already added to the “lucene LDAP group, so
 you now have commit privileges. Please test this by adding yourself to
 the committers section of the Who We Are page on the website:
 http://lucene.apache.org/whoweare.html (use the ASF CMS bookmarklet
 at the bottom of the page here: https://cms.apache.org/#bookmark -
 more info here http://www.apache.org/dev/cms.html).

 Congratulations and welcome!

 --
 Adrien

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: Welcome Upayavira as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Welcome Upayavira!
On 22 Jun 2015 20:02, Steve Rowe sar...@gmail.com wrote:

 I'm pleased to announce that Upayavira has accepted the PMC's invitation
 to become a committer.

 Upayavira, it's tradition that you introduce yourself with a brief bio.

 Mike McCandless, the Lucene PMC chair, has already added your “upayavira
 account to the “lucene LDAP group, so you now have commit privileges.
 Please test this by adding yourself to the committers section of the Who We
 Are page on the website: http://lucene.apache.org/whoweare.html (use
 the ASF CMS bookmarklet at the bottom of the page here: 
 https://cms.apache.org/#bookmark - more info here 
 http://www.apache.org/dev/cms.html).

 Congratulations and welcome!

 Steve
 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 873 - Still Failing

[Ramkumar R. Aiyengar]

On Fri, Jun 12, 2015 at 8:09 PM, Apache Jenkins Server 
jenk...@builds.apache.org wrote:

 FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

 Error Message:
 Invalid content type:

 Stack Trace:
 org.apache.http.ParseException: Invalid content type:
 at
 __randomizedtesting.SeedInfo.seed([F06CDA025DC5E420:7838E5D8F33989D8]:0)
 at org.apache.http.entity.ContentType.parse(ContentType.java:273)



https://issues.apache.org/jira/browse/SOLR-7680 should fix this..


-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: Lucene / Solr 5.2 release notes

[Ramkumar R. Aiyengar]

Could we change the CHANGES.txt links with the .html links instead? They
are much better formatted.

(I just realised I don't have edit access to the wiki, could someone add
me? Name: Ramkumar Aiyengar, Alias: andyetitmoves)
I’ve made drafts for the Lucene and Solr release notes - please feel free
to edit or suggest edits:

Lucene:
https://wiki.apache.org/lucene-java/ReleaseNote52

Solr:
http://wiki.apache.org/solr/ReleaseNote52

-- 
Anshum Gupta

Re: Moving to git?

[Ramkumar R. Aiyengar]

 There is only one good rule though - no merge commmits in the history :)
Ever. Do whatever you want beyond that. A clean, simple history for each
branch is the only sensible use of Git I've seen.

+1


 - Mark


 On Sat, May 30, 2015 at 9:00 AM Adrien Grand jpou...@gmail.com wrote:

 The main benefit I see is that external contributors would get their
 name in the commit log.

 However on the other hand, I'm a bit annoyed that people easily
 disagree on the workflow: some people merge into the maintenance
 branch first and then to master, other people merge into master first
 and then cherry-pick, other people prefer rebasing instead of merging,
 etc. I personally don't really care but if we agree on moving to Git,
 I hope we can agree on the workflow at the same time. At least today
 with svn we have something simple that everybody agrees on.

 -0: I'm not against it but Subversion works well for me today. If
 everybody else agrees on switching to Git I would like us to agree on
 the workflow as well.

 --
 Adrien

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

 --
 - Mark
 about.me/markrmiller

Re: [VOTE] 5.2.0 RC3

[Ramkumar R. Aiyengar]

+1

SUCCESS! [1:45:08.193859]

On Mon, Jun 1, 2015 at 10:49 PM, Shalin Shekhar Mangar 
shalinman...@gmail.com wrote:

 +1

 Java7: SUCCESS! [1:16:45.996357]

 Java8: SUCCESS! [1:29:39.782174]

 On Mon, Jun 1, 2015 at 2:21 AM, Anshum Gupta ans...@anshumgupta.net
 wrote:

 Please vote for the third release candidate for Apache Lucene/Solr 5.2.0.

 The artifacts can be downloaded from:

 https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.2.0-RC3-rev1682753


 You can run the smoke tester directly with this command:

 python3 -u dev-tools/scripts/smokeTestRelease.py
 https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.2.0-RC3-rev1682753/

 Here's my +1

 SUCCESS! [0:34:19.663987]

 Thanks for the patience! :-)

 --
 Anshum Gupta




 --
 Regards,
 Shalin Shekhar Mangar.




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: Moving to git?

[Ramkumar R. Aiyengar]

+1 for git, great for working on multiple things at once.

Side note: git-svn is also not great btw for the kind of merging we need to
do with every commit, it kind of works but with too many caveats.

On the note that git clone is slow, sure, because it fetches a fair amount
of history which svn doesn't. But to compare just them is unfair, since
checkout and clone are not identical. If you want to compare times, you
will also have to add up every log, diff, or annotate you do on the tree
during your development (of which I certainly do a lot and I am sure others
do as well), and git will certainly win if you include all those because it
does no network lookup. Clone and checkout are typically one time
operations, why should their speed be a concern in any case?
I know this has come up a few times in the past but I wanted to bring this
up again.

The lucene-solr ASF git mirror has been behind by about a day. I was
speaking with the infra people and they say that the size of the repo needs
more and more ram. Forcing a sync causes a fork-bomb:

Can't fork: Cannot allocate memory at /usr/share/perl5/Git.pm line 1517.

They tried a few things but it's almost certain that it needs even more
RAM, which still is a band-aid as they'd soon need even more RAM. Also,
adding RAM involves downtime for git.a.o which needs to be planned. As a
stop gap arrangement attached a volume to the instance and are using it as
swap to work around the adding RAM requires restart issue.

FAQ: How would the memory requirement change if we moved to git instead of
mirroring?
Answer: svn - git mirroring is a weird process and has quite the memory
leak. Using git directly is much cleaner.

I personally think git does make things easier to manage when you're
working on multiple overlapping things and so we should re-evaluate moving
to it. I would have been fine had the mirroring worked, as all I want is a
way to be able to work on multiple (local) branches without having to
create and maintain directories like: lucene-solr-trunk1,
lucene-solr-trunk2, or SOLR-, etc.

Opinions?


-- 
Anshum Gupta

Re: Moving to git?

[Ramkumar R. Aiyengar]

Personally, clone for me is 'rare', I did it once years back, and have
never done it since. log, diff and others I do on a daily basis. Same with
svn as well actually, you checkout just once usually..

I think the previous discussion had the agreement that this issue should
focus on committers rather than contributors. And committers by definition
aren't getting started with Solr. If you want to make things more flexible
and faster for contributors, sure, github mirror provides an svn facade
which allows you to check out a subversion wc from it's repos for
read/write (write's not that useful for us though since github is not the
primary repo).
On 31 May 2015 14:25, Robert Muir rcm...@gmail.com wrote:

 You guys totally miss the point on clone.

 The thing is that svn checkout gives you enough, to do what you need
 to do. And yes it does network lookup for more rare things like
 history, but this works just fine in general.

 On the other hand git downloads gigabytes, before you can even get started.

 Something needs to be done about all those jars in the source history,
 I will not let this go.

 On Sun, May 31, 2015 at 9:16 AM, Ramkumar R. Aiyengar
 andyetitmo...@gmail.com wrote:
  +1 for git, great for working on multiple things at once.
 
  Side note: git-svn is also not great btw for the kind of merging we need
 to
  do with every commit, it kind of works but with too many caveats.
 
  On the note that git clone is slow, sure, because it fetches a fair
 amount
  of history which svn doesn't. But to compare just them is unfair, since
  checkout and clone are not identical. If you want to compare times, you
 will
  also have to add up every log, diff, or annotate you do on the tree
 during
  your development (of which I certainly do a lot and I am sure others do
 as
  well), and git will certainly win if you include all those because it
 does
  no network lookup. Clone and checkout are typically one time operations,
 why
  should their speed be a concern in any case?
 
  I know this has come up a few times in the past but I wanted to bring
 this
  up again.
 
  The lucene-solr ASF git mirror has been behind by about a day. I was
  speaking with the infra people and they say that the size of the repo
 needs
  more and more ram. Forcing a sync causes a fork-bomb:
 
  Can't fork: Cannot allocate memory at /usr/share/perl5/Git.pm line 1517.
 
  They tried a few things but it's almost certain that it needs even more
 RAM,
  which still is a band-aid as they'd soon need even more RAM. Also, adding
  RAM involves downtime for git.a.o which needs to be planned. As a stop
 gap
  arrangement attached a volume to the instance and are using it as swap to
  work around the adding RAM requires restart issue.
 
  FAQ: How would the memory requirement change if we moved to git instead
 of
  mirroring?
  Answer: svn - git mirroring is a weird process and has quite the memory
  leak. Using git directly is much cleaner.
 
  I personally think git does make things easier to manage when you're
 working
  on multiple overlapping things and so we should re-evaluate moving to
 it. I
  would have been fine had the mirroring worked, as all I want is a way to
 be
  able to work on multiple (local) branches without having to create and
  maintain directories like: lucene-solr-trunk1, lucene-solr-trunk2, or
  SOLR-, etc.
 
  Opinions?
 
 
  --
  Anshum Gupta

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Fwd: Re: [jetty-users] 9.3.0 RC1 introduces dependency on Java 8?

[Ramkumar R. Aiyengar]

FYI.. We had considered moving to Jetty 9.3 in Solr for a few features
(HTTP/2), looks like that would force us to use Java 8.

May be a step closer to us thinking of a cutover ourselves..
-- Forwarded message --
From: Jesse McConnell jesse.mcconn...@gmail.com
Date: 27 May 2015 18:21
Subject: Re: [jetty-users] 9.3.0 RC1 introduces dependency on Java 8?
To: JETTY user mailing list jetty-us...@eclipse.org
Cc:

David,

Yes, this is the intention for Jetty 9.3.x moving forward.  We had
been operating under the assumption we would support Java 7 but after
looking at what is going on in the SSL areas of the jdk with new
features like SNI, HTTP/2 becoming a standard and requiring Java 8,
and things like we decided to pull that trigger now.  This decision
was also somewhat validated in that most all of our professional
support clients had themselves already switched to Java 8 because of
security issues alone making the decision easier for us.  We will
continue releasing Jetty 9.2.x iterations should the need arise to
keep Java 7 jetty users up to speed, but the adoption of Java 8 over 7
has been dramatically different than 7 over 6...much much faster.

We should have a blog out explaining more of these details sometime soon.

cheers,
Jesse
--
jesse mcconnell
jesse.mcconn...@gmail.com


On Wed, May 27, 2015 at 11:43 AM, David Kellum de...@gravitext.com wrote:
 With the latest RC1 I'm receiving the following error if I use java 7
 (OpenJDK 1.7.0_79, java.class.version: 51.0):

 Unsupported major.minor version 52.0

 This doesn't happen with Java 8, nor did it happen using Java 7 with RC0.
Is
 it the intent now to require Java 8 for jetty 9.3.0, upcoming release?

 Thanks,
 David





 ___
 jetty-users mailing list
 jetty-us...@eclipse.org
 To change your delivery options, retrieve your password, or unsubscribe
from
 this list, visit
 https://dev.eclipse.org/mailman/listinfo/jetty-users
___
jetty-users mailing list
jetty-us...@eclipse.org
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

Re: [JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.8.0_45) - Build # 12644 - Failure!

[Ramkumar R. Aiyengar]

Thanks for sharing Steve, that's indeed impressive :)
SSD: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167299
CPU: http://www.newegg.com/Product/Product.aspx?Item=N82E16819117404
M/B: http://www.newegg.com/Product/Product.aspx?Item=N82E16813132518
RAM: http://www.newegg.com/Product/Product.aspx?Item=N82E16820231820

The mem wasn’t listed as supported by the mobo manufacturer, and it isn’t
detected at its full speed (2800MHz), so currently running at 2400
(“overclocked” from detected 2100 I think).  CPU isn’t overclocked from
stock 3GHz, but I got a liquid cooler, thinking I’d experiment (haven’t
much yet).  Even without overclocking the fans spin faster when all the
cores are busy, and it’s quite the little space heater.

I installed Debian 8, but had to fix the installer in a couple places
because it didn’t know about the new NVMe device naming scheme:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785147
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785149

I also upgraded to the 4.0 Linux kernel, since Debian 8 ships with the 3.16
kernel, and 3.19 contains a bunch of NVMe improvements.

And I turned “swappiness down to zero (thanks to Mike: 
http://blog.mikemccandless.com/2011/04/just-say-no-to-swapping.html) after
seeing a bunch of test stalls while running the Lucene monster tests with 4
JVMs (takes about 2 hours, but I can get it down to 90 minutes or so by
splitting the two tests in Test2BSortedDocValues out into their own suites
- I’ll make an issue).

Steve

 On May 27, 2015, at 5:08 AM, Anshum Gupta ans...@anshumgupta.net wrote:

 8-real-core Haswell-E with 64G DDR4 memory and a NVMe 750-series SSD.

 Can run *all* of the Lucene and Solr tests in 10 minutes by running
multiple ant jobs in parallel!

 On Wed, May 27, 2015 at 1:17 AM, Ramkumar R. Aiyengar 
andyetitmo...@gmail.com wrote:
 Curious.. sarowe, what's the spec?

 On 26 May 2015 20:41, Anshum Gupta ans...@anshumgupta.net wrote:
 The last buch of fixes seems to have fixed this. The tests passed on a
Jenkins that had failing tests earlier.
 Thanks Steve Rowe for lending the super-powerful machine that runs the
entire suite in 8 min!

 I've seen about 20 runs on that box and also runs on Policeman Jenkins
with no issues related to this test since the last commit so I've also
back-ported this to 5x as well.

 On Tue, May 26, 2015 at 9:19 AM, Chris Hostetter hossman_luc...@fucit.org
wrote:

 : Right, as I said, we weren't hitting this issue due to our Kerberos
conf.
 : file. i.e. the only thing that was different on our machines as
compared to
 : everyone else and moving that conf file got the tests to fail for me. It
 : now fails fairly consistently without the patch (from SOLR-7468) and has
 : been looking good with the patch.

 that smells like the kind of thing that sould have some assume sanity
 checks built into it.

 Given:
 * the test setups a special env before running the test
 * the test assumes the specific env will exist
 * the user's machine may already have env properties set before running
ant that affect the expected special env

 therefore: before the test does the setup of the special env, it should
 sanity check that the users basic env doesn't have any properties that
 violate the basic situation.

 so, hypothetical example based on what little i understand the
 authentication stuff: if the purpose of a test is to prove that some
 requests work with (or w/o) kerberos authentication, then before doing any
 setup of a mock kerberos env (or before setting up a mock situation
 where no authentication is required), the test should verify that there
 isn't already an existing kerberos env. (or if possible: unset whatever
 env/properties define that env)


 trivial example of a similar situation is the script engine tests --
 TestBadConfigs.testBogusScriptEngine:  the purpose of the test is to
 ensure that a solrconfig.xml file that refers to a script engine (by
 name) which is not installed on the machine will produce an expeted error
 at Solr init.  before doing the Solr init, we have an whitebox assume that
 asks the JVM directly if a script engine with that name already exists)



 -Hoss
 http://www.lucidworks.com/

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




 --
 Anshum Gupta



 --
 Anshum Gupta


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.8.0_45) - Build # 12644 - Failure!

[Ramkumar R. Aiyengar]

Curious.. sarowe, what's the spec?
On 26 May 2015 20:41, Anshum Gupta ans...@anshumgupta.net wrote:

 The last buch of fixes seems to have fixed this. The tests passed on a
 Jenkins that had failing tests earlier.
 Thanks Steve Rowe for lending the super-powerful machine that runs the
 entire suite in 8 min!

 I've seen about 20 runs on that box and also runs on Policeman Jenkins
 with no issues related to this test since the last commit so I've also
 back-ported this to 5x as well.

 On Tue, May 26, 2015 at 9:19 AM, Chris Hostetter hossman_luc...@fucit.org
  wrote:


 : Right, as I said, we weren't hitting this issue due to our Kerberos
 conf.
 : file. i.e. the only thing that was different on our machines as
 compared to
 : everyone else and moving that conf file got the tests to fail for me. It
 : now fails fairly consistently without the patch (from SOLR-7468) and has
 : been looking good with the patch.

 that smells like the kind of thing that sould have some assume sanity
 checks built into it.

 Given:
 * the test setups a special env before running the test
 * the test assumes the specific env will exist
 * the user's machine may already have env properties set before running
 ant that affect the expected special env

 therefore: before the test does the setup of the special env, it should
 sanity check that the users basic env doesn't have any properties that
 violate the basic situation.

 so, hypothetical example based on what little i understand the
 authentication stuff: if the purpose of a test is to prove that some
 requests work with (or w/o) kerberos authentication, then before doing any
 setup of a mock kerberos env (or before setting up a mock situation
 where no authentication is required), the test should verify that there
 isn't already an existing kerberos env. (or if possible: unset whatever
 env/properties define that env)


 trivial example of a similar situation is the script engine tests --
 TestBadConfigs.testBogusScriptEngine:  the purpose of the test is to
 ensure that a solrconfig.xml file that refers to a script engine (by
 name) which is not installed on the machine will produce an expeted error
 at Solr init.  before doing the Solr init, we have an whitebox assume that
 asks the JVM directly if a script engine with that name already exists)



 -Hoss
 http://www.lucidworks.com/

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




 --
 Anshum Gupta

Re: Welcome Timothy Potter to the PMC

[Ramkumar R. Aiyengar]

Congratulations Tim!
On 26 May 2015 16:10, Steve Rowe sar...@gmail.com wrote:

 I'm pleased to announce that Timothy Potter has accepted the PMC’s
 invitation to join.

 Welcome Tim!

 Steve


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.8.0_60-ea-b12) - Build # 12585 - Still Failing!

[Ramkumar R. Aiyengar]

On Thu, May 21, 2015 at 6:40 PM, Yonik Seeley ysee...@gmail.com wrote:

 It's my opinion that it also wastes everyones time to force them to
 have a pristine checkout before committing


I guess there are two kinds of checks which `ant precommit` tries to
combine -- one is checks which influence the quality/correctness of the
code which is checked in, and that's probably the vast majority. The stray
files check is more of a commit helper, whose impact is limited to the
developers working copy, if you violate it and miss files, it should really
trigger other issues like compilation or tests in jenkins. I don't know if
there are others which fall into this category.

One way out may be to remove the stray files check from precommit and
instead provide a different wrapper script or ant target which wraps around
`svn ci`. That could interactively show unversioned files, and ask
questions like do you really want to go ahead and if you say, commit it
anyway.

Also, have we tried pre-commit hooks before? It certainly won't do all of
pre-commit but way we can move some of the svn checks like eol-style etc.
there and those can be mandatory.

Re: [VOTE] 5.1.0 RC2

[Ramkumar R. Aiyengar]

+1

SUCCESS! [1:28:45.828215]

On Fri, Apr 10, 2015 at 2:05 PM, Joel Bernstein joels...@gmail.com wrote:

 +1

 SUCCESS! [0:40:41.211133]

 Joel Bernstein
 http://joelsolr.blogspot.com/

 On Fri, Apr 10, 2015 at 8:22 AM, Dawid Weiss dawid.we...@cs.put.poznan.pl
  wrote:

 +1

 SUCCESS! [0:45:57.758393]

 Dawid

 P.S. I only ran the build, didn't actually look at the content of the
 distribution.


 On Fri, Apr 10, 2015 at 3:47 AM, Steve Rowe sar...@gmail.com wrote:
  +1
 
  SUCCESS! [2:21:25.391406]
 
  Steve
 
  On Apr 9, 2015, at 2:42 PM, Timothy Potter thelabd...@gmail.com
 wrote:
 
  Please vote for the second release candidate for Lucene/Solr 5.1.0
 
  The artifacts can be downloaded from:
 
 https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.1.0-RC2-rev1672403/
 
  You can run the smoke tester directly with this command:
  python3 -u dev-tools/scripts/smokeTestRelease.py
 
 https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.1.0-RC2-rev1672403/
 
  Here's my +1 SUCCESS! [0:43:35.208102]
 
  Cheers,
  Tim
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  For additional commands, e-mail: dev-h...@lucene.apache.org
 
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  For additional commands, e-mail: dev-h...@lucene.apache.org
 

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org





-- 
Not sent from my iPhone or my Blackberry or anyone else's

Two argument Paths.get

[Ramkumar R. Aiyengar]

May be we should forbid calls to the two argument version of Paths.get.
Especially given other wishes as well apart from Hdfs (like to use a
different filesystem for tests), this call, which uses the default
filesystem, seems to be a mistake waiting to happen..
On 9 Apr 2015 07:48, Varun Thacker (JIRA) j...@apache.org wrote:


 [
 https://issues.apache.org/jira/browse/SOLR-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486843#comment-14486843
 ]

 Varun Thacker commented on SOLR-6637:
 -

 Oh that looks like my mistake. Sorry about that.

 We need to change this {{tmpIndex = Paths.get(solrCore.getDataDir(),
 tmpIdxDirName).toString();}} to
 {{Paths.get(solrCore.getDataDir()).resolve(tmpIdxDirName).toString();}}

 There were two more places which need to be corrected. I'll fix them.

  Solr should have a way to restore a core
  
 
  Key: SOLR-6637
  URL: https://issues.apache.org/jira/browse/SOLR-6637
  Project: Solr
   Issue Type: Improvement
 Reporter: Varun Thacker
 Assignee: Varun Thacker
  Fix For: Trunk, 5.2
 
  Attachments: SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch,
 SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch,
 SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch
 
 
  We have a core backup command which backs up the index. We should have a
 restore command too.
  This would restore any named snapshots created by the replication
 handlers backup command.
  While working on this patch right now I realized that during backup we
 only backup the index. Should we backup the conf files also? Any thoughts?
 I could separate Jira for this.



 --
 This message was sent by Atlassian JIRA
 (v6.3.4#6332)

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Two argument Paths.get

[Ramkumar R. Aiyengar]

I meant the *multiple* arg Paths.get.. (its a vararg function)
On 9 Apr 2015 08:09, Ramkumar R. Aiyengar andyetitmo...@gmail.com wrote:

 May be we should forbid calls to the two argument version of Paths.get.
 Especially given other wishes as well apart from Hdfs (like to use a
 different filesystem for tests), this call, which uses the default
 filesystem, seems to be a mistake waiting to happen..
 On 9 Apr 2015 07:48, Varun Thacker (JIRA) j...@apache.org wrote:


 [
 https://issues.apache.org/jira/browse/SOLR-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486843#comment-14486843
 ]

 Varun Thacker commented on SOLR-6637:
 -

 Oh that looks like my mistake. Sorry about that.

 We need to change this {{tmpIndex = Paths.get(solrCore.getDataDir(),
 tmpIdxDirName).toString();}} to
 {{Paths.get(solrCore.getDataDir()).resolve(tmpIdxDirName).toString();}}

 There were two more places which need to be corrected. I'll fix them.

  Solr should have a way to restore a core
  
 
  Key: SOLR-6637
  URL: https://issues.apache.org/jira/browse/SOLR-6637
  Project: Solr
   Issue Type: Improvement
 Reporter: Varun Thacker
 Assignee: Varun Thacker
  Fix For: Trunk, 5.2
 
  Attachments: SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch,
 SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch,
 SOLR-6637.patch, SOLR-6637.patch, SOLR-6637.patch
 
 
  We have a core backup command which backs up the index. We should have
 a restore command too.
  This would restore any named snapshots created by the replication
 handlers backup command.
  While working on this patch right now I realized that during backup we
 only backup the index. Should we backup the conf files also? Any thoughts?
 I could separate Jira for this.



 --
 This message was sent by Atlassian JIRA
 (v6.3.4#6332)

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr cloud-dev scripts

[Ramkumar R. Aiyengar]

In  SOLR-7349, I have a patch for some of the scripts in cloud-dev:
solrcloud-start.sh, solrcloud-start-existing.sh and stop.sh. clean.sh works
without modifications. If no one has known use cases for anything else, I
will go ahead and delete the remaining scripts in the directory as they
don't work currently anyway. If I shouldn't, please speak up now..

On 4 Apr 2015 00:39, Ramkumar R. Aiyengar andyetitmo...@gmail.com wrote:

 I started looking at porting cloud-dev scripts to the new startup scripts
after the discussion at SOLR-7240, but wasn't quite sure of what the
behaviour should be, having never used them myself. Some of the scripts
there have syntax errors, and I am not sure if some of the others are doing
what was intended even on branch_5x where Jetty 8 is still used. I have a
feeling many of them assume that the stock start.jar starts with a single
collection1 core because of how the solr home used to be set up before,
which is no longer true.

 So how do people use these scripts? Which scripts are used, and for what
purpose?

Solr cloud-dev scripts

[Ramkumar R. Aiyengar]

I started looking at porting cloud-dev scripts to the new startup scripts
after the discussion at SOLR-7240, but wasn't quite sure of what the
behaviour should be, having never used them myself. Some of the scripts
there have syntax errors, and I am not sure if some of the others are doing
what was intended even on branch_5x where Jetty 8 is still used. I have a
feeling many of them assume that the stock start.jar starts with a single
collection1 core because of how the solr home used to be set up before,
which is no longer true.

So how do people use these scripts? Which scripts are used, and for what
purpose?

Re: Make Solr's core admin API internal-only/implementation detail?

[Ramkumar R. Aiyengar]

Sounds good to me, except that we have to reconcile some of the objections
in the past to collection API additions, like with
https://issues.apache.org/jira/SOLR-6278

In short, collection API provides you a way to operate on collections.
Operationally you would often also want functionality based off physical
location (e.g. I need to decommission this machine, so boot and delete
everything on it), core admin appeared to be the place for it..
On 28 Mar 2015 18:28, Erick Erickson erickerick...@gmail.com wrote:

 Mark Haase's comments made me look at the core admin API CREATE
 command and... it's a mess. We've bolted so much stuff on the core
 admin API to support SolrCloud that it's _really_ confusing.

 Plus, we want to move toward ZK as the one source of truth, so
 deprecating the external-facing core admin API seems related at least.

 So a couple of straw-man proposals to kick off the discussion:

 1 Deprecate the core admin API (i.e. documenting it for external
 users) and make it internal-only. Fold any functionality we still want
 to support at a user level into the collections API. I mean a core on
 a machine is really just a single-node collection sans Zookeeper,
 right?

 2 Simplify the user-facing bits of the core admin API, especially
 CREATE, to make it less trappy. Maybe just not even support at all the
 older way of pre-creating a directory with a conf etc. in it then
 creating the core and require configsets? We could nuke instanceDir,
 which is a confusing name now anyway.


 3 Change the docs to support a simple use-case. Label the core admin
 API expert (although that's not really something we put in user's
 faces, more a dev thing but you get the idea).
 3a Split the docs around CREATE into a non-cloud section and an
 additional cloud section.
 3b stop documenting the cloud-specific bits here and consider those
 an implementation detail for SolrCloud.
 3c I'm really not all that enthused about considering this just a doc
 issue that tries to deal with both cloud and non-cloud variants.

 4 ???

 Note that I'm not saying _remove_ the Core Admin API, since it's what
 we use to implement the Collections API. Just don't expose it to the
 user through the documentation; make it an implementation detail

 I find Hasse rather abrasive, but that doesn't alter the fact that he
 happens to be right when he says that this stuff is hard to get right
 unless you're very clear what's going on from a historical
 perspective. Trying to support both the historical usage and the new
 SolrCloud usage in the same API is confusing at best.

 I can raise a JIRA if one hasn't been raised, didn't see one on a quick
 glance.

 Erick

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.7.0_76) - Build # 11952 - Failure!

[Ramkumar R. Aiyengar]

Thanks again Shalin.. You would have thought I would learn from doing this
mistake once, and not repeat it for the merge! :)
On 28 Mar 2015 00:13, Shalin Shekhar Mangar shalinman...@gmail.com
wrote:

 I committed a fix.

 On Fri, Mar 27, 2015 at 4:01 PM, Policeman Jenkins Server 
 jenk...@thetaphi.de wrote:

 Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11952/
 Java: 32bit/jdk1.7.0_76 -server -XX:+UseG1GC

 All tests passed

 Build Log:
 [...truncated 53271 lines...]
 BUILD FAILED
 /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:529: The
 following error occurred while executing this line:
 /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:436: The
 following error occurred while executing this line:
 /home/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:105: The
 following error occurred while executing this line:
 /home/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:204: The
 following files are missing svn:eol-style (or binary svn:mime-type):
 *
 ./solr/test-framework/src/java/org/apache/solr/cloud/StoppableSearchThread.java

 Total time: 57 minutes 21 seconds
 Build step 'Invoke Ant' marked build as failure
 [description-setter] Description set: Java: 32bit/jdk1.7.0_76 -server
 -XX:+UseG1GC
 Archiving artifacts
 Recording test results
 Email was triggered for: Failure - Any
 Sending email for trigger: Failure - Any




 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




 --
 Regards,
 Shalin Shekhar Mangar.

Re: Guidelines for backporting to 4x?

[Ramkumar R. Aiyengar]

Thanks guys. To be clear, I don't think that this change is worth a
release. If at all, I was planning to just go on the back of a 4.10.5 if
there was one (I know at least of one thread where it was mentioned
sometime back, but I realise it might not happen..)
On 25 Mar 2015 15:48, Erick Erickson erickerick...@gmail.com wrote:

 Shawn pretty well nailed it. Especially this bit:

 ...unless you are also volunteering to be the release manager.

 There'll be some push-back even if you're willing to be the RM if it's
 a big change, but that's a discussion for we'll have when there's a
 concrete proposal.

 Best,
 Erick

 On Wed, Mar 25, 2015 at 7:34 AM, Shawn Heisey apa...@elyograg.org wrote:
  On 3/25/2015 3:24 AM, Ramkumar R. Aiyengar wrote:
  I know patches are generally for bug fixes, but given that it is the end
  of line branch for 4x, do we accommodate requests for back porting small
  features, or just ask them to patch it themselves?
 
  I would appreciate a sanity check from someone who has a longer history
  with the project, but this is my understanding:
 
  If there is a critical or trivial bugfix, or a very useful usability
  enhancement that can be considered an easy and safe change (a small
  patch that doesn't affect API compatibility at all), then it might make
  sense to backport, but unless you are also volunteering to be the
  release manager, it may be quite a while before users actually see a
  4.10.x release with the change.
 
  As time passes and branch_5x diverges further from the 4.10 branch
  (which is now considered to be in maintenance mode), it will become
  increasingly difficult to backport.  In some ways, this means that 4x
  will be effectively dead as soon as we have a battle-tested 5.x version
  that we are willing to recommend for even a conservative novice user.
 
  For the most part, unless the patch addresses a serious problem that
  affects a lot of users, the user will need to patch the 4.x code
 themselves.
 
  This info should probably be in the wiki, if it is not already.  I
  didn't look.
 
  Thanks,
  Shawn
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  For additional commands, e-mail: dev-h...@lucene.apache.org
 

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40) - Build # 12076 - Still Failing!

[Ramkumar R. Aiyengar]

Thanks Shalin!
On 25 Mar 2015 01:39, Shalin Shekhar Mangar shalinman...@gmail.com
wrote:

 I committed a fix.

 On Tue, Mar 24, 2015 at 6:02 PM, Policeman Jenkins Server 
 jenk...@thetaphi.de wrote:

 Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12076/
 Java: 64bit/jdk1.8.0_40 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

 All tests passed

 Build Log:
 [...truncated 59397 lines...]
 BUILD FAILED
 /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The
 following error occurred while executing this line:
 /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:426: The
 following error occurred while executing this line:
 /home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:105:
 The following error occurred while executing this line:
 /home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:204:
 The following files are missing svn:eol-style (or binary svn:mime-type):
 *
 ./solr/test-framework/src/java/org/apache/solr/cloud/StoppableSearchThread.java

 Total time: 51 minutes 30 seconds
 Build step 'Invoke Ant' marked build as failure
 [description-setter] Description set: Java: 64bit/jdk1.8.0_40
 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
 Archiving artifacts
 Recording test results
 Email was triggered for: Failure - Any
 Sending email for trigger: Failure - Any




 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




 --
 Regards,
 Shalin Shekhar Mangar.

Guidelines for backporting to 4x?

[Ramkumar R. Aiyengar]

I know patches are generally for bug fixes, but given that it is the end of
line branch for 4x, do we accommodate requests for back porting small
features, or just ask them to patch it themselves?
-- Forwarded message --
From: domenkozar g...@git.apache.org
Date: 25 Mar 2015 09:17
Subject: [GitHub] lucene-solr pull request: Customize number of logs and
records to ...
To: dev@lucene.apache.org
Cc:

Github user domenkozar commented on the pull request:

https://github.com/apache/lucene-solr/pull/83#issuecomment-85947638

@andyetitmoves any chance to backport this to 4.x series? For us this
is a bug since for every rolling restart in solrcloud we have to wait
30min+ per node for full replication since our setup is quite indexing
heavy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Changes to the Solr website - what kind of review process is required?

[Ramkumar R. Aiyengar]

On 16 Mar 2015 22:51, Shawn Heisey apa...@elyograg.org wrote:

 I have some changes I'd like to make to the Solr website, but I'm not
 entirely sure what kind of review process that should go through.
 Should I open an issue in Jira, or just discuss it here?

Given that this is version controlled in the lucene-solr repo, sounds like
a Jira is a good way to go. We have had such issues in the past (like the
new website, further tweaks to it etc.)

 Here are the changes I'd like to do:

 http://apaste.info/pc5

 I think part of the reason that we are having so many people start a
 request for support by opening an issue in Jira is that the info about
 the issue tracker is listed *before* the mailing lists on the resources
 page.  My patch moves that whole section so it's after the mailing list
 info.  This may not entirely solve the problem, but I think it will help.

 There are also some changes on the solr-user mailing list info and the
 IRC channel info.

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Why does QueryParserBase locale default to system locale?

[Ramkumar R. Aiyengar]

Happened to notice that the 'default locale' used by QueryParserBase
is `Locale.getDefault()`, on which we call things like `toLowerCase`
etc. Given that we forbid functions like `toLowerCase` with no args,
does anyone know why we don't use the root locale as the default as
opposed to the system locale?

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Could temporary ZK election / connection loss stall indexing due to LIR?

[Ramkumar R. Aiyengar]

Let me raise an issue for this, ideally all of LIR should happen in the
background..

On Sat, Mar 14, 2015 at 12:10 PM, Shalin Shekhar Mangar 
shalinman...@gmail.com wrote:

 Yes, I have observed this during jepsen tests. The current LIR code tries
 ZK operations with retryOnConnectionLoss set to true which makes the
 indexing thread hang during connection loss events.

 On Sat, Mar 14, 2015 at 4:57 AM, Ramkumar R. Aiyengar 
 andyetitmo...@gmail.com wrote:

 I haven't tested this, but something which I observed looking at the LIR
 code. If there's a ZK election or connection loss, and the leader is unable
 to reach a replica, would it stall till the ZK connection is established,
 due to the LIR process? I can't see it happening in the background, may be
 it should?




 --
 Regards,
 Shalin Shekhar Mangar.




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Could temporary ZK election / connection loss stall indexing due to LIR?

[Ramkumar R. Aiyengar]

I haven't tested this, but something which I observed looking at the LIR
code. If there's a ZK election or connection loss, and the leader is unable
to reach a replica, would it stall till the ZK connection is established,
due to the LIR process? I can't see it happening in the background, may be
it should?

Re: svn commit: r1666186 - in /lucene/dev/branches/branch_5x: ./ solr/ solr/core/ solr/core/src/java/org/apache/solr/parser/ solr/core/src/test/org/apache/solr/search/

[Ramkumar R. Aiyengar]

That explains a lot. Thanks Mike!
On 13 Mar 2015 00:46, Michael McCandless luc...@mikemccandless.com
wrote:

 On Thu, Mar 12, 2015 at 5:38 PM, Yonik Seeley ysee...@gmail.com wrote:
  On Thu, Mar 12, 2015 at 8:04 PM, Ramkumar R. Aiyengar
  andyetitmo...@gmail.com wrote:
  This actually brings me to a question I have had for a while. Why do we
  check in auto generated code? Shouldn't the build system run javacc as a
  prereq to compiling instead?
 
  Historically, the compilation wasn't automated (you had to find +
  install JavaCC yourself, run it yourself, etc).
  I don't know the current reasons however.

 Some discussion about this here:
 https://issues.apache.org/jira/browse/LUCENE-4335

 Mike McCandless

 http://blog.mikemccandless.com

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: svn commit: r1666186 - in /lucene/dev/branches/branch_5x: ./ solr/ solr/core/ solr/core/src/java/org/apache/solr/parser/ solr/core/src/test/org/apache/solr/search/

[Ramkumar R. Aiyengar]

This actually brings me to a question I have had for a while. Why do we
check in auto generated code? Shouldn't the build system run javacc as a
prereq to compiling instead?
On 12 Mar 2015 18:08, Alan Woodward a...@flax.co.uk wrote:

 Ah, OK.  Sorry for the noise!

 On 12 Mar 2015, at 15:19, Yonik Seeley wrote:

  On Thu, Mar 12, 2015 at 11:08 AM, Alan Woodward a...@flax.co.uk wrote:
  Hey Yonik,
 
  I think you've inadvertently added a couple of deprecated methods back
 in here?
 
  Hmmm, but CharStream.java is generated by JavaCC...
  When I got a compile error in FastCharStream.java, I simply copied the
  lucene version.
 
  I built it using the following method:
  $ cd solr/core
  $ ant javacc
 
  -Yonik
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  For additional commands, e-mail: dev-h...@lucene.apache.org
 


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [jira] [Comment Edited] (SOLR-6673) MDC based logging of collection, shard etc.

[Ramkumar R. Aiyengar]

Without commenting on if we should do it (personally I am for it, but I am
also sympathetic to people who feel it's excessive to mandate it), if we
are going down this road, might be worth looking at standalone tools like
astyle which do this rather than an IDE. Some advantages:

* You can easily codify the rules needed instead of coming up with a
configuration for each IDE we use. We could still provide that
configuration but there's at least a golden spec to go by than depending on
IDEs as they evolve.

* You can provide a standard command line way to fix the formatting, or
check it as a part of precommit so that we enforce it as new code comes in.
Let's re-open the let's reformat the entire code base topic again.
Actually, I'd be happy to volunteer to do this if

1 we reached consensus on whether or not it's a good thing.
2 we reached consensus on what to reformat. For instance, I can
convince IntelliJ to reformat any directory of files. So it'd be
possible to reformat, say, just the Solr code. Or just the cloud
sub-folder. Or.
3 we reached consensus on when to do any part of this. The last thing
I'd want to do is screw up someones large complex branch under
development, but we could approach this piecemeal. Say open a JIRA
Reformat the solr/core/src/test directory, give people a chance to
object before doing it, etc.
4 I'd be happy to do the process in the Lucene code base too, but
since I'm personally rarely in that code I'd just as happily leave
that out of the discussion, up to the guys who _are_ in that code IMO.

FWIW,
Erick

On Mon, Mar 9, 2015 at 9:50 AM, Mike Drob mad...@cloudera.com wrote:
 For Eclipse you can get it but only automatically on a save. Preferences 
 Java  Editor  Save Actions.
 If you don't like the changes that it makes, I've found that you can undo
 and save again, and it won't re-format.

 On Mon, Mar 9, 2015 at 11:38 AM, Mark Miller markrmil...@gmail.com
wrote:

 Interesting - never seen such a thing with Eclipse - anyone else?

 I just select the new block of code and shift + control + F. Not bad, but
 would love to have that option as well.

  I usually avoid fixing extra formatting in my patches, but for some of
 the super violations (someone just uses a complete different idea of code
 formatting) I'd rather it be fixed than worry about diffs. This type of
 thing proliferates and lately it feels like it's been going down hill.

 - Mark

 On Mon, Mar 9, 2015 at 10:00 AM Erick Erickson erickerick...@gmail.com
 wrote:

 Ishan:

 Don't know which IDE you use, but IntelliJ has an only vcs changed
 code or some such when reformatting that I find _very_ useful. It's a
 bit dangerous because the _last_ thing you want to do is reformat
 entire files, makes it really hard to look at diffs but I find the
 ability to reformat just what's changed great!.

 Best,
 Erick

 On Mon, Mar 9, 2015 at 12:41 AM, Ishan Chattopadhyaya (JIRA)
 j...@apache.org wrote:
 
  [
 
https://issues.apache.org/jira/browse/SOLR-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352639#comment-14352639
  ]
 
  Ishan Chattopadhyaya edited comment on SOLR-6673 at 3/9/15 7:40 AM:
  
 
  Apologies for the inconsistent formatting; I'll keep this in mind :-)
  Thanks for calling it out!
 
  Updated the patch with changes going to SolrLogLayout, adding the MDC
  values in this format:
 
  [core] [collection] [shard] [replica]
 
 
 
  was (Author: ichattopadhyaya):
  Apologies for the inconsistent formatting; I'll keep this in mind :-)
  Thanks for calling it out!
 
  Updated the patch with changes going to SolrLogLayout, adding the MDC
  values in this format:
  [%X{core}] [%X{collection}] [%X{shard}] [%X{replica}]
 
 
  MDC based logging of collection, shard etc.
  ---
 
  Key: SOLR-6673
  URL: https://issues.apache.org/jira/browse/SOLR-6673
  Project: Solr
   Issue Type: Improvement
 Reporter: Ishan Chattopadhyaya
 Assignee: Noble Paul
   Labels: logging
  Attachments: SOLR-6673.patch, SOLR-6673.patch,
  SOLR-6673.patch, log4j.properties, log4j.properties
 
 
  In cloud mode, the many log items don't contain the collection name,
  shard name, core name etc. Debugging becomes specially difficult
when many
  collections/shards are hosted on the same node.
  The proposed solution adds MDC based stamping of collection, shard,
  core to the thread.
  See also: SOLR-5969, SOLR-5277
 
 
 
  --
  This message was sent by Atlassian JIRA
  (v6.3.4#6332)
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  For additional commands, e-mail: dev-h...@lucene.apache.org
 

 -
 To unsubscribe, e-mail: 

Re: svn commit: r1664126 - in /lucene/dev/trunk/solr: core/src/java/org/apache/solr/core/ core/src/java/org/apache/solr/handler/ core/src/test-files/ core/src/test/org/apache/solr/core/ core/src/test/

[Ramkumar R. Aiyengar]

Thanks all, I have now added the changes entry.

On Wed, Mar 4, 2015 at 10:23 PM, Chris Hostetter hossman_luc...@fucit.org
wrote:


 : The change had no functional impact, hence left it alone.
 :
 : But happy to follow whatever is the existing practice. Should I have one
 : for every change?

 anything non trivial should be noted in CHANGES.txt - the Other
 Changes section is good fit for internal refacotrings that don't fix any
 bugs, but also don't add any features.


 :
 : On Wed, Mar 4, 2015 at 8:29 PM, Alan Woodward a...@flax.co.uk wrote:
 :
 :  Hi Ram, I think you missed a CHANGES.txt entry on this one?
 : 
 :  Alan Woodward
 :  www.flax.co.uk
 : 
 : 
 :  On 4 Mar 2015, at 19:45, andyetitmo...@apache.org wrote:
 : 
 :  Author: andyetitmoves
 :  Date: Wed Mar  4 19:45:09 2015
 :  New Revision: 1664126
 : 
 :  URL: http://svn.apache.org/r1664126
 :  Log:
 :  SOLR-6804: Untangle SnapPuller and ReplicationHandler
 : 
 :  This closes #110
 : 
 :  Added:
 : 
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
 :   - copied, changed from r1663969,
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java
 :  Removed:
 : 
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java
 :  Modified:
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 : 
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
 : 
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapShooter.java
 : lucene/dev/trunk/solr/core/src/test-files/log4j.properties
 : 
 : 
 lucene/dev/trunk/solr/core/src/test/org/apache/solr/core/TestArbitraryIndexDir.java
 : 
 : 
 lucene/dev/trunk/solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
 : 
 : 
 lucene/dev/trunk/solr/test-framework/src/java/org/apache/solr/core/MockDirectoryFactory.java
 : 
 :  Modified:
 :  lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 :  URL:
 : 
 http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java?rev=1664126r1=1664125r2=1664126view=diff
 : 
 : 
 ==
 :  ---
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 :  (original)
 :  +++
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 :  Wed Mar  4 19:45:09 2015
 :  @@ -83,9 +83,9 @@ import org.apache.solr.common.util.IOUti
 :  import org.apache.solr.common.util.NamedList;
 :  import org.apache.solr.common.util.SimpleOrderedMap;
 :  import org.apache.solr.core.DirectoryFactory.DirContext;
 :  +import org.apache.solr.handler.IndexFetcher;
 :  import org.apache.solr.handler.ReplicationHandler;
 :  import org.apache.solr.handler.RequestHandlerBase;
 :  -import org.apache.solr.handler.SnapPuller;
 :  import org.apache.solr.handler.admin.ShowFileRequestHandler;
 :  import org.apache.solr.handler.component.DebugComponent;
 :  import org.apache.solr.handler.component.ExpandComponent;
 :  @@ -291,7 +291,7 @@ public final class SolrCore implements S
 :dir = getDirectoryFactory().get(getDataDir(),
 DirContext.META_DATA,
 :  getSolrConfig().indexConfig.lockType);
 :IndexInput input;
 :try {
 :  -input = dir.openInput(SnapPuller.INDEX_PROPERTIES,
 :  IOContext.DEFAULT);
 :  +input = dir.openInput(IndexFetcher.INDEX_PROPERTIES,
 :  IOContext.DEFAULT);
 :} catch (FileNotFoundException | NoSuchFileException e) {
 :  input = null;
 :}
 :  @@ -307,7 +307,7 @@ public final class SolrCore implements S
 :}
 : 
 :  } catch (Exception e) {
 :  -  log.error(Unable to load  + SnapPuller.INDEX_PROPERTIES,
 e);
 :  +  log.error(Unable to load  +
 IndexFetcher.INDEX_PROPERTIES, e);
 :  } finally {
 :IOUtils.closeQuietly(is);
 :  }
 : 
 :  Copied:
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
 :  (from r1663969,
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java)
 :  URL:
 : 
 http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java?p2=lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.javap1=lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.javar1=1663969r2=1664126rev=1664126view=diff
 : 
 : 
 ==
 :  ---
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java
 :  (original)
 :  +++
 : 
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
 :  Wed Mar  4 19:45:09 2015
 :  @@ -67,11 +67,7 @@ import java.util.concurrent.ExecutionExc
 :  import java.util.concurrent.ExecutorService;
 :  import java.util.concurrent.Executors;
 :  import java.util.concurrent.Future;
 :  -import 

Re: svn commit: r1664126 - in /lucene/dev/trunk/solr: core/src/java/org/apache/solr/core/ core/src/java/org/apache/solr/handler/ core/src/test-files/ core/src/test/org/apache/solr/core/ core/src/test/

[Ramkumar R. Aiyengar]

The change had no functional impact, hence left it alone.

But happy to follow whatever is the existing practice. Should I have one
for every change?

On Wed, Mar 4, 2015 at 8:29 PM, Alan Woodward a...@flax.co.uk wrote:

 Hi Ram, I think you missed a CHANGES.txt entry on this one?

 Alan Woodward
 www.flax.co.uk


 On 4 Mar 2015, at 19:45, andyetitmo...@apache.org wrote:

 Author: andyetitmoves
 Date: Wed Mar  4 19:45:09 2015
 New Revision: 1664126

 URL: http://svn.apache.org/r1664126
 Log:
 SOLR-6804: Untangle SnapPuller and ReplicationHandler

 This closes #110

 Added:


 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
  - copied, changed from r1663969,
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java
 Removed:

lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java
 Modified:
lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java


 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java


 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapShooter.java
lucene/dev/trunk/solr/core/src/test-files/log4j.properties


 lucene/dev/trunk/solr/core/src/test/org/apache/solr/core/TestArbitraryIndexDir.java


 lucene/dev/trunk/solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java


 lucene/dev/trunk/solr/test-framework/src/java/org/apache/solr/core/MockDirectoryFactory.java

 Modified:
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 URL:
 http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java?rev=1664126r1=1664125r2=1664126view=diff

 ==
 --- lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 (original)
 +++ lucene/dev/trunk/solr/core/src/java/org/apache/solr/core/SolrCore.java
 Wed Mar  4 19:45:09 2015
 @@ -83,9 +83,9 @@ import org.apache.solr.common.util.IOUti
 import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.SimpleOrderedMap;
 import org.apache.solr.core.DirectoryFactory.DirContext;
 +import org.apache.solr.handler.IndexFetcher;
 import org.apache.solr.handler.ReplicationHandler;
 import org.apache.solr.handler.RequestHandlerBase;
 -import org.apache.solr.handler.SnapPuller;
 import org.apache.solr.handler.admin.ShowFileRequestHandler;
 import org.apache.solr.handler.component.DebugComponent;
 import org.apache.solr.handler.component.ExpandComponent;
 @@ -291,7 +291,7 @@ public final class SolrCore implements S
   dir = getDirectoryFactory().get(getDataDir(), DirContext.META_DATA,
 getSolrConfig().indexConfig.lockType);
   IndexInput input;
   try {
 -input = dir.openInput(SnapPuller.INDEX_PROPERTIES,
 IOContext.DEFAULT);
 +input = dir.openInput(IndexFetcher.INDEX_PROPERTIES,
 IOContext.DEFAULT);
   } catch (FileNotFoundException | NoSuchFileException e) {
 input = null;
   }
 @@ -307,7 +307,7 @@ public final class SolrCore implements S
   }

 } catch (Exception e) {
 -  log.error(Unable to load  + SnapPuller.INDEX_PROPERTIES, e);
 +  log.error(Unable to load  + IndexFetcher.INDEX_PROPERTIES, e);
 } finally {
   IOUtils.closeQuietly(is);
 }

 Copied:
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
 (from r1663969,
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java)
 URL:
 http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java?p2=lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.javap1=lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.javar1=1663969r2=1664126rev=1664126view=diff

 ==
 ---
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/SnapPuller.java
 (original)
 +++
 lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
 Wed Mar  4 19:45:09 2015
 @@ -67,11 +67,7 @@ import java.util.concurrent.ExecutionExc
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 -import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
 -import java.util.concurrent.atomic.AtomicBoolean;
 -import java.util.regex.Matcher;
 -import java.util.regex.Pattern;
 import java.util.zip.Adler32;
 import java.util.zip.Checksum;
 import java.util.zip.InflaterInputStream;
 @@ -94,7 +90,6 @@ import org.apache.solr.common.SolrExcept
 import org.apache.solr.common.SolrException.ErrorCode;
 import org.apache.solr.common.params.CommonParams;
 import org.apache.solr.common.params.ModifiableSolrParams;
 -import org.apache.solr.common.util.ExecutorUtil;
 import 

RE: reuseAddress default in Solr jetty.xml

[Ramkumar R. Aiyengar]

I agree, sigkill is typically the last resort..
On 3 Mar 2015 00:49, Reitzel, Charles charles.reit...@tiaa-cref.org
wrote:

  My bad.  Too long away from sockets since cleaning up those shutdown
 handlers.  Your point is well taken, on the server side the risks of
 consuming a stray echo packet are fairly low (but non-zero, if you’ve ever
 spent any quality time with tcpdump/wireshark).



 Still, in a production setting, SIGKILL (aka “kill -9”) should be a last
 resort after more reasonable methods (e.g. SIGINT, SIGTERM, SIGSTOP) have
 failed.



 *From:* Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com]
 *Sent:* Monday, March 02, 2015 7:00 PM
 *To:* dev@lucene.apache.org
 *Subject:* RE: reuseAddress default in Solr jetty.xml



 No, reuseAddress doesn't allow you to have two processes, old and new,
 listen to the same port. There's no option which allows you to do that.

 Tl;DR This can happen when you have a connection to a server which gets
 killed hard and comes back up immediately

 So here's what happens.

 When a server normally shuts down, it triggers an active close on all open
 TCP connections it has. That sends a three way msg exchange with the remote
 recipient (FIN, FIN+ACK, ACK) at the end of which the socket is closed and
 the kernel puts it in a TIME_WAIT state for a few minutes in the background
 (depends on the OS, maximum tends to be 4 mins). This is needed to allow
 for reordered older packets to reach the machine just in case. Now
 typically if the server restarts within that period and tries to bind again
 to the same port, the kernel is smart enough to not complain that there is
 an existing socket in TIME_WAIT, because it knows the last sequence number
 it used for the final message in the previous process, and since sequence
 numbers are always increasing, it can reject any messages before that
 sequence number as a new process has now taken the port.

 Trouble is with abnormal shutdown. There's no time for a proper goodbye,
 so the kernel marks the socket to respond to remote packets with a rude RST
 (reset). Since there has been no goodbye with the remote end, it also
 doesn't know the last sequence number to delineate if a new process binds
 to the same port. Hence by default it denies binding to the new port for
 the TIME_WAIT period to avoid the off chance a stray packet gets picked up
 by the new process and utterly confuses it. By setting reuseAddress, you
 are essentially waiving off this protection. Note that this possibility of
 confusion is unbelievably miniscule in the first place (both the source and
 destination host:port should be the same and the client port is generally
 randomly allocated). If the port we are talking of is a local port, it's
 almost impossible -- you have bigger problems if a TCP packet is lost or
 delayed within the same machine!

 As to Shawn's point, for Solr's stop port, you essentially need to be
 trying to actively shutdown the server using the stop port, or be within a
 few minutes of such an attempt while the server is killed. Just the server
 being killed without any active connection to it is not going to cause this
 issue.

 Hi Ram,



 It appears the problem is that the old solr/jetty process is actually
 still running when the new solr/jetty process is started.   That’s the
 problem that needs fixing.



 This is not a rare problem in systems with worker threads dedicated to
 different tasks.   These threads need to wake up in response to the
 shutdown signal/command, as well the normal inputs.



 It’s a bug I’ve created and fixed a couple times over the years … :-)I
 wouldn’t know where to start with Solr.  But, as I say, re-using the port
 is a band-aid.  I’ve yet to see a case where it is the best solution.



 best,

 Charlie



 *From:* Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com]
 *Sent:* Saturday, February 28, 2015 8:15 PM
 *To:* dev@lucene.apache.org
 *Subject:* Re: reuseAddress default in Solr jetty.xml



 Hey Charles, see my explanation above on why this is needed. If Solr has
 to be killed, it would generally be immediately restarted. This would
 normally not the case, except when things are potentially misconfigured or
 if there is a bug, but not doing so makes the impact worse..

 In any case, turns out really that reuseAddress is true by default for the
 connectors we use, so that really isn't the issue. The issue more
 specifically is that the stop port doesn't do it, so the actual port by
 itself starts just fine on a restart, but the stop port fails to bind --
 and there's no way currently in Jetty to configure that.

 Based on my question in the jetty mailing list, I have now created an
 issue for them..

 https://bugs.eclipse.org/bugs/show_bug.cgi?id=461133



 On Fri, Feb 27, 2015 at 3:03 PM, Reitzel, Charles 
 charles.reit...@tiaa-cref.org wrote:

 Disclaimer: I’m not a Solr committer.  But, as a developer, I’ve never
 seen a good case for reusing the listening port.   Better to find and fix

RE: reuseAddress default in Solr jetty.xml

[Ramkumar R. Aiyengar]

No, reuseAddress doesn't allow you to have two processes, old and new,
listen to the same port. There's no option which allows you to do that.

Tl;DR This can happen when you have a connection to a server which gets
killed hard and comes back up immediately

So here's what happens.

When a server normally shuts down, it triggers an active close on all open
TCP connections it has. That sends a three way msg exchange with the remote
recipient (FIN, FIN+ACK, ACK) at the end of which the socket is closed and
the kernel puts it in a TIME_WAIT state for a few minutes in the background
(depends on the OS, maximum tends to be 4 mins). This is needed to allow
for reordered older packets to reach the machine just in case. Now
typically if the server restarts within that period and tries to bind again
to the same port, the kernel is smart enough to not complain that there is
an existing socket in TIME_WAIT, because it knows the last sequence number
it used for the final message in the previous process, and since sequence
numbers are always increasing, it can reject any messages before that
sequence number as a new process has now taken the port.

Trouble is with abnormal shutdown. There's no time for a proper goodbye, so
the kernel marks the socket to respond to remote packets with a rude RST
(reset). Since there has been no goodbye with the remote end, it also
doesn't know the last sequence number to delineate if a new process binds
to the same port. Hence by default it denies binding to the new port for
the TIME_WAIT period to avoid the off chance a stray packet gets picked up
by the new process and utterly confuses it. By setting reuseAddress, you
are essentially waiving off this protection. Note that this possibility of
confusion is unbelievably miniscule in the first place (both the source and
destination host:port should be the same and the client port is generally
randomly allocated). If the port we are talking of is a local port, it's
almost impossible -- you have bigger problems if a TCP packet is lost or
delayed within the same machine!

As to Shawn's point, for Solr's stop port, you essentially need to be
trying to actively shutdown the server using the stop port, or be within a
few minutes of such an attempt while the server is killed. Just the server
being killed without any active connection to it is not going to cause this
issue.

Hi Ram,



It appears the problem is that the old solr/jetty process is actually still
running when the new solr/jetty process is started.   That’s the problem
that needs fixing.



This is not a rare problem in systems with worker threads dedicated to
different tasks.   These threads need to wake up in response to the
shutdown signal/command, as well the normal inputs.



It’s a bug I’ve created and fixed a couple times over the years … :-)I
wouldn’t know where to start with Solr.  But, as I say, re-using the port
is a band-aid.  I’ve yet to see a case where it is the best solution.



best,

Charlie



*From:* Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com]
*Sent:* Saturday, February 28, 2015 8:15 PM
*To:* dev@lucene.apache.org
*Subject:* Re: reuseAddress default in Solr jetty.xml



Hey Charles, see my explanation above on why this is needed. If Solr has to
be killed, it would generally be immediately restarted. This would normally
not the case, except when things are potentially misconfigured or if there
is a bug, but not doing so makes the impact worse..

In any case, turns out really that reuseAddress is true by default for the
connectors we use, so that really isn't the issue. The issue more
specifically is that the stop port doesn't do it, so the actual port by
itself starts just fine on a restart, but the stop port fails to bind --
and there's no way currently in Jetty to configure that.

Based on my question in the jetty mailing list, I have now created an issue
for them..

https://bugs.eclipse.org/bugs/show_bug.cgi?id=461133



On Fri, Feb 27, 2015 at 3:03 PM, Reitzel, Charles 
charles.reit...@tiaa-cref.org wrote:

Disclaimer: I’m not a Solr committer.  But, as a developer, I’ve never seen
a good case for reusing the listening port.   Better to find and fix the
root cause on the zombie state (or just slow shutdown, sometimes) and
release the port.



*From:* Mark Miller [mailto:markrmil...@gmail.com]
*Sent:* Thursday, February 26, 2015 5:28 PM
*To:* dev@lucene.apache.org
*Subject:* Re: reuseAddress default in Solr jetty.xml



+1

- Mark



On Thu, Feb 26, 2015 at 1:54 PM Ramkumar R. Aiyengar 
andyetitmo...@gmail.com wrote:

The jetty.xml we currently ship by default doesn't set reuseAddress=true.
If you are having a bad GC day with things going OOM and resulting in Solr
not even being able to shutdown cleanly (or the oom_solr.sh script killing
it), whatever external service management mechanism you have is probably
going to try respawn it and fail with the default config because the ports
will be in TIME_WAIT. I guess there's the usual disclaimer

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Thanks everyone!

I lead the News Search backend team at the London RD office of Bloomberg.
I have been at this job for a little more than 7 years, and joined straight
from my university at IIT Madras, Chennai, India where I did my bachelors
and masters in CS. I started working with Lucene/Solr around two and a half
years back, when we decided to rewrite our entire search and alerting
backend, something used by every Bloomberg subscriber to get near real-time
access to news with sub-second search/alerting latencies. I have mostly
dabbled with Solr's search distribution and cloud code since, and have had
some fun experiences with it when SolrCloud was, well, let's say less
mature.. :)

I consider myself a Linux geek/evangelist, and outside Java, still use
Emacs and describe Lisp as a beautiful language (As for Java, was Eclipse,
and then thanks Alan for showing me IntelliJ! :) )

On Mon, Mar 2, 2015 at 4:39 AM, Shalin Shekhar Mangar 
shalinman...@gmail.com wrote:

 I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
 invitation to become a committer.

 Ramkumar, it's tradition that you introduce yourself with a brief bio.

 Your handle andyetitmoves has already added to the “lucene LDAP group,
 so you now have commit privileges. Please test this by adding yourself to
 the committers section of the Who We Are page on the website: 
 http://lucene.apache.org/whoweare.html (use the ASF CMS bookmarklet at
 the bottom of the page here: https://cms.apache.org/#bookmark - more
 info here http://www.apache.org/dev/cms.html).

 The ASF dev page also has lots of useful links: 
 http://www.apache.org/dev/.

 Congratulations and welcome!

 --
 Regards,
 Shalin Shekhar Mangar.




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: reuseAddress default in Solr jetty.xml

[Ramkumar R. Aiyengar]

Hey Charles, see my explanation above on why this is needed. If Solr has to
be killed, it would generally be immediately restarted. This would normally
not the case, except when things are potentially misconfigured or if there
is a bug, but not doing so makes the impact worse..

In any case, turns out really that reuseAddress is true by default for the
connectors we use, so that really isn't the issue. The issue more
specifically is that the stop port doesn't do it, so the actual port by
itself starts just fine on a restart, but the stop port fails to bind --
and there's no way currently in Jetty to configure that.

Based on my question in the jetty mailing list, I have now created an issue
for them..

https://bugs.eclipse.org/bugs/show_bug.cgi?id=461133


On Fri, Feb 27, 2015 at 3:03 PM, Reitzel, Charles 
charles.reit...@tiaa-cref.org wrote:

  Disclaimer: I’m not a Solr committer.  But, as a developer, I’ve never
 seen a good case for reusing the listening port.   Better to find and fix
 the root cause on the zombie state (or just slow shutdown, sometimes) and
 release the port.



 *From:* Mark Miller [mailto:markrmil...@gmail.com]
 *Sent:* Thursday, February 26, 2015 5:28 PM
 *To:* dev@lucene.apache.org
 *Subject:* Re: reuseAddress default in Solr jetty.xml



 +1

 - Mark



 On Thu, Feb 26, 2015 at 1:54 PM Ramkumar R. Aiyengar 
 andyetitmo...@gmail.com wrote:

 The jetty.xml we currently ship by default doesn't set reuseAddress=true.
 If you are having a bad GC day with things going OOM and resulting in Solr
 not even being able to shutdown cleanly (or the oom_solr.sh script killing
 it), whatever external service management mechanism you have is probably
 going to try respawn it and fail with the default config because the ports
 will be in TIME_WAIT. I guess there's the usual disclaimer with
 reuseAddress causing stray packets to reach the restarted server, but
 sounds like at least the default should be true..

 I can raise a JIRA, but just wanted to check if anyone has any opinions
 either way..




 *
 This e-mail may contain confidential or privileged information.
 If you are not the intended recipient, please notify the sender
 immediately and then delete it.

 TIAA-CREF
 *




-- 
Not sent from my iPhone or my Blackberry or anyone else's

reuseAddress default in Solr jetty.xml

[Ramkumar R. Aiyengar]

The jetty.xml we currently ship by default doesn't set reuseAddress=true.
If you are having a bad GC day with things going OOM and resulting in Solr
not even being able to shutdown cleanly (or the oom_solr.sh script killing
it), whatever external service management mechanism you have is probably
going to try respawn it and fail with the default config because the ports
will be in TIME_WAIT. I guess there's the usual disclaimer with
reuseAddress causing stray packets to reach the restarted server, but
sounds like at least the default should be true..

I can raise a JIRA, but just wanted to check if anyone has any opinions
either way..

Re: kicking github sync?

[Ramkumar R. Aiyengar]

I have seen this happen before and reached out to infra guys. Invariably it
seems to fix by itself at some point, but till then, git svn rebase ends up
taking all the memory that's available and dies, whatever they try. There
are some known issues with git-svn memory usage on commits to large files,
but it has choked in the past on seemingly innocent commits as well. I was
never able to reproduce it myself though..
On 10 Feb 2015 17:16, Ryan McKinley ryan...@gmail.com wrote:

 I just posted:
 https://issues.apache.org/jira/browse/INFRA-9155



 On Mon, Feb 9, 2015 at 6:48 PM, Mark Miller markrmil...@gmail.com wrote:

 Anyone filed an infra ticket? It should be synced close to right away
 with our git mirror and usually no more than a couple hours on github.
 Neither are updating properly.

 Mark

 On Mon, Feb 9, 2015 at 9:42 PM Ryan McKinley ryan...@gmail.com wrote:

 It looks like the last github sync was 5 days ago :(
 https://github.com/apache/lucene-solr/commits/trunk

 I know this tends to lag the apache mirror, but 5 days is more than usual

 Any idea what (if anything) we can do to kick it?

 Thanks
 ryan

Re: Solr 5 release

[Ramkumar R. Aiyengar]

It would be good to have a link to changes.html (like
https://lucene.apache.org/core/4_10_3/changes/Changes.html) (had suggested
this in previous releases)
On 23 Jan 2015 20:11, Anshum Gupta ans...@anshumgupta.net wrote:

 I’ve made drafts for the Lucene and Solr release notes - please feel free
 to edit or suggest edits:

 Lucene:
 https://wiki.apache.org/lucene-java/ReleaseNote50

 Solr:
 http://wiki.apache.org/solr/ReleaseNote50

 I'm still working on moving things around on the Solr notes.


 On Thu, Jan 22, 2015 at 9:39 AM, Anshum Gupta ans...@anshumgupta.net
 wrote:

 I plan on cutting the first RC later tonight if there are no more
 blockers.

 --
 Anshum Gupta
 http://about.me/anshumgupta




 --
 Anshum Gupta
 http://about.me/anshumgupta

Request for committer attention

[Ramkumar R. Aiyengar]

Would be great if someone is able to pick up the patch for SOLR-6902 soon,
it refactors Solr's distributed tests for easier testing.

The refactor however touches a hundred odd files, and attracts a lot of
conflicts with every passing day. Have just resolved conflicts in 30 odd
files and re-posted the patch..

Re: [JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2399 - Failure

[Ramkumar R. Aiyengar]

Just curious since I see this come up once in a while. Doesn't Apache SVN
offer tweaking pre-commit hooks? Isn't this better checked there than
failing tests after the fact?
On 30 Dec 2014 08:35, Erik Hatcher erik.hatc...@gmail.com wrote:

 I'll fix.  Geez!  :) sorry.


  On Dec 30, 2014, at 00:59, Apache Jenkins Server 
 jenk...@builds.apache.org wrote:
 
  Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2399/
 
  All tests passed
 
  Build Log:
  [...truncated 59402 lines...]
  BUILD FAILED
 
 /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:529:
 The following error occurred while executing this line:
 
 /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:436:
 The following error occurred while executing this line:
 
 /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:105:
 The following error occurred while executing this line:
 
 /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:204:
 The following files are missing svn:eol-style (or binary svn:mime-type):
  * ./solr/bin/post
 
  Total time: 96 minutes 28 seconds
  Build step 'Invoke Ant' marked build as failure
  Archiving artifacts
  Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2398
  Archived 1 artifacts
  Archive block size is 32768
  Received 0 blocks and 464 bytes
  Compression is 0.0%
  Took 0.22 sec
  Recording test results
  Email was triggered for: Failure
  Sending email for trigger: Failure
 
 
 
  -
  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  For additional commands, e-mail: dev-h...@lucene.apache.org

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

RamUsageTester

[Ramkumar R. Aiyengar]

Is RamUsageTester supposed to work for cases where direct memory is
allocated (like with ByteBuffer's)?

I had a made a change to use direct ByteBuffer's in an Accountable object
and adjusted ramBytesUsed to reflect the memory used, but that seems to
upset tests using RamUsageTester to compare results..

Re: IntelliJ build

[Ramkumar R. Aiyengar]

An another thing I found out after a bit of digging is that the language
compliance level setting in the Project SDK had to be set at 8, else it
fails compilation as we are already making use of the default methods
feature.
On 25 Nov 2014 03:38, david.w.smi...@gmail.com david.w.smi...@gmail.com
wrote:

 On trunk I cleaned and re-created my IntelliJ based build (ant clean-idea,
 idea).  IntelliJ didn’t get the memo about Java 8 so I changed that
 (locally).  Then I found that the Solr velocity contrib couldn’t resolve a
 ResourceLoader class in analysis-common.  So I simply checked the “export”
 checkbox on analysis-comon from the Solr-core module, and Solr-core is a
 dependency of velocity, and this it can resolve it.  Export is synonymous
 with transitive resolution.  Now it compiles locally.  It seems like an odd
 thing to go wrong.  Java 8 I expected.

 So if any IntelliJ user has run into issues lately, maybe sharing my
 experience will help.  I should commit the changes but I’ll wait for a
 reply.

 I think the “Export” (transitive resolution) feature could allow us to
 simplify some of the dependency management quite a bit within IntelliJ so
 that it may need less maintenance.

 ~ David Smiley
 Freelance Apache Lucene/Solr Search Consultant/Developer
 http://www.linkedin.com/in/davidwsmiley

Re: Issue with new example layout?

[Ramkumar R. Aiyengar]

Sure, will do..
On 9 Nov 2014 16:22, Erick Erickson erickerick...@gmail.com wrote:

 I'm trying to encourage collecting issues/questions/suggestions for
 using the new layout
 here: https://issues.apache.org/jira/browse/SOLR-6703

 Even if it turns out that it's working as expected (and I don't know
 if it _does_ mind you)
 having  a collection of questions like this will be useful for
 documentation

 So could you add this to that JIRA?

 Thanks!
 Erick

 On Sat, Nov 8, 2014 at 4:24 PM, Ramkumar R. Aiyengar
 andyetitmo...@gmail.com wrote:
  Was trying out the example with the new layout for the first time. Had
 heard
  about issues with stray files lying around with this new layout on the
 list,
  so I started with a fresh checkout.
 
  Did:
 
  $ cd solr
  $ ant dist
  ...
  $ bin/solr start
  Waiting to see Solr listening on port 8983bin/solr: line 1037:
  /home/ramk/projects/lucene-solr/solr/server/logs/solr-8983-console.log:
 No
  such file or directory
 
  waits for a minute, then exits with failure..
 
  after a bit of digging, realized that doing a `mkdir server/logs` does
 the
  trick. Should this be done by the script itself with a `mkdir -p
  server/logs`?
 
  (I initially thought that I was still required to do an `ant example`,
 but
  looks like that doesn't create the logs directory as well..)

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Issue with new example layout?

[Ramkumar R. Aiyengar]

Was trying out the example with the new layout for the first time. Had
heard about issues with stray files lying around with this new layout on
the list, so I started with a fresh checkout.

Did:

$ cd solr
$ ant dist
...
$ bin/solr start
Waiting to see Solr listening on port 8983bin/solr: line 1037:
/home/ramk/projects/lucene-solr/solr/server/logs/solr-8983-console.log: No
such file or directory

waits for a minute, then exits with failure..

after a bit of digging, realized that doing a `mkdir server/logs` does the
trick. Should this be done by the script itself with a `mkdir -p
server/logs`?

(I initially thought that I was still required to do an `ant example`, but
looks like that doesn't create the logs directory as well..)

Re: Help with `ant beast`

[Ramkumar R. Aiyengar]

Raised LUCENE-5968 to explicitly fail on top-level modules, let me know if
I have missed any place..

On Sat, Sep 20, 2014 at 9:32 AM, Uwe Schindler u...@thetaphi.de wrote:

 I would prefer to change the parent modules to have an explicit target.
 See my other message.

 -
 Uwe Schindler
 H.-H.-Meier-Allee 63, D-28213 Bremen
 http://www.thetaphi.de
 eMail: u...@thetaphi.de


  -Original Message-
  From: Chris Hostetter [mailto:hossman_luc...@fucit.org]
  Sent: Friday, September 19, 2014 6:56 PM
  To: dev@lucene.apache.org
  Subject: RE: Help with `ant beast`
 
  :
  : This is correct! Maybe we can improve the error message, but this is
 not
  : so easy... What is the best way to detect if a build file is a parent
  : one? Of course we could add a dummy target to all parent build files -
  : ant test has this to delegate to subant builds, but we don’t want to
  : do this here.
 
  can we just add something like this inside the beast target?...
 
 
  fail message=The Beast wonly orks inside of individual modules (where
  'junit.classpath' is defined)
condition
  notisrefrence refid=junit.classpath//not
/condition
  /fail
 
 
 
  :
  : Uwe
  :
  : -
  : Uwe Schindler
  : H.-H.-Meier-Allee 63, D-28213 Bremen
  : http://www.thetaphi.de
  : eMail: u...@thetaphi.de
  :
  :
  :  -Original Message-
  :  From: Steve Rowe [mailto:sar...@gmail.com]
  :  Sent: Friday, September 19, 2014 5:54 PM
  :  To: dev@lucene.apache.org
  :  Subject: Re: Help with `ant beast`
  : 
  :  I think ‘ant beast’ only works in the directory of the module
 containing the
  :  test, not at a higher level.
  : 
  :  On Sep 19, 2014, at 11:45 AM, Ramkumar R. Aiyengar
  :  andyetitmo...@gmail.com wrote:
  : 
  :   I am trying to use `ant beast` on trunk (per the recommendation in
 test-
  :  help) and getting this error:
  :  
  :   ~/lucene-solr/lucene ant beast -Dbeast.iters=10 -Dtests.dups=6
  :   -Dtestcase=TestBytesStore
  :  
  :   -beast:
  : [beaster] Beast round: 1
  :  
  :   BUILD FAILED
  :   ~/lucene-solr/lucene/common-build.xml:1363: The following error
  :  occurred while executing this line:
  :   ~/lucene-solr/lucene/common-build.xml:1358: The following error
  :  occurred while executing this line:
  :   ~/lucene-solr/lucene/common-build.xml:961: Reference
 junit.classpath
  :  not found.
  :  
  :   `ant test` works just fine. Any idea where the problem might be?
  : 
  : 
  :  -
  :  To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For
  additional
  :  commands, e-mail: dev-h...@lucene.apache.org
  :
  :
  : -
  : To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
  : For additional commands, e-mail: dev-h...@lucene.apache.org
  :
  :
 
  -Hoss
  http://www.lucidworks.com/


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Help with `ant beast`

[Ramkumar R. Aiyengar]

I am trying to use `ant beast` on trunk (per the recommendation in
test-help) and getting this error:

~/lucene-solr/lucene ant beast -Dbeast.iters=10 -Dtests.dups=6
-Dtestcase=TestBytesStore

-beast:
  [beaster] Beast round: 1

BUILD FAILED
~/lucene-solr/lucene/common-build.xml:1363: The following error occurred
while executing this line:
~/lucene-solr/lucene/common-build.xml:1358: The following error occurred
while executing this line:
~/lucene-solr/lucene/common-build.xml:961: Reference junit.classpath not
found.

`ant test` works just fine. Any idea where the problem might be?

Re: Help with `ant beast`

[Ramkumar R. Aiyengar]

Aha.. That worked, thanks!

On Fri, Sep 19, 2014 at 4:53 PM, Steve Rowe sar...@gmail.com wrote:

 I think ‘ant beast’ only works in the directory of the module containing
 the test, not at a higher level.

 On Sep 19, 2014, at 11:45 AM, Ramkumar R. Aiyengar 
 andyetitmo...@gmail.com wrote:

  I am trying to use `ant beast` on trunk (per the recommendation in
 test-help) and getting this error:
 
  ~/lucene-solr/lucene ant beast -Dbeast.iters=10 -Dtests.dups=6
 -Dtestcase=TestBytesStore
 
  -beast:
[beaster] Beast round: 1
 
  BUILD FAILED
  ~/lucene-solr/lucene/common-build.xml:1363: The following error occurred
 while executing this line:
  ~/lucene-solr/lucene/common-build.xml:1358: The following error occurred
 while executing this line:
  ~/lucene-solr/lucene/common-build.xml:961: Reference junit.classpath not
 found.
 
  `ant test` works just fine. Any idea where the problem might be?


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Re: Help with `ant beast`

[Ramkumar R. Aiyengar]

That sounds good. Btw, I first tried it at the toplevel, and that came back
with a beast target not found, hence went to lucene subdir. At least that's
guessable, so less important,  would be to add that failure message to the
toplevel as well..
On 19 Sep 2014 18:02, Ryan Ernst r...@iernst.net wrote:


 On Fri, Sep 19, 2014 at 9:55 AM, Chris Hostetter hossman_luc...@fucit.org
  wrote:

 can we just add something like this inside the beast target?...


 fail message=The Beast wonly orks inside of individual modules (where
 'junit.classpath' is defined)
   condition
 notisrefrence refid=junit.classpath//not
   /condition
 /fail


 +1

Request for committer attention

[Ramkumar R. Aiyengar]

Have a few Solr patches which have been around for a while, could some one
please take a look and commit, or let me know if anything else is needed?

SOLR-6459 https://issues.apache.org/jira/browse/SOLR-6459 Consistently
log all Overseer operations, and queue size
SOLR-6454 https://issues.apache.org/jira/browse/SOLR-6454 Suppress
EOFExceptions in SolrDispatchFilter
SOLR-6453 https://issues.apache.org/jira/browse/SOLR-6453 Stop throwing
from OverseerExitThread on Solr exit
SOLR-6370 https://issues.apache.org/jira/browse/SOLR-6370 Allow tests to
report when many watches are requested parallelly on the same data
SOLR-6359 https://issues.apache.org/jira/browse/SOLR-6359 Customize
records, logs kept by UpdateLog, improve test on log expiry

Thanks! Ramkumar.

Re: Odd test failures

[Ramkumar R. Aiyengar]

Just curious, is there any case where you might genuinely need non-final
static members in a test class? I guess it wouldn't catch all cases, but
something in setup/teardown which bans that using introspection might catch
at least the obvious cases..
On 1 Sep 2014 14:01, Dawid Weiss dawid.we...@cs.put.poznan.pl wrote:

 Hi Erick,

 I filed LUCENE-5916 and attached a patch. Check it out and commit it
 in -- I would like you to understand on a concrete example why the
 patched code will work and the previous code didn't. :)

 Dawid


 On Mon, Sep 1, 2014 at 8:24 AM, Dawid Weiss
 dawid.we...@cs.put.poznan.pl wrote:
  Thanks Dawid! So my take-away is that  this is just pilot error on my
 part,
  not something intrinsic to the code.
 
  I don't know enough about the code to say for sure, but to me that
  FaultyIndexInput's count field should be reset before each test
  (shouldn't propagate from test to test, effectively making each test
  rely on the number of tests before it). As for the exception itself,
  I've no idea -- didn't look at the code; I think it may be assuming
  only one iteration.
 
  Dawid
 
 
  Erick
 
 
  On Sun, Aug 31, 2014 at 12:46 PM, Dawid Weiss 
 dawid.we...@cs.put.poznan.pl
  wrote:
 
  It's because the exception is triggered in a static class
  FaultyIndexInput (initialized in a static context
  TestFieldsReader#beforeClass). When you ask for -Dtests.iters, only
  the tests (methods) are duplicated, the static context remains the
  same. So the count variable in FaultyIndexInput is actually
  propagated from test to test and each repetition is not really atomic/
  isolated from others (search for one of recent e-mail to Ryan, it
  contains a deeper information on why and how this works).
 
  You can repeat the failure if you repeat exactly the same seed for
  each repetition (including test methods):
 
  ant test  -Dtestcase=TestFieldsReader
  -Dtests.seed=DFB0B84C4D087DFD:1DE75618D1B7C867 -Dtests.slow=true
  -Dtests.locale=sr_ME -Dtests.timezone=Asia/Kashgar
  -Dtests.file.encoding=UTF-8 -Dtests.iters=10
 
  This yields:
 
  Tests with failures:
- org.apache.lucene.index.TestFieldsReader.testExceptions {#1
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#2
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#3
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#4
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#5
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#6
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#7
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#8
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
- org.apache.lucene.index.TestFieldsReader.testExceptions {#9
  seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]}
 
  Note I included per-method seed in the -Dtests.seed. Also, #0
  iteration *does pass*; the remaining ones fail because of what I said
  above.
 
  Dawid
 
  On Sun, Aug 31, 2014 at 6:20 AM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
   I'm seeing the fairly easily-reproducible error below on trunk.
   Unfortunately it doesn't reproduce with the seed. I'm wondering if
   anyone
   has an inkling what's going on here?
  
   I did manage to notice that I screwed up the command I was
 _intending_
   and
   actually issued the command below, although I have a hard time
 imagining
   that's the problem, unless it's something like running tests.iters on
   the
   full suite makes this happen. No wonder -Dtests.iters=100 didn't
   finish...
   Siii.
  
   ant -Dtestcasae=TestDistributedSearch -Dtests.iters=10 test
  
   Note I spelled 'testcase' as 'testcasae'...
  
  
   Stack trace:
  
  [junit4] Suite: org.apache.lucene.index.TestFieldsReader
  
  [junit4]   2 NOTE: reproduce with: ant test
   -Dtestcase=TestFieldsReader
   -Dtests.method=testExceptions -Dtests.seed=DFB0B84C4D087DFD
   -Dtests.slow=true -Dtests.locale=sr_ME -Dtests.timezone=Asia/Kashgar
   -Dtests.file.encoding=UTF-8
  
  [junit4] ERROR   0.10s J2 | TestFieldsReader.testExceptions {#1
   seed=[DFB0B84C4D087DFD:1DE75618D1B7C867]} 
  
  [junit4] Throwable #1: java.io.IOException: Simulated network
   outage
  
  [junit4] at
  
 __randomizedtesting.SeedInfo.seed([DFB0B84C4D087DFD:1DE75618D1B7C867]:0)
  
  [junit4] at
  
  
 org.apache.lucene.index.TestFieldsReader$FaultyIndexInput.simOutage(TestFieldsReader.java:156)
  
  [junit4] at
  
  
 org.apache.lucene.index.TestFieldsReader$FaultyIndexInput.readInternal(TestFieldsReader.java:161)
  
  [junit4] at
  
  
 org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
  
  

Re: Odd test failures

[Ramkumar R. Aiyengar]

On Mon, Sep 1, 2014 at 8:54 PM, Dawid Weiss dawid.we...@cs.put.poznan.pl
wrote:

 To be honest you shouldn't have any static initializers at all,
 including final fields (unless they're really immutable, simple data
 types). The reason for this is that static initializers (including
 those for final field assignments) are invoked when the class is
 initialized and this may be happening outside of the scope of the test
 runner. @BeforeClass or class rules are the way to properly
 initialize before-test-suite things.


I was trying to see if I hack something to see if this can be checked and
realized there's a StaticFieldsInvariantRule set up in LuceneTestCase which
was checking for memory leaks on exactly such fields. So I am guessing we
are still using them a bit :) (and a quick grep confirms the same..)

Logging levels in Solr code

[Ramkumar R. Aiyengar]

I am in the process of looking at some of the ERROR level log output coming
from Solr to set up alarms, but currently the severity of what ERROR means
is kind of mixed across the code base. I am happy to fix this where I find,
but some guidance on what the various levels mean would be helpful. This is
what I would have expected:


   - ERROR: Shouldn't happen, indicates a bug or misconfiguration. Leads to
   loss of functionality or some operation failing. Any occurrence indicates
   something needs to be fixed.
   - WARN: Recoverable problem, might genuinely happen in rare cases. If it
   happens too often, might indicate an issue or misconfiguration. Bad input
   data could fall into this category, or should it be INFO?
   - INFO: Informational messages which would help in investigation,
   indicates expected behaviour.

Let me know if this is not accurate..

Re: Logging levels in Solr code

[Ramkumar R. Aiyengar]

Thanks Mark.

I am personally in favour of some record of any request sent to a server
being logged by default to help trace activity when something goes wrong
(which is something successful add logging indirectly achieves), but
unfortunately that currently includes internal distrib=false requests which
adds to the spam.

In any case, I will probably first start with ERROR and WAR...



On Mon, Aug 25, 2014 at 9:48 PM, Mark Miller markrmil...@gmail.com wrote:



  On Aug 25, 2014, at 4:44 PM, Ramkumar R. Aiyengar 
 andyetitmo...@gmail.com wrote:
 
  I am in the process of looking at some of the ERROR level log output
 coming from Solr to set up alarms, but currently the severity of what ERROR
 means is kind of mixed across the code base. I am happy to fix this where I
 find, but some guidance on what the various levels mean would be helpful.
 This is what I would have expected:
 
• ERROR: Shouldn't happen, indicates a bug or misconfiguration.
 Leads to loss of functionality or some operation failing. Any occurrence
 indicates something needs to be fixed.
• WARN: Recoverable problem, might genuinely happen in rare cases.
 If it happens too often, might indicate an issue or misconfiguration. Bad
 input data could fall into this category, or should it be INFO?
• INFO: Informational messages which would help in investigation,
 indicates expected behaviour.
  Let me know if this is not accurate..
 

 Looks right overall. Which is not to say you won’t find an abuse here or
 there…

 bq. Bad input data could fall into this category,

 +1

 I’ve been using more DEBUG as well. I think INFO should not spam (like our
 default successful add logging does) - it should just be useful always
 logged stuff to help with debugging and monitoring and operations.

 DEBUG can be a bit more spammy and just whatever is useful if developing
 in that area.

 - Mark

 http://about.me/markrmiller
 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org




-- 
Not sent from my iPhone or my Blackberry or anyone else's

Solr connection/socket timeouts

[Ramkumar R. Aiyengar]

Currently HttpShardHandlerFactory and DUH2 set socket and connection
timeouts to unlimited by default. Should there be some finite default
instead? With the current defaults, a machine crash for example will lead
to requests in progress hang and occupy threads till the process is shut
down, which doesn't seem desirable. These are all configurable, we could
set some default (60s for conn, 600s for socket?), let people know and have
them up the limits if they really need it..

Thoughts?

Re: Solr connection/socket timeouts

[Ramkumar R. Aiyengar]

I agree we need conservative defaults. I thought 10 mins is reasonable.
Obviously there will be longer requests, but that's where I would have
thought notice in CHANGES.txt would hopefully be good enough, you are
already looking at expert cases in such cases. If some other number is more
reasonable, that's fine, just that infinity isn't reasonable and will
eventually bite people :-) Will raise a JIRA..
On 5 Aug 2014 15:19, Mark Miller markrmil...@gmail.com wrote:

 Part of the issue is picking the right numbers I think. Some of our calls
 can legit take a *long* time.

 I think timeouts become a little more important with SolrCloud over
 ‘legacy mode’ Solr.

 Anyway, IMO, at some point we should introduce large defaults. It’s been
 on my mind before. A good distributed system requires timeouts for proper
 long term “chugging along”.

 --
 Mark Miller
 about.me/markrmiller

 On August 5, 2014 at 7:46:36 AM, Ramkumar R. Aiyengar (
 andyetitmo...@gmail.com) wrote:

 Currently HttpShardHandlerFactory and DUH2 set socket and connection
 timeouts to unlimited by default. Should there be some finite default
 instead? With the current defaults, a machine crash for example will lead
 to requests in progress hang and occupy threads till the process is shut
 down, which doesn't seem desirable. These are all configurable, we could
 set some default (60s for conn, 600s for socket?), let people know and have
 them up the limits if they really need it..

 Thoughts?

Re: Solr checkIfIAmLeader usage from ZK event thread

[Ramkumar R. Aiyengar]

Opened SOLR-6261
On 19 Jul 2014 20:21, Mark Miller markrmil...@gmail.com wrote:

 Put up a patch a lets take a look.

 Most anywhere that holds up the zk processing thread for any decent amount
 of time is probably something waiting to be fixed.

 --
 Mark Miller
 about.me/markrmiller

 On July 15, 2014 at 10:09:56 AM, Ramkumar R. Aiyengar (
 andyetitmo...@gmail.com) wrote:
  Currently when a replica is watching the current leader's ephemeral node
  and the leader disappears, it runs the leadership check along with its
 two
  way peer sync, ZK update etc. on the ZK event thread where the watch was
  fired.
 
  What this means is that for instances with lots of cores, you would be
  serializing leadership elections and the last in the list could take a
 long
  time to have a replacement elected (during which you will have no
 leader).
 
  I did a quick change to make the checkIfIAmLeader call async, but Solr
  cloud tests being what they are (thanks Shalin for cleaning them up btw
 :)
  ), I wanted to check if I am doing something stupid. If not, I will
 raise a
  JIRA.
 
  One contention could be if you might end up with two elections for the
 same
  shard, but I can't see how that might happen..
 


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Solr checkIfIAmLeader usage from ZK event thread

[Ramkumar R. Aiyengar]

Currently when a replica is watching the current leader's ephemeral node
and the leader disappears, it runs the leadership check along with its two
way peer sync, ZK update etc. on the ZK event thread where the watch was
fired.

What this means is that for instances with lots of cores, you would be
serializing leadership elections and the last in the list could take a long
time to have a replacement elected (during which you will have no leader).

I did a quick change to make the checkIfIAmLeader call async, but Solr
cloud tests being what they are (thanks Shalin for cleaning them up btw :)
), I wanted to check if I am doing something stupid. If not, I will raise a
JIRA.

One contention could be if you might end up with two elections for the same
shard, but I can't see how that might happen..

Re: HttpShardHandlerFactory - should max total connections be configurable?

[Ramkumar R. Aiyengar]

+1. We are probably already breaching this with 256 shards, something we
are still investigating.

On the question of other limits, there's also the limit on how many threads
your OS can support, we have in the past hit 65k threads and errored out
because of 64 shards per machine, not sure as yet if this is configurable
(needless to say a scalability problem nevertheless.. I know there are some
nascent ideas floating about with doing away with the thread per connection
model, but none concrete afaik).

We are currently migrating to 4.6.1 from 4.4, planning to attack this once
that's done. The mess you've got to clear up with the 4.4 SolrCloud after
its screwed makes you think twice before attempting to test it again :-)
On 18 Feb 2014 06:12, Shawn Heisey s...@elyograg.org wrote:

 While working on SOLR-5741, I noticed that HttpShardHandlerFactory did
 not have an option to configure the max total connections.  I *think*
 that this value ultimately gets configured at 1.  Do we need to make
 it configurable?  Does this get used for SolrCloud as well as the old
 distributed search?


 http://wiki.apache.org/solr/SolrConfigXml#Configuration_of_Shard_Handlers_for_Distributed_searches

 The default of 1 is well beyond what mere mortals will require, but
 if there are 1000 shards and the search application manages to crank up
 the simultaneous requests to 11, that won't be enough.

 There are other things that would need changes to handle that scenario,
 like maxThreads and the HTTP header size on the container.

 If it doesn't already exist, we probably need a reference guide section
 that covers what needs to be changed from defaults when you're going for
 extreme scalability.

 Thanks,
 Shawn


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Fwd: [jira] [Created] (SOLR-5620) Race condition while setting ZkStateReader.aliases

[Ramkumar R. Aiyengar]

Would someone have a few minutes to validate this race condition fix?
Thanks!
-- Forwarded message --
From: Ramkumar Aiyengar (JIRA) j...@apache.org
Date: 8 Jan 2014 16:49
Subject: [jira] [Created] (SOLR-5620) Race condition while setting
ZkStateReader.aliases
To: dev@lucene.apache.org
Cc:

Ramkumar Aiyengar created SOLR-5620:
 ---

  Summary: Race condition while setting ZkStateReader.aliases
  Key: SOLR-5620
  URL: https://issues.apache.org/jira/browse/SOLR-5620
  Project: Solr
   Issue Type: Bug
   Components: SolrCloud
 Affects Versions: 4.6
 Reporter: Ramkumar Aiyengar
 Priority: Minor


 Noticed while working on SOLR-5615,
 https://github.com/apache/lucene-solr/pull/15 for a patch.



 --
 This message was sent by Atlassian JIRA
 (v6.1.5#6160)

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [ANNOUNCE] Apache Solr 4.6.1 released.

[Ramkumar R. Aiyengar]

Any time there's a release, I tend to look at the Changes.html file (
https://lucene.apache.org/solr/4_6_1/changes/Changes.html), it's a handy
link to look up the changes as soon as you get the mail, especially on
mobile. It would be nice if our release announcements had the link instead
of the 'please look up CHANGES.txt'..
On 28 Jan 2014 19:13, Robert Muir rcm...@gmail.com wrote:

 January 2014, Apache Solr™ 4.6.1 available The Lucene PMC is pleased to 
 announce the release of Apache Solr 4.6.1Solr is the popular, blazing fast, 
 open source NoSQL search platform from the Apache Lucene project. Its major 
 features include powerful full-text search, hit highlighting, faceted search, 
 dynamic clustering, database integration, rich document (e.g., Word, PDF) 
 handling, and geospatial search. Solr is highly scalable, providing fault 
 tolerant distributed search and indexing, and powers the search and 
 navigation features of many of the world's largest internet sites.Solr 4.6.1 
 is available for immediate download at:
 http://lucene.apache.org/solr/mirrors-solr-latest-redir.html Solr 4.6.1 
 includes 29 bug fixes and one optimization as well as Lucene 4.6.1 and its 
 bug fixes.See the CHANGES.txt file included with the release for a full list 
 of changes and further details.Please report any feedback to the mailing 
 lists (http://lucene.apache.org/solr/discussion.html)Note: The Apache 
 Software Foundation uses an extensive mirroring network for distributing 
 releases. It is possible that the mirror you are using may not have 
 replicated the release yet. If that is the case, please try another mirror. 
 This also goes for Maven access.