Re: [ANNOUNCE] Apache Solr 8.8.1 released

[Timothy Potter]

Awesome! Thank you David and Tobias ;-)

On Sat, Feb 27, 2021 at 12:21 PM David Smiley  wrote:
>
> The corresponding docker image has been released as well:
> https://hub.docker.com/_/solr
> (credit to Tobias Kässmann for helping)
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Feb 23, 2021 at 10:39 AM Timothy Potter 
> wrote:
>
> > The Lucene PMC is pleased to announce the release of Apache Solr 8.8.1.
> >
> >
> > Solr is the popular, blazing fast, open source NoSQL search platform from
> > the Apache Lucene project. Its major features include powerful full-text
> > search, hit highlighting, faceted search, dynamic clustering, database
> > integration, rich document handling, and geospatial search. Solr is highly
> > scalable, providing fault tolerant distributed search and indexing, and
> > powers the search and navigation features of many of the world's largest
> > internet sites.
> >
> >
> > Solr 8.8.1 is available for immediate download at:
> >
> >
> >   <https://lucene.apache.org/solr/downloads.html>
> >
> >
> > ### Solr 8.8.1 Release Highlights:
> >
> >
> > Fix for a SolrJ backwards compatibility issue when upgrading the server to
> > 8.8.0 without upgrading SolrJ to 8.8.0.
> >
> >
> > Please refer to the Upgrade Notes in the Solr Ref Guide for information on
> > upgrading from previous Solr versions:
> >
> >
> >   <https://lucene.apache.org/solr/guide/8_8/solr-upgrade-notes.html>
> >
> >
> > Please read CHANGES.txt for a full list of bugfixes:
> >
> >
> >   <https://lucene.apache.org/solr/8_8_1/changes/Changes.html>
> >
> >
> > Solr 8.8.1 also includes bugfixes in the corresponding Apache Lucene
> > release:
> >
> >
> >   <https://lucene.apache.org/core/8_8_1/changes/Changes.html>
> >
> >
> >
> > Note: The Apache Software Foundation uses an extensive mirroring network
> > for
> >
> > distributing releases. It is possible that the mirror you are using may not
> > have
> >
> > replicated the release yet. If that is the case, please try another mirror.
> >
> > This also applies to Maven access.
> >
> > 
> >

[ANNOUNCE] Apache Solr 8.8.1 released

[Timothy Potter]

The Lucene PMC is pleased to announce the release of Apache Solr 8.8.1.


Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project. Its major features include powerful full-text
search, hit highlighting, faceted search, dynamic clustering, database
integration, rich document handling, and geospatial search. Solr is highly
scalable, providing fault tolerant distributed search and indexing, and
powers the search and navigation features of many of the world's largest
internet sites.


Solr 8.8.1 is available for immediate download at:


  


### Solr 8.8.1 Release Highlights:


Fix for a SolrJ backwards compatibility issue when upgrading the server to
8.8.0 without upgrading SolrJ to 8.8.0.


Please refer to the Upgrade Notes in the Solr Ref Guide for information on
upgrading from previous Solr versions:


  


Please read CHANGES.txt for a full list of bugfixes:


  


Solr 8.8.1 also includes bugfixes in the corresponding Apache Lucene
release:


  



Note: The Apache Software Foundation uses an extensive mirroring network for

distributing releases. It is possible that the mirror you are using may not
have

replicated the release yet. If that is the case, please try another mirror.

This also applies to Maven access.

Re: Is 8.8.x going be stabilized and finalized?

[Timothy Potter]

To add to what Shawn said, RC's are made available to anyone
interested in testing them and that helps us find bugs before release.

RC2 for 8.8.1 is available for testing now, see dev mailing list for location.

Please download it and verify it is stable for your use cases and environment.

Tim

On Tue, Feb 16, 2021 at 9:03 PM Shawn Heisey  wrote:
>
> On 2/16/2021 7:57 PM, Subhajit Das wrote:
> > I am planning to use 8.8 line-up for production use.
> >
> > But recently, a lot of people are complaining on 8.7 and 8.8. Also, there 
> > is a clearly known issue on 8.8 as well.
> >
> > Following trends of earlier versions (5.x, 6.x and 7.x), will 8.8 will also 
> > be finalized?
> > For 5.x, 5.5.x was last version. For 6.x, 6.6.x was last version. For 7.x, 
> > 7.7.x was last version. It would match the pattern, it seems.
> > And 9.x is already planned and under development.
> > And it seems, we require some stability.
>
> All released versions are considered stable.  Sometimes problems are
> uncovered after release.  Sometimes BIG problems.  We try our very best
> to avoid bugs, but achieving that kind of perfection is nearly
> impossible for any software project.
>
> 8.8.0 is the most current release.  The 8.8.1 release is underway, but
> there's no way I can give you a concrete date.  The announcement MIGHT
> come in the next few days, but it's always possible it could get pushed
> back.  At this time, the changelog for 8.8.1 has five bugfixes
> mentioned.  It should be more stable than 8.8.0, but it's impossible for
> me to tell you whether you will have any problems with it.
>
> On the dev list, the project is discussing the start of work on the 9.0
> release, but that work has not yet begun.  Even if it started tomorrow,
> it would be several weeks, maybe even a few months, before 9.0 is
> actually released.  On top of the "normal" headaches involved in any new
> major version release, there are some other things going on that might
> further delay 9.0 and future 8.x versions:
>
> * Solr is being promoted from a subproject of Lucene to it's own
> top-level project at Apache.  This involves a LOT of work.  Much of that
> work is administrative in nature, which is going to occupy us and take
> away from time that we might spend working on the code and new releases.
> * The build system for the master branch, which is currently versioned
> as 9.0.0-SNAPSHOT, was recently switched from Ant+Ivy to Gradle.  It's
> going to take some time to figure out all the fallout from that migration.
> * Some of the devs have been involved in an effort to greatly simplify
> and rewrite how SolrCloud does internal management of a cluster.  The
> intent is much better stability and better performance.  You might have
> seen public messages referring to a "reference implementation."  At this
> time, it is unclear how much of that work will make it into 9.0 and how
> much will be revealed in later releases.  We would like very much to
> include at least the first phase in 9.0 if we can.
>
>  From what I have seen over the last several years as one of the
> developers on this project, it is likely that 8.9 and possibly even 8.10
> and 8.11 will be released before we see 9.0.  Releases are NOT made on a
> specific schedule, so I cannot tell you which versions you will see or
> when they might happen.
>
> I am fully aware that despite typing quite a lot of text here, that I
> provided almost nothing in the way of concrete information that you can
> use.  Sorry about that.
>
> Thanks,
> Shawn

Re: Unable to connect to an 8.8.0 Solr Cloud database via API

[Timothy Potter]

Hi Matthew,

Ok, that's great to hear. Thanks for reporting back!

Cheers,
Tim

On Mon, Feb 8, 2021 at 12:26 PM Flowerday, Matthew J <
matthew.flower...@gb.unisys.com> wrote:

> Hi Tim
>
>
>
> Upgrading to solrJ 8.8.0 fixed the issue.
>
>
>
> Many Thanks for your help!
>
>
>
> Matthew
>
>
>
> *Matthew Flowerday* | Consultant | ULEAF
>
> Unisys | 01908 774830| matthew.flower...@unisys.com
>
> Address Enigma | Wavendon Business Park | Wavendon | Milton Keynes | MK17
> 8LX
>
>
>
> [image: unisys_logo] <http://www.unisys.com/>
>
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all devices.
>
> [image: Grey_LI] <http://www.linkedin.com/company/unisys>  [image:
> Grey_TW] <http://twitter.com/unisyscorp> [image: Grey_YT]
> <http://www.youtube.com/theunisyschannel>[image: Grey_FB]
> <http://www.facebook.com/unisyscorp>[image: Grey_Vimeo]
> <https://vimeo.com/unisys>[image: Grey_UB] <http://blogs.unisys.com/>
>
>
>
> *From:* Timothy Potter 
> *Sent:* 08 February 2021 15:44
> *To:* solr-user@lucene.apache.org
> *Subject:* Re: Unable to connect to an 8.8.0 Solr Cloud database via API
>
>
>
> *EXTERNAL EMAIL - Be cautious of all links and attachments.*
>
> Thanks for the additional details Matthew. I created this JIRA to track
> this problem: https://issues.apache.org/jira/browse/SOLR-15145. Please
> add any additional information to that ticket if needed.
>
>
>
> Are you able to upgrade your SolrJ client JAR to 8.8.0? If not, I
> understand but that would be a quick work-around for this problem if
> possible in your env. Otherwise, we'll have to include a fix into 8.8.1
>
>
>
> Cheers,
>
> Tim
>
>
>
> On Mon, Feb 8, 2021 at 8:27 AM Timothy Potter 
> wrote:
>
> What version of SolrJ is embedded in your uleaf.ear file? There have been
> changes in how we deal with URLs stored in ZK in 8.8 --> SOLR-12182
>
>
>
> On Fri, Feb 5, 2021 at 2:34 AM Flowerday, Matthew J <
> matthew.flower...@gb.unisys.com> wrote:
>
> Hi There
>
>
>
> I have been checking out the latest (8.8.0) SolrCloud database (using
> Zookeeper 3.6.2) against our application which talks to Solr via the Solr
> API (I am not too sure of the details as I am not a java developer
> unfortunately!). The software has Solr 8.7.0/ZooKeeper 3.6.2 libraries
> ‘embedded’.
>
>
>
> When our application sends a query to the SolrCloud 8.8.0 database it
> generates this error:
>
>
>
> 2021-02-05 05:36:42,644 INFO
> [org.apache.solr.common.cloud.ConnectionManager] (default task-7) Client is
> connected to ZooKeeper
>
> 2021-02-05 05:36:42,685 INFO  [org.apache.solr.common.cloud.ZkStateReader]
> (default task-7) Updated live nodes from ZooKeeper... (0) -> (1)
>
> 2021-02-05 05:36:42,752 INFO
> [org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider] (default
> task-7) Cluster at localhost:2185 ready
>
> 2021-02-05 05:36:42,785 INFO
> [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7)
> request was not communication error it seems
>
> 2021-02-05 05:36:42,788 INFO
> [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7)
> Request to collection [holmes] failed due to (0)
> java.lang.NullPointerException, retry=0 maxRetries=5 commError=false
> errorCode=0
>
> 2021-02-05 05:36:42,789 ERROR
> [com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl]
> (default task-7) Solr exception on query 'test':
> org.apache.solr.client.solrj.SolrServerException:
> java.lang.NullPointerException
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:1053)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl.search(SolrConnectionManagerImpl.java:191)
>
>   at
> deployment.uleaf.

Re: Unable to connect to an 8.8.0 Solr Cloud database via API

[Timothy Potter]

Thanks for the additional details Matthew. I created this JIRA to track
this problem: https://issues.apache.org/jira/browse/SOLR-15145. Please add
any additional information to that ticket if needed.

Are you able to upgrade your SolrJ client JAR to 8.8.0? If not, I
understand but that would be a quick work-around for this problem if
possible in your env. Otherwise, we'll have to include a fix into 8.8.1

Cheers,
Tim

On Mon, Feb 8, 2021 at 8:27 AM Timothy Potter  wrote:

> What version of SolrJ is embedded in your uleaf.ear file? There have been
> changes in how we deal with URLs stored in ZK in 8.8 --> SOLR-12182
>
> On Fri, Feb 5, 2021 at 2:34 AM Flowerday, Matthew J <
> matthew.flower...@gb.unisys.com> wrote:
>
>> Hi There
>>
>>
>>
>> I have been checking out the latest (8.8.0) SolrCloud database (using
>> Zookeeper 3.6.2) against our application which talks to Solr via the Solr
>> API (I am not too sure of the details as I am not a java developer
>> unfortunately!). The software has Solr 8.7.0/ZooKeeper 3.6.2 libraries
>> ‘embedded’.
>>
>>
>>
>> When our application sends a query to the SolrCloud 8.8.0 database it
>> generates this error:
>>
>>
>>
>> 2021-02-05 05:36:42,644 INFO
>> [org.apache.solr.common.cloud.ConnectionManager] (default task-7) Client is
>> connected to ZooKeeper
>>
>> 2021-02-05 05:36:42,685 INFO
>> [org.apache.solr.common.cloud.ZkStateReader] (default task-7) Updated live
>> nodes from ZooKeeper... (0) -> (1)
>>
>> 2021-02-05 05:36:42,752 INFO
>> [org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider] (default
>> task-7) Cluster at localhost:2185 ready
>>
>> 2021-02-05 05:36:42,785 INFO
>> [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7)
>> request was not communication error it seems
>>
>> 2021-02-05 05:36:42,788 INFO
>> [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7)
>> Request to collection [holmes] failed due to (0)
>> java.lang.NullPointerException, retry=0 maxRetries=5 commError=false
>> errorCode=0
>>
>> 2021-02-05 05:36:42,789 ERROR
>> [com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl]
>> (default task-7) Solr exception on query 'test':
>> org.apache.solr.client.solrj.SolrServerException:
>> java.lang.NullPointerException
>>
>>   at
>> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:1053)
>>
>>   at
>> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
>>
>>   at
>> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
>>
>>   at
>> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003)
>>
>>   at
>> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018)
>>
>>   at
>> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl.search(SolrConnectionManagerImpl.java:191)
>>
>>   at
>> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.search(SolrSearchServiceImpl.java:223)
>>
>>   at
>> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.retrieveSearchPages(SolrSearchServiceImpl.java:163)
>>
>>   at
>> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.retrieveResults(SolrSearchServiceImpl.java:147)
>>
>>   at
>> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.search(SolrSearchServiceImpl.java:132)
>>
>>   at
>> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.search(SolrSearchServiceImpl.java:118)
>>
>>   at
>> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController.search(SolrController.java:163)
>>
>>   at
>> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController.searchValid(SolrController.java:127)
>>
>>   at
>> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController.search(SolrController.java:108)
>>
>>   at
>> deployment.ule

Re: Unable to connect to an 8.8.0 Solr Cloud database via API

[Timothy Potter]

What version of SolrJ is embedded in your uleaf.ear file? There have been
changes in how we deal with URLs stored in ZK in 8.8 --> SOLR-12182

On Fri, Feb 5, 2021 at 2:34 AM Flowerday, Matthew J <
matthew.flower...@gb.unisys.com> wrote:

> Hi There
>
>
>
> I have been checking out the latest (8.8.0) SolrCloud database (using
> Zookeeper 3.6.2) against our application which talks to Solr via the Solr
> API (I am not too sure of the details as I am not a java developer
> unfortunately!). The software has Solr 8.7.0/ZooKeeper 3.6.2 libraries
> ‘embedded’.
>
>
>
> When our application sends a query to the SolrCloud 8.8.0 database it
> generates this error:
>
>
>
> 2021-02-05 05:36:42,644 INFO
> [org.apache.solr.common.cloud.ConnectionManager] (default task-7) Client is
> connected to ZooKeeper
>
> 2021-02-05 05:36:42,685 INFO  [org.apache.solr.common.cloud.ZkStateReader]
> (default task-7) Updated live nodes from ZooKeeper... (0) -> (1)
>
> 2021-02-05 05:36:42,752 INFO
> [org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider] (default
> task-7) Cluster at localhost:2185 ready
>
> 2021-02-05 05:36:42,785 INFO
> [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7)
> request was not communication error it seems
>
> 2021-02-05 05:36:42,788 INFO
> [org.apache.solr.client.solrj.impl.BaseCloudSolrClient] (default task-7)
> Request to collection [holmes] failed due to (0)
> java.lang.NullPointerException, retry=0 maxRetries=5 commError=false
> errorCode=0
>
> 2021-02-05 05:36:42,789 ERROR
> [com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl]
> (default task-7) Solr exception on query 'test':
> org.apache.solr.client.solrj.SolrServerException:
> java.lang.NullPointerException
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:1053)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003)
>
>   at
> deployment.uleaf.ear//org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrConnectionManagerImpl.search(SolrConnectionManagerImpl.java:191)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.search(SolrSearchServiceImpl.java:223)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.retrieveSearchPages(SolrSearchServiceImpl.java:163)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.retrieveResults(SolrSearchServiceImpl.java:147)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.search(SolrSearchServiceImpl.java:132)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.solr.business.impl.SolrSearchServiceImpl.search(SolrSearchServiceImpl.java:118)
>
>   at
> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController.search(SolrController.java:163)
>
>   at
> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController.searchValid(SolrController.java:127)
>
>   at
> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController.search(SolrController.java:108)
>
>   at
> deployment.uleaf.ear.solr-ui-2.0-SNAPSHOT.war//com.unisys.holmes2.h2ng.solr.presentation.SolrController$$FastClassBySpringCGLIB$$82e1da1b.invoke()
>
>   at
> deployment.uleaf.ear//org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>
>   at
> deployment.uleaf.ear//org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
>
>   at
> deployment.uleaf.ear//org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>
>   at
> deployment.uleaf.ear//org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.persistence.support.AbstractRequestLogger.proceedWithoutDebug(AbstractRequestLogger.java:60)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.persistence.support.AbstractRequestLogger.logAndInvoke(AbstractRequestLogger.java:36)
>
>   at
> deployment.uleaf.ear//com.unisys.holmes2.h2ng.presentation.support.RequestLogger.request(RequestLogger.java:48)
>

Re: Session expired when executing streaming expression, but no long GC pauses ...

[Timothy Potter]

No, not every time, but there was no GC pause on the Solr side (no
gaps in the log, nothing in the gc log) ... in the zk log, I do see
this around the same time:

2017-05-05T13:59:52,362 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:9983:NIOServerCnxn@1007] -
Closed socket connection for client /127.0.0.1:54140 which had
sessionid 0x15bd8bdd3500022
2017-05-05T13:59:52,818 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:9983:NIOServerCnxn@357] - caught
end of stream exception
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to
read additional data from client sessionid 0x15bd8bdd3500023, likely
client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[zookeeper-3.4.6.jar:3.4.6-1569965]
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[zookeeper-3.4.6.jar:3.4.6-1569965]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
2017-05-05T13:59:52,819 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:9983:NIOServerCnxn@1007] -
Closed socket connection for client /127.0.0.1:54200 which had
sessionid 0x15bd8bdd3500023
...

2017-05-05T14:00:00,001 - INFO  [SessionTracker:ZooKeeperServer@347] -
Expiring session 0x15bd8bdd3500023, timeout of 1ms exceeded

On Fri, May 19, 2017 at 9:48 AM, Joel Bernstein <joels...@gmail.com> wrote:
> You get this every time you run the expression?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Fri, May 19, 2017 at 10:44 AM, Timothy Potter <thelabd...@gmail.com>
> wrote:
>
>> I'm executing a streaming expr and get this error:
>>
>> Caused by: org.apache.solr.common.SolrException: Could not load
>> collection from ZK:
>> MovieLens_Ratings_f2e6f8b0_3199_11e7_b8ab_0242ac110002
>> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
>> ZkStateReader.java:1098)
>> at org.apache.solr.common.cloud.ZkStateReader$
>> LazyCollectionRef.get(ZkStateReader.java:638)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.
>> getDocCollection(CloudSolrClient.java:1482)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.
>> requestWithRetryOnStaleState(CloudSolrClient.java:1092)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
>> CloudSolrClient.java:1057)
>> at org.apache.solr.client.solrj.io.stream.FacetStream.open(
>> FacetStream.java:356)
>> ... 38 more
>> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for
>> /collections/MovieLens_Ratings_f2e6f8b0_3199_11e7_
>> b8ab_0242ac110002/state.json
>> at org.apache.zookeeper.KeeperException.create(
>> KeeperException.java:127)
>> at org.apache.zookeeper.KeeperException.create(
>> KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>> at org.apache.solr.common.cloud.SolrZkClient$7.execute(
>> SolrZkClient.java:356)
>> at org.apache.solr.common.cloud.SolrZkClient$7.execute(
>> SolrZkClient.java:353)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
>> ZkCmdExecutor.java:60)
>> at org.apache.solr.common.cloud.SolrZkClient.getData(
>> SolrZkClient.java:353)
>> at org.apache.solr.common.cloud.ZkStateReader.
>> fetchCollectionState(ZkStateReader.java:1110)
>> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
>> ZkStateReader.java:1096)
>> ... 43 more
>>
>> I've scoured the GC logs for solr and there are no long pauses
>> (nothing even over 1 second) ... any ideas why that session could be
>> expired?
>>

Session expired when executing streaming expression, but no long GC pauses ...

[Timothy Potter]

I'm executing a streaming expr and get this error:

Caused by: org.apache.solr.common.SolrException: Could not load
collection from ZK:
MovieLens_Ratings_f2e6f8b0_3199_11e7_b8ab_0242ac110002
at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1098)
at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:638)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1482)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1092)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1057)
at 
org.apache.solr.client.solrj.io.stream.FacetStream.open(FacetStream.java:356)
... 38 more
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for
/collections/MovieLens_Ratings_f2e6f8b0_3199_11e7_b8ab_0242ac110002/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1110)
at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1096)
... 43 more

I've scoured the GC logs for solr and there are no long pauses
(nothing even over 1 second) ... any ideas why that session could be
expired?

Re: Possible regression in Parallel SQL in 6.5.1?

[Timothy Potter]

cool, thanks ... easy enough to fix the SQL statement for now ;-)


On Tue, May 16, 2017 at 6:27 PM, Kevin Risden <compuwizard...@gmail.com> wrote:
> Well didn't take as long as I thought:
> https://issues.apache.org/jira/browse/CALCITE-1306
>
> Once Calcite 1.13 is released we should upgrade and get support for this
> again.
>
> Kevin Risden
>
> On Tue, May 16, 2017 at 7:23 PM, Kevin Risden <compuwizard...@gmail.com>
> wrote:
>
>> Yea this came up on the calcite mailing list. Not sure if aliases in the
>> having clause were going to be added. I'll have to see if I can find that
>> discussion or JIRA.
>>
>> Kevin Risden
>>
>> On May 16, 2017 18:54, "Joel Bernstein" <joels...@gmail.com> wrote:
>>
>>> Yeah, Calcite doesn't support field aliases in the having clause. The
>>> query
>>> should work if you use count(*). We could consider this a regression, but
>>> I
>>> think this will be a won't fix.
>>>
>>> Joel Bernstein
>>> http://joelsolr.blogspot.com/
>>>
>>> On Tue, May 16, 2017 at 12:51 PM, Timothy Potter <thelabd...@gmail.com>
>>> wrote:
>>>
>>> > This SQL used to work pre-calcite:
>>> >
>>> > SELECT movie_id, COUNT(*) as num_ratings, avg(rating) as aggAvg FROM
>>> > ratings GROUP BY movie_id HAVING num_ratings > 100 ORDER BY aggAvg ASC
>>> > LIMIT 10
>>> >
>>> > Now I get:
>>> > Caused by: java.io.IOException: -->
>>> > http://192.168.1.4:8983/solr/ratings_shard2_replica1/:Failed to
>>> > execute sqlQuery 'SELECT movie_id, COUNT(*) as num_ratings,
>>> > avg(rating) as aggAvg FROM ratings GROUP BY movie_id HAVING
>>> > num_ratings > 100 ORDER BY aggAvg ASC LIMIT 10' against JDBC
>>> > connection 'jdbc:calcitesolr:'.
>>> > Error while executing SQL "SELECT movie_id, COUNT(*) as num_ratings,
>>> > avg(rating) as aggAvg FROM ratings GROUP BY movie_id HAVING
>>> > num_ratings > 100 ORDER BY aggAvg ASC LIMIT 10": From line 1, column
>>> > 103 to line 1, column 113: Column 'num_ratings' not found in any table
>>> > at org.apache.solr.client.solrj.io.stream.SolrStream.read(
>>> > SolrStream.java:235)
>>> > at com.lucidworks.spark.query.TupleStreamIterator.fetchNextTupl
>>> e(
>>> > TupleStreamIterator.java:82)
>>> > at com.lucidworks.spark.query.TupleStreamIterator.hasNext(
>>> > TupleStreamIterator.java:47)
>>> > ... 31 more
>>> >
>>>
>>

Possible regression in Parallel SQL in 6.5.1?

[Timothy Potter]

This SQL used to work pre-calcite:

SELECT movie_id, COUNT(*) as num_ratings, avg(rating) as aggAvg FROM
ratings GROUP BY movie_id HAVING num_ratings > 100 ORDER BY aggAvg ASC
LIMIT 10

Now I get:
Caused by: java.io.IOException: -->
http://192.168.1.4:8983/solr/ratings_shard2_replica1/:Failed to
execute sqlQuery 'SELECT movie_id, COUNT(*) as num_ratings,
avg(rating) as aggAvg FROM ratings GROUP BY movie_id HAVING
num_ratings > 100 ORDER BY aggAvg ASC LIMIT 10' against JDBC
connection 'jdbc:calcitesolr:'.
Error while executing SQL "SELECT movie_id, COUNT(*) as num_ratings,
avg(rating) as aggAvg FROM ratings GROUP BY movie_id HAVING
num_ratings > 100 ORDER BY aggAvg ASC LIMIT 10": From line 1, column
103 to line 1, column 113: Column 'num_ratings' not found in any table
at 
org.apache.solr.client.solrj.io.stream.SolrStream.read(SolrStream.java:235)
at 
com.lucidworks.spark.query.TupleStreamIterator.fetchNextTuple(TupleStreamIterator.java:82)
at 
com.lucidworks.spark.query.TupleStreamIterator.hasNext(TupleStreamIterator.java:47)
... 31 more

Re: Example of posting to /stream in SolrJ?

[Timothy Potter]

Sorry I was the one that initiated the cross-post ;-) Thanks for the
pointer, works great!



On Tue, Jul 26, 2016 at 3:32 PM, Joel Bernstein <joels...@gmail.com> wrote:
> I posted this also to another thread, but I'll cross post to this ticket:
>
> Take a look at org.apache.solr.client.solrj.io.sql.
> StatementImpl.constructStream()
>
> This uses a SolrStream to connect to the /sql handler. You can use the same
> approach to send a request to the /stream handler just by changing the
> parameters. Then you can open and read the SolrStream.
>
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jul 26, 2016 at 3:58 PM, Timothy Potter <thelabd...@gmail.com>
> wrote:
>
>> Does anyone have an example of just POST'ing a streaming expression to
>> the /stream handler from SolrJ client code? i.e. I don't want to parse
>> and execute the streaming expression on the client side, rather, I
>> want to post the expression to the server side.
>>
>> Currently, my client code is a big copy and paste of the /stream
>> request handler, but I'd rather not do that. Specifically, I wasn't
>> able to figure out how to parse the tuple
>> stream coming back using SolrJ code if I just post the expression to
>> /stream.
>>
>> Thanks.
>>

Example of posting to /stream in SolrJ?

[Timothy Potter]

Does anyone have an example of just POST'ing a streaming expression to
the /stream handler from SolrJ client code? i.e. I don't want to parse
and execute the streaming expression on the client side, rather, I
want to post the expression to the server side.

Currently, my client code is a big copy and paste of the /stream
request handler, but I'd rather not do that. Specifically, I wasn't
able to figure out how to parse the tuple
stream coming back using SolrJ code if I just post the expression to /stream.

Thanks.

Re: Streaming expression - workers is zero somehow?

[Timothy Potter]

Ok, makes sense now, thanks Joel. We should probably add some earlier
error checking vs. letting the code get all the way into the
HashQParser.

So this raises a separate question that I haven't been able to figure
out, namely, do we have an example of just POST'ing the expression to
the /stream handler from SolrJ client code? i.e. I don't want to parse
and execute the streaming expression on the client side, rather, I
want to post it to the server side. Currently, my client code is a big
copy and paste of the /stream request handler (with 1 obvious omission
;-) Specifically, I wasn't able to figure out how to parse the tuple
stream coming back using SolrJ code if I just post the expression to
/stream.

Tim

On Tue, Jul 26, 2016 at 8:54 AM, Joel Bernstein <joels...@gmail.com> wrote:
> The difference would be if you are compiling and running the expression in
> a java class or sending it to the /stream handler to be compiled.
>
> If you're compiling it and running it locally you could get this error
> because the StreamContext would not have the numWorkers variable set.
>
> The /stream handler always sets the numWorkers variable, so in theory you
> would never see this error if the /stream handler is executing the
> expression.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jul 26, 2016 at 10:44 AM, Timothy Potter <thelabd...@gmail.com>
> wrote:
>
>> it's from a unit test, but not sure why that matters? If I wrap the
>> expression in a parallel expression with explicit workers=1, then it
>> works
>>
>> On Thu, Jul 21, 2016 at 11:13 AM, Joel Bernstein <joels...@gmail.com>
>> wrote:
>> > Are you getting this error from a test case you've setup or from a manual
>> > call to the /stream handler?
>> >
>> > Joel Bernstein
>> > http://joelsolr.blogspot.com/
>> >
>> > On Thu, Jul 21, 2016 at 12:28 PM, Timothy Potter <thelabd...@gmail.com>
>> > wrote:
>> >
>> >> I'm working with 6.1.0 release and I have a single SolrCloud instance
>> >> with 1 shard / 1 replica. Somehow I'm triggering this, which from what
>> >> I can see, means workers == 0, but how? Shouldn't workers default to 1
>> >>
>> >> I should mention that my streaming expression doesn't include any
>> >> workers, i.e. it is simply:
>> >>
>> >> val hashJoinExpr =
>> >>   s"""
>> >>  | hashJoin(
>> >>  |search(${ratingsCollection},
>> >>  |   q="*:*",
>> >>  |   fl="movie_id,user_id,rating",
>> >>  |   sort="movie_id asc",
>> >>  |   qt="/export",
>> >>  |   partitionKeys="movie_id"),
>> >>  |hashed=search(${moviesCollection},
>> >>  |  q="*:*",
>> >>  |  fl="movie_id,title",
>> >>  |  sort="movie_id asc",
>> >>  |  qt="/export",
>> >>  |  partitionKeys="movie_id"),
>> >>  |on="movie_id"
>> >>  |  )
>> >>""".stripMargin
>> >>
>> >>
>> >>
>> >> 2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
>> >> - java.io.IOException: java.lang.RuntimeException:
>> >> java.lang.ArithmeticException: / by zero
>> >> at
>> >>
>> org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
>> >> at
>> >>
>> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
>> >> at
>> >>
>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
>> >> at
>> >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
>> >> at
>> >>
>> org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
>> >> at
>> org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSear

Re: Streaming expression - workers is zero somehow?

[Timothy Potter]

it's from a unit test, but not sure why that matters? If I wrap the
expression in a parallel expression with explicit workers=1, then it
works

On Thu, Jul 21, 2016 at 11:13 AM, Joel Bernstein <joels...@gmail.com> wrote:
> Are you getting this error from a test case you've setup or from a manual
> call to the /stream handler?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Jul 21, 2016 at 12:28 PM, Timothy Potter <thelabd...@gmail.com>
> wrote:
>
>> I'm working with 6.1.0 release and I have a single SolrCloud instance
>> with 1 shard / 1 replica. Somehow I'm triggering this, which from what
>> I can see, means workers == 0, but how? Shouldn't workers default to 1
>>
>> I should mention that my streaming expression doesn't include any
>> workers, i.e. it is simply:
>>
>> val hashJoinExpr =
>>   s"""
>>  | hashJoin(
>>  |search(${ratingsCollection},
>>  |   q="*:*",
>>  |   fl="movie_id,user_id,rating",
>>  |   sort="movie_id asc",
>>  |   qt="/export",
>>  |   partitionKeys="movie_id"),
>>  |hashed=search(${moviesCollection},
>>  |  q="*:*",
>>  |  fl="movie_id,title",
>>  |  sort="movie_id asc",
>>  |  qt="/export",
>>  |  partitionKeys="movie_id"),
>>  |on="movie_id"
>>  |  )
>>""".stripMargin
>>
>>
>>
>> 2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
>> - java.io.IOException: java.lang.RuntimeException:
>> java.lang.ArithmeticException: / by zero
>> at
>> org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
>> at
>> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
>> at
>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
>> at
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
>> at
>> org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
>> at org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1234)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1751)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1627)
>> at
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:643)
>> at
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:529)
>> at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:293)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>> at
>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:10

Streaming expression - workers is zero somehow?

[Timothy Potter]

I'm working with 6.1.0 release and I have a single SolrCloud instance
with 1 shard / 1 replica. Somehow I'm triggering this, which from what
I can see, means workers == 0, but how? Shouldn't workers default to 1

I should mention that my streaming expression doesn't include any
workers, i.e. it is simply:

val hashJoinExpr =
  s"""
 | hashJoin(
 |search(${ratingsCollection},
 |   q="*:*",
 |   fl="movie_id,user_id,rating",
 |   sort="movie_id asc",
 |   qt="/export",
 |   partitionKeys="movie_id"),
 |hashed=search(${moviesCollection},
 |  q="*:*",
 |  fl="movie_id,title",
 |  sort="movie_id asc",
 |  qt="/export",
 |  partitionKeys="movie_id"),
 |on="movie_id"
 |  )
   """.stripMargin



2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
- java.io.IOException: java.lang.RuntimeException:
java.lang.ArithmeticException: / by zero
at 
org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
at 
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
at 
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
at 
org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
at org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
at 
org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
at 
org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
at 
org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1234)
at 
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1751)
at 
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1627)
at 
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:643)
at 
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:529)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:293)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.ArithmeticException: / by zero
at 
org.apache.solr.search.HashQParserPlugin$HashQuery$SegmentPartitioner.run(HashQParserPlugin.java:215)
at 

Re: DateMath parsing change?

[Timothy Potter]

Got an answer from Hossman in another channel ... this syntax was not
officially supported and is no longer valid, i.e. my code must change
;-)

On Mon, Jul 18, 2016 at 8:02 AM, Timothy Potter <thelabd...@gmail.com> wrote:
> I have code that uses the DateMathParser and this used to work in 5.x
> but is no longer accepted in 6.x:
>
> time:[NOW-2DAY TO 2016-07-19Z]
>
> org.apache.solr.common.SolrException: Invalid Date in Date Math
> String:'2016-07-19Z'
> at org.apache.solr.util.DateMathParser.parseMath(DateMathParser.java:241)
>
> This is in a unit test, so I can change it, but seems like regression to me?

DateMath parsing change?

[Timothy Potter]

I have code that uses the DateMathParser and this used to work in 5.x
but is no longer accepted in 6.x:

time:[NOW-2DAY TO 2016-07-19Z]

org.apache.solr.common.SolrException: Invalid Date in Date Math
String:'2016-07-19Z'
at org.apache.solr.util.DateMathParser.parseMath(DateMathParser.java:241)

This is in a unit test, so I can change it, but seems like regression to me?

Re: Streaming expression not hitting all replicas?

[Timothy Potter]

Thanks Joel, that cleared things up nicely ... using 4 workers against
4 shards resulted in 16 queries to the collection. However, not all
replicas were used for all shards, so it's not as balanced as I
thought it would be, but we're dealing with small numbers of shards
and replicas here.

On Mon, May 23, 2016 at 12:58 PM, Joel Bernstein <joels...@gmail.com> wrote:
> Streaming expressions will utilize all replicas of a cluster when the
> number of workers >= the number of replicas.
>
> For example if there are 40 workers and 40 shards and 5 replicas.
>
> For a single parallel request:
>
> Each worker will send 1 query to a random replica in each shard. This is
> 1600 hundreds requests. The 1600 requests will be spread evenly across all
> 200 nodes in the cluster, with each node handling 8 requests. Each request
> will return 1/1600 of the result set.
>
> If you add another row of replicas the 1600 hundred requests will be
> handled by 240 nodes.
>
> -
>
> In streaming expressions you use the parallel function to send requests to
> workers.
>
> In SQL you specify aggregationMode=map_reduce and workers=X. The SQL
> interface only goes into parallel mode for GROUP BY and SELECT DISTINCT
> queries.
>
>
>
>
>
>
>
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Mon, May 23, 2016 at 7:17 PM, Joel Bernstein <joels...@gmail.com> wrote:
>
>> The image is the correct flow. Are you using workers?
>>
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Mon, May 23, 2016 at 7:16 PM, Timothy Potter <thelabd...@gmail.com>
>> wrote:
>>
>>> This image from the wiki kind of gives that impression to me:
>>>
>>>
>>> https://cwiki.apache.org/confluence/download/attachments/61311194/cluster.png?version=1=1447365789000=v2
>>>
>>> On Mon, May 23, 2016 at 11:50 AM, Erick Erickson
>>> <erickerick...@gmail.com> wrote:
>>> > I _think_ this is a distinction between
>>> > serving the query and processing the results. The
>>> > query is the standard Solr processing returning
>>> > results from one replica per shard.
>>> >
>>> > Those results can be partitioned out to N Solr instances
>>> > for sub-processing, where N is  however many worker
>>> > nodes you specified that may or may not be host
>>> > to any replicas of that collection.
>>> >
>>> > At least I think that's what's up, but then again this is
>>> > new to me too.
>>> >
>>> > Which bits of the doc anyway? Sounds like some
>>> > clarification is in order.
>>> >
>>> > Best,
>>> > Erick
>>> >
>>> > On Mon, May 23, 2016 at 9:32 AM, Timothy Potter <thelabd...@gmail.com>
>>> wrote:
>>> >> I've seen docs and diagrams that seem to indicate a streaming
>>> >> expression can utilize all replicas of a shard but I'm seeing only 1
>>> >> replica per shard (I have 2) being queried.
>>> >>
>>> >> All replicas are on the same host for my experimentation, could that
>>> >> be the issue? What are the circumstances where all replicas will be
>>> >> utilized?
>>> >>
>>> >> Or is this a mis-understanding of the docs?
>>>
>>
>>

Re: Streaming expression not hitting all replicas?

[Timothy Potter]

This image from the wiki kind of gives that impression to me:

https://cwiki.apache.org/confluence/download/attachments/61311194/cluster.png?version=1=1447365789000=v2

On Mon, May 23, 2016 at 11:50 AM, Erick Erickson
<erickerick...@gmail.com> wrote:
> I _think_ this is a distinction between
> serving the query and processing the results. The
> query is the standard Solr processing returning
> results from one replica per shard.
>
> Those results can be partitioned out to N Solr instances
> for sub-processing, where N is  however many worker
> nodes you specified that may or may not be host
> to any replicas of that collection.
>
> At least I think that's what's up, but then again this is
> new to me too.
>
> Which bits of the doc anyway? Sounds like some
> clarification is in order.
>
> Best,
> Erick
>
> On Mon, May 23, 2016 at 9:32 AM, Timothy Potter <thelabd...@gmail.com> wrote:
>> I've seen docs and diagrams that seem to indicate a streaming
>> expression can utilize all replicas of a shard but I'm seeing only 1
>> replica per shard (I have 2) being queried.
>>
>> All replicas are on the same host for my experimentation, could that
>> be the issue? What are the circumstances where all replicas will be
>> utilized?
>>
>> Or is this a mis-understanding of the docs?

Streaming expression not hitting all replicas?

[Timothy Potter]

I've seen docs and diagrams that seem to indicate a streaming
expression can utilize all replicas of a shard but I'm seeing only 1
replica per shard (I have 2) being queried.

All replicas are on the same host for my experimentation, could that
be the issue? What are the circumstances where all replicas will be
utilized?

Or is this a mis-understanding of the docs?

Re: Parallel SQL doesn't support >, >=, <, <= syntax?

[Timothy Potter]

Thanks Joel, see: SOLR-9146

On Sat, May 21, 2016 at 3:39 PM, Joel Bernstein <joels...@gmail.com> wrote:
> Also agreed we should throw an exception in this scenario until it's
> implemented.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Sat, May 21, 2016 at 5:34 PM, Joel Bernstein <joels...@gmail.com> wrote:
>
>> Yes, currently only Solr range syntax is supported. The SQL greater, less
>> than syntax isn't yet supported.
>>
>> This isn't explicitly stated in the docs, which would be a good first step.
>>
>> Supporting SQL greater and less then predicates should not be too
>> difficult. Feel free to create a jira ticket for this.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Sat, May 21, 2016 at 10:55 AM, Timothy Potter <thelabd...@gmail.com>
>> wrote:
>>
>>> this gives expected result:
>>>
>>>  SELECT title_s, COUNT(*) as cnt
>>> FROM movielens
>>>  WHERE genre_ss='action' AND rating_i='[4 TO 5]'
>>> GROUP BY title_s
>>> ORDER BY cnt desc
>>>  LIMIT 5
>>>
>>> but using >= 4 doesn't give same results (my ratings are 1-5):
>>>
>>>   SELECT title_s, COUNT(*) as cnt
>>>  FROM movielens
>>>   WHERE genre_ss='action' AND rating_i >= 4
>>> GROUP BY title_s
>>> ORDER BY cnt desc
>>>   LIMIT 5
>>>
>>> on the Solr side, I see queries forumlated as:
>>>
>>> 2016-05-21 14:53:43.096 INFO  (qtp1435804085-1419) [c:movielens
>>> s:shard1 r:core_node1 x:movielens_shard1_replica1] o.a.s.c.S.Request
>>> [movielens_shard1_replica1]  webapp=/solr path=/export
>>>
>>> params={q=((genre_ss:"action")+AND+(rating_i:"4"))=false=title_s=title_s+desc=json=2.2}
>>> hits=2044 status=0 QTime=0
>>>
>>> which is obviously wrong ... known issue or should I open a JIRA?
>>>
>>> In general, rather than crafting an incorrect query that gives the
>>> wrong results, we should throw an exception stating that the syntax is
>>> not supported.
>>>
>>
>>

Parallel SQL and function queries?

[Timothy Potter]

How would I do something like: find all docs using a geofilt, e.g.

SELECT title_s
  FROM movielens
WHERE location_p='{!geofilt d=90 pt=37.773972,-122.431297 sfield=location_p}'

This fails with:

{"result-set":{"docs":[
{"EXCEPTION":"java.util.concurrent.ExecutionException:
java.io.IOException: -->
http://ec2-54-165-55-141.compute-1.amazonaws.com:8984/solr/movielens_shard2_replica1/:Can't
parse point '{!geofilt d=90 pt=37.773972,-122.431297
sfield=location_p}' because: java.lang.NumberFormatException: For
input string: \"{!geofilt d=90
pt=37.773972\"","EOF":true,"RESPONSE_TIME":4}]}}

In general, I wasn't able to find much about using functions with
local params in the SQL syntax?

Parallel SQL doesn't support >, >=, <, <= syntax?

[Timothy Potter]

this gives expected result:

 SELECT title_s, COUNT(*) as cnt
FROM movielens
 WHERE genre_ss='action' AND rating_i='[4 TO 5]'
GROUP BY title_s
ORDER BY cnt desc
 LIMIT 5

but using >= 4 doesn't give same results (my ratings are 1-5):

  SELECT title_s, COUNT(*) as cnt
 FROM movielens
  WHERE genre_ss='action' AND rating_i >= 4
GROUP BY title_s
ORDER BY cnt desc
  LIMIT 5

on the Solr side, I see queries forumlated as:

2016-05-21 14:53:43.096 INFO  (qtp1435804085-1419) [c:movielens
s:shard1 r:core_node1 x:movielens_shard1_replica1] o.a.s.c.S.Request
[movielens_shard1_replica1]  webapp=/solr path=/export
params={q=((genre_ss:"action")+AND+(rating_i:"4"))=false=title_s=title_s+desc=json=2.2}
hits=2044 status=0 QTime=0

which is obviously wrong ... known issue or should I open a JIRA?

In general, rather than crafting an incorrect query that gives the
wrong results, we should throw an exception stating that the syntax is
not supported.

Re: Jetty Vs Tomcat (Performance issue)

[Timothy Potter]

I hope 256MB of Xss is a typo and you really meant 256k right?


On Mon, Nov 16, 2015 at 4:58 AM, Behzad Qureshi
 wrote:
> Hi All,
>
> I am using Tomcat server with solr 4.10.3. I want to shift to Jetty as
> replacement of Tomcat server but I am not getting any good results with
> respect to performance. I have tried solr 4.10.3 on both Jetty 8 and Jetty
> 9 with java 8. Below are configurations I have used.
>
> Can anyone please tell me if I am missing anything?
>
> *Jetty:*
>
> Xms: 5GB
> Xmx: 50GB
> Xss: 256MB
>
>
> *Tomcat:*
>
> Xms: 5GB
> Xmx: 50GB
> Xss: Default
>
>
> *Index Size:*
>
> 1TB (20 cores)
>
>
>
> --
>
> Regards,
>
> Behzad Qureshi

Re: Closing Windows CMD kills Solr

[Timothy Potter]

would launching the java process with javaw help here?

On Thu, Oct 29, 2015 at 4:03 AM, Zheng Lin Edwin Yeo
 wrote:
> Yes, this is the expected behaviour. Once you close the command window,
> Solr will stop running. This has happened to me several times. Just to
> check, which version of Solr are you using?
>
> I have tried NSSM before, and it works for Solr 5.0 and Solr 5.1. However,
> when I move up to Solr 5.3.0, I wasn't able to use the same method to start
> it as a service using NSSM.
>
> Do let me know if you managed to set up NSSM for Solr 5.3.0?
>
> Regards,
> Edwin
>
>
> On 29 October 2015 at 17:22, Charlie Hull  wrote:
>
>> On 29/10/2015 00:12, Steven White wrote:
>>
>>> Hi Folks,
>>>
>>> I don't understand if this is an expected behavior or not.
>>>
>>> On Windows, I start Solr from a command prompt like so:
>>>
>>>  bin\solr start -p 8983 -s C:\MySolrIndex
>>>
>>> Now, once I close the command prompt the Java process that started Solr is
>>> killed.  is this expected?  How do I keep Solr alive when I close the
>>> command prompt?
>>>
>>> This dose not happen on Linux.
>>>
>>> Thank in advanced.
>>>
>>> Steve
>>>
>>> Yes, this is expected behaviour. If you want to prevent it you either
>> need to keep the command window open, or run Solr as a Windows Service
>> (kinda like a Unix daemon, but more annoying). To do the latter I would
>> recommend NSSM - here are some links that will help:
>>
>> http://mikerobbins.co.uk/2014/11/07/install-solr-as-windows-service-for-sitecore-content-search/
>> (from step 2)
>> http://www.norconex.com/how-to-run-solr5-as-a-service-on-windows/
>>
>> HTH
>>
>> Charlie
>>
>>
>>
>> --
>> Charlie Hull
>> Flax - Open Source Enterprise Search
>>
>> tel/fax: +44 (0)8700 118334
>> mobile:  +44 (0)7767 825828
>> web: www.flax.co.uk
>>

Re: `cat /dev/null > solr-8983-console.log` frees host's memory

[Timothy Potter]

You should fix your log4j.properties file to no log to console ...
it's there for the initial getting started experience, but you don't
need to send log messages to 2 places.

On Tue, Oct 20, 2015 at 10:42 AM, Shawn Heisey  wrote:
> On 10/20/2015 9:19 AM, Eric Torti wrote:
>> I had a 52GB solr-8983-console.log on my Solr 5.2.1 Amazon Linux
>> 64-bit box and decided to `cat /dev/null > solr-8983-console.log` to
>> free space.
>>
>> The weird thing is that when I checked Sematext I noticed the OS had
>> freed a lot of memory at the same exact instant I did that.
>
> On that memory graph, the legend doesn't indicate which of the graph
> colors represent each of the four usage types at the top -- they all
> have blue checkboxes, so I can't tell for sure what changed.
>
> If the number that dropped is "cached" (which I think is likely) then
> everything is working exactly as it should.  The OS had simply cached a
> large chunk of the logfile, exactly as it is designed to do, and once
> the file was deleted, it stopped reserving that memory and made it
> available.
>
> https://en.wikipedia.org/wiki/Page_cache
>
> Thanks,
> Shawn
>

Re: FW: Issue while setting Solr on Slider / YARN

[Timothy Potter]

Hi Vijay,

I'm not sure what's wrong here ... have you posted to the Slider
mailing list? Also, which version of Java are you using when
interacting with Slider? I know it had some issues with Java 8 at one
point. Which version of Slider so I can try to reproduce ...

Cheers,
Tim

On Thu, Aug 27, 2015 at 11:05 AM, Vijay Bhoomireddy
vijaya.bhoomire...@whishworks.com wrote:


 Hi Tim,



 For some reason, I was not receiving messages from Solr mailing list, though
 I could post it to the list. Now I got that sorted. For my below query, I
 saw your response on the mailing list. Below is the snippet of your
 response:



 Hi Vijay,



 Verify the ResourceManager URL and try passing the --manager param to

 explicitly set the ResourceManager URL during the create step.



 Cheers,

 Tim





 I verified the Resource Manager URL and its pointing correctly. In my case,
 it's myhdpcluster.com:8032 I even tried passing --manager param to the
 slider create solr command, but without any luck. So not sure where it's
 getting wrong. Can you please help me in understanding if I need to modify
 something else to get this working? I am just wondering whether
 ${AGENT_WORK_ROOT} in step below has any impact? I haven't changed this
 line in the json file. Should this be modified?



 I am able to login to Ambari console and see all the services running fine ,
 including YARN related and ZooKeeper. Also, I could login to Resource
 Manager's web UI as well and its working fine. Can you please let me know
 what / where it could have gone wrong?





 Thanks  Regards

 Vijay



 From: Vijay Bhoomireddy [mailto:vijaya.bhoomire...@whishworks.com]
 Sent: 17 August 2015 11:37
 To: solr-user@lucene.apache.org mailto:solr-user@lucene.apache.org
 Subject: Issue while setting Solr on Slider / YARN



 Hi,



 Any help on this please?



 Thanks  Regards

 Vijay



 From: Vijay Bhoomireddy [mailto:vijaya.bhoomire...@whishworks.com]
 Sent: 14 August 2015 18:03
 To: solr-user@lucene.apache.org mailto:solr-user@lucene.apache.org
 Subject: Issue while setting Solr on Slider / YARN



 Hi,



 We have a requirement of setting up of Solr Cloud to work along with Hadoop.
 Earlier, I could setup a SolrCloud cluster separately alongside the Hadoop
 cluster i.e. it looks like two logical  clusters sitting next to each other,
 both relying on HDFS.



 However, the experiment now I am trying to do is to install SolrCloud on
 YARN using Apache Slider. I am following LucidWorks blog at
 https://github.com/LucidWorks/solr-slider for the same. I already have a
 Hortonworks HDP cluster. When I try to setup Solr on my HDP cluster using
 Slider, I am facing some issues.



 As per the blog, I have performed the below steps:



 1.   I have setup a single node HDP cluster for which the hostname is
 myhdpcluster.com with all the essential services including ZooKeeper and
 Slider running on it.

 2.   Updated the resource manager address and port in slider-client.xml
 present under /usr/hdp/current/slider-client/conf

 property

 nameyarn.resourcemanager.address/name

 value myhdpcluster.com:8032/value

 /property

 3.   Cloned the LucidWorks git and moved it under /home/hdfs/solr-slider

 4.   Downloaded solr latest stable distribution and renamed it as
 solr.tgz and placed it under /home/hdfs/solr-slider/package/files/solr.tgz

 5.   Next ran the following command from within the
 /home/hdfs/solr-slider folder

 zip -r solr-on-yarn.zip metainfo.xml package/

 6.   Next ran the following command as hdfs user

 slider install-package --replacepkg --name solr --package
 /home/hdfs/solr-slider/solr-on-yarn.zip

 7.   Modified the following settings in the
 /home/hdfs/solr-slider/appConfig-default.json file

 java_home: MY_JAVA_HOME_LOCATION

 site.global.app_root: ${AGENT_WORK_ROOT}/app/install/solr-5.2.1,
 (Should this be changed to any other value?)

 site.global.zk_host:  myhdpcluster.com:2181,

 8.   Set yarn.component.instances to 1 in resources-default.json file

 9.   Next ran the following command

 slider create solr --template /home/hdfs/solr-slider/appConfig-default.json
 --resources /home/hdfs/solr-slider/resources-default.json



 During this step, I am seeing an message INFO client.RMProxy - Connecting to
 ResourceManager at myhdpcluster.com/10.0.2.15:8032


 INFO ipc.Client - Retrying connect to server:
 myhdpcluster.com/10.0.2.15:8032. Already tried 0 time(s);



 This message keeps repeating for 50 times and then pauses for a couple of
 seconds and then prints the same message in a loop eternally. Not sure on
 where the problem is.



 Can anyone please help me out to get away from this issue and help me setup
 Solr on Slider/YARN?



 Thanks  Regards

 Vijay


 --
 The contents of this e-mail are confidential and for the exclusive use of
 the intended recipient. If you receive this e-mail in error please delete
 it from your system immediately and notify us either by e-mail or
 telephone. 

Re: Issue while setting Solr on Slider / YARN

[Timothy Potter]

Hi Vijay,

Verify the ResourceManager URL and try passing the --manager param to
explicitly set the ResourceManager URL during the create step.

Cheers,
Tim

On Mon, Aug 17, 2015 at 4:37 AM, Vijay Bhoomireddy
vijaya.bhoomire...@whishworks.com wrote:
 Hi,



 Any help on this please?



 Thanks  Regards

 Vijay



 From: Vijay Bhoomireddy [mailto:vijaya.bhoomire...@whishworks.com]
 Sent: 14 August 2015 18:03
 To: solr-user@lucene.apache.org
 Subject: Issue while setting Solr on Slider / YARN



 Hi,



 We have a requirement of setting up of Solr Cloud to work along with Hadoop.
 Earlier, I could setup a SolrCloud cluster separately alongside the Hadoop
 cluster i.e. it looks like two logical  clusters sitting next to each other,
 both relying on HDFS.



 However, the experiment now I am trying to do is to install SolrCloud on
 YARN using Apache Slider. I am following LucidWorks blog at
 https://github.com/LucidWorks/solr-slider for the same. I already have a
 Hortonworks HDP cluster. When I try to setup Solr on my HDP cluster using
 Slider, I am facing some issues.



 As per the blog, I have performed the below steps:



 1.   I have setup a single node HDP cluster for which the hostname is
 myhdpcluster.com with all the essential services including ZooKeeper and
 Slider running on it.

 2.   Updated the resource manager address and port in slider-client.xml
 present under /var/hdp/current/slider/conf

 property

 nameyarn.resourcemanager.address/name

 value myhdpcluster.com:8032/value

 /property

 3.   Cloned the LucidWorks git and moved it under /user/hdfs/solr-slider

 4.   Downloaded solr latest stable distribution and renamed it as
 solr.tgz and placed it under /user/hdfs/solr-slider/package/files/solr.tgz

 5.   Next ran the following command from within the
 /user/hdfs/solr-slider folder

 zip -r solr-on-yarn.zip metainfo.xml package/

 6.   Next ran the following command as hdfs user

 slider install-package --replacepkg --name solr --package
 /user/hdfs/solr-slider/solr-on-yarn.zip

 7.   Modified the following settings in the
 /user/hdfs/solr-slider/appConfig-default.json file

 java_home: MY_JAVA_HOME_LOCATION

 site.global.app_root: ${AGENT_WORK_ROOT}/app/install/solr-5.2.1,
 (Should this be changed to any other value?)

 site.global.zk_host:  myhdpcluster.com:2181,

 8.   Set yarn.component.instances to 1 in resources-default.json file

 9.   Next ran the following command

 slider create solr --template /user/hdfs/solr-slider/appConfig-default.json
 --resources /user/hdfs/solr-slider/resources-default.json



 During this step, I am seeing an message INFO client.RMProxy - Connecting to
 ResourceManager at myhdpcluster.com/10.0.2.15:8032


 INFO ipc.Client - Retrying connect to server:
 myhdpcluster.com/10.0.2.15:8032. Already tried 0 time(s);



 This message keeps repeating for 50 times and then pauses for a couple of
 seconds and then prints the same message in a loop eternally. Not sure on
 where the problem is.



 Can anyone please help me out to get away from this issue and help me setup
 Solr on Slider/YARN?



 Thanks  Regards

 Vijay


 --
 The contents of this e-mail are confidential and for the exclusive use of
 the intended recipient. If you receive this e-mail in error please delete
 it from your system immediately and notify us either by e-mail or
 telephone. You should not copy, forward or otherwise disclose the content
 of the e-mail. The views expressed in this communication may not
 necessarily be the view held by WHISHWORKS.

Re: Leader election

[Timothy Potter]

Hi Olivier,

Can you look at the collections to see if there are leader initiated
recovery nodes in the ZooKeeper tree? Go into the Solr Admin UI -
Cloud panel - Tree view and drill into one of the collections that's
not recovering /collections/collection/leader_initiated_recovery/

You could try deleting the znodes for one of the shards under that and
see if that shard recovers.

Let me know what shakes out as there still may be a bug in this area
of the recovery logic.


On Wed, Jul 29, 2015 at 1:49 PM, Olivier Damiot
olivier.dam...@gmail.com wrote:
 Hello everybody,

 I use solr 5.2.1 and am having a big problem.
 I have about 1200 collections, 3 shards, replicationfactor = 3,
 MaxShardPerNode=3.
 I have 3 boxes of 64G (32 JVM).
 I have no problems with the creation of collection or indexing, but when I
 lose a node (VMY full or kill) and I restart, all my collections are down.
 I look in the logs I can see problems of leader election, eg:
   - Checking if I (core = test339_shard1_replica1, coreNodeName =
 core_node5) shoulds try and be the leader.
 - Cloud says we are still state leader.

 I feel that all server pass the buck!

 I do not understand this error especially as if I read the mailing list I
 have the impression that this bug is solved long ago.

 what should I do to start my collections properly?

 Is someone could help me ?

 thank you a lot

 Olivier

Re: Possible memory leak? Help!

[Timothy Potter]

What are your cache sizes? Max doc?

Also, what GC settings are you using? 6GB isn't all that much for a
memory-intensive app like Solr, esp. given the number of facet fields
you have. Lastly, are you using docvalues for your facet fields? That
should help reduce the amount of heap needed to compute facets.

On Tue, Jul 14, 2015 at 2:33 PM, Yael Gurevich yae...@gmail.com wrote:
 Hi,

 We're running Solr 4.10.1 on Linux using Tomcat. Distributed environment,
 40 virtual servers with high resources. Concurrent queries that are quite
 complex (may be hundreds of terms), NRT indexing and a few hundreds of
 facet fields which might have many (hundreds of thousands) distinct values.

 We've configured a 6GB JVM heap, and after quite a bit of work, it seems to
 be pretty well configured GC parameter-wise (we're using CMS and ParNew).

 The following problem occurs -
 Once every couple of hours, suddenly start getting
 concurrent-mode-failure on one or more servers, the memory starts
 climbing up further and further and concurrent-mode-failure continues.
 Naturally, during this time, SOLR is unresponsive and the queries are
 timed-out. Eventually it might pass (GC will succeed), after 5-10 minutes.
 Sometimes this phenomenon can occur for a great deal of time, one server
 goes up and then another and so forth.

 Memory dumps point to ConcurrentLRUCache (used in filterCache and
 fieldValueCache). Mathematically speaking, the sizes I see in the dumps do
 not make sense. The configured sizes shouldn't take up more than a few
 hunderds of MBs.

 Any ideas? Anyone seen this kind of problem?

Re: Solr 5.2.1 setup zookeeper ensemble problem

[Timothy Potter]

can you try with double-quotes around the zk connect string?

bin\solr.cmd -e cloud -z localhost:2181,localhost:2182,localhost:2183

On Mon, Jul 6, 2015 at 2:59 AM, Adrian Liew adrian.l...@avanade.com wrote:
 Hi David,

 When I run the command below on a Windows machine using Powershell window:

 .\solr.cmd -e cloud -z localhost:2181,localhost:2182,localhost:2183

 I get the following error:

 Invalid command-line option: localhost:2182

 Somehow it does not recognize comma separated between the localhost. As far 
 as I know if you are trying to run SolrCloud against your ZooKeeper Ensemble, 
 you will need to specifify all three ZK server addresses (according to . 
 https://cwiki.apache.org/confluence/display/solr/Setting+Up+an+External+ZooKeeper+Ensemble)
  However, some blogs say you can just connect to only just one instance for a 
 ZK Ensemble (http://solr.pl/en/2013/03/11/solrcloud-howto-2/). So I am not 
 sure which one is correct now.

 Thoughts on the above?

 Regards,
 Adrian

 -Original Message-
 From: davidphilip cherian [mailto:davidphilipcher...@gmail.com]
 Sent: Monday, July 6, 2015 4:35 PM
 To: solr-user@lucene.apache.org
 Subject: Re: Solr 5.2.1 setup zookeeper ensemble problem

 Hi Adrian,

 What is the error that you are getting?
 In order to  upload configs files, you could use zkcli.sh script that will be 
 shipped with solr and use the upconfig command.

 ./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 \
-cmd upconfig -confname my_new_config -confdir 
 server/solr/configsets/basic_configs/conf

 https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities



 On Mon, Jul 6, 2015 at 1:43 PM, Adrian Liew adrian.l...@avanade.com wrote:

 There seems to be an issue running the following command using
 solr.cmd as
 below:

  - bin\solr.cmd -e cloud -z
 localhost:2181,localhost:2182,localhost:2183

 Anyone can please advise.

 Also, Is there a way to upload a configuration file (containing
 schema.xml and solrconfig.xml) to ZooKeeper easily using solr.cmd in solr 
 5.2.1?

 Best regards,

 Adrian Liew |  Consultant Application Developer Avanade Malaysia Sdn.
 Bhd..| Consulting Services
 (: Direct: +(603) 2382 5668
 È: +6010-2288030

Re: Migrating from Solr 5.1 to Solr 5.2.1

[Timothy Potter]

Hi Edwin,

You'll need to use the bin\solr.cmd to start Solr as it now requires
some additional system properties to be set. Put simply, starting solr
using java -jar start.jar is not supported. Please try bin\solr.cmd
and let us know if you run into any issues. You can set any additional
system properties (-D) you need in the bin\solr.in.cmd script

Cheers,
Tim

On Fri, Jul 3, 2015 at 2:07 AM, Zheng Lin Edwin Yeo
edwinye...@gmail.com wrote:
 Hi,

 I'm trying to migrate from Solr 5.1 to Solr 5.2.1. However, I faced some
 problems when I'm trying to migrate my index over, and when I'm trying to
 link up the external ZooKeeper to Solr.
 I'm using ZooKeeper 3.4.6

 In Solr 5.1, I used this command to start Solr for both Shard1 and Shard2:

 java -D64 -Dsolr.clustering.enabled=true -Xms512M -Xmx4096M
 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapDumps
 -DzkHost=localhost:2181,localhost:2182,localhost:2183 -jar start.jar

 However, I get the following error in Solr 5.2.1:
 *Error: Unable to access jarfile start.jar*

 For ZooKeeper, I've been using the zkCli.bat file
 under server\scripts\cloud-scripts:

 zkcli.bat -zkhost localhost:2181 \ -cmd upconfig -confname collection1
 -confdir
 C:\Users\edwin_000\Desktop\edwin\edm-5.2.1\configuration\collection1\conf

 However, I get the following error in Solr 5.2.1
 *Error: Could not find or load main class org.apache.solr.cloud.ZkCLI*

 Is there any changes to the code structure in Solr 5.2.1 as compared to the
 older versions?


 Regards,
 Edwin

Re: Solr advanced StopFilterFactory

[Timothy Potter]

Seems like you should be able to use the ManagedStopFilterFactory with
a custom StorageIO impl that pulls from your db:

http://lucene.apache.org/solr/5_1_0/solr-core/index.html?org/apache/solr/rest/ManagedResourceStorage.StorageIO.html

On Thu, May 28, 2015 at 7:03 AM, Alessandro Benedetti
benedetti.ale...@gmail.com wrote:
 As Alex initially specified , the custom stop filter factory is the right
 way !
 So is mainly related to the suggester ?
 Anyway with a  custom stop filter, it can be possible and actually can be a
 nice contribution as well.

 Cheers


 2015-05-28 13:01 GMT+01:00 Rupali rupali@gmail.com:

 sylkaalex sylkaalex at gmail.com writes:

 
  The main goal to allow each user use own stop words list. For example
 user
  type th
  now he will see next results in his terms search:
  the
  the one
  the then
  then
  then and
 
  But user has stop word the and he want get next results:
  then
  then and
 
  --
  View this message in context: http://lucene.472066.n3.nabble.com/Solr-
 advanced-StopFilterFactory-tp4195797p4195855.html
  Sent from the Solr - User mailing list archive at Nabble.com.
 
 

 Hi,

 Is there any way to get the stopwords from database? Checking options to
 get the list from database and use that list as stopwords into
 spellcheck component.

 Thanks in advance.






 --
 --

 Benedetti Alessandro
 Visiting card : http://about.me/alessandro_benedetti

 Tyger, tyger burning bright
 In the forests of the night,
 What immortal hand or eye
 Could frame thy fearful symmetry?

 William Blake - Songs of Experience -1794 England

Re: Running Solr 5.1.0 as a Service on Windows

[Timothy Potter]

Hi Edwin,

Are there changes you recommend to bin/solr.cmd to make it easier to
work with NSSM? If so, please file a JIRA as I'd like to help make
that process easier.

Thanks.
Tim

On Mon, May 25, 2015 at 3:34 AM, Zheng Lin Edwin Yeo
edwinye...@gmail.com wrote:
 I've managed to get the Solr started as a Windows service after
 re-configuring the startup script, as I've previously missed out some of
 the custom configurations there.

 However, I still couldn't get the zookeeper to start the same way too. Are
 we able to use NSSM to start up zookeeper as a Microsoft Windows service
 too?


 Regards,
 Edwin



 On 25 May 2015 at 12:16, Zheng Lin Edwin Yeo edwinye...@gmail.com wrote:

 Hi,

 Has anyone tried to run Solr 5.1.0 as a Microsoft Windows service?

 i've tried to follow the steps from this website
 http://www.norconex.com/how-to-run-solr5-as-a-service-on-windows/, which
 uses NSSM.

 However, when I tried to start the service from the Component Services in
 the Windows Control Panel Administrative tools, I get the following message:
 Windows could not start the Solr5 service on Local Computer. The service
 did not return an error. This could be an internal Windows error or an
 internal service error.

 Is this the correct way to set it up, or is there other methods?


 Regards,
 Edwin

Re: Solr 5.1 ignores SOLR_JAVA_MEM setting

[Timothy Potter]

Yes, same bug. Fixed in 5.2

On Tue, May 26, 2015 at 9:15 AM, Clemens Wyss DEV clemens...@mysign.ch wrote:
 I also noticed that (see my post this morning)
 ...
 SOLR_OPTS=$SOLR_OPTS -Dsolr.allow.unsafe.resourceloading=true
 ...
 Is not taken into consideration (anymore). Same bug?


 -Ursprüngliche Nachricht-
 Von: Ere Maijala [mailto:ere.maij...@helsinki.fi]
 Gesendet: Mittwoch, 15. April 2015 09:25
 An: solr-user
 Betreff: Solr 5.1 ignores SOLR_JAVA_MEM setting

 Folks, just a quick heads-up that apparently Solr 5.1 introduced a change in 
 bin/solr that overrides SOLR_JAVA_MEM setting from solr.in.sh or environment. 
 I just filed https://issues.apache.org/jira/browse/SOLR-7392. The problem can 
 be circumvented by using SOLR_HEAP setting, e.g. SOLR_HEAP=32G, but it's 
 not mentioned in solr.in.sh by default.

 --Ere

 --
 Ere Maijala
 Kansalliskirjasto / The National Library of Finland

Confused about whether Real-time Gets must be sent to leader?

[Timothy Potter]

I'm seeing that RTG requests get routed to any active replica of the
shard hosting the doc requested by /get ... I was thinking only the
leader should handle that request since there's a brief window of time
where the latest update may not be on the replica (albeit usually very
brief) and the latest update is definitely on the leader. Am I
overthinking this since we've always maintained that Solr is
eventually consistent or ???

Cheers,
Tim

Re: Start Solr with multiple external zookeepers on Windows Server?

[Timothy Potter]

Can you try defining the ZK_HOST in bin\solr.in.cmd instead of passing
it on the command-line?

On Mon, Apr 27, 2015 at 12:10 PM, Erick Erickson
erickerick...@gmail.com wrote:
 What version of Solr are you using? 4.10.3? 5.1?

 And can we see the full output of your attempt to start Solr? There
 might be some more informative bits above the help response.

 Best,
 Erick

 On Mon, Apr 27, 2015 at 9:49 AM, Stephan Schubert
 stephan.schub...@sick.de wrote:
 Hi everyone,

 how is it possible to start solr with an external set of zookeeper
 instances (quorum of 3 servers) on a windows server (2008R2)?

 From the wiki I got (
 https://cwiki.apache.org/confluence/display/solr/Setting+Up+an+External+ZooKeeper+Ensemble
 )

 bin\solr restart -c -p 8983 -z
 samplehost1:9983,samplehost2:9983,samplehost3:9983 -m 6g

 But this seems not to work on windows. After I start this command, the
 start skript prints out the help option only. With only one zookeeper
 Instance it works. Any suggestions?

 Regards

 Stephan
 
 Solr Version: 5.1

 SICK AG - Sitz: Waldkirch i. Br. - Handelsregister: Freiburg i. Br. HRB
 280355
 Vorstand: Dr. Robert Bauer (Vorsitzender)  -  Reinhard Bösl  -  Dr. Mats
 Gökstorp  -  Dr. Martin Krämer  -  Markus Vatter
 Aufsichtsrat: Gisela Sick (Ehrenvorsitzende) - Klaus M. Bukenberger
 (Vorsitzender)

[ANNOUNCE] Apache Solr 5.1.0 released

[Timothy Potter]

14 April 2015 - The Lucene PMC is pleased to announce the release of
Apache Solr 5.1.0.

Solr 5.1.0 is available for immediate download at:
http://www.apache.org/dyn/closer.cgi/lucene/solr/5.1.0

Solr 5.1.0 includes 39 new features, 40 bug fixes, and 36 optimizations
/ other changes from over 60 unique contributors.

For detailed information about what is included in 5.1.0 release,
please see: http://lucene.apache.org/solr/5_1_0/changes/Changes.html

Enjoy!

Re: [ANNOUNCE] Apache Solr 5.1.0 released

[Timothy Potter]

I apologize - Yonik prepared these nice release notes for 5.1 and I
neglected to include them:

Solr 5.1 Release Highlights:

 * The new Facet Module, including the JSON Facet API.
   This module is currently marked as experimental to allow for
further API feedback and improvements.

 * A new JSON request API.
   This feature is currently marked as experimental to allow for
further API feedback and improvements.

 * The ability to upload and download Solr configurations via SolrJ
(CloudSolrClient).

 * First-class support for Real-Time Get in SolrJ.

 * Spatial 2D heat-map faceting.

 * EnumField now has docValues support.

 * API to dynamically add Jars to Solr's classpath for plugins.

 * Ability to enable/disable individual stats in the StatsComponent.

 * lucene/solr query syntax to give any query clause a constant score.

 * Schema API enhancements to remove or replace fields, dynamic
fields, field types and copy fields.

 * When posting XML or JSON to Solr with curl, there is no need to
specify the content type.

 * A list of update processors to be used for an update can be
specified dynamically for any given request.

 * StatsComponent now supports Percentiles.

 * facet.contains option to limit which constraints are returned.

 * Streaming Aggregation for SolrCloud.

 * The admin UI now visualizes Lucene segment information.

 * Parameter substitution / macro expansion across entire request


On Tue, Apr 14, 2015 at 11:42 AM, Timothy Potter thelabd...@gmail.com wrote:
 14 April 2015 - The Lucene PMC is pleased to announce the release of
 Apache Solr 5.1.0.

 Solr 5.1.0 is available for immediate download at:
 http://www.apache.org/dyn/closer.cgi/lucene/solr/5.1.0

 Solr 5.1.0 includes 39 new features, 40 bug fixes, and 36 optimizations
 / other changes from over 60 unique contributors.

 For detailed information about what is included in 5.1.0 release,
 please see: http://lucene.apache.org/solr/5_1_0/changes/Changes.html

 Enjoy!

Re: Backup within SolrCloud

[Timothy Potter]

I wrote a simple backup utility for a Collection that uses the
replication handler, see:
https://github.com/LucidWorks/solr-scale-tk/blob/master/src/main/java/com/lucidworks/SolrCloudTools.java#L614
feel free to borrow / steal if useful.

On Mon, Apr 6, 2015 at 12:42 PM, Davis, Daniel (NIH/NLM) [C]
daniel.da...@nih.gov wrote:
 I withdraw this question - it is covered in the Solr 5 reference manual.   
 The suggestion is to use the replication handler, which suggests that this 
 scheme still works.   That's how I will go.

 From: Davis, Daniel (NIH/NLM) [C]
 Sent: Monday, April 06, 2015 2:29 PM
 To: solr-user@lucene.apache.org
 Subject: Backup within SolrCloud

 So, we have replication, but what if something bad is indexed into the 
 cluster, or someone accidentally deletes *:* on some collection?
 How do people manage backup in SolrCloud?

 I'm primarily interested in smaller indexes where backup is at all feasible.  
  I imagine a system such as Facebook really has to hope it stays up and maybe 
 do something similar to log shipping, e.g. keep intermediate results that 
 were fed into Solr.   My applications are likely to be a bit smaller, and 
 classic system backups apply.

 Thanks,

 Dan Davis, Systems/Applications Architect (Contractor),
 Office of Computer and Communications Systems,
 National Library of Medicine, NIH

Re: Spark-Solr in python

[Timothy Potter]

You'll need a python lib that uses a python ZooKeeper client to be
SolrCloud-aware so that you can do RDD like things, such as reading
from all shards in a collection in parallel. I'm not aware of any Solr
py libs that are cloud-aware yet, but it would be a good contribution
to upgrade https://github.com/toastdriven/pysolr to be SolrCloud-aware

On Mon, Mar 30, 2015 at 11:31 PM, Chaushu, Shani
shani.chau...@intel.com wrote:
 Hi,
 I saw there is a tool for reading solr into Spark RDD in JAVA
 I want to do something like this in python, is there any package in python 
 for reading solr into spark RDD?

 Thanks ,
 Shani


 -
 Intel Electronics Ltd.

 This e-mail and any attachments may contain confidential material for
 the sole use of the intended recipient(s). Any review or distribution
 by others is strictly prohibited. If you are not the intended
 recipient, please contact the sender and delete all copies.

Re: NoNode for /clusterstate.json in solr5.0.0 cloud

[Timothy Potter]

Anything in the server-side Solr logs? Also, if you go to the Solr admin
console at http://localhost:8983/solr, do you see the gettingstarted
collection in the cloud panel?



On Mon, Mar 30, 2015 at 1:12 PM, Purohit, Sumit sumit.puro...@pnnl.gov
wrote:

 I have a basic Solr 5.0.0 cloud setup after following
 http://lucene.apache.org/solr/quickstart.html

 I am trying to read data from spark and index it into solr using following
 lib:
 https://github.com/LucidWorks/spark-solr

 I am getting following error when my code try to make request to solr


 Exception in thread main org.apache.spark.SparkException: Job aborted
 due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent
 failure: Lost task 0.0 in stage 0.0 (TID 0, localhost):
 org.apache.solr.common.cloud.ZooKeeperException:

 at
 org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:465)

 .

 ..

 ..

 Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
 KeeperErrorCode = NoNode for /clusterstate.json

 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)

 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)

 at
 org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:500)



 I am not sure how (and when) to create nodes for /clusterstate.json

 I am using solr 5.0.0, sorlj5.0.0 spark-core_2.10_2.12.jar



 Thanks for the help.

 Sumit Purohit

Re: NoNode for /clusterstate.json in solr5.0.0 cloud

[Timothy Potter]

Ok, let me upgrade my version of spark-solr to 5 to see what I get ...

On Mon, Mar 30, 2015 at 2:26 PM, Purohit, Sumit sumit.puro...@pnnl.gov
wrote:

 yes there is getting started collection..
 and on admin webpage  console--cloud---tree---/clusterstate.json  shows
 me this table

 version =1
 aversion=0
 children_count=0
 ctimeFri= Mar 27 19:20:21 UTC 2015 (1427484021901)
 cversion=0
 czxid=32
 ephemeralOwner=0
 mtime=Fri Mar 27 19:20:36 UTC 2015 (1427484036453)
 mzxid=110
 pzxid=32
 dataLength=2

 children_count=0  seems related to no node error.

 thanks
 sumit
 
 From: Timothy Potter [thelabd...@gmail.com]
 Sent: Monday, March 30, 2015 2:18 PM
 To: solr-user@lucene.apache.org
 Subject: Re: NoNode for /clusterstate.json in solr5.0.0 cloud

 Anything in the server-side Solr logs? Also, if you go to the Solr admin
 console at http://localhost:8983/solr, do you see the gettingstarted
 collection in the cloud panel?



 On Mon, Mar 30, 2015 at 1:12 PM, Purohit, Sumit sumit.puro...@pnnl.gov
 wrote:

  I have a basic Solr 5.0.0 cloud setup after following
  http://lucene.apache.org/solr/quickstart.html
 
  I am trying to read data from spark and index it into solr using
 following
  lib:
  https://github.com/LucidWorks/spark-solr
 
  I am getting following error when my code try to make request to solr
 
 
  Exception in thread main org.apache.spark.SparkException: Job aborted
  due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent
  failure: Lost task 0.0 in stage 0.0 (TID 0, localhost):
  org.apache.solr.common.cloud.ZooKeeperException:
 
  at
 
 org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:465)
 
  .
 
  ..
 
  ..
 
  Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
  KeeperErrorCode = NoNode for /clusterstate.json
 
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
 
  at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 
  at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
 
  at
 
 org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:500)
 
 
 
  I am not sure how (and when) to create nodes for /clusterstate.json
 
  I am using solr 5.0.0, sorlj5.0.0 spark-core_2.10_2.12.jar
 
 
 
  Thanks for the help.
 
  Sumit Purohit
 
 

Re: NoNode for /clusterstate.json in solr5.0.0 cloud

[Timothy Potter]

I upgraded the spark-solr project to solrj-5.0.0 and was able to index into
the gettingstarted collection using Solr 5.0.0, so seems like it may be
environmental. Almost seems like the spark project is looking at the wrong
ZooKeeper? Are you using the default -zkHost localhost:9983

On Mon, Mar 30, 2015 at 2:32 PM, Purohit, Sumit sumit.puro...@pnnl.gov
wrote:

 Thanks Tim,

 i had to make some changes in my local spark-solr clone to build it for
 sorl5.
 If its ok, i can commit these to github.

 thanks
 sumit
 
 From: Timothy Potter [thelabd...@gmail.com]
 Sent: Monday, March 30, 2015 2:27 PM
 To: solr-user@lucene.apache.org
 Subject: Re: NoNode for /clusterstate.json in solr5.0.0 cloud

 Ok, let me upgrade my version of spark-solr to 5 to see what I get ...

 On Mon, Mar 30, 2015 at 2:26 PM, Purohit, Sumit sumit.puro...@pnnl.gov
 wrote:

  yes there is getting started collection..
  and on admin webpage  console--cloud---tree---/clusterstate.json
 shows
  me this table
 
  version =1
  aversion=0
  children_count=0
  ctimeFri= Mar 27 19:20:21 UTC 2015 (1427484021901)
  cversion=0
  czxid=32
  ephemeralOwner=0
  mtime=Fri Mar 27 19:20:36 UTC 2015 (1427484036453)
  mzxid=110
  pzxid=32
  dataLength=2
 
  children_count=0  seems related to no node error.
 
  thanks
  sumit
  
  From: Timothy Potter [thelabd...@gmail.com]
  Sent: Monday, March 30, 2015 2:18 PM
  To: solr-user@lucene.apache.org
  Subject: Re: NoNode for /clusterstate.json in solr5.0.0 cloud
 
  Anything in the server-side Solr logs? Also, if you go to the Solr admin
  console at http://localhost:8983/solr, do you see the gettingstarted
  collection in the cloud panel?
 
 
 
  On Mon, Mar 30, 2015 at 1:12 PM, Purohit, Sumit sumit.puro...@pnnl.gov
  wrote:
 
   I have a basic Solr 5.0.0 cloud setup after following
   http://lucene.apache.org/solr/quickstart.html
  
   I am trying to read data from spark and index it into solr using
  following
   lib:
   https://github.com/LucidWorks/spark-solr
  
   I am getting following error when my code try to make request to solr
  
  
   Exception in thread main org.apache.spark.SparkException: Job aborted
   due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent
   failure: Lost task 0.0 in stage 0.0 (TID 0, localhost):
   org.apache.solr.common.cloud.ZooKeeperException:
  
   at
  
 
 org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:465)
  
   .
  
   ..
  
   ..
  
   Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
   KeeperErrorCode = NoNode for /clusterstate.json
  
   at
 org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
  
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
  
   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
  
   at
  
 
 org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:500)
  
  
  
   I am not sure how (and when) to create nodes for /clusterstate.json
  
   I am using solr 5.0.0, sorlj5.0.0 spark-core_2.10_2.12.jar
  
  
  
   Thanks for the help.
  
   Sumit Purohit
  
  
 

Re: CloudSolrServer : Could not find collection : gettingstarted

[Timothy Potter]

Are you using a SolrJ client from 4.x to connect to a Solr 5 cluster?

On Wed, Mar 18, 2015 at 1:32 PM, Adnan Yaqoob itsad...@gmail.com wrote:

 I'm getting following exception while trying to upload document on
 SolrCloud using CloudSolrServer.

 Exception in thread main org.apache.solr.common.SolrException:
 *Could not find collection :* gettingstarted
 at
 org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:162)
 at
 org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:305)
 at
 org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:533)
 at
 org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
 at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
 at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
 at Test.addDocumentSolrCloud(Test.java:265)
 at Test.main(Test.java:284)

 I can query through Solr admin, able to upload document using
 HttpSolrServer (single instance - non cloud mode) but CloudSolrServer. I've
 also verified the collection exists on zookeeper using zkCli command.

 Following is the code snippet

 CloudSolrServer server = new CloudSolrServer(localhost:2181);
 server.setDefaultCollection(gettingstarted);
 SolrInputDocument doc = new SolrInputDocument();
 doc.addField(id, id);
 doc.addField(name, name);

 server.add(doc);

 server.commit();

 Not sure what I'm missing. My Zookeeper is running externally with two solr
 nodes on same mac

 --
 Regards,
 *Adnan Yaqoob*

Num docs, block join, and dupes?

[Timothy Potter]

Before I open a JIRA, I wanted to put this out to solicit feedback on what
I'm seeing and what Solr should be doing. So I've indexed the following 8
docs into a 2-shard collection (Solr 4.8'ish - internal custom branch
roughly based on 4.8) ... notice that the 3 grand-children of 2-1 have
dup'd keys:

[
  {
id:1,
name:parent,
_childDocuments_:[
  {
id:1-1,
name:child
  },
  {
id:1-2,
name:child
  }
]
  },
  {
id:2,
name:parent,
_childDocuments_:[
  {
id:2-1,
name:child,
_childDocuments_:[
  {
id:2-1-1,
name:grandchild
  },
  {
id:2-1-1,
name:grandchild2
  },
  {
id:2-1-1,
name:grandchild3
  }
]
  }
]
  }
]

When I query this collection, using:

http://localhost:8984/solr/blockjoin2_shard2_replica1/select?q=*%3A*wt=jsonindent=trueshards.info=truerows=10

I get:

{
  responseHeader:{
status:0,
QTime:9,
params:{
  indent:true,
  q:*:*,
  shards.info:true,
  wt:json,
  rows:10}},
  shards.info:{

http://localhost:8984/solr/blockjoin2_shard1_replica1/|http://localhost:8985/solr/blockjoin2_shard1_replica2/:{
  numFound:3,
  maxScore:1.0,
  shardAddress:http://localhost:8984/solr/blockjoin2_shard1_replica1;,
  time:4},

http://localhost:8984/solr/blockjoin2_shard2_replica1/|http://localhost:8985/solr/blockjoin2_shard2_replica2/:{
  numFound:5,
  maxScore:1.0,
  shardAddress:http://localhost:8985/solr/blockjoin2_shard2_replica2;,
  time:4}},
  response:{numFound:6,start:0,maxScore:1.0,docs:[
  {
id:1-1,
name:child},
  {
id:1-2,
name:child},
  {
id:1,
name:parent,
_version_:1495272401329455104},
  {
id:2-1-1,
name:grandchild},
  {
id:2-1,
name:child},
  {
id:2,
name:parent,
_version_:1495272401361960960}]
  }}


So Solr has de-duped the results.

If I execute this query against the shard that has the dupes (distrib=false):

http://localhost:8984/solr/blockjoin2_shard2_replica1/select?q=*%3A*wt=jsonindent=trueshards.info=truerows=10distrib=false

Then the dupes are returned:

{
  responseHeader:{
status:0,
QTime:0,
params:{
  indent:true,
  q:*:*,
  shards.info:true,
  distrib:false,
  wt:json,
  rows:10}},
  response:{numFound:5,start:0,docs:[
  {
id:2-1-1,
name:grandchild},
  {
id:2-1-1,
name:grandchild2},
  {
id:2-1-1,
name:grandchild3},
  {
id:2-1,
name:child},
  {
id:2,
name:parent,
_version_:1495272401361960960}]
  }}

So I guess my question is why doesn't the non-distrib query do
de-duping? Mainly confirming this is how it's supposed to work and
this behavior doesn't strike anyone else as odd ;-)

Cheers,

Tim

Re: Solr 5.0.0 - Multiple instances sharing Solr server *read-only* dir

[Timothy Potter]

I think the next step here is to ship Solr with the war already extracted
so that Jetty doesn't need to extract it on first startup -
https://issues.apache.org/jira/browse/SOLR-7227

On Tue, Mar 10, 2015 at 10:15 AM, Erick Erickson erickerick...@gmail.com
wrote:

 If I'm understanding your problem correctly, I think you want the -d
 option,
 then all the -s guys would be under that.

 Just to check, though, why are you running multiple Solrs? There are
 sometimes
 very good reasons, just checking that you're not making things more
 difficult
 than necessary

 Best,
 Erick

 On Mon, Mar 9, 2015 at 4:59 PM, Damien Dykman damien.dyk...@gmail.com
 wrote:
  Hi all,
 
  Quoted from
 
 https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
 
  When running multiple instances of Solr on the same host, it is more
  common to use the same server directory for each instance and use a
  unique Solr home directory using the -s option.
 
  Is there a way to achieve this without making *any* changes to the
  extracted content of solr-5.0.0.tgz and only use runtime parameters? I
  other words, make the extracted folder solr-5.0.0 strictly read-only?
 
  By default, the Solr web app is deployed under server/solr-webapp, as
  per solr-jetty-context.xml. So unless I change solr-jetty-context.xml, I
  cannot make folder sorl-5.0.0 read-only to my Solr instances.
 
  I've figured out how to make the log files and pid file to be located
  under the Solr data dir by doing:
 
  export SOLR_PID_DIR=mySolrDataDir/logs; \
  export SOLR_LOGS_DIR=mySolrDataDir/logs; \
  bin/solr start -c -z localhost:32101/solr \
   -s mySolrDataDir \
   -a -Dsolr.log=mySolrDataDir/logs \
   -p 31100 -h localhost
 
  But if there was a way to not have to change solr-jetty-context.xml that
  would be awesome! Thoughts?
 
  Thanks,
  Damien

Re: 43sec commit duration - blocked by index merge events?

[Timothy Potter]

I think Mark found something similar -
https://issues.apache.org/jira/browse/SOLR-6838

On Sat, Feb 14, 2015 at 2:05 AM, Erick Erickson erickerick...@gmail.com
wrote:

 Exactly how are you issuing the commit? I'm assuming you're
 using SolrJ. the server.commit(whatever, true) waits for the searcher
 to be opened before returning. This includes (I believe) warmup
 times. It could be that the warmup times are huge in your case, the
 solr logs should show you the autowarm times for a new searcher.

 Best,
 Erick

 On Fri, Feb 13, 2015 at 2:53 PM, Jack Krupansky
 jack.krupan...@gmail.com wrote:
  I wasn't able to follow Otis' answer but... the purpose of commit is to
  make make recent document changes (since the last commit) visible to
  queries, and has nothing to do with merging of segments. IOW, take the
 new
  segment that is being created and not yet ready for use by query, and
  finish it so that query can access it. Soft commit vs. hard commit is
  simply a matter of whether Solr will wait for the I/O to write the new
  segment to disk to complete. Merging is an independent, background
  procedure (thread) that merges existing segments. It does seem odd that
 the
  cited doc does say that soft commit waits for background merges! (Hoss??)
 
  -- Jack Krupansky
 
  On Fri, Feb 13, 2015 at 4:47 PM, Otis Gospodnetic 
  otis.gospodne...@gmail.com wrote:
 
  Check
  http://search-lucene.com/?q=commit+wait+blockfc_type=mail+_hash_+user
 
  e.g. http://search-lucene.com/m/QTPa7Sqx81
 
  Otis
  --
  Monitoring * Alerting * Anomaly Detection * Centralized Log Management
  Solr  Elasticsearch Support * http://sematext.com/
 
 
  On Fri, Feb 13, 2015 at 8:50 AM, Gili Nachum gilinac...@gmail.com
 wrote:
 
   Thanks Otis, can you confirm that a commit call will wait for merges
 to
   complete before returning?
  
   On Thu, Feb 12, 2015 at 8:46 PM, Otis Gospodnetic 
   otis.gospodne...@gmail.com wrote:
  
If you are using Solr and SPM for Solr, you can check a report that
  shows
the # of files in an index and the report that shows you the max
  docs-num
docs delta.  If you see the # of files drop during a commit, that's
 a
merge.  If you see a big delta change, that's probably a merge, too.
   
You could also jstack or kill -3 the JVM and see where it's spending
  its
time to give you some ideas what's going on inside.
   
HTH.
   
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log
 Management
Solr  Elasticsearch Support * http://sematext.com/
   
   
On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum gilinac...@gmail.com
   wrote:
   
 Hello,

 During a load test I noticed a commit that took 43 seconds to
  complete
 (client hard complete).
 Is this to be expected? What's causing it?
 I have a pair of machines hosting a 128M docs collection (8
 shards,
 replication factor=2).

 Could it be merges? In Lucene merges happen async of commit
  statements,
but
 reading Solr's doc for Update Hanlder
 

   
  
 
 https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
 
 it sounds like hard commits do wait for merges to occur: * The
   tradeoff
is
 that a soft commit gives you faster visibility because it's not
  waiting
for
 background merges to finish.*
 Thanks.

   
  
 

Re: Solrcloud performance issues

[Timothy Potter]

Hi Vijay,


We're working on SOLR-6816 ... would love for you to be a test site for any
improvements we make ;-)

Curious if you've experimented with changing the mergeFactor to a higher
value, such as 25 and what happens if you set soft-auto-commits to
something lower like 15 seconds? Also, make sure your indexing clients are
not sending hard-commits as well, i.e. just rely on auto-commits.

re: When the number of replicas increases the bulk indexing time increase
almost exponentially ... ugh ... I'm wondering what your CPU utilization /
thread counts are? the Leader sends updates to all replicas in parallel, so
it shouldn't be a huge impact if you're doing 1 replica or 15 (probably a
little more overhead with 15, but not exponential for sure) ... what are
threads waiting on when this huge slow down occurs? jstack -l PID should
give you some idea.

Lastly, do you have GC logging enabled and have you ruled out GC pauses
causing the big slow down?

On Thu, Feb 12, 2015 at 4:07 PM, Vijay Sekhri sekhrivi...@gmail.com wrote:

 Hi Erick,
 We have following configuration of our solr cloud

1. 10 Shards
2. 15 replicas per shard
3. 9 GB of index size per shard
4. a total of around 90 mil documents
5. 2 collection viz search1 serving live traffic and search 2 for
indexing. We swap collection when indexing finishes
6. On 150 hosts we have 2 JVMs running one for search1 collection and
other for search2 collection
7. Each jvm has 12 GB of heap assigned to it while the host has 50GB in
total
8. Each host has 16 processors
9. Linux XXX 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
10. We have two ways to index data.
1. Bulk indexing . All 90 million docs pumped in from 14 parallel
   process (on 14 different client hosts). This is done on
 collection that is
   not serving live traffic
   2.  Incremental indexing . Only delta changes (Range from 100K to 5
   Mil) every two hours. This is done on collection also serving live
 traffic
11. The request per second count on live collection is around 300 TPS
12. Hard commit setting is every 30 second with open searcher false and
soft commit setting is every 15 minutes . We have tried a lot of
 different
setting here BTW.




 Now we have two issues with indexing
 1) Solr just could not keep up with the bulk indexing when replicas are
 also active. We have concluded this by changing the number of replicas to
 just 2 , to 4 and then to 15. When the number of replicas increases the
 bulk indexing time increase almost exponentially
 We seem to have encountered the same issue reported here
 https://issues.apache.org/jira/browse/SOLR-6816
 It gets to a point that even to index 100 docs the solr cluster would take
 300 second. It would start of indexing 100 docs in 55 millisecond and
 slowly increase over time and within hour and a half just could not keep
 up. We have a workaround for this and i.e we stop all the replicas , do the
 bulk indexing and bring all the replicas up one by one . This sort of
 defeats the purpose of solr cloud but we can still work with this
 workaround. We can do this because , bulk indexing happen on the collection
 that is not serving live traffic. However we would love to have a solution
 from the solr cloud itself like ask it to stop replication and start via an
 API at the end of indexing.

 2) This issues is related to soft commit with incremental indexing . When
 we do incremental indexing, it is done on the same collection serving live
 traffic with 300 request per second throughput.  Everything is fine except
 whenever the soft commit happens. Each time soft commit (autosoftcommit in
 sorlconfig.xml) happens which BTW happens almost at the same time
 throughout the cluster , there is a spike in the response times and
 throughput decreases almost to 150 tps. The spike continues for 2 minutes
 and then it happens again at the exact interval when the soft commit
 happens. We have monitored the logs and found a direct co relation when the
 soft commit happens and when the response time tanks.

 Now the latter issue is quite disturbing , because it is serving live
 traffic and we cannot sustain these periodic degradation. We have played
 around with different soft commit setting . Interval ranging from 2 minutes
 to 30 minutes . Auto warming half cache  , auto warming full cache, auto
 warming only 10 %. Doing warm up queries on every new searcher , doing NONE
 warm up queries on every new searching and all the different setting yields
 the same results . As and when soft commit happens the response time tanks
 and throughput deceases. The difference is almost 50 % in response times
 and 50 % in throughput


 Our workaround for this solution is to also do incremental delta indexing
 on the collection not serving live traffic and swap when it is done. As you
 can see that this also defeats the purpose of solr 

Re: Solr on Tomcat

[Timothy Potter]

Correct. Solr 5.0 is not a Web application; any WAR or Web app'ish things
in Solr 5 are implementation details that may change in the future. The ref
guide will include some content about how to migrate to Solr 5 from 4.

On Tue, Feb 10, 2015 at 9:48 AM, Matt Kuiper matt.kui...@issinc.com wrote:

 I am starting to look in to Solr 5.0.  I have been running Solr 4.* on
 Tomcat.   I was surprised to find the following notice on
 https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+Tomcat
  (Marked as Unreleased)

  Beginning with Solr 5.0, Support for deploying Solr as a WAR in
 servlet containers like Tomcat is no longer supported.

 I want to verify that it is true that Solr 5.0 will not be able to run on
 Tomcat, and confirm that the recommended way to deploy Solr 5.0 is as a
 Linux service.

 Thanks,
 Matt

Re: log location when using bin/start

[Timothy Potter]

The bin/solr script in 4 didn't do a good job at allowing you to control
the location of the redirected console log or gc log, so you'll probably
have to hack that script a bit. The location of the main Solr log can be
configured in the example/resources/log4j.properties

This has been improved in the 5.0 scripts.

On Mon, Feb 9, 2015 at 6:36 PM, Benson Margulies ben...@basistech.com
wrote:

 Running bin/start with a command like:

 /data/solr-4.10.3/bin/solr start -s $PWD/solr_home -a
 -Djava.library.path=$libdir -Dbt.root=$bt_root\
  $@

 I note that the logs are ending up in the solr install
 dir/examples/logs. Can I move them?

Dealing with bad apples in a SolrCloud cluster

[Timothy Potter]

Just soliciting some advice from the community ...

Let's say I have a 10-node SolrCloud cluster and have a single collection
with 2 shards with replication factor 10, so basically each shard has one
replica on each of my nodes.

Now imagine one of those nodes starts getting into a bad state and starts
to be slow about serving queries (not bad enough to crash outright though)
... I'm sure we could ponder any number of ways a box might slow down
without crashing.

From my calculations, about 2/10ths of the queries will now be affected
since

1/10 queries from client apps will hit the bad apple
  +
1/10 queries from other replicas will hit the bad apple (distrib=false)


If QPS is high enough and the bad apple is slow enough, things can start to
get out of control pretty fast, esp. since we've set max threads so high to
avoid distributed dead-lock.

What have others done to mitigate this risk? Anything we can do in Solr to
help deal with this? It seems reasonable that nodes can identify a bad
apple by keeping track of query times and looking for nodes that are
significantly outside (=2 stddev) what the other nodes are doing. Then
maybe mark the node as being down in ZooKeeper so clients and other nodes
stop trying to send requests to it; or maybe a simple policy of just don't
send requests to that node for a few minutes.

Re: Changed behavior in solr 4 ??

[Timothy Potter]

Indeed - Hoss is correct ... it's a problem with the example in the
book ... my apologies for the confusion!

On Tue, Sep 30, 2014 at 3:57 PM, Chris Hostetter
hossman_luc...@fucit.org wrote:

 : Thanks for the response, yes the way you describe I know it works and is
 : how I get it to work but then what does mean the snippet of the
 : documentation I see on the documentation about overriding the default

 It means that there is implicitly a set of search components that have
 default behavior, and there is an implicit list of component *names*
 used by default by SearchHandler -- and if you override one of those
 implicit searchComponent instances by declaring your own with the same
 name, then it will be used by default in SerachHandler.

 a very concrete example of this is HighlightComponent -- if you have no
 HighlightComponent declared in your solrconfig.xml, then an implicit
 instance exists with the name highlight  and SearchHandler by default
 includes that component.

 If you want to declare your own HighlightComponent instance with special
 initialization logic, you can either declare it with it's own unique name,
 and edit the components list on a SerachHandler declatarion to include
 that name, or you can just name it highlight and it will override the
 default instance -- this is in fact done in the example solrconfig.xml
 (grep for HighlightComponent)

 : components shipped with Solr? Even on the book Solr in Action in chapter
 : 7 listing 7.3 I saw something similar to what I wanted to do:
 :
 : searchComponent name=query class=solr.QueryComponent
 :   lst name=invariants
 ...

 That appears to be a mistake in Solr in Action ... the QueryComponent
 class does nothing with it's init params (the nested XML inside the
 searchComponent declaration) so that syntax does nothing.



 -Hoss
 http://www.lucidworks.com/

Re: solr/lucene 4.10 out of memory issues

[Timothy Potter]

Probably need to look at it running with a profiler to see what's up.
Here's a few additional flags that might help the GC work better for
you (which is not to say there isn't a leak somewhere):

-XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=40

This should lead to a nice up-and-down GC profile over time.

On Thu, Sep 11, 2014 at 10:52 AM, Luis Carlos Guerrero
lcguerreroc...@gmail.com wrote:
 hey guys,

 I'm running a solrcloud cluster consisting of five nodes. My largest index
 contains 2.5 million documents and occupies about 6 gigabytes of disk
 space. We recently switched to the latest solr version (4.10) from version
 4.4.1 which we ran successfully for about a year without any major issues.
 From the get go we started having memory problems caused by the CMS old
 heap usage being filled up incrementally. It starts out with a very low
 memory consumption and after 12 hours or so it ends up using up all
 available heap space. We thought it could be one of the caches we had
 configured, so we reduced our main core filter cache max size from 1024 to
 512 elements. The only thing we accomplished was that the cluster ran for a
 longer time than before.

 I generated several heapdumps and basically what is filling up the heap is
 lucene's field cache. it gets bigger and bigger until it fills up all
 available memory.

 My jvm memory settings are the following:

 -Xms15g -Xmx15g -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=5g
 -XX:MaxNewSize=5g
 -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps
 -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
 What's weird to me is that we didn't have this problem before, I'm thinking
 this is some kind of memory leak issue present in the new lucene. We ran
 our old cluster for several weeks at a time without having to redeploy
 because of config changes or other reasons. Was there some issue reported
 related to elevated memory consumption by the field cache?

 any help would be greatly appreciated.

 regards,

 --
 Luis Carlos Guerrero
 about.me/luis.guerrero

Re: Solr API for getting shard's leader/replica status

[Timothy Potter]

You need to also verify the node hosting the replica is a live node
(/live_nodes). From SolrJ, you can call:
clusterState.getLiveNodes().contains(node).

As for API, there is CLUSTERSTATE provided by the Collection API, but
it's not consulting /live_nodes (which is a bug) - I'll open a ticket.

On Thu, Sep 4, 2014 at 4:37 AM, manohar211 manohar.srip...@gmail.com wrote:
 Solr has a Admin UI where we can check each and every Collections that were
 deployed to Solr Cloud. For example, I can see a Slice/Shard in a collection
 up or not in the mentioned diagram.
 http://lucene.472066.n3.nabble.com/file/n4156902/cloud-graph.png

 Our production environment doesn't provide access to this Admin UI due to
 security reasons. I need to provide an API to get the status of each and
 every collection, and its shards and each shard's replica. I am using Solr
 APIs to do that

 http://lucene.apache.org/solr/4_7_2/solr-solrj/index.html

 /CloudSolrServer server = new CloudSolrServer(zk quorum);
 ZkStateReader reader = server.getZkStateReader();
 CollectionSlice slices = reader.getClusterState().getSlices(collection);
 IteratorSlice iter = slices.iterator();
 while (iter.hasNext()) {
 Slice slice = iter.next();
 System.out.println(slice.getName());
 System.out.println(slice.getState());
 }/

 The above piece of code is always returning Active only as the state of
 shard, even its replica is showing down in the UI. I assume this returns
 only the state of a shard, not the state of shard's leader or replica.

 How can I get the replicas status through Solr APIs? is there any API for
 this? And what is the API being using by Solr Admin UI for getting shard's
 replicas/leader status?

 Thanks



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-API-for-getting-shard-s-leader-replica-status-tp4156902.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr API for getting shard's leader/replica status

[Timothy Potter]

https://issues.apache.org/jira/browse/SOLR-6481

On Thu, Sep 4, 2014 at 12:32 PM, Timothy Potter thelabd...@gmail.com wrote:
 You need to also verify the node hosting the replica is a live node
 (/live_nodes). From SolrJ, you can call:
 clusterState.getLiveNodes().contains(node).

 As for API, there is CLUSTERSTATE provided by the Collection API, but
 it's not consulting /live_nodes (which is a bug) - I'll open a ticket.

 On Thu, Sep 4, 2014 at 4:37 AM, manohar211 manohar.srip...@gmail.com wrote:
 Solr has a Admin UI where we can check each and every Collections that were
 deployed to Solr Cloud. For example, I can see a Slice/Shard in a collection
 up or not in the mentioned diagram.
 http://lucene.472066.n3.nabble.com/file/n4156902/cloud-graph.png

 Our production environment doesn't provide access to this Admin UI due to
 security reasons. I need to provide an API to get the status of each and
 every collection, and its shards and each shard's replica. I am using Solr
 APIs to do that

 http://lucene.apache.org/solr/4_7_2/solr-solrj/index.html

 /CloudSolrServer server = new CloudSolrServer(zk quorum);
 ZkStateReader reader = server.getZkStateReader();
 CollectionSlice slices = reader.getClusterState().getSlices(collection);
 IteratorSlice iter = slices.iterator();
 while (iter.hasNext()) {
 Slice slice = iter.next();
 System.out.println(slice.getName());
 System.out.println(slice.getState());
 }/

 The above piece of code is always returning Active only as the state of
 shard, even its replica is showing down in the UI. I assume this returns
 only the state of a shard, not the state of shard's leader or replica.

 How can I get the replicas status through Solr APIs? is there any API for
 this? And what is the API being using by Solr Admin UI for getting shard's
 replicas/leader status?

 Thanks



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-API-for-getting-shard-s-leader-replica-status-tp4156902.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to set storageDir for managed resource?

[Timothy Potter]

You can set the storageDir init-arg in the solrconfig.xml for the
RestManager for each core. However, since it is at the core config
level, you can't have a different storageDir per language. Here's an
example of how to configure the RestManager in solrconfig.xml to
customize the storageDir:

  restManager
str name=storageDirSOME_PATH_HERE/str
  /restManager

If this is not flexible enough, you can supply your own StorageIO
implementation, doing:

  restManager
str name=storageIOSOME CLASS HERE (implements
ManagedResourceStorage$StorageIO)/str
  /restManager



On Wed, Aug 27, 2014 at 2:55 AM, Phuong Doan phuong.d...@dkd.de wrote:
 Hi,

 currently I am using REST Api for managing synonyms and stopwords. My setup 
 for Apache Solr 4.8.1 looks like:

 SOLR_HOME/conf/
 _schema_analysis_synonyms_english.json
 _schema_analysis_stopwords_english.json
 solrconfig.xml
 english/schema.xml
 german/schema.xml
 etc.

 My goal is to store the _schema_analysis_synonyms_english.json and 
 _schema_analysis_stopwords_english.json in the language folder, not on the 
 same level as solrconfig.xml. I’m using ClassicIndexSchemaFactory.

 Setting storageDir as an attribute for ManagedSynonymFilterFactory occurs an 
 error.

 How can I set the storageDir for each language/core?


 Thanks
 Phuong

Replication of full index to replica after merge index into leader not working

[Timothy Potter]

Hi,

Using the coreAdmin mergeindexes command to merge an index into a
leader (SolrCloud mode on 4.9.0) and the replica does not do a snap
pull from the leader as I would have expected. The merge into the
leader worked like a charm except I had to send a hard commit after
that (which makes sense).

I'm guessing the replica would snap pull from the leader if I
restarted it, but reloading the collection or core does not trigger
the replica to pull from the leader. This seems like an oversight in
the mergeindex interaction with SolrCloud. Seems like the simplest
would be for the leader to send all replicas a request recovery
command after performing the merge.

Advice?

Cheers,
Tim

Re: Replication of full index to replica after merge index into leader not working

[Timothy Potter]

Was able to get around it for now sending the REQUESTRECOVERY command
to the replica. Will open an improvement JIRA but not sure if it's
worth it as the work-around is pretty clean (IMO).

Tim

On Tue, Aug 19, 2014 at 5:33 PM, Mark Miller markrmil...@gmail.com wrote:
 I’d just file a JIRA. Merge, like optimize and a few other things, were never 
 tested or considered in early SolrCloud days. It’s used in the HDFS stuff, 
 but in that case, the index is merged to all replicas and no recovery is 
 necessary.

 If you want to make the local filesystem merge work well with SolrCloud, 
 sounds like we should write a test and make it work.

 --
 Mark Miller
 about.me/markrmiller

 On August 19, 2014 at 1:20:54 PM, Timothy Potter (thelabd...@gmail.com) wrote:
 Hi,

 Using the coreAdmin mergeindexes command to merge an index into a
 leader (SolrCloud mode on 4.9.0) and the replica does not do a snap
 pull from the leader as I would have expected. The merge into the
 leader worked like a charm except I had to send a hard commit after
 that (which makes sense).

 I'm guessing the replica would snap pull from the leader if I
 restarted it, but reloading the collection or core does not trigger
 the replica to pull from the leader. This seems like an oversight in
 the mergeindex interaction with SolrCloud. Seems like the simplest
 would be for the leader to send all replicas a request recovery
 command after performing the merge.

 Advice?

 Cheers,
 Tim

Re: Stand alone Solr - no zookeeper?

[Timothy Potter]

It will always be supported under the current architecture as
SolrCloud uses master/slave style replication to bring replicas back
in-sync with leaders if a replica is too far out of date (currently,
too far  100 missed updates). So if it fits your architecture better,
then use it!

On Mon, Aug 4, 2014 at 10:53 AM, Joel Cohen jco...@grubhub.com wrote:
 Thanks for the input. For how long will the 'old style' of replication be
 supported? Is it set to go away in Solr 5? I don't want to be stuck on
 using an old version because we designed our application the wrong way.


 On Mon, Aug 4, 2014 at 10:22 AM, Michael Della Bitta 
 michael.della.bi...@appinions.com wrote:

 Hi Joel,

 You're sort of describing the classic replication scenario, which you can
 get started on by reading this:
 http://wiki.apache.org/solr/SolrReplication

 Although I believe this is handled in the reference guide, too.

 Generally speaking, the sorts of issues you mention are general issues that
 you have to deal with when using Solr at scale, no matter how you
 replicate. Proper GC tuning is a must. You can seriously diminish the
 impact of GC with some tuning.

 Etsy has done some interesting things regarding implementing an API that's
 resilient to garbage collecting nodes. Take a look at this:

 http://www.lucenerevolution.org/sites/default/files/Living%20with%20Garbage.pdf


 Michael Della Bitta

 Applications Developer

 o: +1 646 532 3062

 appinions inc.

 “The Science of Influence Marketing”

 18 East 41st Street

 New York, NY 10017

 t: @appinions https://twitter.com/Appinions | g+:
 plus.google.com/appinions
 
 https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
 
 w: appinions.com http://www.appinions.com/


 On Fri, Aug 1, 2014 at 10:48 AM, Joel Cohen jco...@grubhub.com wrote:

  Hi,
 
  We're in the development phase of a new application and the current dev
  team mindset leans towards running Solr (4.9) in AWS without Zookeeper.
 The
  theory is that we can add nodes quickly to our load balancer
  programmatically and get a dump of the indexes from another node and copy
  them over to the new one. A RESTful API would handle other applications
  talking to Solr without the need for each of them to have to use SolrJ.
  Data ingestion happens nightly in bulk by way of ActiveMQ which each
 server
  subscribes to and pulls its own copy of the indexes. Incremental updates
  are very few during the day, but we would have some mechanism of getting
 a
  new server to 'catch up' to the live servers before making it active in
 the
  load balancer.
 
  The only thing so far that I see as a hurdle here is the data set size
 vs.
  heap size. If the index grows too large, then we have to increase the
 heap
  size, which could lead to longer GC times. Servers could pop in and out
 of
  the load balancer if they are unavailable for too long when a major GC
  happens.
 
  Current stats:
  11 Gb of data (and growing)
  4 Gb java heap
  4 CPU, 16 Gb RAM nodes (maybe more needed?)
 
  All thoughts are welcomed.
 
  Thanks.
  --
  *Joel Cohen*
  Devops Engineer
 
  *GrubHub Inc.*
  *jco...@grubhub.com jco...@grubhub.com*
  646-527-7771
  1065 Avenue of the Americas
  15th Floor
  New York, NY 10018
 
  grubhub.com | *fb http://www.facebook.com/grubhub* | *tw
  http://www.twitter.com/grubhub*
  seamless.com | *fb http://www.facebook.com/seamless* | *tw
  http://www.twitter.com/seamless*
 




 --
 *Joel Cohen*
 Senior Devops Engineer

 *GrubHub Inc.*
 *jco...@grubhub.com jco...@grubhub.com*
 646-527-7771
 1065 Avenue of the Americas
 15th Floor
 New York, NY 10018

 grubhub.com | *fb http://www.facebook.com/grubhub* | *tw
 http://www.twitter.com/grubhub*
 seamless.com | *fb http://www.facebook.com/seamless* | *tw
 http://www.twitter.com/seamless*

Re: How to sync lib directory in SolrCloud?

[Timothy Potter]

You'll need to scp the JAR files to all nodes in the cluster. ZK is
not a great distribution mechanism for large binary files since it has
a 1MB znode size limit (by default)

On Thu, Jul 31, 2014 at 10:26 AM, P Williams
williams.tricia.l...@gmail.com wrote:
 Hi,

 I have an existing collection that I'm trying to add to a new SolrCloud.
  This collection has all the normal files in conf but also has a lib
 directory to support the filters schema.xml uses.

 wget
 https://github.com/projectblacklight/blacklight-jetty/archive/v4.9.0.zip
 unzip v4.9.0.zip

 I add the configuration to Zookeeper

 cd /solr-4.9.0/example/scripts
 cloud-scripts/zkcli.sh -cmd upconfig -confname blacklight -zkhost
 zk1:2181,zk2:2181,zk3:2181 -confdir
 ~/blacklight-jetty-4.9.0/solr/blacklight-core/conf/

 I try to create the collection
 curl 
 http://solr1:8080/solr/admin/collections?action=CREATEname=blacklightnumShards=3collection.configName=blacklightreplicationFactor=2maxShardsPerNode=2
 

 but it looks like the jars in the lib directory aren't available and this
 is what is causing my collection creation to fail.  I guess this makes
 sense because it's not one of the files that I added to Zookeeper to share.
  How do I share the lib directory via Zookeeper?

 Thanks,
 Tricia

 [pjenkins@solr1 scripts]$ cloud-scripts/zkcli.sh -cmd upconfig -zkhost
 zk1:2181,zk2:2181,zk3:2181 -confdir
 ~/blacklight-jetty-4.9.0/solr/blacklight-core/conf/ -confname blacklight
 INFO  - 2014-07-31 09:28:06.289; org.apache.zookeeper.Environment; Client
 environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
 INFO  - 2014-07-31 09:28:06.292; org.apache.zookeeper.Environment; Client
 environment:host.name=solr1.library.ualberta.ca
 INFO  - 2014-07-31 09:28:06.295; org.apache.zookeeper.Environment; Client
 environment:java.version=1.7.0_65
 INFO  - 2014-07-31 09:28:06.295; org.apache.zookeeper.Environment; Client
 environment:java.vendor=Oracle Corporation
 INFO  - 2014-07-31 09:28:06.295; org.apache.zookeeper.Environment; Client
 environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre
 INFO  - 2014-07-31 09:28:06.295; org.apache.zookeeper.Environment; Client
 

Re: Scaling Issues

[Timothy Potter]

Hi Ameya,

Tough to say without more information about what's slow. In general,
when I've seen Solr index that slow, it's usually related to some
complex text analysis, for instance, are you doing any phonetic
analysis? Best thing to do is attach a Java profiler (e.g. JConsole or
VisualVM) using rmi and see where the hotspots are (this will also
give you hints about GC activity).

Here are the flags I use to setup my Solr server to allow JConsole or
VisualVm to attach. You can also try out YourKit (it's better but a
little more involved to use).

-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.port=1084 \
-Dcom.sun.management.jmxremote.rmi.port=1084
-Djava.rmi.server.hostname=$SOLR_HOST

Note: Have a chat with your sysadmin about these before using in production ;-)



On Tue, Jul 29, 2014 at 11:49 AM, Ameya Aware ameya.aw...@gmail.com wrote:
 Hi,

 I am running Solr with below parameters:

 -XX:MaxPermSize=128m -Xms5120m -Xmx5120m -XX:+UseConcMarkSweepGC
 -XX:CMSInitiatingOccupancyFraction=70 -XX:NewRatio=3
 -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled
 -XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts
 -XX:-UseGCOverheadLimit


 I need to index around 30 documents but with above parameters
 performance is coming very poor around 15000-2 documents per hour.

 This would take a lot of time to index all the documents.

 Also, my autocommit in solrconfig.xml is as below:

 autoCommit
  maxTime15/maxTime
   openSearcherfalse/openSearcher
 /autoCommit

 documentCache class=solr.LRUCache autowarmCount=0 initialSize=256
 size=256/



 I am running Solr on machine having 12GB RAM.


 Please advice on how can i improve the performance.


 Thanks,
 Ameya

Re: Slow inserts when using Solr Cloud

[Timothy Potter]

Hi Ian,

What's the CPU doing on the leader? Have you tried attaching a
profiler to the leader while running and then seeing if there are any
hotspots showing. Not sure if this is related but we recently fixed an
issue in the area of leader forwarding to replica that used too many
CPU cycles inefficiently - see SOLR-6136.

Tim

On Wed, Jul 16, 2014 at 7:49 AM, ian ian.willi...@wales.nhs.uk wrote:
 That's useful to know, thanks very much.   I'll look into using
 CloudSolrServer, although I'm using solrnet at present.

 That would reduce some of the overhead - but not the extra 200ms I'm getting
 for forwarding to the replica when the replica is switched on.

 It does seem a very high overhead.  When I consider that it takes 20ms to
 insert a new document to Solr with replicas disabled (if I route to the
 correct shard), you might expect it to take two to three times longer if it
 has to forward to one replica and then wait for a response, but an increase
 of 200ms seems really high doesn't it?  Is there a forum where I should
 raise that?

 Thanks again for your help
 Ian


 Shalin Shekhar Mangar wrote
 You can use CloudSolrServer (if you're using Java) which will route
 documents correctly to the leader of the appropriate shard.


 On Tue, Jul 15, 2014 at 3:04 PM, ian lt;

 Ian.Williams@.nhs

 gt; wrote:

 Hi Mark

 Thanks for replying to my post.  Would you know whether my findings are
 consistent with what other people see when using SolrCloud?

 One thing I want to investigate is whether I can route my updates to the
 correct shard in the first place, by having my client using the same
 hashing
 logic as Solr, and working out in advance which shard my inserts should
 be
 sent to.  Do you know whether that's an approach that others have used?

 Thanks again
 Ian



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Slow-inserts-when-using-Solr-Cloud-tp4146087p4147183.html
 Sent from the Solr - User mailing list archive at Nabble.com.




 --
 Regards,
 Shalin Shekhar Mangar.





 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Slow-inserts-when-using-Solr-Cloud-tp4146087p4147481.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Parallel optimize of index on SolrCloud.

[Timothy Potter]

Hi Modassar,

Have you tried hitting the cores for each replica directly (instead of
using the collection)? i.e. if you had col_shard1_replica1 on node1,
then send the optimize command to that core URL directly:

curl -i -v http://host:port/solr/col_shard1_replica1/update; -H
'Content-type:application/xml' \
  --data-binary optimize/

I haven't tried this myself but might work ;-)

Tim

On Wed, Jul 9, 2014 at 12:59 AM, Modassar Ather modather1...@gmail.com wrote:
 Hi All,

 Thanks for your kind suggestions and inputs.

 We have been going the optimize way and it has helped. There have been
 testing and benchmarking already done around memory and performance.
 So while optimizing we see a scope of improvement on it by doing it
 parallel so kindly suggest in what way it can be achieved.

 Thanks,
 Modassar


 On Wed, Jul 9, 2014 at 11:48 AM, Shalin Shekhar Mangar 
 shalinman...@gmail.com wrote:

 Hi Walter,

 I wonder why you think SolrCloud isn't necessary if you're indexing once
 per week. Isn't the automatic failover and auto-sharding still useful? One
 can also do custom sharding with SolrCloud if necessary.


 On Wed, Jul 9, 2014 at 11:38 AM, Walter Underwood wun...@wunderwood.org
 wrote:

  More memory or faster disks will make a much bigger improvement than a
  forced merge.
 
  What are you measuring? If it is average query time, that is not a good
  measure. Look at 90th or 95th percentile. Test with queries from logs.
 
  No user can see a 10% or 20% difference. If your managers are watching
  that, they are watching the wrong thing.
 
  If you are indexing once per week, you don't really need the complexity
 of
  Solr Cloud. You can do manual sharding.
 
  wunder
 
  On Jul 8, 2014, at 10:55 PM, Modassar Ather modather1...@gmail.com
  wrote:
 
   Our index has almost 100M documents running on SolrCloud of 3 shards
 and
   each shard has an index size of about 700GB (for the record, we are not
   using stored fields - our documents are pretty large). We perform a
 full
   indexing every weekend and during the week there are no updates made to
  the
   index. Most of the queries that we run are pretty complex with hundreds
  of
   terms using PhraseQuery, BooleanQuery, SpanQuery, Wildcards, boosts
 etc.
   and take many minutes to execute. A difference of 10-20% is also a big
   advantage for us.
  
   We have been optimizing the index after indexing for years and it has
   worked well for us. Every once in a while, we upgrade Solr to the
 latest
   version and try without optimizing so that we can save the many hours
 it
   take to optimize such a huge index, but it does not work well.
  
   Kindly provide your suggestion.
  
   Thanks,
   Modassar
  
  
   On Wed, Jul 9, 2014 at 10:47 AM, Walter Underwood 
 wun...@wunderwood.org
  
   wrote:
  
   I seriously doubt that you are required to force merge.
  
   How much improvement? And is the big performance cost also OK?
  
   I have worked on search engines that do automatic merges and offer
  forced
   merges for over fifteen years. For all that time, forced merges have
   usually caused problems.
  
   Stop doing forced merges.
  
   wunder
  
   On Jul 8, 2014, at 10:09 PM, Modassar Ather modather1...@gmail.com
   wrote:
  
   Thanks Walter for your inputs.
  
   Our use case and performance benchmark requires us to invoke
 optimize.
  
   Here we see a chance of improvement in performance of optimize() if
   invoked
   in parallel.
   I found that if* distrib=false *is used, the optimization will happen
  in
   parallel.
  
   But I could not find a way to set it using
   HttpSolrServer/CloudSolrServer.
   Also with the parameter setting as given in my mail above does not
  seems
   to
   work.
  
   Please let me know in what ways I can achieve the parallel optimize
 on
   SolrCloud.
  
   Thanks,
   Modassar
  
   On Tue, Jul 8, 2014 at 7:53 PM, Walter Underwood 
  wun...@wunderwood.org
   wrote:
  
   You probably do not need to force merge (mistakenly called
 optimize)
   your index.
  
   Solr does automatic merges, which work just fine.
  
   There are only a few situations where a forced merge is even a good
   idea.
   The most common one is a replicated (non-cloud) setup with a full
   reindex
   every night.
  
   If you need Solr Cloud, I cannot think of a situation where you
 would
   want
   a forced merge.
  
   wunder
  
   On Jul 8, 2014, at 2:01 AM, Modassar Ather modather1...@gmail.com
   wrote:
  
   Hi,
  
   Need to optimize index created using CloudSolrServer APIs under
   SolrCloud
   setup of 3 instances on separate machines. Currently it optimizes
   sequentially if I invoke cloudSolrServer.optimize().
  
   To make it parallel I tried making three separate HttpSolrServer
   instances
   and invoked httpSolrServer.opimize() on them parallely but still it
   seems
   to be doing optimization sequentially.
  
   I tried invoking optimize directly using HttpPost with following
 url
   and
   parameters but still it seems 

Re: Planning ahead for Solr Cloud and Scaling

[Timothy Potter]

Hi Zane,

re 1: as an alternative to shard splitting, you can just overshard the
collection from the start and then migrate existing shards to new
hardware as needed. The migrate can happen online, see collection API
ADDREPLICA. Once the new replica is online on the new hardware, you
can unload the older replica on your original hardware. There are
other benefits to oversharding, such as increased parallelism during
indexing and query execution (provided you have the CPU capacity,
which is typically the case on modern hardware).

re 2: mainly depends on how the Java GC and heap are affected by
colocating the cores on the same JVM ... if heap is stable and the GC
is keeping up and qps / latency times are acceptable, I wouldn't
change it.

re 3: read Trey's chapter 14 in Solr in Action ;-)

Cheers,
Tim

On Tue, Jul 8, 2014 at 10:09 PM, Zane Rockenbaugh z...@navigo.com wrote:
 I'm working on a product hosted with AWS that uses Elastic Beanstalk
 auto-scaling to good effect and we are trying to set up similar (more or
 less) runtime scaling support with Solr. I think I understand how to set
 this up, and wanted to check I was on the right track.

 We currently run 3 cores on a single host / Solr server / shard. This is
 just fine for now, and we have overhead for the near future. However, I
 need to have a plan, and then test, for a higher capacity future.

 1) I gather that if I set up SolrCloud, and then later load increases, I
 can spin up a second host / Solr server, create a new shard, and then split
 the first shard:

 https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api3

 And doing this, we no longer have to commit to shards out of the gate.

 2) I'm not clear whether there's a big advantage splitting up the cores or
 not. Two of the three cores will have about the same number of documents,
 though only one contains large amounts of text. The third core is much
 smaller in both bytes and documents (2 orders of magnitude).

 3) We are also looking at moving multi-lingual. The current plan is to
 store the localized text in fields within the same core. The languages will
 be added over time. We can update the schema (as each will be optional).
 This seems easier than adding a core for each language. Is there a downside?

 Thanks for any pointers.

Re: SolrCloud leaders using more disk space

[Timothy Potter]

Hi Greg,

Sorry for the slow response. The general thinking is that you
shouldn't worry about which nodes host leaders vs. replicas because A)
that can change, and B) as you say, the additional responsibilities
for leader nodes is quite minimal (mainly per-doc version management
and then distributing updates to replicas). The segment merging all
happens at the Lucene level, which has no knowledge of SolrCloud
leaders / replicas. Since this is SolrCloud, all nodes pull the config
from ZooKeeper so should be running the same settings. Can you diff
the listings of the index data directories on a leader vs. replica?
Might give us some insights to what files the leader has that the
replicas don't have.

Cheers,
Tim

On Tue, Jun 3, 2014 at 8:32 PM, Greg Pendlebury
greg.pendleb...@gmail.com wrote:
 Hi all,

 We launched our new production instance of SolrCloud last week and since
 then have noticed a trend with regards to disk usage. The non-leader
 replicas all seem to be self-optimizing their index segments as expected,
 but the leaders have (on average) around 33% more data on disk. My
 assumption is that leader's are not self-optimising (or not to the same
 extent)... but it is still early days of course.

 If it helps, there are 45 JVMs in the cloud, with 15 shards and 3 replicas
 per shard. Each non-leader shard is sitting at between 59GB and 87GB on
 their SSD, but the leaders are between 84GB and 116GB.

 We have pretty much constant read and write traffic 24x7, with just 'slow'
 periods overnight when write traffic is  1 document per second and
 searches are between 1 and 2 per second. Is this light level of traffic
 still too much for the leaders to self-optimise?

 I'd also be curious to hear about what others are doing in terms of
 operating procedures. We load test before launch what would happen if we
 turned off JVMs and forced recovery events. I know that these things all
 work, just that customers will experience slower search responses whilst
 they occur. For example, a restore from a leader to a replica under load
 testing for us takes around 30 minutes and response times drop from around
 200-300ms average to 1.5s average.

 Bottleneck appears to be network I/O on the servers. We haven't explored
 whether this is specific to the servers replicating, or saturation of the
 of the infrastructure that all the servers share, because...

 This performance is acceptable for us, but I'm not sure if I'd like to
 force that event to occur unless required... this is following the line of
 reasoning proposed internally that we should periodically rotate leaders by
 turning them off briefly. We aren't going to do that unless we have a
 strong reason though. Does anyone try to manipulate production instances
 that way?

 Vaguely related to this is leader distribution. We have 9 physical servers
 and 5 JVMs running on each server. By virtue of the deployment procedures
 the first 3 servers to come online are all running 5 leaders each. Is there
 any merit in 'moving' these around (by reboots)?

 Our planning up to launch was based on lots of mailing list response we'd
 seen that indicated leaders had no significant performance difference to
 normal replicas, and all of our testing has agreed with that. The disk size
 'issue' (which we aren't worried about... yet. It hasn't been in prod long
 enough to know for certain) may be the only thing we've seen so far.

 Ta,
 Greg

Re: Solr Scale Toolkit Access Denied Error

[Timothy Potter]

Hi Mark,

Sorry for the trouble! I've now made the ami-1e6b9d76 AMI public;
total oversight on my part :-(. Please try again. Thanks Hoss for
trying to help out on this one.

Cheers,
Tim

On Fri, Jun 6, 2014 at 6:46 PM, Mark Gershman montan...@gmail.com wrote:
 Thanks, Hoss.

 I did substitute the previous AMI ID from the mid-May release of the
 toolkit and the build process does proceed further; however, it appears the
 the AMI changed enough that it is not compatible with the new toolkit
 release.  In doing a little more research, I'm inclined to believe that the
 permissions on the AMI may be the source of the problem and will post to
 the issue tracker per your suggestion.


 Mark Gershman


 On Fri, Jun 6, 2014 at 7:41 PM, Chris Hostetter hossman_luc...@fucit.org
 wrote:


 : My guess is that the customized toolkit AMI (ami-1e6b9d76) at AWS is not
 : accessible by my AWS credentials.  Is this an AMI permissioning issue or
 is
 : it a problem with my particular account or how it is configured at AWS.
  I
 : did not experience this specific problem when working with the previous
 : iteration of the Solr Scale Toolkit back toward the latter part of May.
  It
 : appears that the AMI was updated from ami-96779efe to ami-1e6b9d76 with
 the
 : newest version of the toolkit.

 I'm not much of an AWS expert, but i seem to recall that if you don't
 have your AWS security group setup properly this type of error can
 happen? is it possible that when you were trying out solr-scale-tk before
 you had this setup, but now you don't?

 https://github.com/LucidWorks/solr-scale-tk

  You'll need to setup a security group named solr-scale-tk (or update the
  fabfile.py to change the name).
 
  At a minimum you should allow TCP traffic to ports: 8983, 8984-8989,
  SSH, and 2181 (ZooKeeper). However, it is your responsibility to review
  the security configuration of your cluster and lock it down
 appropriately.
 
  You'll also need to create an keypair (using the Amazon console) named
  solr-scale-tk (you can rename the key used by the framework, see:
  AWS_KEY_NAME). After downloading the keypair file (solr-scale-tk.pem),
  save it to ~/.ssh/ and change permissions: chmod 600
  ~/.ssh/solr-scale-tk.pem

 ...if I'm wrong, and there really is a problem with the security on the
 AMI, the best place to report that would be in the project's issue
 tracker...

 https://github.com/LucidWorks/solr-scale-tk/issues



 -Hoss
 http://www.lucidworks.com/

Possible UI regression in the Analysis form in 4.7?

[Timothy Potter]

I have an example in Solr In Action that uses the
PatternReplaceCharFilterFactory and now it doesn't work in 4.7.0.
Specifically, the fieldType is:

fieldType name=text_microblog class=solr.TextField
positionIncrementGap=100
  analyzer
charFilter class=solr.PatternReplaceCharFilterFactory
pattern=([a-zA-Z])\1+
replacement=$1$1/
tokenizer class=solr.WhitespaceTokenizerFactory/
filter class=solr.WordDelimiterFilterFactory
generateWordParts=1
splitOnCaseChange=0
splitOnNumerics=0
stemEnglishPossessive=1
preserveOriginal=0
catenateWords=1
generateNumberParts=1
catenateNumbers=0
catenateAll=0
types=wdfftypes.txt/
filter class=solr.StopFilterFactory
ignoreCase=true
words=lang/stopwords_en.txt
/
filter class=solr.LowerCaseFilterFactory/
filter class=solr.ASCIIFoldingFilterFactory/
filter class=solr.KStemFilterFactory/
  /analyzer
/fieldType

The PatternReplaceCharFilterFactory (PRCF) is used to collapse
repeated letters in a term down to a max of 2, such as #yu would
be #yumm

When I run some text through this analyzer using the Analysis form,
the output is as if the resulting text is unavailable to the
tokenizer. In other words, the only results being displayed in the
output on the form is for the PRCF

This example stopped working in 4.7.0 and I've verified it worked
correctly in 4.6.1.

Initially, I thought this might be an issue with the actual analysis,
but the analyzer actually works when indexing / querying. Then,
looking at the JSON response in the Developer console with Chrome, I
see the JSON that comes back includes output for all the components in
my chain (see below) ... so looks like a UI issue to me?

Anyone have any ideas on what might be going on?

If not, I'll create a JIRA.

{responseHeader:{status:0,QTime:24},analysis:{field_types:{text_microblog:{index:[org.apache.lucene.analysis.pattern.PatternReplaceCharFilter,#Yumm
:) Drinking a latte at Caffe Grecco in SF's historic North Beach...
Learning text analysis with #SolrInAction by @ManningBooks on my i-Pad
foo5,org.apache.lucene.analysis.core.WhitespaceTokenizer,[{text:#Yumm,raw_bytes:[23
59 75 6d 
6d],start:0,end:6,position:1,positionHistory:[1],type:word},{text::),raw_bytes:[3a
29],start:7,end:9,position:2,positionHistory:[2],type:word},{text:Drinking,raw_bytes:[44
72 69 6e 6b 69 6e
67],start:10,end:18,position:3,positionHistory:[3],type:word},{text:a,raw_bytes:[61],start:19,end:20,position:4,positionHistory:[4],type:word},{text:latte,raw_bytes:[6c
61 74 74 
65],start:21,end:26,position:5,positionHistory:[5],type:word},{text:at,raw_bytes:[61
74],start:27,end:29,position:6,positionHistory:[6],type:word},{text:Caffe,raw_bytes:[43
61 66 66 
65],start:30,end:35,position:7,positionHistory:[7],type:word},{text:Grecco,raw_bytes:[47
72 65 63 63 
6f],start:36,end:42,position:8,positionHistory:[8],type:word},{text:in,raw_bytes:[69
6e],start:43,end:45,position:9,positionHistory:[9],type:word},{text:SF's,raw_bytes:[53
46 27 
73],start:46,end:50,position:10,positionHistory:[10],type:word},{text:historic,raw_bytes:[68
69 73 74 6f 72 69
63],start:51,end:59,position:11,positionHistory:[11],type:word},{text:North,raw_bytes:[4e
6f 72 74 
68],start:60,end:65,position:12,positionHistory:[12],type:word},{text:Beach...,raw_bytes:[42
65 61 63 68 2e 2e
2e],start:66,end:74,position:13,positionHistory:[13],type:word},{text:Learning,raw_bytes:[4c
65 61 72 6e 69 6e
67],start:75,end:83,position:14,positionHistory:[14],type:word},{text:text,raw_bytes:[74
65 78 
74],start:84,end:88,position:15,positionHistory:[15],type:word},{text:analysis,raw_bytes:[61
6e 61 6c 79 73 69
73],start:89,end:97,position:16,positionHistory:[16],type:word},{text:with,raw_bytes:[77
69 74 
68],start:98,end:102,position:17,positionHistory:[17],type:word},{text:#SolrInAction,raw_bytes:[23
53 6f 6c 72 49 6e 41 63 74 69 6f
6e],start:103,end:116,position:18,positionHistory:[18],type:word},{text:by,raw_bytes:[62
79],start:117,end:119,position:19,positionHistory:[19],type:word},{text:@ManningBooks,raw_bytes:[40
4d 61 6e 6e 69 6e 67 42 6f 6f 6b
73],start:120,end:133,position:20,positionHistory:[20],type:word},{text:on,raw_bytes:[6f
6e],start:134,end:136,position:21,positionHistory:[21],type:word},{text:my,raw_bytes:[6d
79],start:137,end:139,position:22,positionHistory:[22],type:word},{text:i-Pad,raw_bytes:[69
2d 50 61 
64],start:140,end:145,position:23,positionHistory:[23],type:word},{text:foo5,raw_bytes:[66
6f 6f 
35],start:146,end:150,position:24,positionHistory:[24],type:word}],org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter,[{text:#Yumm,raw_bytes:[23
59 75 6d 
6d],start:0,end:6,type:word,position:1,positionHistory:[1,1]},{text:Drinking,raw_bytes:[44
72 69 6e 6b 69 6e

Re: Possible UI regression in the Analysis form in 4.7?

[Timothy Potter]

Thanks for confirming Shawn. I created SOLR-5800 for this. I'd dig
into the UI code further but my JavaScript skills are a bit rusty to
say the least ... bummer that this regression breaks a very nice
example in the book though :-(

On Sat, Mar 1, 2014 at 1:28 PM, Shawn Heisey s...@elyograg.org wrote:
 On 3/1/2014 12:15 PM, Timothy Potter wrote:
 The PatternReplaceCharFilterFactory (PRCF) is used to collapse
 repeated letters in a term down to a max of 2, such as #yu would
 be #yumm

 When I run some text through this analyzer using the Analysis form,
 the output is as if the resulting text is unavailable to the
 tokenizer. In other words, the only results being displayed in the
 output on the form is for the PRCF

 This example stopped working in 4.7.0 and I've verified it worked
 correctly in 4.6.1.

 Initially, I thought this might be an issue with the actual analysis,
 but the analyzer actually works when indexing / querying. Then,
 looking at the JSON response in the Developer console with Chrome, I
 see the JSON that comes back includes output for all the components in
 my chain (see below) ... so looks like a UI issue to me?

 I confirmed this with branch_4x.  The UI breaks, but if a document is
 indexed with that value, the Schema Browser shows the term in the index.

 Please file an issue in Jira on the 'web gui' component.

 Thanks,
 Shawn

Re: What is the right way to list top terms for a given field?

[Timothy Potter]

Hi Dave,

Have you looked at the TermsComponent?
http://wiki.apache.org/solr/TermsComponent It is easy to wire into an
existing request handler and allows you to return the top terms for a
field. Example server even includes an example request handler that
uses it:

  searchComponent name=terms class=solr.TermsComponent/

  requestHandler name=/terms class=solr.SearchHandler startup=lazy
 lst name=defaults
  bool name=termstrue/bool
  bool name=distribfalse/bool
/lst
arr name=components
  strterms/str
/arr
  /requestHandler

Cheers,
Tim

On Wed, Nov 27, 2013 at 10:07 AM, Dave Seltzer dselt...@tveyes.com wrote:
 It's certainly seems to be faster (in my limited testing).

 I just don't want to base my software on the Luke scripts if they're
 prone to changing in the future.

 And yes, I realize there are ways to make this secure. I just wanted
 to know if it's something I should avoid doing (perhaps for reasons
 beyond my comprehension.)

 Thanks!

 -D

 On Nov 27, 2013, at 11:46 AM, Stefan Matheis matheis.ste...@gmail.com 
 wrote:

 Since your users shouldn't be allowed at any time to access Solr directly, 
 it's up to you to implement that on the client side anyway?

 I can't tell if there is a technical difference between the two calls you 
 named, but i'd guess that the second might be a more direct way to access 
 this information (and probably a bit faster?).

 -Stefan


 On Wednesday, November 27, 2013 at 5:22 PM, Dave Seltzer wrote:

 Hello,

 I'm trying to get a list of top terms for a field called Tags.

 One way to do this would be to query all data *:* and then facet by the
 Tags column:
 /solr/collection/admin/select?q=*:*rows=0facet=truefacet.field=Tags

 I've noticed another way to do this is using the luke interface like this:
 /solr/collection/admin/luke?fl=TagsnumTerms=20

 One problem I see with the luke interface is that its inside the /admin/
 path, which to me means that my users shouldn't be able to access it.

 Whats the most SOLRy way to do this?

 Thanks!

 -D

Re: removing dead replicas in solrcloud 4.4

[Timothy Potter]

Yes, I've done this ... but I had to build my own utility to update
clusterstate.json (for reasons I can't recall now). So make your
changes to clusterstate.json manually and then do something like the
following with SolrJ:

public static void updateClusterstateJsonInZk(CloudSolrServer
cloudSolrServer, CommandLine cli) throws Exception {
String updateClusterstateJson =
cli.getOptionValue(updateClusterstateJson);

ZkStateReader zkStateReader = cloudSolrServer.getZkStateReader();
SolrZkClient zkClient = zkStateReader.getZkClient();

File jsonFile = new File(updateClusterstateJson);
if (!jsonFile.isFile()) {
System.err.println(jsonFile.getAbsolutePath()+ not found.);
return;
}

byte[] clusterstateJson = readFile(jsonFile);

// validate the user is passing is valid JSON
InputStreamReader bytesReader = new InputStreamReader(new
ByteArrayInputStream(clusterstateJson), UTF-8);
JSONParser parser = new JSONParser(bytesReader);
parser.toString();

zkClient.setData(/clusterstate.json, clusterstateJson, true);
System.out.println(Updated /clusterstate.json with data from
+jsonFile.getAbsolutePath());
}

On Fri, Nov 22, 2013 at 12:33 PM, Eric Parish
eric.par...@sherpaanalytics.com wrote:
 My 4.4 sorlcloud cluster has several down replicas that need to be removed. I 
 am looking for a solution to clean them up like the deletereplica api 
 available in 4.6.

 Will manually removing the replicas from the clusterstate.json file in 
 zookeeper accomplish my needs?

 Thanks,
 Eric

Re: Option to enforce a majority quorum approach to accepting updates in SolrCloud?

[Timothy Potter]

Hi Otis,

I think these are related problems but giving the ability to enforce a
majority quorum among the total replica set for a shard is not the
same as hinted handoff in the Cassandra sense. Cass's hinted handed
allows you to say it's ok to send the write somewhere and somehow
it'll make its way to the correct node, eventually. But, from the docs
A hinted write does not count towards ConsistencyLevel requirements
of ONE, QUORUM, or ALL. It's useful for environments that need
extreme write-availability.

Today, SolrCloud uses a consistency level of ALL, where ALL is based
on the number of *active* replicas for a shard, which could be as
small as one. What I'm proposing is to give a knob to allow a
SolrCloud user to enforce a consistency level of QUORUM, where QUORUM
is based on the entire replica set (down + active replicas) and not
just active replicas as it is today. However, we'll need a better
vocabulary for this because in my scenario, QUORUM is stronger than
ALL which will confuse even the most seasoned of distributed systems
engineers ;-)

Cheers,
Tim


On Tue, Nov 19, 2013 at 9:25 PM, Otis Gospodnetic
otis.gospodne...@gmail.com wrote:
 Btw. isn't the situation Timothy is describing what hinted handoff is all
 about?

 http://wiki.apache.org/cassandra/HintedHandoff
 http://www.datastax.com/dev/blog/modern-hinted-handoff

 Check this:
 http://www.jroller.com/otis/entry/common_distributed_computing_routines

 Otis
 --
 Performance Monitoring * Log Analytics * Search Analytics
 Solr  Elasticsearch Support * http://sematext.com/


 On Tue, Nov 19, 2013 at 1:58 PM, Mark Miller markrmil...@gmail.com wrote:

 Mostly a lot of other systems already offer these types of things, so they
 were hard not to think about while building :) Just hard to get back to a
 lot of those things, even though a lot of them are fairly low hanging
 fruit. Hardening takes the priority :(

 - Mark

 On Nov 19, 2013, at 12:42 PM, Timothy Potter thelabd...@gmail.com wrote:

  You're thinking is always one-step ahead of me! I'll file the JIRA
 
  Thanks.
  Tim
 
 
  On Tue, Nov 19, 2013 at 10:38 AM, Mark Miller markrmil...@gmail.com
 wrote:
 
  Yeah, this is kind of like one of many little features that we have just
  not gotten to yet. I’ve always planned for a param that let’s you say
 how
  many replicas an update must be verified on before responding success.
  Seems to make sense to fail that type of request early if you notice
 there
  are not enough replicas up to satisfy the param to begin with.
 
  I don’t think there is a JIRA issue yet, fire away if you want.
 
  - Mark
 
  On Nov 19, 2013, at 12:14 PM, Timothy Potter thelabd...@gmail.com
 wrote:
 
  I've been thinking about how SolrCloud deals with write-availability
  using
  in-sync replica sets, in which writes will continue to be accepted so
  long
  as there is at least one healthy node per shard.
 
  For a little background (and to verify my understanding of the process
 is
  correct), SolrCloud only considers active/healthy replicas when
  acknowledging a write. Specifically, when a shard leader accepts an
  update
  request, it forwards the request to all active/healthy replicas and
 only
  considers the write successful if all active/healthy replicas ack the
  write. Any down / gone replicas are not considered and will sync up
 with
  the leader when they come back online using peer sync or snapshot
  replication. For instance, if a shard has 3 nodes, A, B, C with A being
  the
  current leader, then writes to the shard will continue to succeed even
  if B
   C are down.
 
  The issue is that if a shard leader continues to accept updates even if
  it
  loses all of its replicas, then we have acknowledged updates on only 1
  node. If that node, call it A, then fails and one of the previous
  replicas,
  call it B, comes back online before A does, then any writes that A
  accepted
  while the other replicas were offline are at risk to being lost.
 
  SolrCloud does provide a safe-guard mechanism for this problem with the
  leaderVoteWait setting, which puts any replicas that come back online
  before node A into a temporary wait state. If A comes back online
 within
  the wait period, then all is well as it will become the leader again
 and
  no
  writes will be lost. As a side note, sys admins definitely need to be
  made
  more aware of this situation as when I first encountered it in my
  cluster,
  I had no idea what it meant.
 
  My question is whether we want to consider an approach where SolrCloud
  will
  not accept writes unless there is a majority of replicas available to
  accept the write? For my example, under this approach, we wouldn't
 accept
  writes if both BC failed, but would if only C did, leaving A  B
 online.
  Admittedly, this lowers the write-availability of the system, so may be
  something that should be tunable? Just wanted to put this out there as
  something I've been thinking about lately ...
 
  Cheers,
  Tim
 
 

Option to enforce a majority quorum approach to accepting updates in SolrCloud?

[Timothy Potter]

I've been thinking about how SolrCloud deals with write-availability using
in-sync replica sets, in which writes will continue to be accepted so long
as there is at least one healthy node per shard.

For a little background (and to verify my understanding of the process is
correct), SolrCloud only considers active/healthy replicas when
acknowledging a write. Specifically, when a shard leader accepts an update
request, it forwards the request to all active/healthy replicas and only
considers the write successful if all active/healthy replicas ack the
write. Any down / gone replicas are not considered and will sync up with
the leader when they come back online using peer sync or snapshot
replication. For instance, if a shard has 3 nodes, A, B, C with A being the
current leader, then writes to the shard will continue to succeed even if B
 C are down.

The issue is that if a shard leader continues to accept updates even if it
loses all of its replicas, then we have acknowledged updates on only 1
node. If that node, call it A, then fails and one of the previous replicas,
call it B, comes back online before A does, then any writes that A accepted
while the other replicas were offline are at risk to being lost.

SolrCloud does provide a safe-guard mechanism for this problem with the
leaderVoteWait setting, which puts any replicas that come back online
before node A into a temporary wait state. If A comes back online within
the wait period, then all is well as it will become the leader again and no
writes will be lost. As a side note, sys admins definitely need to be made
more aware of this situation as when I first encountered it in my cluster,
I had no idea what it meant.

My question is whether we want to consider an approach where SolrCloud will
not accept writes unless there is a majority of replicas available to
accept the write? For my example, under this approach, we wouldn't accept
writes if both BC failed, but would if only C did, leaving A  B online.
Admittedly, this lowers the write-availability of the system, so may be
something that should be tunable? Just wanted to put this out there as
something I've been thinking about lately ...

Cheers,
Tim

Re: Option to enforce a majority quorum approach to accepting updates in SolrCloud?

[Timothy Potter]

I got to thinking about this particular question while watching this
presentation, which is well worth 45 minutes if you can spare it:

http://www.infoq.com/presentations/partitioning-comparison

I created SOLR-5468 for this.


On Tue, Nov 19, 2013 at 11:58 AM, Mark Miller markrmil...@gmail.com wrote:

 Mostly a lot of other systems already offer these types of things, so they
 were hard not to think about while building :) Just hard to get back to a
 lot of those things, even though a lot of them are fairly low hanging
 fruit. Hardening takes the priority :(

 - Mark

 On Nov 19, 2013, at 12:42 PM, Timothy Potter thelabd...@gmail.com wrote:

  You're thinking is always one-step ahead of me! I'll file the JIRA
 
  Thanks.
  Tim
 
 
  On Tue, Nov 19, 2013 at 10:38 AM, Mark Miller markrmil...@gmail.com
 wrote:
 
  Yeah, this is kind of like one of many little features that we have just
  not gotten to yet. I’ve always planned for a param that let’s you say
 how
  many replicas an update must be verified on before responding success.
  Seems to make sense to fail that type of request early if you notice
 there
  are not enough replicas up to satisfy the param to begin with.
 
  I don’t think there is a JIRA issue yet, fire away if you want.
 
  - Mark
 
  On Nov 19, 2013, at 12:14 PM, Timothy Potter thelabd...@gmail.com
 wrote:
 
  I've been thinking about how SolrCloud deals with write-availability
  using
  in-sync replica sets, in which writes will continue to be accepted so
  long
  as there is at least one healthy node per shard.
 
  For a little background (and to verify my understanding of the process
 is
  correct), SolrCloud only considers active/healthy replicas when
  acknowledging a write. Specifically, when a shard leader accepts an
  update
  request, it forwards the request to all active/healthy replicas and
 only
  considers the write successful if all active/healthy replicas ack the
  write. Any down / gone replicas are not considered and will sync up
 with
  the leader when they come back online using peer sync or snapshot
  replication. For instance, if a shard has 3 nodes, A, B, C with A being
  the
  current leader, then writes to the shard will continue to succeed even
  if B
   C are down.
 
  The issue is that if a shard leader continues to accept updates even if
  it
  loses all of its replicas, then we have acknowledged updates on only 1
  node. If that node, call it A, then fails and one of the previous
  replicas,
  call it B, comes back online before A does, then any writes that A
  accepted
  while the other replicas were offline are at risk to being lost.
 
  SolrCloud does provide a safe-guard mechanism for this problem with the
  leaderVoteWait setting, which puts any replicas that come back online
  before node A into a temporary wait state. If A comes back online
 within
  the wait period, then all is well as it will become the leader again
 and
  no
  writes will be lost. As a side note, sys admins definitely need to be
  made
  more aware of this situation as when I first encountered it in my
  cluster,
  I had no idea what it meant.
 
  My question is whether we want to consider an approach where SolrCloud
  will
  not accept writes unless there is a majority of replicas available to
  accept the write? For my example, under this approach, we wouldn't
 accept
  writes if both BC failed, but would if only C did, leaving A  B
 online.
  Admittedly, this lowers the write-availability of the system, so may be
  something that should be tunable? Just wanted to put this out there as
  something I've been thinking about lately ...
 
  Cheers,
  Tim
 
 

Re: Zookeeper down question

[Timothy Potter]

Good questions ... From my understanding, queries will work if Zk goes down
but writes do not work w/o Zookeeper. This works because the clusterstate
is cached on each node so Zookeeper doesn't participate directly in queries
and indexing requests. Solr has to decide not to allow writes if it loses
its connection to Zookeeper, which is a safe guard mechanism. In other
words, Solr assumes it's pretty safe to allow reads if the cluster doesn't
have a healthy coordinator, but chooses to not allow writes to be safe.

If a Solr nodes goes down while ZK is not available, since Solr no longer
accepts writes, leader / replica doesn't really matter. I'd venture to
guess there is some failover logic built in when executing distributing
queries but I'm not as familiar with that part of the code (I'll brush up
on it though as I'm now curious as well).

Cheers,
Tim


On Tue, Nov 19, 2013 at 11:58 AM, Garth Grimm 
garthgr...@averyranchconsulting.com wrote:

 Given a 4 solr node instance (i.e. 2 shards, 2 replicas per shard), and a
 standalone zookeeper.

 Correct me if any of my understanding is incorrect on the following:
 If ZK goes down, most normal operations will still function, since my
 understanding is that ZK isn't involved on a transaction by transaction
 basis for each of these.
 Document adds, updates, and deletes on existing collection will still work
 as expected.
 Queries will still get processed as expected.
 Is the above correct?

 But adding new collections, changing configs, etc., will all fail while ZK
 is down (or at least, place things in an inconsistent state?)
 Is that correct?

 If, while ZK is down, one of the 4 solr nodes also goes down, will all
 normal operations fail?  Will they all continue to succeed?  I.e. will each
 of the nodes realize which node is down and route indexing and query
 requests around them, or is that impossible while ZK is down?  Will some
 queries succeed (because they were lucky enough to get routed to the one
 replica on the one shard that is still functional) while other queries fail
 (they aren't so lucky and get routed to the one replica that is down on the
 one shard)?

 Thanks,
 Garth Grimm

Re: Is Solr can create temporary sub-index ?

[Timothy Potter]

Hi Bruno,

Have you looked into Solr's facet support? If I'm reading your post
correctly, this sounds like the classic case for facets. Each time the user
selects a facet, you add a filter query (fq clause) to the original query.
http://wiki.apache.org/solr/SolrFacetingOverview

Tim


On Wed, Oct 23, 2013 at 8:16 AM, Bruno Mannina bmann...@free.fr wrote:

 Dear Solr User,

 We have to do a new web project which is : Connect our SOLR database to a
 web plateform.

 This Web Plateform will be used by several users at the same time.
 They do requests on our SOLR and they can apply filter on the result.

 i.e.:
 Our SOLR contains 87M docs
 An user do requests, result is around few hundreds to several thousands.
 On the Web Plateform, user will see first 20 results (or more by using
 Next Page button)
 But he will need also to filter the whole result by additional terms.
 (Terms that our plateform will propose him)

 Is SOLR can create temporary index (manage by SOLR himself during a web
 session) ?

 My goal is to not download the whole result on local computer to provide
 filter, or to re-send
 the same request several times added to the new criterias.

 Many thanks for your comment,

 Regards,
 Bruno

Re: Having two document sets in one index, separated by filter query.

[Timothy Potter]

Sounds correct - you probably want to use an invariant parameter in
solrconfig.xml, something along the lines of:

lst name=invariants str name=fqdocset:0/str /lst

Where docset is the new field you add to the schema to determine which set
a document belongs to. You might also consider adding a newSearcher warming
query that includes this fq so that the filter gets cached everytime you
open a new searcher.

On Wed, Oct 23, 2013 at 7:09 AM, Achim Domma do...@procoders.net wrote:

 Hi,

 I have two document sets, both having the same schema. On set is the
 larger reference set (lets say a few hundred thousand documents) and the
 smaller set is some user generated content (a few hundreds or thousands).
 In most cases, I just want to search on the larger reference sets but some
 functionality also works on both sets.

 Is the following assumption correct?

 I could add a field to the schema which defines what type a document is
 (large set or small set). Now I configure two search handlers. For one
 handler I add a filter query, which filters on the just defined type field.
 If I use this handler in my application, I should only see content from the
 large set and it should be impossible to get results from the small set
 back.

 cheers,
 Achim

Re: Is Solr can create temporary sub-index ?

[Timothy Potter]

Yes, absolutely you resend the q= each time, optionally with any facets
selected by the user using fq=


On Wed, Oct 23, 2013 at 10:00 AM, Bruno Mannina bmann...@free.fr wrote:

 Hello Tim,

 Yes solr's facet could be a solution, but I need to re-send the q= each
 time.
 I'm asking me just if an another solution exists.

 Facet seems to be the good solution.

 Bruno



 Le 23/10/2013 17:03, Timothy Potter a écrit :

  Hi Bruno,

 Have you looked into Solr's facet support? If I'm reading your post
 correctly, this sounds like the classic case for facets. Each time the
 user
 selects a facet, you add a filter query (fq clause) to the original query.
 http://wiki.apache.org/solr/**SolrFacetingOverviewhttp://wiki.apache.org/solr/SolrFacetingOverview

 Tim


 On Wed, Oct 23, 2013 at 8:16 AM, Bruno Mannina bmann...@free.fr wrote:

  Dear Solr User,

 We have to do a new web project which is : Connect our SOLR database to a
 web plateform.

 This Web Plateform will be used by several users at the same time.
 They do requests on our SOLR and they can apply filter on the result.

 i.e.:
 Our SOLR contains 87M docs
 An user do requests, result is around few hundreds to several thousands.
 On the Web Plateform, user will see first 20 results (or more by using
 Next Page button)
 But he will need also to filter the whole result by additional terms.
 (Terms that our plateform will propose him)

 Is SOLR can create temporary index (manage by SOLR himself during a web
 session) ?

 My goal is to not download the whole result on local computer to provide
 filter, or to re-send
 the same request several times added to the new criterias.

 Many thanks for your comment,

 Regards,
 Bruno

Need help understanding the use cases behind core auto-discovery

[Timothy Potter]

Trying to add some information about core.properties and auto-discovery in
Solr in Action and am at a loss for what to tell the reader is the purpose
of this feature.

Can anyone point me to any background information about core
auto-discovery? I'm not interested in the technical implementation details.
Mainly I'm trying to understand the motivation behind having this feature
as it seems unnecessary with the Core Admin API. Best I can tell is it
removes a manual step of firing off a call to the Core Admin API or loading
a core from the Admin UI. If that's it and I'm overthinking it, then cool
but was expecting more of an ah-ha moment with this feature ;-)

Any insights you can share are appreciated.

Thanks.
Tim

Re: Need help understanding the use cases behind core auto-discovery

[Timothy Potter]

Exactly the insight I was looking for! Thanks Yonik ;-)


On Fri, Sep 20, 2013 at 10:37 AM, Yonik Seeley yo...@lucidworks.com wrote:

 On Fri, Sep 20, 2013 at 11:56 AM, Timothy Potter thelabd...@gmail.com
 wrote:
  Trying to add some information about core.properties and auto-discovery
 in
  Solr in Action and am at a loss for what to tell the reader is the
 purpose
  of this feature.

 IMO, it was more a removal of unnecessary central configuration.
 You previously had to list the core in solr.xml, and now you don't.
 Cores should be fully self-describing so that it should be easy to
 move them in the future just by moving the core directory (although
 that may not yet work...)

 -Yonik
 http://lucidworks.com

  Can anyone point me to any background information about core
  auto-discovery? I'm not interested in the technical implementation
 details.
  Mainly I'm trying to understand the motivation behind having this feature
  as it seems unnecessary with the Core Admin API. Best I can tell is it
  removes a manual step of firing off a call to the Core Admin API or
 loading
  a core from the Admin UI. If that's it and I'm overthinking it, then cool
  but was expecting more of an ah-ha moment with this feature ;-)
 
  Any insights you can share are appreciated.
 
  Thanks.
  Tim

Re: Data Centre recovery/replication, does this seem plausible?

[Timothy Potter]

I've been thinking about this one too and was curious about using the Solr
Entity support in the DIH to do the import from one DC to another (for the
lost docs). In my mind, one configures the DIH to use the
SolrEntityProcessor with a query to capture the docs in the DC that stayed
online, most likely using a timestamp in the query (see:
http://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor).

Would that work? If so, any downsides? I've only used DIH /
SolrEntityProcessor to populate a staging / dev environment from prod but
have had good success with it.

Thanks.
Tim


On Wed, Aug 28, 2013 at 6:59 AM, Erick Erickson erickerick...@gmail.comwrote:

 The separate DC problem has been lurking for a while. But your
 understanding it a little off. When a replica discovers that
 it's too far out of date, it does an old-style replication. IOW, the
 tlog doesn't contain the entire delta. Eventually, the old-style
 replications catch up to close enough and _then_ the remaining
 docs in the tlog are replayed. The target number of updates in the
 tlog is 100 so it's a pretty small window that's actually replayed in
 the normal case.

 None of which helps your problem. The simplest way (and on the
 expectation that DC outages were pretty rare!) would be to have your
 indexing process fire the missed updates at the DC after it came
 back up.

 Copying from one DC to another is tricky. You'd have to be very,
 very sure that you copied indexes to the right shard. Ditto for any
 process that tried to have, say, a single node from the recovering
 DC temporarily join the good DC, at least long enough to synch.

 Not a pretty problem, we don't really have any best practices yet
 that I know of.

 FWIW,
 Erick


 On Wed, Aug 28, 2013 at 8:13 AM, Daniel Collins danwcoll...@gmail.com
 wrote:

  We have 2 separate data centers in our organisation, and in order to
  maintain the ZK quorum during any DC outage, we have 2 separate Solr
  clouds, one in each DC with separate ZK ensembles but both are fed with
 the
  same indexing data.
 
  Now in the event of a DC outage, all our Solr instances go down, and when
  they come back up, we need some way to recover the lost data.
 
  Our thought was to replicate from the working DC, but is there a way to
 do
  that whilst still maintaining an online presence for indexing purposes?
 
  In essence, we want to do what happens within Solr cloud's recovery, so
 (as
  I understand cloud recovery) a node starts up, (I'm assuming worst case
 and
  peer sync has failed) then buffers all updates into the transaction log,
  replicates from the leader, and replays the transaction log to get
  everything in sync.
 
  Is it conceivable to do the same by extending Solr, so on the activation
 of
  some handler (user triggered), we initiated a replicate from other DC,
  which puts all the leaders into buffering updates, replicate from some
  other set of servers and then replay?
 
  Our goal is to try to minimize the downtime (beyond the initial outage),
 so
  we would ideally like to be able to start up indexing before this
  replicate/clone has finished, that's why I thought to enable buffering
 on
  the transaction log.  Searches shouldn't be sent here, but if they do we
  have a valid (albeit old) index to serve those until the new one swaps
 in.
 
  Just curious how any other DC-aware setups handle this kind of scenario?
   Or other concerns, issues with this type of approach.
 

Re: Trying to determine the benefit of spellcheck-based suggester vs. using terms component?

[Timothy Potter]

Thanks for the reply Erick. I'm looking for type-ahead support; using
spell checking too via the DirectSolrSpellChecker. Seems like the
spell check based suggester is designed for type-head or am I not
understanding something? Here's my config:

requestHandler
class=org.apache.solr.handler.component.SearchHandler
name=/suggest
lst name=defaults
str name=echoParamsexplicit/str
str name=wtjson/str
str name=indenttrue/str
str name=dfsuggest/str
str name=spellchecktrue/str
str name=spellcheck.dictionarysuggestDictionary/str
str name=spellcheck.onlyMorePopulartrue/str
str name=spellcheck.count5/str
str name=spellcheck.collatefalse/str
/lst
arr name=components
strsuggest/str
/arr
/requestHandler

searchComponent class=solr.SpellCheckComponent name=suggest
lst name=spellchecker
str name=namesuggestDictionary/str
str
name=classnameorg.apache.solr.spelling.suggest.Suggester/str
str
name=lookupImplorg.apache.solr.spelling.suggest.tst.TSTLookup/str
str name=fieldsuggest/str
float name=threshold0./float
str name=buildOnCommittrue/str
/lst
/searchComponent

I was confused why this approach was needed because using terms
component is so easy and doesn't require any build step. From your
answer, it seems like either approach is valid in Solr 4.4 but the
spellcheck based suggester has more knobs, such as loading in an
external dictionary in addition to data in my index, etc.

Cheers,
Tim


On Wed, Jul 31, 2013 at 5:08 AM, Erick Erickson erickerick...@gmail.com wrote:
 The biggest thing is that the spellchecker has lots of knobs
 to tune, all the stuff in
 http://wiki.apache.org/solr/SpellCheckComponent#spellcheck.collate

 TermsComponent, on the other hand, just gives you
 what's in the index with essentially no knobs to tune.

 So it depends on your goal. Typeahead or spelling
 correction? In the first case I'd go for TermsComponent
 and the second spell check as an example.

 Best
 Erick

 On Tue, Jul 30, 2013 at 2:07 PM, Timothy Potter thelabd...@gmail.com wrote:
 Going over the comments in SOLR-1316, I seemed to have lost the
 forrest for the trees. What is the benefit of using the spellcheck
 based suggester over something like the terms component to get
 suggestions as the user types?

 Maybe it is faster because it builds the in-memory data structure on
 commit? Seems like the terms component is pretty fast too.

 I'd appreciate any additional insights about this. There are so many
 solutions to auto-suggest for Solr, it's hard to decide what
 approach to take.

 Cheers,
 Tim

Trying to determine the benefit of spellcheck-based suggester vs. using terms component?

[Timothy Potter]

Going over the comments in SOLR-1316, I seemed to have lost the
forrest for the trees. What is the benefit of using the spellcheck
based suggester over something like the terms component to get
suggestions as the user types?

Maybe it is faster because it builds the in-memory data structure on
commit? Seems like the terms component is pretty fast too.

I'd appreciate any additional insights about this. There are so many
solutions to auto-suggest for Solr, it's hard to decide what
approach to take.

Cheers,
Tim

Re: Solr Cloud Questions

[Timothy Potter]

1) Depends on your document routing strategy. It sounds like you could
be using the compositeId strategy and if so, there's still a hash
range assigned to each shard, so you can split the big shards into
smaller shards.

2) Since you're replicating in 2 places, when one of your servers
crash, there will still be one copy of the shard. When the new server
comes online, you should specify -Dshard=ID and the data will be
pulled over from the healthy replica using good ol' snapshot
replication. btw: ID is the shard Id of the server that crashed

3) No comment on this ;-) Post back if you find a good one.

On Tue, Jul 30, 2013 at 2:46 PM, AdityaR aditya.ravinuth...@gmail.com wrote:
 Hi,

 I have created a solr cloud and have started using it , I have few questions
 regarding the set up and it would be really helpful if someone can answer
 these.

 Use Case: We have many clients and each clients data is in his own
 collection, we currently have 10 server cloud and have distributed clients
 on them ( 5 shards for each collection and replicationFactor =2)

 1) Since we are adding new clients, it will only be matter of time before we
 have no space on the 10 servers, when this happens what options do I have?

 2) Lets say one of the 10 servers crashes and had to be replaced by a
 different server, if I add a new server into the cloud will this new server
 have the same data as the one that crashed. Meaning does the data get moved
 into the new server.

 3) Are there any monitoring tools for monitoring solr cloud, I have looked
 at SPM by sematext and new relic , but am having issues with both the tools.

 Thanks,
 Aditya



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-Cloud-Questions-tp4081441.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Does Solr continue to processing queries and index documents during a collection reload?

[Timothy Potter]

Quick behavior check on whether Solr continues to process queries and
index documents during a collection reload?

For example, after I upload new config documents to Zookeeper, I issue
a reload command using the collections API. Of course this propagates
a core reload across all nodes in the collection, which re-open
searchers, etc. This can be expensive in my environment so I'm curious
if the soon-to-be-closed core stays active until the new core is open?
I assume yes but want to make sure.

Thanks.
Tim

Re: Solr 4.3.1 - SolrCloud nodes down and lost documents

[Timothy Potter]

Log messages?

On Wed, Jul 24, 2013 at 1:37 AM, Neil Prosser neil.pros...@gmail.com wrote:
 Great. Thanks for your suggestions. I'll go through them and see what I can
 come up with to try and tame my GC pauses. I'll also make sure I upgrade to
 4.4 before I start. Then at least I know I've got all the latest changes.

 In the meantime, does anyone have any idea why I am able to get leaders who
 are marked as down? I've just had the situation where of two nodes hosting
 replicas of the same shard the leader was alive and marked as down and the
 other replica was gone. I could perform searches directly on the two nodes
 (with distrib=false) and once I'd restarted the node which was down the
 leader sprung into live. I assume that since there was a change in
 clusterstate.json it forced the leader to reconsider what it was up to.

 Does anyone know the hole my nodes are falling into? Is it likely to be
 tied up in my GC woes?


 On 23 July 2013 13:06, Otis Gospodnetic otis.gospodne...@gmail.com wrote:

 Hi,

 On Tue, Jul 23, 2013 at 8:02 AM, Erick Erickson erickerick...@gmail.com
 wrote:
  Neil:
 
  Here's a must-read blog about why allocating more memory
  to the JVM than Solr requires is a Bad Thing:
  http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
 
  It turns out that you actually do yourself harm by allocating more
  memory to the JVM than it really needs. Of course the problem is
  figuring out how much it really needs, which if pretty tricky.
 
  Your long GC pauses _might_ be ameliorated by allocating _less_
  memory to the JVM, counterintuitive as that seems.

 or by using G1 :)

 See http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/

 Otis
 --
 Solr  ElasticSearch Support -- http://sematext.com/
 Performance Monitoring -- http://sematext.com/spm


  On Mon, Jul 22, 2013 at 5:05 PM, Neil Prosser neil.pros...@gmail.com
 wrote:
  I just have a little python script which I run with cron (luckily that's
  the granularity we have in Graphite). It reads the same JSON the admin
 UI
  displays and dumps numeric values into Graphite.
 
  I can open source it if you like. I just need to make sure I remove any
  hacks/shortcuts that I've taken because I'm working with our cluster!
 
 
  On 22 July 2013 19:26, Lance Norskog goks...@gmail.com wrote:
 
  Are you feeding Graphite from Solr? If so, how?
 
 
  On 07/19/2013 01:02 AM, Neil Prosser wrote:
 
  That was overnight so I was unable to track exactly what happened (I'm
  going off our Graphite graphs here).
 
 
 

Can the admins of this list please boot wired...@yahoo.com

[Timothy Potter]

Apologize if this is not the correct way to request mailing list admin
support but it's pretty clear that wired...@yahoo.com is spamming this
list and should be booted out.

Tim

Re: Solr 4.3.1 - SolrCloud nodes down and lost documents

[Timothy Potter]

One thing I'm seeing in your logs is the leaderVoteWait safety
mechanism that I mentioned previously:


2013-07-24 07:06:19,856 INFO  o.a.s.c.ShardLeaderElectionContext -
Waiting until we see more replicas up: total=2 found=1 timeoutin=45792


From Mark M: This is a safety mechanism - you can turn it off by
configuring leaderVoteWait to 0 in solr.xml. This is meant to protect
the case where you stop a shard or it fails and then the first node to
get started back up has stale data - you don't want it to just become
the leader. So we wait to see everyone we know about in the shard up
to 3 or 5 min by default. Then we know all the shards participate in
the leader election and the leader will end up with all updates it
should have. You can lower that wait or turn it off with 0.

NOTE: I tried setting it to 0 and my cluster went haywire, so consider
just lowering it but not making it zero ;-)


From what I can tell, server04 heads into this leaderVoteWait loop
before it declares success as the leader which is why it is registered
as down. Down is different than gone, so it's likely it can
respond to queries.



On Wed, Jul 24, 2013 at 10:33 AM, Neil Prosser neil.pros...@gmail.com wrote:
 Sorry, good point...

 https://gist.github.com/neilprosser/d75a13d9e4b7caba51ab

 I've included the log files for two servers hosting the same shard for the
 same time period. The logging settings exclude anything below WARN
 for org.apache.zookeeper, org.apache.solr.core.SolrCore
 and org.apache.solr.update.processor.LogUpdateProcessor. That said, there's
 still a lot of spam there.

 The log for server09 starts with it throwing OutOfMemoryErrors. At this
 point I externally have it listed as recovering. Unfortunately I haven't
 got the GC logs for either box in that time period.

 The key times that I know of are:

 2013-07-24 07:14:08,560 - server04 registers its state as down.
 2013-07-24 07:17:38,462 - server04 says it's the new leader (this ties in
 with my external Graphite script observing that at 07:17 server04 was both
 leader and down).
 2013-07-24 07:31:21,667 - I get involved and server09 is restarted.
 2013-07-24 07:31:42,408 - server04 updates its cloud state from ZooKeeper
 and realises that it's the leader.
 2013-07-24 07:31:42,449 - server04 registers its state as active.

 I'm sorry there's so much there. I'm still getting used to what's important
 for people. Both servers were running 4.3.1. I've since upgraded to 4.4.0.

 If you need any more information or want me to do any filtering let me know.



 On 24 July 2013 15:50, Timothy Potter thelabd...@gmail.com wrote:

 Log messages?

 On Wed, Jul 24, 2013 at 1:37 AM, Neil Prosser neil.pros...@gmail.com
 wrote:
  Great. Thanks for your suggestions. I'll go through them and see what I
 can
  come up with to try and tame my GC pauses. I'll also make sure I upgrade
 to
  4.4 before I start. Then at least I know I've got all the latest changes.
 
  In the meantime, does anyone have any idea why I am able to get leaders
 who
  are marked as down? I've just had the situation where of two nodes
 hosting
  replicas of the same shard the leader was alive and marked as down and
 the
  other replica was gone. I could perform searches directly on the two
 nodes
  (with distrib=false) and once I'd restarted the node which was down the
  leader sprung into live. I assume that since there was a change in
  clusterstate.json it forced the leader to reconsider what it was up to.
 
  Does anyone know the hole my nodes are falling into? Is it likely to be
  tied up in my GC woes?
 
 
  On 23 July 2013 13:06, Otis Gospodnetic otis.gospodne...@gmail.com
 wrote:
 
  Hi,
 
  On Tue, Jul 23, 2013 at 8:02 AM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
   Neil:
  
   Here's a must-read blog about why allocating more memory
   to the JVM than Solr requires is a Bad Thing:
  
 http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
  
   It turns out that you actually do yourself harm by allocating more
   memory to the JVM than it really needs. Of course the problem is
   figuring out how much it really needs, which if pretty tricky.
  
   Your long GC pauses _might_ be ameliorated by allocating _less_
   memory to the JVM, counterintuitive as that seems.
 
  or by using G1 :)
 
  See http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/
 
  Otis
  --
  Solr  ElasticSearch Support -- http://sematext.com/
  Performance Monitoring -- http://sematext.com/spm
 
 
   On Mon, Jul 22, 2013 at 5:05 PM, Neil Prosser neil.pros...@gmail.com
 
  wrote:
   I just have a little python script which I run with cron (luckily
 that's
   the granularity we have in Graphite). It reads the same JSON the
 admin
  UI
   displays and dumps numeric values into Graphite.
  
   I can open source it if you like. I just need to make sure I remove
 any
   hacks/shortcuts that I've taken because I'm working with our cluster!
  
  
   On 22 July 2013 19:26, Lance Norskog goks

Re: Node down, but not out

[Timothy Potter]

Hi Jim,

Based on our discussion, I cooked up this solution for my book Solr in
Action and would appreciate you looking it over to see if it meets
your needs. The basic idea is to extend Solr's built-in
PingRequestHandler to verify a replica is connected to Zookeeper and
is in the active state. To enable this, install the custom JAR and
then update your solrconfig.xml to use this class instead of the
built-in one for the /admin/ping request handler:

requestHandler name=/admin/ping
class=sia.ch13.ClusterStateAwarePingRequestHandler



 Code 

package sia.ch13;

import org.apache.solr.cloud.CloudDescriptor;
import org.apache.solr.cloud.ZkController;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.cloud.ClusterState;
import org.apache.solr.common.cloud.Slice;
import org.apache.solr.core.CoreContainer;
import org.apache.solr.core.CoreDescriptor;
import org.apache.solr.core.SolrCore;
import org.apache.solr.handler.PingRequestHandler;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.response.SolrQueryResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Extends Solr's PingRequestHandler to check a replica's cluster
status as part of the health check.
 */
public class ClusterStateAwarePingRequestHandler extends PingRequestHandler {

public static Logger log =
LoggerFactory.getLogger(ClusterStateAwarePingRequestHandler.class);

@Override
public void handleRequestBody(SolrQueryRequest solrQueryRequest,
SolrQueryResponse solrQueryResponse) throws Exception {
// delegate to the base class to check the status of this local index
super.handleRequestBody(solrQueryRequest, solrQueryResponse);

// if ping status is OK, then check cluster state of this core
if (OK.equals(solrQueryResponse.getValues().get(status))) {
verifyThisReplicaIsActive(solrQueryRequest.getCore());
}
}

/**
 * Verifies this replica is active.
 */
protected void verifyThisReplicaIsActive(SolrCore solrCore) throws
SolrException {
String replicaState = unknown;
String nodeName = ?;
String shardName = ?;
String collectionName = ?;
String role = ?;
Exception exc = null;
try {
CoreDescriptor coreDescriptor = solrCore.getCoreDescriptor();
CoreContainer coreContainer = coreDescriptor.getCoreContainer();
CloudDescriptor cloud = coreDescriptor.getCloudDescriptor();

shardName = cloud.getShardId();
collectionName = cloud.getCollectionName();
role = (cloud.isLeader() ? Leader : Replica);

ZkController zkController = coreContainer.getZkController();
if (zkController != null) {
nodeName = zkController.getNodeName();
if (zkController.isConnected()) {
ClusterState clusterState = zkController.getClusterState();
Slice slice =
clusterState.getSlice(collectionName, shardName);
replicaState = (slice != null) ? slice.getState() : gone;
} else {
replicaState = not connected to Zookeeper;
}
} else {
replicaState = Zookeeper not enabled/configured;
}
} catch (Exception e) {
replicaState = error determining cluster state;
exc = e;
}

if (active.equals(replicaState)) {
log.info(String.format(%s at %s for %s in the %s
collection is active.,
role, nodeName, shardName, collectionName));
} else {
// fail the ping by raising an exception
String errMsg = String.format(%s at %s for %s in the %s
collection is not active! State is: %s,
role, nodeName, shardName, collectionName, replicaState);
if (exc != null) {
throw new
SolrException(SolrException.ErrorCode.SERVER_ERROR, errMsg, exc);
} else {
throw new
SolrException(SolrException.ErrorCode.SERVER_ERROR, errMsg);
}
}
}
}

On Tue, Jul 23, 2013 at 1:46 PM, jimtronic jimtro...@gmail.com wrote:
 I think the best bet here would be a ping like handler that would simply
 return the state of only this box in the cluster:

 Something like /admin/state which would return
 down,active,leader,recovering

 I'm not really sure where to begin however. Any ideas?

 jim

 On Mon, Jul 22, 2013 at 12:52 PM, Timothy Potter [via Lucene] 
 ml-node+s472066n4079518...@n3.nabble.com wrote:

 There is but I couldn't get it to work in my environment on Jetty, see:


 http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201306.mbox/%3CCAJt9Wnib+p_woYODtrSPhF==v8Vx==mDBd_qH=x_knbw-BnPXQ@...%3Ehttp://mail-archives.apache.org/mod_mbox/lucene-solr-user/201306.mbox/%3CCAJt9Wnib+p_woYODtrSPhF==v8Vx==mDBd_qH=x_knbw-bn...@mail.gmail.com%3E

 Let me know if you have any

Re: Start independent Zookeeper from within Solr install

[Timothy Potter]

Curious what the use case is for this? Zookeeper is not an HTTP
service so loading it in Jetty by itself doesn't really make sense. I
also think this creates more work for the Solr team especially since
setting up a production ensemble shouldn't take more than a few
minutes once you have the nodes provisioned.

On Tue, Jul 23, 2013 at 7:05 AM, Upayavira u...@odoko.co.uk wrote:
 Assumptions:

  * you currently have two choices to start Zookeeper: run it embedded
  within Solr, or download it from the ZooKeeper site and start it
  independently.
  * everything you need to run ZooKeeper (embedded or not) is included
  within the Solr distribution

 Assuming I've got the above right, then currently starting an embedded
 ZooKeeper is easy (-DzkRun), and starting an ensemble is irritatingly
 complex.

 So, my question is, how hard would it be to start Zookeeper without
 Solr, but from within the Solr codebase? -DensembleOnly or some such,
 causes Solr not to load, but Zookeeper still starts. I'm assuming that
 Jetty would still listen on port 8983, but it wouldn't initialise the
 Solr webapp:

 java -DzkRun -DzkEnsembleOnly
 -DzkHosts=zkhost01:9983,zkhost02:9983,zkhost03:9983 -jar start.jar

 Is this possible? If it is, I'm happy to have a go at making it happen.

 Upayavira

Re: Processing a lot of results in Solr

[Timothy Potter]

Hi Matt,

This feature is commonly known as deep paging and Lucene and Solr have
issues with it ... take a look at
http://solr.pl/en/2011/07/18/deep-paging-problem/ as a potential
starting point using filters to bucketize a result set into sets of
sub result sets.

Cheers,
Tim

On Tue, Jul 23, 2013 at 3:04 PM, Matt Lieber mlie...@impetus.com wrote:
 Hello Solr users,

 Question regarding processing a lot of docs returned from a query; I
 potentially have millions of documents returned back from a query. What is
 the common design to deal with this ?

 2 ideas I have are:
 - create a client service that is multithreaded to handled this
 - Use the Solr pagination to retrieve a batch of rows at a time (start,
 rows in Solr Admin console )

 Any other ideas that I may be missing ?

 Thanks,
 Matt


 






 NOTE: This message may contain information that is confidential, proprietary, 
 privileged or otherwise protected by law. The message is intended solely for 
 the named addressee. If received in error, please destroy and notify the 
 sender. Any use of this email is prohibited when received in error. Impetus 
 does not represent, warrant and/or guarantee, that the integrity of this 
 communication has been maintained nor that the communication is free of 
 errors, virus, interception or interference.

Re: Problem instatanting a ValueSourceParser plugin in 4.3.1

[Timothy Potter]

I saw something similar and used an absolute path to my JAR file in
solrconfig.xml vs. a relative path and it resolved the issue for me.
Not elegant but worth trying, at least to rule that out.


Tim

On Mon, Jul 22, 2013 at 7:51 AM, Abeygunawardena, Niran
niran.abeygunaward...@proquest.co.uk wrote:
 Hi,

 I'm trying to migrate to Solr 4.3.1 from Solr 4.0.0. I have a Solr Plugin 
 which extends ValueSourceParser and it works under Solr 4.0.0 but it does not 
 work under Solr 4.3.1. I compiled the plugin using the solr-4.3.1*.jars and 
 lucene-4.3.1*.jars but I get the following stacktrace error when starting up 
 a core referencing this plugin...seen below. Does anyone know why it might be 
 giving me a ClassCastException under 4.3.1?

 Thanks,
 Niran

 2458 [coreLoadExecutor-3-thread-2] ERROR org.apache.solr.core.CoreContainer   
 Unable to create core: example_core
 org.apache.solr.common.SolrException: Error Instantiating ValueSourceParser, 
 com.example.HitsValueSourceParser failed to instanti
 ate org.apache.solr.search.ValueSourceParser
 at org.apache.solr.core.SolrCore.init(SolrCore.java:821)
 at org.apache.solr.core.SolrCore.init(SolrCore.java:618)
 at 
 org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
 at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
 at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: org.apache.solr.common.SolrException: Error Instantiating 
 ValueSourceParser, com.example.HitsValueSourceParser failed
 to instantiate org.apache.solr.search.ValueSourceParser
 at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:539)
 at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:575)
 at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2088)
 at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2082)
 at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2115)
 at 
 org.apache.solr.core.SolrCore.initValueSourceParsers(SolrCore.java:2027)
 at org.apache.solr.core.SolrCore.init(SolrCore.java:749)
 ... 13 more
 Caused by: java.lang.ClassCastException: class 
 com.example.HitsValueSourceParser
 at java.lang.Class.asSubclass(Unknown Source)
 at 
 org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:448)
 at 
 org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:396)
 at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:518)
 ... 19 more
 2466 [coreLoadExecutor-3-thread-2] ERROR org.apache.solr.core.CoreContainer   
 null:org.apache.solr.common.SolrException: Unable to create core: example_core
 at 
 org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1450)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:993)
 at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
 at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
 Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
 Caused by: org.apache.solr.common.SolrException: Error Instantiating 
 ValueSourceParser, com.example.HitsValueSourceParser failed
 to instantiate org.apache.solr.search.ValueSourceParser
 at org.apache.solr.core.SolrCore.init(SolrCore.java:821)
 at org.apache.solr.core.SolrCore.init(SolrCore.java:618)
 at 
 org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
 ... 10 more
 Caused by: org.apache.solr.common.SolrException: Error Instantiating 
 ValueSourceParser, com.example.HitsValueSourceParser failed
 to instantiate org.apache.solr.search.ValueSourceParser
   

Re: Solr 4.3.1 - SolrCloud nodes down and lost documents

[Timothy Potter]

A couple of things I've learned along the way ...

I had a similar architecture where we used fairly low numbers for
auto-commits with openSearcher=false. This keeps the tlog to a
reasonable size. You'll need something on the client side to send in
the hard commit request to open a new searcher every N docs or M
minutes.

Be careful with raising the Zk timeout as that also determines how
quickly Zk can detect a node has crashed (afaik). In other words, it
takes the zk client timeout seconds for Zk to consider an ephemeral
znode as gone, so I caution you in increasing this value too much.

The other thing to be aware of is this leaderVoteWait safety mechanism
... might see log messages that look like:

2013-06-24 18:12:40,408 [coreLoadExecutor-4-thread-1] INFO
solr.cloud.ShardLeaderElectionContext  - Waiting until we see more
replicas up: total=2 found=1 timeoutin=139368

From Mark M: This is a safety mechanism - you can turn it off by
configuring leaderVoteWait to 0 in solr.xml. This is meant to protect
the case where you stop a shard or it fails and then the first node to
get started back up has stale data - you don't want it to just become
the leader. So we wait to see everyone we know about in the shard up
to 3 or 5 min by default. Then we know all the shards participate in
the leader election and the leader will end up with all updates it
should have. You can lower that wait or turn it off with 0.

NOTE: I tried setting it to 0 and my cluster went haywire, so consider
just lowering it but not making it zero ;-)

Max heap of 8GB seems overly large to me for 8M docs per shard esp.
since you're using MMapDirectory to cache the primary data structures
of your index in OS cache. I have run shards with 40M docs with 6GB
max heap and chose to have more aggressive cache eviction by using a
smallish LFU filter cache. This approach seems to spread the cost of
GC out over time vs. massive amounts of clean-up when a new searcher
is opened. With 8M docs, each cached filter will require about 1M of
memory, so it seems like you could run with a smaller heap. I'm not a
GC expert but found that having smaller heap and more aggressive cache
evictions reduced full GC's (and how long they run for) on my Solr
instances.

On Mon, Jul 22, 2013 at 8:09 AM, Shawn Heisey s...@elyograg.org wrote:
 On 7/22/2013 6:45 AM, Markus Jelsma wrote:
 You should increase your ZK time out, this may be the issue in your case. 
 You may also want to try the G1GC collector to keep STW under ZK time out.

 When I tried G1, the occasional stop-the-world GC actually got worse.  I
 tried G1 after trying CMS with no other tuning parameters.  The average
 GC time went down, but when it got into a place where it had to do a
 stop-the-world collection, it was worse.

 Based on the GC statistics in jvisualvm and jstat, I didn't think I had
 a problem.  The way I discovered that I had a problem was by looking at
 my haproxy load balancer -- sometimes requests would be sent to a backup
 server instead of my primary, because the ping request handler was
 timing out on the LB health check.  The LB was set to time out after
 five seconds.  When I went looking deeper with the GC log and some other
 tools, I was seeing 8-10 second GC pauses.  G1 was showing me pauses of
 12 seconds.

 Now I use a heavily tuned CMS config, and there are no more LB switches
 to a backup server.  I've put some of my own information about my GC
 settings on my personal Solr wiki page:

 http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning

 I've got an 8GB heap on my systems running 3.5.0 (one copy of the index)
 and a 6GB heap on those running 4.2.1 (the other copy of the index).

 Summary: Just switching to the G1 collector won't solve GC pause
 problems.  There's not a lot of G1 tuning information out there yet.  If
 someone can come up with a good set of G1 tuning parameters, G1 might
 become better than CMS.

 Thanks,
 Shawn

Re: Node down, but not out

[Timothy Potter]

Why was it down? e.g. did it OOM? If so, the recommended approach is
kill the process on OOM vs. leaving it in the cluster in a zombie
state. I had similar issues when my nodes OOM'd is why I ask. That
said, you can get the /clusterstate.json which contains Zk's status of
a node using a request like:
http://localhost:8983/solr/zookeeper?detail=truepath=%2Fclusterstate.json
Although that would require some basic JSON processing to dig into the
response to get the status of the node of interest, so you may want to
implement a custom request handler.

On Mon, Jul 22, 2013 at 9:55 AM, jimtronic jimtro...@gmail.com wrote:
 I've run into a problem recently that's difficult to debug and search for:

 I have three nodes in a cluster and this weekend one of the nodes went
 partially down. It no longer responds to distributed updates and it is
 marked as GONE in the Cloud view of the admin screen. That's not ideal, but
 there's still two boxes up so not the end of the world.

 The problem is that it is still responding to ping requests and returning
 queries successfully. In my setup, I have the three servers on an haproxy
 load balancer so that I can distribute requests and have clients stick to a
 specific solr box. Because the bad node is still returning OK to the ping
 requests and still returns results for simple queries, the load balancer
 does not remove it from the group.

 Is there a ping like request handler that would tell me whether the given
 box I'm hitting is still in the cloud?

 Thanks!
 Jim Musil



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Node-down-but-not-out-tp4079495.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Node down, but not out

[Timothy Potter]

There is but I couldn't get it to work in my environment on Jetty, see:

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201306.mbox/%3CCAJt9Wnib+p_woYODtrSPhF==v8Vx==mDBd_qH=x_knbw-bn...@mail.gmail.com%3E

Let me know if you have any better luck. I had to resort to something
hacky but was out of time I could devote to such unproductive
endeavors ;-)

On Mon, Jul 22, 2013 at 10:49 AM, jimtronic jimtro...@gmail.com wrote:
 I'm not sure why it went down exactly -- I restarted the process and lost the
 logs. (d'oh!)

 An OOM seems likely, however. Is there a setting for killing the processes
 when solr encounters an OOM?

 Thanks!

 Jim



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Node-down-but-not-out-tp4079495p4079507.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Ability to specify the server where shard splits go?

[Timothy Potter]

This is not a problem per se, just want to verify that we're not able
to specify which server shard splits are created as of 4.3.1? From
what I've seen, the new cores for the sub-shards are created on the
leader of the shard being split.

Of course it's easy enough to migrate the new sub-shards to another
node after the fact especially since replication occurs automatically
for the splits.

Seems like if the shard being split is large enough that doing the
split on the same node could cause some resource issues so might be
better to do the split on another server. Or is my assumption that the
split operation is pretty expensive incorrect?

Lastly, also seems like we don't have control over where the replicas
of the split shards go?

Cheers,
Tim

Re: Ability to specify the server where shard splits go?

[Timothy Potter]

Ok, thanks for the answer Yonik. After looking closer at the index
splitting code, definitely seems like you wouldn't want to pay the
network I/O cost when creating the sub-shard indexes. Might be cool to
be able to specify a different local disk path for the new cores so
that we can get some extra disks working in parallel during the split
(icing on the cake of course).

Cheers,
Tim

On Wed, Jul 17, 2013 at 10:40 AM, Yonik Seeley yo...@lucidworks.com wrote:
 On Wed, Jul 17, 2013 at 12:26 PM, Timothy Potter thelabd...@gmail.com wrote:
 This is not a problem per se, just want to verify that we're not able
 to specify which server shard splits are created as of 4.3.1? From
 what I've seen, the new cores for the sub-shards are created on the
 leader of the shard being split.

 Of course it's easy enough to migrate the new sub-shards to another
 node after the fact especially since replication occurs automatically
 for the splits.

 Seems like if the shard being split is large enough that doing the
 split on the same node could cause some resource issues so might be
 better to do the split on another server. Or is my assumption that the
 split operation is pretty expensive incorrect?

 I think it will be mostly IO - it may or may not be expensive
 depending on how IO bound your box already is.

 Splitting directly to a different servers would be cool, but would
 seem to require some sort of Directory implementation that streams
 things over the network rather than just locally store on disk.  It's
 something I think we want in the future, but was a bit too much to
 bite off for the first iteration of this feature.

 Lastly, also seems like we don't have control over where the replicas
 of the split shards go?

 Seems like a good idea to optionally allow this...

 -Yonik
 http://lucidworks.com

OOM killer script woes

[Timothy Potter]

Recently upgraded to 4.3.1 but this problem has persisted for a while now ...

I'm using the following configuration when starting Jetty:

-XX:OnOutOfMemoryError=/home/solr/oom_killer.sh 83 %p

If an OOM is triggered during Solr web app initialization (such as by
me lowering -Xmx to a value that is too low to initialize Solr with),
then the script gets called and does what I expect!

However, once the Solr webapp initializes and Solr is happily
responding to updates and queries. When an OOM occurs in this
situation, then the script doesn't actually get invoked! All I see is
the following in the stdout/stderr log of my process:

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError=/home/solr/oom_killer.sh 83 %p
#   Executing /bin/sh -c /home/solr/oom_killer.sh 83 21358...

The oom_killer.sh script doesn't actually get called!

So to recap, it works if an OOM occurs during initialization but once
Solr is running, the OOM killer doesn't fire correctly. This leads me
to believe my script is fine and there's something else going wrong.
Here's the oom_killer.sh script (pretty basic):

#!/bin/bash
SOLR_PORT=$1
SOLR_PID=$2
NOW=$(date +%Y%m%d_%H%M)
(
echo Running OOM killer script for process $SOLR_PID for Solr on port
89$SOLR_PORT
kill -9 $SOLR_PID
echo Killed process $SOLR_PID
exec /home/solr/solr-dg/dg-solr.sh recover $SOLR_PORT 
echo Restarted Solr on 89$SOLR_PORT after OOM
) | tee oom_killer-89$SOLR_PORT-$NOW.log

Anyone see anything like this before? Suggestions on where to begin
tracking down this issue?

Cheers,
Tim

Re: OOM killer script woes

[Timothy Potter]

Thanks for the feedback Daniel ... For now, I've opted to just kill
the JVM with System.exit(1) in the SolrDispatchFilter code and will
restart it with a Linux supervisor. Not elegant but the alternative of
having a zombie Solr instance walking around my cluster is much worse
;-) Will try to dig into the code that is trapping this error but for
now I've lost too many hours on this problem.

Cheers,
Tim

On Wed, Jun 26, 2013 at 2:43 PM, Daniel Collins danwcoll...@gmail.com wrote:
 Ooh, I guess Jetty is trapping that java.lang.OutOfMemoryError, and
 throwing it/packaging it as a java.lang.RuntimeException.  The -XX option
 assumes that the application doesn't handle the Errors and so they would
 reach the JVM and thus invoke the handler.
 Since Jetty has an exception handler that is dealing with anything
 (included Errors), they never reach the JVM, hence no handler.

 Not much we can do short of not using Jetty?

 That's a pain, I'd just written a nice OOM handler too!


 On 26 June 2013 20:37, Timothy Potter thelabd...@gmail.com wrote:

 A little more to this ...

 Just on chance this was a weird Jetty issue or something, I tried with
 the latest 9 and the problem still occurs :-(

 This is on Java 7 on debian:

 java version 1.7.0_21
 Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
 Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)

 Here is an example stack trace from the log

 2013-06-26 19:31:33,801 [qtp632640515-62] ERROR
 solr.servlet.SolrDispatchFilter Q:22 -
 null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap
 space
 at
 org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:670)
 at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:380)
 at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
 at
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1423)
 at
 org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:450)
 at
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
 at
 org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564)
 at
 org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)
 at
 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1083)
 at
 org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
 at
 org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
 at
 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
 at
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
 at
 org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:258)
 at
 org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
 at
 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
 at org.eclipse.jetty.server.Server.handle(Server.java:445)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
 at
 org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
 at
 org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
 at
 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596)
 at
 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527)
 at java.lang.Thread.run(Thread.java:722)
 Caused by: java.lang.OutOfMemoryError: Java heap space

 On Wed, Jun 26, 2013 at 12:27 PM, Timothy Potter thelabd...@gmail.com
 wrote:
  Recently upgraded to 4.3.1 but this problem has persisted for a while
 now ...
 
  I'm using the following configuration when starting Jetty:
 
  -XX:OnOutOfMemoryError=/home/solr/oom_killer.sh 83 %p
 
  If an OOM is triggered during Solr web app initialization (such as by
  me lowering -Xmx to a value that is too low to initialize Solr with),
  then the script gets called and does what I expect!
 
  However, once the Solr webapp initializes and Solr is happily
  responding to updates and queries. When an OOM occurs in this
  situation, then the script doesn't actually get invoked! All I see is
  the following in the stdout/stderr log of my process:
 
  #
  # java.lang.OutOfMemoryError: Java heap space
  # -XX:OnOutOfMemoryError=/home/solr/oom_killer.sh 83 %p
  #   Executing /bin/sh -c /home/solr/oom_killer.sh 83 21358...
 
  The oom_killer.sh script doesn't actually get called!
 
  So to recap, it works if an OOM occurs during initialization but once
  Solr is running, the OOM killer doesn't fire correctly. This leads me
  to believe my script is fine and there's something else going wrong.
  Here's the oom_killer.sh script (pretty basic):
 
  #!/bin/bash
  SOLR_PORT=$1
  SOLR_PID=$2
  NOW=$(date +%Y%m%d_%H%M)
  (
  echo Running OOM killer script for process $SOLR_PID for Solr on port
  89$SOLR_PORT
  kill -9 $SOLR_PID
  echo Killed process

Waiting until we see more replicas up message??

[Timothy Potter]

I'm seeing this message in the logs and it seems weird to me that the
instance needs to wait to see more replicas.

2013-06-24 18:12:40,408 [coreLoadExecutor-4-thread-1] INFO
solr.cloud.ShardLeaderElectionContext  - Waiting until we see more
replicas up: total=2 found=1 timeoutin=139368

Can someone elaborate on what this means? This messages seems to occur
after all replicas of a shard fail and I restart one of the nodes.
Shouldn't it just proceed as if it is the only replica and assume the
leader role?

This is on 4.2.0 btw.

Cheers,
Tim