Re: Solr Cloud existing shards down after enabling SSL

[Zheng Lin Edwin Yeo]

When you generate the keystore, did you include the IP address of both
servers?

Regards,
Edwin

On Mon, 11 Feb 2019 at 21:06, Rakesh Enjala 
wrote:

> Please help
> *Regards,*
> *Rakesh Enjala*
>
>
> On Wed, Feb 6, 2019 at 2:59 PM Rakesh Enjala 
> wrote:
>
> > Hi,
> >
> > We have a solr cloud with  4 nodes installed in two different servers( 1
> > on on server and 3 on other server)and  a collection with data in 4
> shards.
> > We have enabled SSL for solrcloud by using below link
> >
> > https://lucene.apache.org/solr/guide/7_4/enabling-ssl.html
> >
> > Successfully enabled SSL and we are able to access the solr GUI, But for
> > existing collection only one shard is in active state and rest 3 are in
> > down state. When i click cloud in GUI all 3 shards are down and showing
> > port to 8984 instead different ports.
> >
> > Solr Version:7.4
> >
> > Environment: Centos 7.2
> >
> > Please help me out !!
> >
> > Thanks in advance
> > *Regards,*
> > *Rakesh Enjala*
> >
>
> --
> *Disclaimer**
> **The information contained in this electronic message and
> any attachments to this message are intended for the exclusive use of the
> addressee(s) and may contain proprietary, confidential or privileged
> information. If you are not the intended recipient, you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately and destroy all copies of this message and any attachments.
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> The company accepts no liability for any damage caused by any virus
> transmitted by this email. **www.solix.com
> <
> https://www.google.com/url?q=http://www.solix.com=D=hangouts=1527924614994000=AFQjCNF6JdJTwnvKO4xgbMJbOjUo4g4hiA
> >.*
>
>

[search > edismax] compound words different result issue

[유정인]

Hi 

I use 'edismax'. 

Our main language uses compound words.

There is an issue here. 

For example, assume that 'ab' => 'a' and 'b' are analyzed. 

The results are different when searching with 'ab' and 'a b'. 

I want to get the same result as searching 'a b' when searching 'ab'.

Is there a way? 

 



 

Re: Document Score seen in debug section and in main results section dont match

[Baloo]

Thanks Erick  to answer your question "What is "Y"?"
Score that we see in debug section actually looks correct and if we order
documents by that score we can get similar ranking of results that we were
getting for solr 6.4.2. 

But With the score field that we get with each record it looks like boost
queries are getting ignored  (if there is more than one boost parameter to
solr, for single boost parameter it works fine, is something broken here
regarding multiple boost queries to solr)





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Relevancy Score Calculation

[Ashish Bisht]

Thanks.I Agree.

Regards
Ashish



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Document Score seen in debug section and in main results section dont match

[Erick Erickson]

No workarounds that I know of, but I have to ask:
"Why do you care?". This feels like an XY problem.
You are saying that "X" doesn't work, in this case the
scores are different in the debug section. But this
implies that there is a problem "Y" that you're having.

What is "Y"? What is the problem that these differing
scores is germane to? Perhaps there's another approach
that'll work to solve "Y" other than scores.

Best,
Erick

On Mon, Feb 11, 2019 at 6:04 AM Baloo  wrote:
>
> HI All,
>
> Currently I am migrating  Solr 6.4.2 to Solr 7.4.  We pass multiple boost
> queries (multiplicative boost queries, Solr's 'boost' parameter) to Solr
> with each query. We have migrated all our custom components and solr
> configurations to Solr 7.4.2 but during verification we have seen different
> score in debug section which is not matching with the score that we see when
> we return score field using fl parameter.
>
> After spending half of my day analysing issue I found similar issue is
> already open. Below issue is exactly same as what I am also observing with
> Solr 7.4.
>
> https://issues.apache.org/jira/browse/SOLR-13126
>
> Is there and solution/workaround available for this issue?
> or going back to Solr 7.2.1 is the only solution - As per comments in above
> issue (https://issues.apache.org/jira/browse/LUCENE-8099 these changes are
> not there in 7.2.1)
>
> Any help will be appreciated.
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Full index replication upon service restart

[Erick Erickson]

bq. To answer your question about index size on
disk, it is 3 TB on every node. As mentioned it's a 32 GB machine and I
allocated 24GB to Java heap.

This is massively undersized in terms of RAM in my experience. You're
trying to cram 3TB of index into 32GB of memory. Frankly, I don't think
there's much you can do to increase stability in this situation, too many
things are going on. In particular, you're indexing during node restart.

That means that
1> you'll almost inevitably get a full sync on start given your update
 rate.
2> while you're doing the full sync, all new updates are sent to the
  recovering replica and put in the tlog.
3> When the initial replication is done, the documents sent to the
 tlog while recovering are indexed. This is 7 hours of accumulated
 updates.
4> If much goes wrong in this situation, then you're talking another full
 sync.
5> rinse, repeat.

There are no magic tweaks here. You really have to rethink your
architecture. I'm actually surprised that your queries are performant.
I expect you're getting a _lot_ of I/O, that is the relevant parts of your
index are swapping in and out of the OS memory space. A _lot_.
Or you're only using a _very_ small bit of your index.

Sorry to be so negative, but this is not a situation that's amenable to
a quick fix.

Best,
Erick




On Mon, Feb 11, 2019 at 4:10 PM Rahul Goswami  wrote:
>
> Thanks for the response Eric. To answer your question about index size on
> disk, it is 3 TB on every node. As mentioned it's a 32 GB machine and I
> allocated 24GB to Java heap.
>
> Further monitoring the recovery, I see that when the follower node is
> recovering, the leader node (which is NOT recovering) almost freezes with
> 100% CPU usage and 80%+ memory usage. Follower node's memory usage is 80%+
> but CPU is very healthy. Also Follower node's log is filled up with updates
> forwarded from the leader ("...PRE_UPDATE FINISH
> {update.distrib=FROMLEADER=...") and replication starts much
> afterwards.
> There have been instances when complete recovery took 10+ hours. We have
> upgraded to a 4 Gbps NIC between the nodes to see if it helps.
>
> Also, a few followup questions:
>
> 1) Is  there a configuration which would start throttling update requests
> if the replica falls behind a certain number of updates so as to not
> trigger an index replication later?  If not, would it be a worthy
> enhancement?
> 2) What would be a recommended hard commit interval for this kind of setup
> ?
> 3) What are some of the improvements in 7.5 with respect to recovery as
> compared to 7.2.1?
> 4) What do the below peersync failure logs lines mean?  This would help me
> better understand the reasons for peersync failure and maybe devise some
> alert mechanism to start throttling update requests from application
> program if feasible.
>
> *PeerSync Failure type 1*:
> --
> 2019-02-04 20:43:50.018 INFO
> (recoveryExecutor-4-thread-2-processing-n:indexnode1:2_solr
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
> s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
> [c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
> org.apache.solr.update.PeerSync Fingerprint comparison: 1
>
> 2019-02-04 20:43:50.018 INFO
> (recoveryExecutor-4-thread-2-processing-n:indexnode1:2_solr
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
> s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
> [c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
> org.apache.solr.update.PeerSync Other fingerprint:
> {maxVersionSpecified=1624579878580912128,
> maxVersionEncountered=1624579893816721408, maxInHash=1624579878580912128,
> versionsHash=-8308981502886241345, numVersions=32966082, numDocs=32966165,
> maxDoc=1828452}, Our fingerprint: {maxVersionSpecified=1624579878580912128,
> maxVersionEncountered=1624579975760838656, maxInHash=1624579878580912128,
> versionsHash=4017509388564167234, numVersions=32966066, numDocs=32966165,
> maxDoc=1828452}
>
> 2019-02-04 20:43:50.018 INFO
> (recoveryExecutor-4-thread-2-processing-n:indexnode1:2_solr
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
> s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
> [c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
> org.apache.solr.update.PeerSync PeerSync:
> core=DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42 url=
> http://indexnode1:8983/solr DONE. sync failed
>
> 2019-02-04 20:43:50.018 INFO
> (recoveryExecutor-4-thread-2-processing-n:indexnode1:8983_solr
> x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
> s:shard11 

Re: Load balance writes

[Boban Acimovic]

OK, thank you guys :)

Regards,
Boban

Re: Full index replication upon service restart

[Rahul Goswami]

Thanks for the response Eric. To answer your question about index size on
disk, it is 3 TB on every node. As mentioned it's a 32 GB machine and I
allocated 24GB to Java heap.

Further monitoring the recovery, I see that when the follower node is
recovering, the leader node (which is NOT recovering) almost freezes with
100% CPU usage and 80%+ memory usage. Follower node's memory usage is 80%+
but CPU is very healthy. Also Follower node's log is filled up with updates
forwarded from the leader ("...PRE_UPDATE FINISH
{update.distrib=FROMLEADER=...") and replication starts much
afterwards.
There have been instances when complete recovery took 10+ hours. We have
upgraded to a 4 Gbps NIC between the nodes to see if it helps.

Also, a few followup questions:

1) Is  there a configuration which would start throttling update requests
if the replica falls behind a certain number of updates so as to not
trigger an index replication later?  If not, would it be a worthy
enhancement?
2) What would be a recommended hard commit interval for this kind of setup
?
3) What are some of the improvements in 7.5 with respect to recovery as
compared to 7.2.1?
4) What do the below peersync failure logs lines mean?  This would help me
better understand the reasons for peersync failure and maybe devise some
alert mechanism to start throttling update requests from application
program if feasible.

*PeerSync Failure type 1*:
--
2019-02-04 20:43:50.018 INFO
(recoveryExecutor-4-thread-2-processing-n:indexnode1:2_solr
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
[c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
org.apache.solr.update.PeerSync Fingerprint comparison: 1

2019-02-04 20:43:50.018 INFO
(recoveryExecutor-4-thread-2-processing-n:indexnode1:2_solr
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
[c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
org.apache.solr.update.PeerSync Other fingerprint:
{maxVersionSpecified=1624579878580912128,
maxVersionEncountered=1624579893816721408, maxInHash=1624579878580912128,
versionsHash=-8308981502886241345, numVersions=32966082, numDocs=32966165,
maxDoc=1828452}, Our fingerprint: {maxVersionSpecified=1624579878580912128,
maxVersionEncountered=1624579975760838656, maxInHash=1624579878580912128,
versionsHash=4017509388564167234, numVersions=32966066, numDocs=32966165,
maxDoc=1828452}

2019-02-04 20:43:50.018 INFO
(recoveryExecutor-4-thread-2-processing-n:indexnode1:2_solr
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
[c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
org.apache.solr.update.PeerSync PeerSync:
core=DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42 url=
http://indexnode1:8983/solr DONE. sync failed

2019-02-04 20:43:50.018 INFO
(recoveryExecutor-4-thread-2-processing-n:indexnode1:8983_solr
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42
s:shard11 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node45)
[c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard11 r:core_node45
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard11_replica_n42]
org.apache.solr.cloud.RecoveryStrategy PeerSync Recovery was not successful
- trying replication.


*PeerSync Failure type 1*:
-
2019-02-02 20:26:56.256 WARN
(recoveryExecutor-4-thread-11-processing-n:indexnode1:2_solr
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard12_replica_n46
s:shard12 c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 r:core_node49)
[c:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66 s:shard12 r:core_node49
x:DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard12_replica_n46]
org.apache.solr.update.PeerSync PeerSync:
core=DataIndex_1C6F947C-6673-4778-847D-2DE0FDE56C66_shard12_replica_n46 url=
http://indexnode1:2/solr too many updates received since start -
startingUpdates no longer overlaps with our currentUpdates


Regards,
Rahul

On Thu, Feb 7, 2019 at 12:59 PM Erick Erickson 
wrote:

> bq. We have a heavy indexing load of about 10,000 documents every 150
> seconds.
> Not so heavy query load.
>
> It's unlikely that changing numRecordsToKeep will help all that much if
> your
> maintenance window is very large. Rather, that number would have to be
> _very_
> high.
>
> 7 hours is huge. How big are your indexes on disk? You're essentially
> going to get a
> full copy from the leader for each replica, so network bandwidth may
> be the 

Delete by id

[Dwane Hall]

Hey Solr community,

I’m having an issue deleting documents from my Solr index and am seeking some 
community advice when somebody gets a spare minute. It seems really like a 
really simple problem …a requirement to delete a document by its id.

Here’s how my documents are mapped in solr

DOC_ID


My json format to delete the document (all looks correct according to 
https://lucene.apache.org/solr/guide/7_6/uploading-data-with-index-handlers.html
 “The JSON update format allows for a simple delete-by-id. The value of a 
delete can be an array which contains a list of zero or more specific document 
id’s (not a range) to be deleted. For example, a single document”)

Attempt 1 – “shorthand”
{“delete”:”123!12345”}

Attempt 2 – “longhand”
{“delete”:“DOC_ID”:”123!12345”}
{“delete”:{“DOC_ID”:”123!12345”}}

..the error is the same in all instances “org.apache.solr.common.SolrException: 
Document is missing mandatory uniqueKey field: DOC_ID”

Can anyone see any obvious details I’m overlooking?

I’ve tried all the update handlers below (both curl and through admin ui)

/update/
/update/json
/update/json/docs

My environment
Solr cloud 7.6
Single node

As always any advice would be greatly appreciated,

Thanks,

Dwane

Re: Load balance writes

[Jason Gerlowski]

> On the other hand, the CloudSolrClient ignores errors from Solr, which makes 
> it unacceptable for production use.

Did you mean "ConcurrentUpdateSolrClient"?  I don't think
CloudSolrClient does this, though I've been surprised before and
possible I just missed something.  Just wondering.

Jason

On Mon, Feb 11, 2019 at 2:14 PM Walter Underwood  wrote:
>
> The update router would also need to look for failures indexing at each 
> leader,
> then re-read the cluster state to see if the leader had changed. Also re-send 
> any
> failed updates, and so on.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Feb 11, 2019, at 11:07 AM, lstusr 5u93n4  wrote:
> >
> > Hi Boban,
> >
> > First of all: I agree with Walter here. Because the bottleneck is during
> > indexing on the leader, a basic round robin load balancer will perform just
> > as well as a custom solution. With far less headache. A custom solution
> > will be far more work than it's worth.
> >
> > But, should you really want to write this yourself, you can get all of the
> > information you need from zookeeper, from the path:
> >
> > /collections//state.json
> >
> > There, for each shard you'll see:
> >  - the "range" parameter that tells  you which subset of documents this
> > shard is responsible for (see
> > https://lucene.apache.org/solr/guide/7_6/shards-and-indexing-data-in-solrcloud.html#document-routing
> > for details on routing)
> >  - the list of all replicas. On each replica it will tell you:
> >  - the host name (base_url)
> >  - if it is the leader (has the property leader: true)
> >
> > So your go-based solution would be to watch the state.json file from
> > zookeeper, and build up a function that, given the proper routing structure
> > for your document (the hash of the id by default, I think) will return the
> > hostname of the replica that's the leader.
> >
> > Kyle
> >
> > On Mon, 11 Feb 2019 at 13:30, Boban Acimovic  wrote:
> >
> >> Like I said before, nginx is not a load balancer or at least not a clever
> >> load balancer. It does not talk to ZK. Please give me advanced solutions.
> >>
> >>
> >>
> >>
> >>> On 11. Feb 2019, at 18:32, Walter Underwood 
> >> wrote:
> >>>
> >>> I haven’t used Kubernetes, but a web search for “helm nginx” seems to
> >> give some useful pages.
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wun...@wunderwood.org
> >>> http://observer.wunderwood.org/  (my blog)
> >>>
>  On Feb 11, 2019, at 9:13 AM, Davis, Daniel (NIH/NLM) [C] <
> >> daniel.da...@nih.gov> wrote:
> 
>  I think that the container orchestration framework takes care of that
> >> for you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress
> >> controller, and as long as the services are running within the Kubernetes
> >> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a
> >> "Load Balancer" appears to be a concept for accessing services outside the
> >> cluster.
> 
>  I presume you are using Kubernetes because of your reference to helm,
> >> but for what it's worth, here's an official haproxy image -
> >> https://hub.docker.com/_/haproxy
> >>
>

Re: Load balance writes

[Walter Underwood]

The update router would also need to look for failures indexing at each leader,
then re-read the cluster state to see if the leader had changed. Also re-send 
any
failed updates, and so on.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 11, 2019, at 11:07 AM, lstusr 5u93n4  wrote:
> 
> Hi Boban,
> 
> First of all: I agree with Walter here. Because the bottleneck is during
> indexing on the leader, a basic round robin load balancer will perform just
> as well as a custom solution. With far less headache. A custom solution
> will be far more work than it's worth.
> 
> But, should you really want to write this yourself, you can get all of the
> information you need from zookeeper, from the path:
> 
> /collections//state.json
> 
> There, for each shard you'll see:
>  - the "range" parameter that tells  you which subset of documents this
> shard is responsible for (see
> https://lucene.apache.org/solr/guide/7_6/shards-and-indexing-data-in-solrcloud.html#document-routing
> for details on routing)
>  - the list of all replicas. On each replica it will tell you:
>  - the host name (base_url)
>  - if it is the leader (has the property leader: true)
> 
> So your go-based solution would be to watch the state.json file from
> zookeeper, and build up a function that, given the proper routing structure
> for your document (the hash of the id by default, I think) will return the
> hostname of the replica that's the leader.
> 
> Kyle
> 
> On Mon, 11 Feb 2019 at 13:30, Boban Acimovic  wrote:
> 
>> Like I said before, nginx is not a load balancer or at least not a clever
>> load balancer. It does not talk to ZK. Please give me advanced solutions.
>> 
>> 
>> 
>> 
>>> On 11. Feb 2019, at 18:32, Walter Underwood 
>> wrote:
>>> 
>>> I haven’t used Kubernetes, but a web search for “helm nginx” seems to
>> give some useful pages.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
 On Feb 11, 2019, at 9:13 AM, Davis, Daniel (NIH/NLM) [C] <
>> daniel.da...@nih.gov> wrote:
 
 I think that the container orchestration framework takes care of that
>> for you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress
>> controller, and as long as the services are running within the Kubernetes
>> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a
>> "Load Balancer" appears to be a concept for accessing services outside the
>> cluster.
 
 I presume you are using Kubernetes because of your reference to helm,
>> but for what it's worth, here's an official haproxy image -
>> https://hub.docker.com/_/haproxy
>> 

Re: Load balance writes

[lstusr 5u93n4]

Hi Boban,

First of all: I agree with Walter here. Because the bottleneck is during
indexing on the leader, a basic round robin load balancer will perform just
as well as a custom solution. With far less headache. A custom solution
will be far more work than it's worth.

But, should you really want to write this yourself, you can get all of the
information you need from zookeeper, from the path:

/collections//state.json

There, for each shard you'll see:
  - the "range" parameter that tells  you which subset of documents this
shard is responsible for (see
https://lucene.apache.org/solr/guide/7_6/shards-and-indexing-data-in-solrcloud.html#document-routing
for details on routing)
  - the list of all replicas. On each replica it will tell you:
  - the host name (base_url)
  - if it is the leader (has the property leader: true)

So your go-based solution would be to watch the state.json file from
zookeeper, and build up a function that, given the proper routing structure
for your document (the hash of the id by default, I think) will return the
hostname of the replica that's the leader.

Kyle

On Mon, 11 Feb 2019 at 13:30, Boban Acimovic  wrote:

> Like I said before, nginx is not a load balancer or at least not a clever
> load balancer. It does not talk to ZK. Please give me advanced solutions.
>
>
>
>
> > On 11. Feb 2019, at 18:32, Walter Underwood 
> wrote:
> >
> > I haven’t used Kubernetes, but a web search for “helm nginx” seems to
> give some useful pages.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >> On Feb 11, 2019, at 9:13 AM, Davis, Daniel (NIH/NLM) [C] <
> daniel.da...@nih.gov> wrote:
> >>
> >> I think that the container orchestration framework takes care of that
> for you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress
> controller, and as long as the services are running within the Kubernetes
> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a
> "Load Balancer" appears to be a concept for accessing services outside the
> cluster.
> >>
> >> I presume you are using Kubernetes because of your reference to helm,
> but for what it's worth, here's an official haproxy image -
> https://hub.docker.com/_/haproxy
>

Re: Load balance writes

[Walter Underwood]

For the fourth time, ignore the shard leaders until you have measurements that 
prove the complexity is worth it.

We can index a million documents per minute by sending batched updates to a 
dumb load balancer.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 11, 2019, at 10:29 AM, Boban Acimovic  wrote:
> 
> Like I said before, nginx is not a load balancer or at least not a clever 
> load balancer. It does not talk to ZK. Please give me advanced solutions.
> 
> 
> 
> 
>> On 11. Feb 2019, at 18:32, Walter Underwood  wrote:
>> 
>> I haven’t used Kubernetes, but a web search for “helm nginx” seems to give 
>> some useful pages.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Feb 11, 2019, at 9:13 AM, Davis, Daniel (NIH/NLM) [C] 
>>>  wrote:
>>> 
>>> I think that the container orchestration framework takes care of that for 
>>> you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress 
>>> controller, and as long as the services are running within the Kubernetes 
>>> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a 
>>> "Load Balancer" appears to be a concept for accessing services outside the 
>>> cluster.
>>> 
>>> I presume you are using Kubernetes because of your reference to helm, but 
>>> for what it's worth, here's an official haproxy image - 
>>> https://hub.docker.com/_/haproxy

Re: Load balance writes

[Boban Acimovic]

Like I said before, nginx is not a load balancer or at least not a clever load 
balancer. It does not talk to ZK. Please give me advanced solutions.




> On 11. Feb 2019, at 18:32, Walter Underwood  wrote:
> 
> I haven’t used Kubernetes, but a web search for “helm nginx” seems to give 
> some useful pages.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Feb 11, 2019, at 9:13 AM, Davis, Daniel (NIH/NLM) [C] 
>>  wrote:
>> 
>> I think that the container orchestration framework takes care of that for 
>> you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress 
>> controller, and as long as the services are running within the Kubernetes 
>> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a 
>> "Load Balancer" appears to be a concept for accessing services outside the 
>> cluster.
>> 
>> I presume you are using Kubernetes because of your reference to helm, but 
>> for what it's worth, here's an official haproxy image - 
>> https://hub.docker.com/_/haproxy

Re: Load balance writes

[Boban Acimovic]

But like I said in the previous message, nginx is not aware of the status of 
Solr nodes. I can easily write Go load balancer but not considering the shards. 
The only problem I have here is how to figure out which shard master is 
responsible of a document I want to insert to the index. How does Solr sharing 
works? Which values are used to determine the shard?




> On 11. Feb 2019, at 18:13, Davis, Daniel (NIH/NLM) [C]  
> wrote:
> 
> I think that the container orchestration framework takes care of that for 
> you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress 
> controller, and as long as the services are running within the Kubernetes 
> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a 
> "Load Balancer" appears to be a concept for accessing services outside the 
> cluster.
> 
> I presume you are using Kubernetes because of your reference to helm, but for 
> what it's worth, here's an official haproxy image - 
> https://hub.docker.com/_/haproxy
> 
>> -Original Message-
>> From: Boban Acimovic 
>> Sent: Monday, February 11, 2019 11:58 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Load balance writes
>> 
>> Can you mention one dockerized load balancer? Or even better one with
>> Helm chart?
>> 
>> 
>> Like I said, I send all updates at the moment just to one out of 12 nodes.
>> 
>> 
>> 
>> 
>>> On 11. Feb 2019, at 17:52, Walter Underwood
>>  wrote:
>>> 
>>> Why would you want to write a load balancer when there are so many that
>> are free and very fast?
>>> 
>>> For update traffic, there is very little benefit in sending updates 
>>> directly to
>> the shard leader. Forwarding an update to the leader is fast. Indexing is 
>> slow.
>> So the bottleneck is always at the leader.
>>> 
>>> Before you build anything, measure. Collect a large update and send that
>> directly to the leader. Then do the same to a non-leader shard. Compare the
>> speed. If you are batching and indexing with multiple threads, I doubt you’ll
>> see a meaningful difference. I commonly see 10% difference in identical load
>> benchmarks, so the speedup has to be much larger than that to be real.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)

Re: Load balance writes

[Boban Acimovic]

This is naive load balancing because it is not aware of ZK.




> On 11. Feb 2019, at 18:05, Walter Underwood  wrote:
> 
> nginx
> 
> http://nginx.org/en/docs/http/load_balancing.html
> https://hub.docker.com/_/nginx
> 
> We run in Amazon AWS, so we use their Application Load Balaner (ALB). We do 
> use nginx for other things.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)

Re: Load balance writes

[Walter Underwood]

I haven’t used Kubernetes, but a web search for “helm nginx” seems to give some 
useful pages.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 11, 2019, at 9:13 AM, Davis, Daniel (NIH/NLM) [C] 
>  wrote:
> 
> I think that the container orchestration framework takes care of that for 
> you, but I am not an expert.  In Kubernetes, NGINX is often the Ingress 
> controller, and as long as the services are running within the Kubernetes 
> cluster, it can also serve as a load balancer, AFAICT.   In Kubernetes, a 
> "Load Balancer" appears to be a concept for accessing services outside the 
> cluster.
> 
> I presume you are using Kubernetes because of your reference to helm, but for 
> what it's worth, here's an official haproxy image - 
> https://hub.docker.com/_/haproxy
> 
>> -Original Message-
>> From: Boban Acimovic 
>> Sent: Monday, February 11, 2019 11:58 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Load balance writes
>> 
>> Can you mention one dockerized load balancer? Or even better one with
>> Helm chart?
>> 
>> 
>> Like I said, I send all updates at the moment just to one out of 12 nodes.
>> 
>> 
>> 
>> 
>>> On 11. Feb 2019, at 17:52, Walter Underwood
>>  wrote:
>>> 
>>> Why would you want to write a load balancer when there are so many that
>> are free and very fast?
>>> 
>>> For update traffic, there is very little benefit in sending updates 
>>> directly to
>> the shard leader. Forwarding an update to the leader is fast. Indexing is 
>> slow.
>> So the bottleneck is always at the leader.
>>> 
>>> Before you build anything, measure. Collect a large update and send that
>> directly to the leader. Then do the same to a non-leader shard. Compare the
>> speed. If you are batching and indexing with multiple threads, I doubt you’ll
>> see a meaningful difference. I commonly see 10% difference in identical load
>> benchmarks, so the speedup has to be much larger than that to be real.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)

RE: Load balance writes

[Davis, Daniel (NIH/NLM) [C]]

I think that the container orchestration framework takes care of that for you, 
but I am not an expert.  In Kubernetes, NGINX is often the Ingress controller, 
and as long as the services are running within the Kubernetes cluster, it can 
also serve as a load balancer, AFAICT.   In Kubernetes, a "Load Balancer" 
appears to be a concept for accessing services outside the cluster.

I presume you are using Kubernetes because of your reference to helm, but for 
what it's worth, here's an official haproxy image - 
https://hub.docker.com/_/haproxy

> -Original Message-
> From: Boban Acimovic 
> Sent: Monday, February 11, 2019 11:58 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Load balance writes
> 
> Can you mention one dockerized load balancer? Or even better one with
> Helm chart?
> 
> 
> Like I said, I send all updates at the moment just to one out of 12 nodes.
> 
> 
> 
> 
> > On 11. Feb 2019, at 17:52, Walter Underwood
>  wrote:
> >
> > Why would you want to write a load balancer when there are so many that
> are free and very fast?
> >
> > For update traffic, there is very little benefit in sending updates 
> > directly to
> the shard leader. Forwarding an update to the leader is fast. Indexing is 
> slow.
> So the bottleneck is always at the leader.
> >
> > Before you build anything, measure. Collect a large update and send that
> directly to the leader. Then do the same to a non-leader shard. Compare the
> speed. If you are batching and indexing with multiple threads, I doubt you’ll
> see a meaningful difference. I commonly see 10% difference in identical load
> benchmarks, so the speedup has to be much larger than that to be real.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)

Re: Load balance writes

[Walter Underwood]

nginx

http://nginx.org/en/docs/http/load_balancing.html
https://hub.docker.com/_/nginx

We run in Amazon AWS, so we use their Application Load Balaner (ALB). We do use 
nginx for other things.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 11, 2019, at 8:57 AM, Boban Acimovic  wrote:
> 
> Can you mention one dockerized load balancer? Or even better one with Helm 
> chart?
> 
> 
> Like I said, I send all updates at the moment just to one out of 12 nodes.
> 
>> On 11. Feb 2019, at 17:52, Walter Underwood  wrote:
>> 
>> Why would you want to write a load balancer when there are so many that are 
>> free and very fast?
>> 
>> For update traffic, there is very little benefit in sending updates directly 
>> to the shard leader. Forwarding an update to the leader is fast. Indexing is 
>> slow. So the bottleneck is always at the leader.
>> 
>> Before you build anything, measure. Collect a large update and send that 
>> directly to the leader. Then do the same to a non-leader shard. Compare the 
>> speed. If you are batching and indexing with multiple threads, I doubt 
>> you’ll see a meaningful difference. I commonly see 10% difference in 
>> identical load benchmarks, so the speedup has to be much larger than that to 
>> be real.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)

Re: Load balance writes

[Boban Acimovic]

Can you mention one dockerized load balancer? Or even better one with Helm 
chart?


Like I said, I send all updates at the moment just to one out of 12 nodes.




> On 11. Feb 2019, at 17:52, Walter Underwood  wrote:
> 
> Why would you want to write a load balancer when there are so many that are 
> free and very fast?
> 
> For update traffic, there is very little benefit in sending updates directly 
> to the shard leader. Forwarding an update to the leader is fast. Indexing is 
> slow. So the bottleneck is always at the leader.
> 
> Before you build anything, measure. Collect a large update and send that 
> directly to the leader. Then do the same to a non-leader shard. Compare the 
> speed. If you are batching and indexing with multiple threads, I doubt you’ll 
> see a meaningful difference. I commonly see 10% difference in identical load 
> benchmarks, so the speedup has to be much larger than that to be real.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)

Re: Load balance writes

[Walter Underwood]

Why would you want to write a load balancer when there are so many that are 
free and very fast?

For update traffic, there is very little benefit in sending updates directly to 
the shard leader. Forwarding an update to the leader is fast. Indexing is slow. 
So the bottleneck is always at the leader.

Before you build anything, measure. Collect a large update and send that 
directly to the leader. Then do the same to a non-leader shard. Compare the 
speed. If you are batching and indexing with multiple threads, I doubt you’ll 
see a meaningful difference. I commonly see 10% difference in identical load 
benchmarks, so the speedup has to be much larger than that to be real.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 11, 2019, at 8:38 AM, Boban Acimovic  wrote:
> 
> I would actually like to write a load balancer itself, but I want it to be 
> able to send the data as efficiently as possible. I know how to read ZK data, 
> but I don’t know how can I figure out which shard is responsible upon data 
> that I have in a document that I want to index.
> 
> 
> 
> 
>> On 11. Feb 2019, at 17:23, Walter Underwood  wrote:
>> 
>> We send all updates to the load balancer, so they’ll end up on the wrong 
>> shard, not on the leader, etc. Indexing speed is still limited by the CPU 
>> available on each leader. I don’t think that sending the update to the right 
>> leader makes any improvement in throughput.
>> 
>> On the other hand, the CloudSolrClient ignores errors from Solr, which makes 
>> it unacceptable for production use.
>> 
>> I would stay with your current indexing client and worry about something 
>> else.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)

Createsnapshot null pointer exception

[SOLR4189]

Hi all,

I use SOLR-6.5.1. When I run this command:

*http://my_server_name:8983/solr/admin/collections?action=CREATESNAPSHOT=collection_name=MYCommit*

I got this exception:
Collection: collection_name operation: createsnapshot failed:
java.lang.NullPointerException
  at
org.apache.solr.cloud.CreateSnapshotCmd.call(CreateSnapshotCmd.java:128)
  at
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java)
 ...

Does somebody know what's a problem?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Load balance writes

[Boban Acimovic]

I would actually like to write a load balancer itself, but I want it to be able 
to send the data as efficiently as possible. I know how to read ZK data, but I 
don’t know how can I figure out which shard is responsible upon data that I 
have in a document that I want to index.




> On 11. Feb 2019, at 17:23, Walter Underwood  wrote:
> 
> We send all updates to the load balancer, so they’ll end up on the wrong 
> shard, not on the leader, etc. Indexing speed is still limited by the CPU 
> available on each leader. I don’t think that sending the update to the right 
> leader makes any improvement in throughput.
> 
> On the other hand, the CloudSolrClient ignores errors from Solr, which makes 
> it unacceptable for production use.
> 
> I would stay with your current indexing client and worry about something else.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)

Re: Load balance writes

[Boban Acimovic]

Thank you again Emir. I can make my code ZK aware, that is no problem, but I 
can’t make it shard leader aware.  Can you point me to a document how are Solr 
shards created?  I already use ZK to get stuff, but I don’ t understand how to 
distinguish between shards from information I can get from a document that has 
to be indexes.

At the moment I send everything to one node, but I am pretty much sure it would 
help to send data to collection nodes. However, it would be even better it I 
can send data directly to shard leader. If you can’t describe this easily, I 
will check Soltj implementation.

Regards,
Boban




> On 11. Feb 2019, at 17:15, Emir Arnautović  
> wrote:
> 
> Hi Boban,
> Not sure if there is Solrj port to Go, but you can take that as model to 
> build your ZK aware client that groups and sends updates to shard leaders. I 
> see that there are couple of Solr Go clients, so you might first check if 
> some already supports it or if it makes sense that you contribute that part 
> to one of your choice.
> 
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/

Re: Load balance writes

[Walter Underwood]

We send all updates to the load balancer, so they’ll end up on the wrong shard, 
not on the leader, etc. Indexing speed is still limited by the CPU available on 
each leader. I don’t think that sending the update to the right leader makes 
any improvement in throughput.

On the other hand, the CloudSolrClient ignores errors from Solr, which makes it 
unacceptable for production use.

I would stay with your current indexing client and worry about something else.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 11, 2019, at 8:15 AM, Emir Arnautović  
> wrote:
> 
> Hi Boban,
> Not sure if there is Solrj port to Go, but you can take that as model to 
> build your ZK aware client that groups and sends updates to shard leaders. I 
> see that there are couple of Solr Go clients, so you might first check if 
> some already supports it or if it makes sense that you contribute that part 
> to one of your choice.
> 
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> 
> 
> 
>> On 11 Feb 2019, at 16:09, Boban Acimovic  wrote:
>> 
>> Thank you Emir for quick reply. I use home brewed Go client and write just 
>> to one of 12 available nodes. I believe I should find out this smart way to 
>> handle this :)
>> 
>> 
>> 
>> 
>>> On 11. Feb 2019, at 15:21, Emir Arnautović  
>>> wrote:
>>> 
>>> Hi Boban,
>>> If you use SolrCloud  Solrj client and initialise it with ZK, it should be 
>>> aware of masters and send documents in a smart way.
>>> 
>>> HTH,
>>> Emir
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>> 
> 

Re: Load balance writes

[Emir Arnautović]

Hi Boban,
Not sure if there is Solrj port to Go, but you can take that as model to build 
your ZK aware client that groups and sends updates to shard leaders. I see that 
there are couple of Solr Go clients, so you might first check if some already 
supports it or if it makes sense that you contribute that part to one of your 
choice.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Feb 2019, at 16:09, Boban Acimovic  wrote:
> 
> Thank you Emir for quick reply. I use home brewed Go client and write just to 
> one of 12 available nodes. I believe I should find out this smart way to 
> handle this :)
> 
> 
> 
> 
>> On 11. Feb 2019, at 15:21, Emir Arnautović  
>> wrote:
>> 
>> Hi Boban,
>> If you use SolrCloud  Solrj client and initialise it with ZK, it should be 
>> aware of masters and send documents in a smart way.
>> 
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> 

Re: RegexReplaceProcessorFactory pattern to detect multiple \n

[Zheng Lin Edwin Yeo]

Hi,

Should we report this as a bug in Solr?

Regards,
Edwin

On Fri, 8 Feb 2019 at 22:18, Zheng Lin Edwin Yeo 
wrote:

> Hi Paul,
>
> Regarding the regex (\n\s*){2,} that we are using, when we try in on
> https://regex101.com/, it is able to give us the correct result for all
> the examples (ie: All of them will only have , and not more than
> that like what we are getting in Solr in our earlier examples).
>
> Could there be a possibility of a bug in Solr?
>
> Regards,
> Edwin
>
> On Fri, 8 Feb 2019 at 00:33, Zheng Lin Edwin Yeo 
> wrote:
>
>> Hi Paul,
>>
>> We have tried it with the space preceeding the \n i.e. > name="pattern">(\s*\n){2,}, with the following regex pattern:
>>
>> 
>>content
>>(\s*\n){2,}
>>brbr
>> 
>>
>> However, we are also getting the exact same results as the earlier
>> Example 1, 2 and 3.
>>
>> As for your point 2 on perhaps in the data you have other (non printing)
>> characters than \n, we have find that there are no non printing characters.
>> It is just next line with a space. You can refer to the original content in
>> the same examples below.
>>
>>
>> Example 1: The sentence that the above regex pattern is working correctly
>> *Original content in EML file:*
>> Dear Sir,
>>
>>
>> I am terminating
>> *Original content:*Dear Sir,  \n\n \n \n\n I am terminating
>> *Index content: *Dear Sir,  I am terminating
>>
>> Example 2: The sentence that the above regex pattern is partially working
>> (as you can see, instead of 2 , there are 4 )
>> *Original content in EML file:*
>>
>> *exalted*
>>
>> *Psalm 89:17*
>>
>>
>> 3 Choa Chu Kang Avenue 4
>> *Original content:* exalted  \n \n\n   Psalm 89:17   \n\n   \n\n  3 Choa
>> Chu Kang Avenue 4, Singapore
>> *Index content: *exalted  Psalm 89:17 3 Choa
>> Chu Kang Avenue 4, Singapore
>>
>> Example 3: The sentence that the above regex pattern is partially working
>> (as you can see, instead of 2 , there are 4 )
>> *Original content in EML file:*
>>
>> http://www.concordpri.moe.edu.sg/
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Dec 18, 2018 at 10:07 AM
>> *Original content:* http://www.concordpri.moe.edu.sg/   \n\n   \n\n \n
>> \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n  On Tue, Dec 18,
>> 2018 at 10:07 AM
>> *Index content: *http://www.concordpri.moe.edu.sg/   
>> On Tue, Dec 18, 2018 at 10:07 AM
>>
>>
>> Appreciate any other ideas or suggestions that you may have.
>>
>> Thank you.
>>
>> Regards,
>> Edwin
>>
>> On Thu, 7 Feb 2019 at 22:49,  wrote:
>>
>>> Hi Edwin
>>>
>>>
>>>
>>>   1.  Sorry, the pattern was wrong, the space should preceed the \n i.e.
>>> (\s*\n){2,}
>>>   2.  Perhaps in the data you have other (non printing) characters than
>>> \n?
>>>
>>>
>>>
>>> Gesendet von Mail für
>>> Windows 10
>>>
>>>
>>>
>>> Von: Zheng Lin Edwin Yeo
>>> Gesendet: Donnerstag, 7. Februar 2019 15:23
>>> An: solr-user@lucene.apache.org
>>> Betreff: Re: RegexReplaceProcessorFactory pattern to detect multiple \n
>>>
>>>
>>>
>>> Hi Paul,
>>>
>>> We have tried this suggested regex pattern as follow:
>>> 
>>>content
>>>(\n\s*){2,}
>>>brbr
>>> 
>>>
>>> But we still have exactly the same problem of Example 1,2 and 3 below.
>>>
>>> Example 1: The sentence that the above regex pattern is working correctly
>>> *Original content:*Dear Sir,  \n\n \n \n\n I am terminating
>>> *Index content: *Dear Sir,  I am terminating
>>>
>>> Example 2: The sentence that the above regex pattern is partially working
>>> (as you can see, instead of 2 , there are 4 )
>>> *Original content:* exalted  \n \n\n   Psalm 89:17   \n\n   \n\n  3 Choa
>>> Chu Kang Avenue 4, Singapore
>>> *Index content: *exalted  Psalm 89:17 3 Choa
>>> Chu Kang Avenue 4, Singapore
>>>
>>> Example 3: The sentence that the above regex pattern is partially working
>>> (as you can see, instead of 2 , there are 4 )
>>> *Original content:* http://www.concordpri.moe.edu.sg/   \n\n   \n\n \n
>>> \n\n
>>> \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n  On Tue, Dec 18,
>>> 2018
>>> at 10:07 AM
>>> *Index content: *http://www.concordpri.moe.edu.sg/   
>>> On
>>> Tue, Dec 18, 2018 at 10:07 AM
>>>
>>> Any further suggestion?
>>>
>>> Thank you.
>>>
>>> Regards,
>>> Edwin
>>>
>>> On Thu, 7 Feb 2019 at 22:20,  wrote:
>>>
>>> > To avoid the «\n+\s*» matching too many \n and then failing on the {2,}
>>> > part you could try
>>> >
>>> >
>>> >
>>> > (\n\s*){2,}
>>> >
>>> >
>>> >
>>> > If you also want to match CRLF then
>>> >
>>> > (\r?\n\s*){2,}
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Gesendet von Mail für
>>> > Windows 10
>>> >
>>> >
>>> >
>>> > Von: Zheng Lin Edwin Yeo
>>> > Gesendet: Donnerstag, 7. Februar 2019 15:10
>>> > An: solr-user@lucene.apache.org
>>> > Betreff: Re: RegexReplaceProcessorFactory pattern to detect multiple \n
>>> >
>>> >
>>> 

Re: Load balance writes

[Boban Acimovic]

Thank you Emir for quick reply. I use home brewed Go client and write just to 
one of 12 available nodes. I believe I should find out this smart way to handle 
this :)




> On 11. Feb 2019, at 15:21, Emir Arnautović  
> wrote:
> 
> Hi Boban,
> If you use SolrCloud  Solrj client and initialise it with ZK, it should be 
> aware of masters and send documents in a smart way.
> 
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/

[ANNOUNCE] Luke 7.7.0 released

[Tomoko Uchida]

Hi,

Luke 7.7.0 is out.
Zip archive can be downloaded at here:
https://github.com/DmitryKey/luke/releases/tag/luke-swing-7.7.0

In this release,
- Lucene version was upgraded to 7.7.0.
- Some trivial UI bugs were fixed.

Regards,
Tomoko

RE: Re: Enable SSL for the existing SOLR Cloud Cluster

[rakesh.enjala]

Hi,

I am facing the same issue. If u get any solution please post



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Load balance writes

[Emir Arnautović]

Hi Boban,
If you use SolrCloud  Solrj client and initialise it with ZK, it should be 
aware of masters and send documents in a smart way.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 11 Feb 2019, at 12:18, Boban Acimovic  wrote:
> 
> I am wondering would I get performance benefits if I distribute writes to 
> Solr nodes by sending documents exactly to the master of collection where the 
> document belongs? My idea is that this would save some load between the 
> cluster nodes and improve performances. How to do writes in the best way? 
> Thank you in advance.

Document Score seen in debug section and in main results section dont match

[Baloo]

HI All,

Currently I am migrating  Solr 6.4.2 to Solr 7.4.  We pass multiple boost
queries (multiplicative boost queries, Solr's 'boost' parameter) to Solr
with each query. We have migrated all our custom components and solr
configurations to Solr 7.4.2 but during verification we have seen different
score in debug section which is not matching with the score that we see when
we return score field using fl parameter. 

After spending half of my day analysing issue I found similar issue is
already open. Below issue is exactly same as what I am also observing with
Solr 7.4.  

https://issues.apache.org/jira/browse/SOLR-13126

Is there and solution/workaround available for this issue?
or going back to Solr 7.2.1 is the only solution - As per comments in above
issue (https://issues.apache.org/jira/browse/LUCENE-8099 these changes are
not there in 7.2.1)

Any help will be appreciated. 




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Cloud existing shards down after enabling SSL

[Rakesh Enjala]

Please help
*Regards,*
*Rakesh Enjala*


On Wed, Feb 6, 2019 at 2:59 PM Rakesh Enjala 
wrote:

> Hi,
>
> We have a solr cloud with  4 nodes installed in two different servers( 1
> on on server and 3 on other server)and  a collection with data in 4 shards.
> We have enabled SSL for solrcloud by using below link
>
> https://lucene.apache.org/solr/guide/7_4/enabling-ssl.html
>
> Successfully enabled SSL and we are able to access the solr GUI, But for
> existing collection only one shard is in active state and rest 3 are in
> down state. When i click cloud in GUI all 3 shards are down and showing
> port to 8984 instead different ports.
>
> Solr Version:7.4
>
> Environment: Centos 7.2
>
> Please help me out !!
>
> Thanks in advance
> *Regards,*
> *Rakesh Enjala*
>

-- 
*Disclaimer**
**The information contained in this electronic message and 
any attachments to this message are intended for the exclusive use of the 
addressee(s) and may contain proprietary, confidential or privileged 
information. If you are not the intended recipient, you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately and destroy all copies of this message and any attachments. 
WARNING: Computer viruses can be transmitted via email. The recipient 
should check this email and any attachments for the presence of viruses. 
The company accepts no liability for any damage caused by any virus 
transmitted by this email. **www.solix.com 
.*

Load balance writes

[Boban Acimovic]

I am wondering would I get performance benefits if I distribute writes to Solr 
nodes by sending documents exactly to the master of collection where the 
document belongs? My idea is that this would save some load between the cluster 
nodes and improve performances. How to do writes in the best way? Thank you in 
advance.

Re: Ignore accent in a request

[elisabeth benoit]

Thanks for the hint. We've been using the char filter for full unidecode
normalization. Is the ICUFoldingFilter supposed to be faster? Or just
simpler to use?

Le lun. 11 févr. 2019 à 09:58, Ere Maijala  a
écrit :

> Please note that mapping characters works well for a small set of
> characters, but if you want full UNICODE normalization, take a look at
> the ICUFoldingFilter:
>
> https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-ICUFoldingFilter
>
> --Ere
>
> elisabeth benoit kirjoitti 8.2.2019 klo 22.47:
> > yes you do
> >
> > and use the char filter at index and query time
> >
> > Le ven. 8 févr. 2019 à 19:20, SAUNIER Maxence  a
> écrit :
> >
> >> For the charFilter, I need to reindex all documents ?
> >>
> >> -Message d'origine-
> >> De : Erick Erickson 
> >> Envoyé : vendredi 8 février 2019 18:03
> >> À : solr-user 
> >> Objet : Re: Ignore accent in a request
> >>
> >> Elisabeth's suggestion is spot on for the accent.
> >>
> >> One other thing I noticed. You are using KeywordTokenizerFactory
> combined
> >> with EdgeNGramFilterFactory. This implies that you can't search for
> >> individual _words_, only prefix queries, i.e.
> >> je
> >> je s
> >> je su
> >> je sui
> >> je suis
> >>
> >> You can't search for "suis" for instance.
> >>
> >> basically this is an efficient way to search anything starting with
> >> three-or-more letter prefixes at the expense of index size. You might be
> >> better off just using wildcards (restrict to three letters at the prefix
> >> though).
> >>
> >> This is perfectly valid, I'm mostly asking if it's your intent.
> >>
> >> Best,
> >> Erick
> >>
> >> On Fri, Feb 8, 2019 at 9:35 AM SAUNIER Maxence 
> wrote:
> >>>
> >>> Thanks you !
> >>>
> >>> -Message d'origine-
> >>> De : elisabeth benoit  Envoyé : vendredi 8
> >>> février 2019 14:12 À : solr-user@lucene.apache.org Objet : Re: Ignore
> >>> accent in a request
> >>>
> >>> Hello,
> >>>
> >>> We use solr 7 and use
> >>>
> >>>  >>> mapping="mapping-ISOLatin1Accent.txt"/>
> >>>
> >>> with mapping-ISOLatin1Accent.txt
> >>>
> >>> containing lines like
> >>>
> >>> # À => A
> >>> "\u00C0" => "A"
> >>>
> >>> # Á => A
> >>> "\u00C1" => "A"
> >>>
> >>> # Â => A
> >>> "\u00C2" => "A"
> >>>
> >>> # Ã => A
> >>> "\u00C3" => "A"
> >>>
> >>> # Ä => A
> >>> "\u00C4" => "A"
> >>>
> >>> # Å => A
> >>> "\u00C5" => "A"
> >>>
> >>> # Ā Ă Ą =>
> >>> "\u0100" => "A"
> >>> "\u0102" => "A"
> >>> "\u0104" => "A"
> >>>
> >>> # Æ => AE
> >>> "\u00C6" => "AE"
> >>>
> >>> # Ç => C
> >>> "\u00C7" => "C"
> >>>
> >>> # é => e
> >>> "\u00E9" => "e"
> >>>
> >>> Best regards,
> >>> Elisabeth
> >>>
> >>> Le ven. 8 févr. 2019 à 11:18, Gopesh Sharma  >
> >> a écrit :
> >>>
>  We have fixed this type of issue by using Synonyms by adding
>  SynonymFilterFactory(Before Solr 7).
> 
>  -Original Message-
>  From: SAUNIER Maxence 
>  Sent: Friday, February 8, 2019 3:36 PM
>  To: solr-user@lucene.apache.org
>  Subject: RE: Ignore accent in a request
> 
>  Hello,
> 
>  Thanks for you answer.
> 
>  I have test :
> 
>  select?defType=dismax=je suis avarié=content
>  90.000 results
> 
>  select?defType=dismax=je suis avarie=content
>  60.000 results
> 
>  With avarié, I dont find documents with avarie and with avarie, I
>  don't find documents with avarié.
> 
>  I want to find they 150.000 documents with avarié or avarie.
> 
>  Thanks
> 
>  -Message d'origine-
>  De : Erick Erickson  Envoyé : jeudi 7
>  février
>  2019 19:37 À : solr-user  Objet : Re:
>  Ignore accent in a request
> 
>  exactly _how_ is it "not working"?
> 
>  Try building your parameters _up_ rather than starting with a lot,
> e.g.
>  select?defType=dismax=je suis avarié=title ^^ assumes you
>  expect a match on title. Then:
>  select?defType=dismax=je suis avarié=title subject
> 
>  etc.
> 
>  Because mm=757 looks really wrong. From the docs:
>  Defines the minimum number of clauses that must match, regardless of
>  how many clauses there are in total.
> 
>  edismax is used much more than dismax as it's more flexible, but
>  that's not germane here.
> 
>  finally, try adding =query to the url to see exactly how the
>  query is parsed.
> 
>  Best,
>  Erick
> 
>  On Mon, Feb 4, 2019 at 9:09 AM SAUNIER Maxence 
> >> wrote:
> >
> > Hello,
> >
> > How can I ignore accent in the query result ?
> >
> > Request :
> > http://*:8983/solr/***/select?defType=dismax=je+suis+avarié;
> > qf
> > =t
> > itle%5e20+subject%5e15+category%5e1+content%5e0.5=757
> >
> > I want to have doc with avarié and avarie.
> >
> > I have add this in my schema :
> >
> >   {
> > "name": "string",
> > "positionIncrementGap": "100",
> > "analyzer": {
> >   

[ANNOUNCE] Apache Solr 7.7.0 released

[jim ferenczi]

11 February 2019, Apache Solr™ 7.7.0 available

The Lucene PMC is pleased to announce the release of Apache Solr 7.7.0

Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project. Its major features include powerful full-text
search, hit highlighting, faceted search, dynamic clustering, database
integration, rich document (e.g., Word, PDF) handling, and geospatial
search. Solr is highly scalable, providing fault tolerant distributed
search and indexing, and powers the search and navigation features of many
of the world's largest internet sites.

Solr 7.7.0 is available for immediate download at:
http://lucene.apache.org/solr/downloads.html

See http://lucene.apache.org/solr/7_7_0/changes/Changes.html for a full
list of details.

Solr 7.7.0 Release Highlights:

Bug Fixes:
  * URI Too Long with large streaming expressions in SolrJ.
  * A failure while reloading a SolrCore can result in the SolrCore not
being closed.
  * Spellcheck parameters not working in new UI.
  * New Admin UI Query does not URL-encode the query produced in the URL
box.
  * Rule-base Authorization plugin skips authorization if querying node
does not have collection replica.
  * Solr installer fails on SuSE linux.
  * Fix incorrect SOLR_SSL_KEYSTORE_TYPE variable in solr start script.

Improvements:
  * JSON 'terms' Faceting now supports a 'prelim_sort' option to use when
initially selecting the top ranking buckets, prior to the final 'sort'
option used after refinement.
  * Add a login page to Admin UI, with initial support for Basic Auth and
Kerberos.
  * New Node-level health check handler at /admin/info/healthcheck and
/node/health paths that checks if the node is live, connected to zookeeper
and not shutdown.
  * It is now possible to configure a host whitelist for distributed search.

You are encouraged to thoroughly read the "Upgrade Notes" at
http://lucene.apache.org/solr/7_7_0/changes/Changes.html or in the
CHANGES.txt file accompanying the release.

Solr 7.7 also includes many other new features as well as numerous
optimizations and bugfixes of the corresponding Apache Lucene release.

Please report any feedback to the mailing lists (
http://lucene.apache.org/solr/community.html#mailing-lists-irc)

Note: The Apache Software Foundation uses an extensive mirroring network
for distributing releases. It is possible that the mirror you are using may
not have replicated the release yet. If that is the case, please try
another mirror. This also goes for Maven access.

Re: Ignore accent in a request

[Ere Maijala]

Please note that mapping characters works well for a small set of
characters, but if you want full UNICODE normalization, take a look at
the ICUFoldingFilter:
https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-ICUFoldingFilter

--Ere

elisabeth benoit kirjoitti 8.2.2019 klo 22.47:
> yes you do
> 
> and use the char filter at index and query time
> 
> Le ven. 8 févr. 2019 à 19:20, SAUNIER Maxence  a écrit :
> 
>> For the charFilter, I need to reindex all documents ?
>>
>> -Message d'origine-
>> De : Erick Erickson 
>> Envoyé : vendredi 8 février 2019 18:03
>> À : solr-user 
>> Objet : Re: Ignore accent in a request
>>
>> Elisabeth's suggestion is spot on for the accent.
>>
>> One other thing I noticed. You are using KeywordTokenizerFactory combined
>> with EdgeNGramFilterFactory. This implies that you can't search for
>> individual _words_, only prefix queries, i.e.
>> je
>> je s
>> je su
>> je sui
>> je suis
>>
>> You can't search for "suis" for instance.
>>
>> basically this is an efficient way to search anything starting with
>> three-or-more letter prefixes at the expense of index size. You might be
>> better off just using wildcards (restrict to three letters at the prefix
>> though).
>>
>> This is perfectly valid, I'm mostly asking if it's your intent.
>>
>> Best,
>> Erick
>>
>> On Fri, Feb 8, 2019 at 9:35 AM SAUNIER Maxence  wrote:
>>>
>>> Thanks you !
>>>
>>> -Message d'origine-
>>> De : elisabeth benoit  Envoyé : vendredi 8
>>> février 2019 14:12 À : solr-user@lucene.apache.org Objet : Re: Ignore
>>> accent in a request
>>>
>>> Hello,
>>>
>>> We use solr 7 and use
>>>
>>> >> mapping="mapping-ISOLatin1Accent.txt"/>
>>>
>>> with mapping-ISOLatin1Accent.txt
>>>
>>> containing lines like
>>>
>>> # À => A
>>> "\u00C0" => "A"
>>>
>>> # Á => A
>>> "\u00C1" => "A"
>>>
>>> # Â => A
>>> "\u00C2" => "A"
>>>
>>> # Ã => A
>>> "\u00C3" => "A"
>>>
>>> # Ä => A
>>> "\u00C4" => "A"
>>>
>>> # Å => A
>>> "\u00C5" => "A"
>>>
>>> # Ā Ă Ą =>
>>> "\u0100" => "A"
>>> "\u0102" => "A"
>>> "\u0104" => "A"
>>>
>>> # Æ => AE
>>> "\u00C6" => "AE"
>>>
>>> # Ç => C
>>> "\u00C7" => "C"
>>>
>>> # é => e
>>> "\u00E9" => "e"
>>>
>>> Best regards,
>>> Elisabeth
>>>
>>> Le ven. 8 févr. 2019 à 11:18, Gopesh Sharma 
>> a écrit :
>>>
 We have fixed this type of issue by using Synonyms by adding
 SynonymFilterFactory(Before Solr 7).

 -Original Message-
 From: SAUNIER Maxence 
 Sent: Friday, February 8, 2019 3:36 PM
 To: solr-user@lucene.apache.org
 Subject: RE: Ignore accent in a request

 Hello,

 Thanks for you answer.

 I have test :

 select?defType=dismax=je suis avarié=content
 90.000 results

 select?defType=dismax=je suis avarie=content
 60.000 results

 With avarié, I dont find documents with avarie and with avarie, I
 don't find documents with avarié.

 I want to find they 150.000 documents with avarié or avarie.

 Thanks

 -Message d'origine-
 De : Erick Erickson  Envoyé : jeudi 7
 février
 2019 19:37 À : solr-user  Objet : Re:
 Ignore accent in a request

 exactly _how_ is it "not working"?

 Try building your parameters _up_ rather than starting with a lot, e.g.
 select?defType=dismax=je suis avarié=title ^^ assumes you
 expect a match on title. Then:
 select?defType=dismax=je suis avarié=title subject

 etc.

 Because mm=757 looks really wrong. From the docs:
 Defines the minimum number of clauses that must match, regardless of
 how many clauses there are in total.

 edismax is used much more than dismax as it's more flexible, but
 that's not germane here.

 finally, try adding =query to the url to see exactly how the
 query is parsed.

 Best,
 Erick

 On Mon, Feb 4, 2019 at 9:09 AM SAUNIER Maxence 
>> wrote:
>
> Hello,
>
> How can I ignore accent in the query result ?
>
> Request :
> http://*:8983/solr/***/select?defType=dismax=je+suis+avarié;
> qf
> =t
> itle%5e20+subject%5e15+category%5e1+content%5e0.5=757
>
> I want to have doc with avarié and avarie.
>
> I have add this in my schema :
>
>   {
> "name": "string",
> "positionIncrementGap": "100",
> "analyzer": {
>   "filters": [
> {
>   "class": "solr.LowerCaseFilterFactory"
> },
> {
>   "class": "solr.ASCIIFoldingFilterFactory"
> },
> {
>   "class": "solr.EdgeNGramFilterFactory",
>   "minGramSize": "3",
>   "maxGramSize": "50"
> }
>   ],
>   "tokenizer": {
> "class": "solr.KeywordTokenizerFactory"
>   }
> },
> "stored": true,
> "indexed": true,
> "sortMissingLast": true,
> 

Re: Unable to create collection with custom queryParser Plugin

[Aroop Ganguly]

Thanks Erick, Jörn danke sehr.

tldr;
gradle trickery and thriftiness helped here.

detail:
To make things easier, for our deployment systems, I created a plugin gradle 
task which is economical and yet brings in the right number of jars.
in my case scala-lang jars were required, everything else was compile only. I 
used the compileOnly gradle dependency directive for all dependencies except 
scala-lang.
also had to remove shading.



> On Feb 10, 2019, at 11:37 PM, Erick Erickson  wrote:
> 
> What Jörn said.
> 
> Your jar should be nothing but your custom code. Usually I cheat in
> IntelliJ and check the box for artifacts that says something like
> "only compiled output"
> 
> On Sun, Feb 10, 2019 at 10:37 PM Jörn Franke  wrote:
>> 
>> You can put all solr dependencies as provided. They are already on the class 
>> path - no need to put them in the fat jar.
>> 
>>> Am 11.02.2019 um 05:59 schrieb Aroop Ganguly :
>>> 
>>> Thanks Erick!
>>> 
>>> I see. Yes it is a fat jar post shadowJar process (in the order of MBs).
>>> It contains solrj and solr-core dependencies plus a few more scala related 
>>> ones.
>>> I guess the solr-core dependencies are unavoidable (right ?), let me try to 
>>> trim the others.
>>> 
>>> Regards
>>> Aroop
>>> 
 On Feb 10, 2019, at 8:44 PM, Erick Erickson  
 wrote:
 
 Aroop:
 
 How big is your custom jar file? The name "test-plugins-aroop-all.jar"
 makes me suspicious. It should be very small and should _not_ contain
 any of the Solr distribution jar files, just your compiled custom
 code. I'm grasping at straws a bit, but it may be that you have the
 same jar files from the Solr distro and also included in your custom
 jar and it's confusing the classloader. "Very small" here is on the
 order of 10K given it does very little. If it's much bigger than, say,
 15K it's a red flag. If you do a "jar -dvf your_custom_jar" there
 should be _very_ few classes in it.
 
 Best,
 Erick
 
 On Sun, Feb 10, 2019 at 8:33 PM Aroop Ganguly
  wrote:
> 
> [resending due to bounce warning from the other email]
> 
> 
> Hi Team
> 
> I thought this was simple, but I am just missing something here. Any 
> guidance would be very appreciated.
> 
> What have I done so far:
>  1. I have created a custom querParser (class SamplePluggin extends 
> QParserPlugin { ), which right now does nothing but logs an info message, 
> and returns a new LuceneQParser() instance with the same parameters.
>  2. I am on solr 7.5 and I have added the path to the jar and 
> referenced the plugin in the following ways in my solrconfig.xml:
> 
>  
>   class="com.aroop.plugins.SamplePluggin"/>
> 
> Now when I create a collection with this solrconfig, I keep getting this 
> exception stack:
> I have tried debugging the live solr instance and for the life of me, I 
> cannot understand why am I getting this cast exception
> 2019-02-11 03:57:10.410 ERROR (qtp1594873248-62) [c:cvp2 s:shard1 
> r:core_node2 x:testCollection_shard1_replica_n1] 
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error 
> CREATEing SolrCore 'testCollection_shard1_replica_n1': Unable to create 
> core [testCollection_shard1_replica_n1] Caused by: class 
> com.aroop.plugins.SamplePluggin
>  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1087)
>  at 
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$247(CoreAdminOperation.java:92)
>  at 
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
>  at 
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
>  at 
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
>  at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>  at 
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
>  at 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
>  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
>  at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
>  at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
>  at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
>  at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>  at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>  at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>  at 
>