Re: Explicit OR in edismax query with mm=100%

[Yasufumi Mizoguchi]

Hi,

It looks that edismax respects the mm parameter in your case.
You should set "mm=1", if you want to obtain the results of OR search.
"mm=100%" means that all terms in your query should match.

Regards,
Yasufumi


On 2017/04/20 10:40, Nguyen Manh Tien wrote:

Hi,

I run a query "Solr OR Lucene" with defType=edismax and mm=100%.
The search result show that query works similar to "Solr AND Lucene" (all
terms required)

Does edismax ignore mm parameter because i already use OR explicitly here?

Thanks,
Tien

Explicit OR in edismax query with mm=100%

[Nguyen Manh Tien]

Hi,

I run a query "Solr OR Lucene" with defType=edismax and mm=100%.
The search result show that query works similar to "Solr AND Lucene" (all
terms required)

Does edismax ignore mm parameter because i already use OR explicitly here?

Thanks,
Tien

Questions about the read and write speed of solr

[hu.xiaodong]

Hi , all

 What is the speed of reading and writing about solr?

Can someone give me some data of Performance?




Thanks,

hu xiaodong.

Re: Solr Stream Content from URL

[Furkan KAMACI]

Hi Alexandre,

My content is protected via Basic Authentication. Is it possible to use
Basic Authentication with Solr Content Streams?

Kind Regards,
Furkan KAMACI

On Wed, Apr 19, 2017 at 9:13 PM, Alexandre Rafalovitch 
wrote:

> Have you tried stream.url parameter after enabling the
> enableRemoteStreaming flag?
> https://cwiki.apache.org/confluence/display/solr/Content+Streams
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 19 April 2017 at 13:27, Furkan KAMACI  wrote:
> > Hi,
> >
> > Is it possible to stream a CSV content from URL to Solr?
> >
> > I've tried URLDataSource but could not figure out about what to use as
> > document.
> >
> > Kind Regards,
> > Furkan KAMACI
>

Re: Solr Stream Content from URL

[Alexandre Rafalovitch]

Have you tried stream.url parameter after enabling the
enableRemoteStreaming flag?
https://cwiki.apache.org/confluence/display/solr/Content+Streams

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 19 April 2017 at 13:27, Furkan KAMACI  wrote:
> Hi,
>
> Is it possible to stream a CSV content from URL to Solr?
>
> I've tried URLDataSource but could not figure out about what to use as
> document.
>
> Kind Regards,
> Furkan KAMACI

Solr Stream Content from URL

[Furkan KAMACI]

Hi,

Is it possible to stream a CSV content from URL to Solr?

I've tried URLDataSource but could not figure out about what to use as
document.

Kind Regards,
Furkan KAMACI

Re: SolrIndexSearcher accumulation

[Elodie Sannier]

Yes, I didn't copy all our code but we also do extraReq.close(); in a
finally block. It was not the problem.

On 04/19/2017 11:53 AM, Mikhail Khludnev wrote:

If you create SolrQueryRequest make sure you close it then, since it's
necessary to release a searcher.

On Wed, Apr 19, 2017 at 12:35 PM, Elodie Sannier 
wrote:


Hello,

We have found how to fix the problem.
When we update the original SolrQueryResponse object, we need to create
a new BasicResultContext object with the extra response.

Simplified code :

public class CustomSearchHandler extends SearchHandler {

public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
rsp) throws Exception {

   SolrQueryRequest extraReq = createExtraRequest();
   SolrQueryResponse extraRsp = new SolrQueryResponse();

   super.handleRequestBody(extraReq, extraRsp);

   ResultContext extraRc = (ResultContext) extraRsp.getResponse();

   // code with memory leak !!
   rsp.addResponse(extraRc);

   // code without memory leak
   ResultContext extraRcClone = new BasicResultContext(extraRc.get
DocList(),
 rsp.getReturnFields(), req.getSearcher(),
extraRc.getQuery(), req);
   rsp.addResponse(extraRcClone);

}

}

We don't know why we need to create a new BasicResultContext to properly
manage searchers. Do you know why ?

Elodie


On 04/07/2017 04:14 PM, Rick Leir wrote:


Hi Gerald
The best solution in my mind is to look at the custom code and try to
find a way to remove it from your system. Solr queries can be complex, and
I hope there is a way to get the results you need. Would you like to say
what results you want to get, and what Solr queries you have tried?
I realize that in large organizations it is difficult to suggest change.
Cheers -- Rick

On April 7, 2017 9:08:19 AM EDT, Shawn Heisey 
wrote:


On 4/7/2017 3:09 AM, Gerald Reinhart wrote:


 We have some custom code that extends SearchHandler to be able to


:


  - do an extra request
  - merge/combine the original request and the extra request
results

 On Solr 5.x, our code was working very well, now with Solr 6.x we
have the following issue:  the number of SolrIndexSearcher are
increasing (we can see them in the admin view > Plugins/ Stats > Core


).


As SolrIndexSearcher are accumulating, we have the following issues :
 - the memory used by Solr is increasing => OOM after a long
period of time in production
 - some files in the index has been deleted from the system but
the Solr JVM still hold them => ("fake") Full disk after a long


period


of time in production

 We are wondering,
- what has changed between Solr 5.x and Solr 6.x in the
management of the SolrIndexSearcher ?
- what would be the best way, in a Solr plugin, to perform 2
queries and merge the results to a single SolrQueryResponse ?


I hesitated to send a reply because when it comes right down to it, I
do
not know a whole lot about deep Solr internals.  I tend to do my work
with the code at a higher level, and don't dive down in the depths all
that often.  I am slowly learning, though.  You may need to wait for a
reply from someone who really knows those internals.

It looks like you and I participated in a discussion last month where
you were facing a similar problem with searchers -- deleted index files
being held open.  How did that turn out?  Seems like if that problem
were solved, it would also solve this problem.

Very likely, the fact that the plugin worked correctly in 5.x was
actually a bug in Solr related to reference counting, one that has been
fixed in later versions.

You may need to use a paste website or a file-sharing website to share
all your plugin code so that people can get a look at it.  The list has
a habit of deleting attachments.

Thanks,
Shawn


Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à
l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
destinataire de ce message, merci de le détruire et d'en avertir
l'expéditeur.







Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: BUILD FAILED solr 6.5.0

[Steve Rowe]

Hi Bernd,

Your Ivy may be outdated - the project requires minimum 2.3.

Try removing all pre-2.3 ivy-*.jar files from ~/.ant/lib/, then running “ant 
ivy-bootstrap”.

--
Steve
www.lucidworks.com

> On Apr 19, 2017, at 10:55 AM, Bernd Fehling  
> wrote:
> 
> Tried today to have a look at solr 6.5.0.
> - download solr-6.5.0-src.tgz from apache.org and extracted to workspace
> - ant eclipse
> - imported to eclipse neon as new project
> - from eclipse in lucene subdir clicked on build.xml and selected
>  "Run As" --> "Ant Build..."
> - selected "package" and "Run"
> 
> Result:
> ...
>  [javadoc] Loading source files for package org.apache.lucene.search.spans...
>  [javadoc] Loading source files for package org.apache.lucene.store...
>  [javadoc] Loading source files for package org.apache.lucene.util...
>  [javadoc] Loading source files for package 
> org.apache.lucene.util.automaton...
>  [javadoc] Loading source files for package org.apache.lucene.util.fst...
>  [javadoc] Constructing Javadoc information...
>  [javadoc] Standard Doclet version 1.8.0_121
>  [javadoc] Building tree for all the packages and classes...
>  [javadoc] Building index for all the packages and classes...
>  [javadoc] Building index for all classes...
> [exec] Result: 128
>  [jar] Building jar:
> /srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/test-framework/lucene-test-framework-6.5.0-SNAPSHOT-javadoc.jar
> javadocs:
> changes-to-html:
>[mkdir] Created dir: 
> /srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/docs/changes
>   [delete] Deleting: 
> /srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/doap.lucene.version.dates.csv
> [copy] Copying 3 files to 
> /srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/docs/changes
> ivy-availability-check:
> ivy-fail:
> ivy-configure:
> [ivy:configure] :: loading settings :: file = 
> /srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/top-level-ivy-settings.xml
> resolve-groovy:
> [ivy:cachepath] :: resolving dependencies :: 
> org.codehaus.groovy#groovy-all-caller;working
> [ivy:cachepath]   confs: [default]
> [ivy:cachepath]   found org.codehaus.groovy#groovy-all;2.4.8 in public
> [ivy:cachepath] :: resolution report :: resolve 10ms :: artifacts dl 0ms
>   -
>   |  |modules||   artifacts   |
>   |   conf   | number| search|dwnlded|evicted|| number|dwnlded|
>   -
>   |  default |   1   |   0   |   0   |   0   ||   1   |   0   |
>   -
> resolve-markdown:
> 
> BUILD FAILED
> /srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/common-build.xml:2415:
>  ivy:cachepath doesn't support the nested "dependency" element.
> 
> 
> Any idea what is going wrong?
> 
> Something with ivy:dependency within ivy:cachepath, but how to fix it?
> 
> Regards
> Bernd

CDCR Source Errors

[Webster Homer]

I am seeing these errors in the log:

java.lang.Exception: Submitter stack trace
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:204)
at 
org.apache.solr.handler.CdcrReplicatorScheduler.lambda$start$1(CdcrReplicatorScheduler.java:76)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)



This had been working. The source cdcr looks like this

"requestHandler": {"/cdcr": {"name": "/cdcr","class":
"solr.CdcrRequestHandler","replica": {"zkHost":
"ae1b-ecom-mzk01:2181,ae1b-ecom-mzk02:2181,ae1b-ecom-mzk03:2181/solr","source":
"sial-catalog-product","target": "sial-catalog-product"},"replicator":
{"threadPoolSize": 4,"schedule": 10,"batchSize":
128},"updateLogSynchronizer": {"schedule": 6}}}


The target looks like this:

"/cdcr": {"name": "/cdcr","class": "solr.CdcrRequestHandler","buffer":
{"defaultState": "disabled"}}

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

BUILD FAILED solr 6.5.0

[Bernd Fehling]

Tried today to have a look at solr 6.5.0.
- download solr-6.5.0-src.tgz from apache.org and extracted to workspace
- ant eclipse
- imported to eclipse neon as new project
- from eclipse in lucene subdir clicked on build.xml and selected
  "Run As" --> "Ant Build..."
- selected "package" and "Run"

Result:
...
  [javadoc] Loading source files for package org.apache.lucene.search.spans...
  [javadoc] Loading source files for package org.apache.lucene.store...
  [javadoc] Loading source files for package org.apache.lucene.util...
  [javadoc] Loading source files for package org.apache.lucene.util.automaton...
  [javadoc] Loading source files for package org.apache.lucene.util.fst...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.8.0_121
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
 [exec] Result: 128
  [jar] Building jar:
/srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/test-framework/lucene-test-framework-6.5.0-SNAPSHOT-javadoc.jar
javadocs:
changes-to-html:
[mkdir] Created dir: 
/srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/docs/changes
   [delete] Deleting: 
/srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/doap.lucene.version.dates.csv
 [copy] Copying 3 files to 
/srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/build/docs/changes
ivy-availability-check:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = 
/srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/top-level-ivy-settings.xml
resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: 
org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.codehaus.groovy#groovy-all;2.4.8 in public
[ivy:cachepath] :: resolution report :: resolve 10ms :: artifacts dl 0ms
-
|  |modules||   artifacts   |
|   conf   | number| search|dwnlded|evicted|| number|dwnlded|
-
|  default |   1   |   0   |   0   |   0   ||   1   |   0   |
-
resolve-markdown:

BUILD FAILED
/srv/www/solr/workspace_neon_solr_6_5_0/solr-6.5.0/lucene/common-build.xml:2415:
 ivy:cachepath doesn't support the nested "dependency" element.


Any idea what is going wrong?

Something with ivy:dependency within ivy:cachepath, but how to fix it?

Regards
Bernd

Re: log4j configuration reload

[Shawn Heisey]

On 4/19/2017 1:27 AM, Vincenzo D'Amore wrote:
> I'm investigating around latest version of Solr (6.4.1) if it is possible
> reload the log4j.properties configuration file without Solr restart.
>
> As far as I understood it isn't possible, or at least, it's impossible
> without develop an ad hoc component.
>
> Do you know if there are workarounds or best practices about?

This would depend on exactly what you changed.  If the change involves
logging levels at the top level or for a specific class, you can simply
make the same change in the logging tab of the admin UI, and it will
take effect immediately.  If you changed something else, you will need
to restart Solr.

The log4j library itself may have the ability to make changes without a
restart, but since you're running an application that somebody else
wrote, you're stuck with what that application can do.  Logging levels
are the only thing that Solr currently has code to change on the fly.

Thanks,
Shawn

Re: Getting error while excuting full import

[Shawn Heisey]

On 4/18/2017 11:21 PM, ankur.168 wrote:
> I thought DIH does parallel db request for all the entities defined in a
> document.

I do not know anything about that.  It *could* be possible for all the
sub-entities just below another entity to run in parallel, but I've got
no idea whether this is the case.  At the top level, there is only one
thread handling documents one at a time, this I am sure of.

> I do believe that DIH is easier to use that's why I am trying to find a way 
> to use this in my current system. But as I explained above since I have so 
> many sub entities,each returns list of response which will be joined in to 
> parent. for more than 2 lacs document, full import is taking forever.
>
> What I am looking for is a way to speed up my full import using DIH only. To 
> achieve this I tried to split the document in 2 and do full import parallely. 
> but with this approach latest import overrides other document indexed data, 
> since unique key(property_id) is same for both documents.

The way to achieve top speed with DIH is to *not* define nested
entities.  Only define one entity with a single SELECT statement.  Let
the database handle all the JOIN work.  In my DIH config, I do "SELECT *
FROM X WHERE Y" ... X is a view defined on the database server that
handles all the JOINs, and Y is a fairly detailed conditional.

> One way I could think of is to keep document in different core which will 
> maintain different index files and merge the search results from both cores 
> while performing search on indexed data. But is this a good approach?

In order to do a sharded query, the uniqueKey field would need to be
unique across all cores.  My index is sharded manually, each shard does
a separate import when fully rebuilding the index.  The sharding
algorithm is coded into the SQL statement.

Thanks,
Shawn

Re: SolrIndexSearcher accumulation

[Mikhail Khludnev]

If you create SolrQueryRequest make sure you close it then, since it's
necessary to release a searcher.

On Wed, Apr 19, 2017 at 12:35 PM, Elodie Sannier 
wrote:

> Hello,
>
> We have found how to fix the problem.
> When we update the original SolrQueryResponse object, we need to create
> a new BasicResultContext object with the extra response.
>
> Simplified code :
>
> public class CustomSearchHandler extends SearchHandler {
>
> public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
> rsp) throws Exception {
>
>   SolrQueryRequest extraReq = createExtraRequest();
>   SolrQueryResponse extraRsp = new SolrQueryResponse();
>
>   super.handleRequestBody(extraReq, extraRsp);
>
>   ResultContext extraRc = (ResultContext) extraRsp.getResponse();
>
>   // code with memory leak !!
>   rsp.addResponse(extraRc);
>
>   // code without memory leak
>   ResultContext extraRcClone = new BasicResultContext(extraRc.get
> DocList(),
> rsp.getReturnFields(), req.getSearcher(),
> extraRc.getQuery(), req);
>   rsp.addResponse(extraRcClone);
>
> }
>
> }
>
> We don't know why we need to create a new BasicResultContext to properly
> manage searchers. Do you know why ?
>
> Elodie
>
>
> On 04/07/2017 04:14 PM, Rick Leir wrote:
>
>> Hi Gerald
>> The best solution in my mind is to look at the custom code and try to
>> find a way to remove it from your system. Solr queries can be complex, and
>> I hope there is a way to get the results you need. Would you like to say
>> what results you want to get, and what Solr queries you have tried?
>> I realize that in large organizations it is difficult to suggest change.
>> Cheers -- Rick
>>
>> On April 7, 2017 9:08:19 AM EDT, Shawn Heisey 
>> wrote:
>>
>>> On 4/7/2017 3:09 AM, Gerald Reinhart wrote:
>>>
 We have some custom code that extends SearchHandler to be able to

>>> :
>>>
  - do an extra request
  - merge/combine the original request and the extra request
 results

 On Solr 5.x, our code was working very well, now with Solr 6.x we
 have the following issue:  the number of SolrIndexSearcher are
 increasing (we can see them in the admin view > Plugins/ Stats > Core

>>> ).
>>>
 As SolrIndexSearcher are accumulating, we have the following issues :
 - the memory used by Solr is increasing => OOM after a long
 period of time in production
 - some files in the index has been deleted from the system but
 the Solr JVM still hold them => ("fake") Full disk after a long

>>> period
>>>
 of time in production

 We are wondering,
- what has changed between Solr 5.x and Solr 6.x in the
 management of the SolrIndexSearcher ?
- what would be the best way, in a Solr plugin, to perform 2
 queries and merge the results to a single SolrQueryResponse ?

>>> I hesitated to send a reply because when it comes right down to it, I
>>> do
>>> not know a whole lot about deep Solr internals.  I tend to do my work
>>> with the code at a higher level, and don't dive down in the depths all
>>> that often.  I am slowly learning, though.  You may need to wait for a
>>> reply from someone who really knows those internals.
>>>
>>> It looks like you and I participated in a discussion last month where
>>> you were facing a similar problem with searchers -- deleted index files
>>> being held open.  How did that turn out?  Seems like if that problem
>>> were solved, it would also solve this problem.
>>>
>>> Very likely, the fact that the plugin worked correctly in 5.x was
>>> actually a bug in Solr related to reference counting, one that has been
>>> fixed in later versions.
>>>
>>> You may need to use a paste website or a file-sharing website to share
>>> all your plugin code so that people can get a look at it.  The list has
>>> a habit of deleting attachments.
>>>
>>> Thanks,
>>> Shawn
>>>
>>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>



-- 
Sincerely yours
Mikhail Khludnev

Aliases feature scales?

[Yago Riveiro]

Hi, 

Does Anyone know if there is any theoretical limit related to the number of
aliases that a Solr cluster can handle?

If I create like 10K aliases would I experiment any kind of bottleneck?

Regards





-
Best regards

/Yago
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Aliases-feature-scales-tp4330721.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrIndexSearcher accumulation

[Elodie Sannier]

Hello,

We have found how to fix the problem.
When we update the original SolrQueryResponse object, we need to create
a new BasicResultContext object with the extra response.

Simplified code :

public class CustomSearchHandler extends SearchHandler {

public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse
rsp) throws Exception {

  SolrQueryRequest extraReq = createExtraRequest();
  SolrQueryResponse extraRsp = new SolrQueryResponse();

  super.handleRequestBody(extraReq, extraRsp);

  ResultContext extraRc = (ResultContext) extraRsp.getResponse();

  // code with memory leak !!
  rsp.addResponse(extraRc);

  // code without memory leak
  ResultContext extraRcClone = new BasicResultContext(extraRc.getDocList(),
rsp.getReturnFields(), req.getSearcher(),
extraRc.getQuery(), req);
  rsp.addResponse(extraRcClone);

}

}

We don't know why we need to create a new BasicResultContext to properly
manage searchers. Do you know why ?

Elodie

On 04/07/2017 04:14 PM, Rick Leir wrote:

Hi Gerald
The best solution in my mind is to look at the custom code and try to find a 
way to remove it from your system. Solr queries can be complex, and I hope 
there is a way to get the results you need. Would you like to say what results 
you want to get, and what Solr queries you have tried?
I realize that in large organizations it is difficult to suggest change.
Cheers -- Rick

On April 7, 2017 9:08:19 AM EDT, Shawn Heisey  wrote:

On 4/7/2017 3:09 AM, Gerald Reinhart wrote:

We have some custom code that extends SearchHandler to be able to

:

 - do an extra request
 - merge/combine the original request and the extra request
results

On Solr 5.x, our code was working very well, now with Solr 6.x we
have the following issue:  the number of SolrIndexSearcher are
increasing (we can see them in the admin view > Plugins/ Stats > Core

).

As SolrIndexSearcher are accumulating, we have the following issues :
- the memory used by Solr is increasing => OOM after a long
period of time in production
- some files in the index has been deleted from the system but
the Solr JVM still hold them => ("fake") Full disk after a long

period

of time in production

We are wondering,
   - what has changed between Solr 5.x and Solr 6.x in the
management of the SolrIndexSearcher ?
   - what would be the best way, in a Solr plugin, to perform 2
queries and merge the results to a single SolrQueryResponse ?

I hesitated to send a reply because when it comes right down to it, I
do
not know a whole lot about deep Solr internals.  I tend to do my work
with the code at a higher level, and don't dive down in the depths all
that often.  I am slowly learning, though.  You may need to wait for a
reply from someone who really knows those internals.

It looks like you and I participated in a discussion last month where
you were facing a similar problem with searchers -- deleted index files
being held open.  How did that turn out?  Seems like if that problem
were solved, it would also solve this problem.

Very likely, the fact that the plugin worked correctly in 5.x was
actually a bug in Solr related to reference counting, one that has been
fixed in later versions.

You may need to use a paste website or a file-sharing website to share
all your plugin code so that people can get a look at it.  The list has
a habit of deleting attachments.

Thanks,
Shawn



Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 158 Ter Rue du Temple 75003 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention 
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce 
message, merci de le détruire et d'en avertir l'expéditeur.

Re: log4j configuration reload

[Vincenzo D'Amore]

An update, I've found this:

http://logging.apache.org/log4j/1.2/faq.html#a3.6

*Is there a way to get log4j to automatically reload a configuration file
> if it changes?*



>
> Yes. Both the DOMConfigurator and the PropertyConfigurator support
> automatic reloading through the configureAndWatch method. See the API
> documentation for more details.
> Because the configureAndWatch launches a separate wathdog thread, and
> because there is no way to stop this thread in log4j 1.2, the
> configureAndWatch method is unsafe for use in J2EE envrironments where
> applications are recycled.


Anyone has tried it?


On Wed, Apr 19, 2017 at 9:27 AM, Vincenzo D'Amore 
wrote:

> Hi all,
>
> I'm investigating around latest version of Solr (6.4.1) if it is possible
> reload the log4j.properties configuration file without Solr restart.
>
> As far as I understood it isn't possible, or at least, it's impossible
> without develop an ad hoc component.
>
> Do you know if there are workarounds or best practices about?
>
> Best regards,
> Vincenzo
>
>
> --
> Vincenzo D'Amore
> email: v.dam...@gmail.com
> skype: free.dev
> mobile: +39 349 8513251 <349%20851%203251>
>



-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: Index and query time suggester behavior in a SolrCloud environment

[Andrea Gazzarini]

Hi,
any help out there?

BTW I forgot the Solr version: 6.5.0

Thanks,
Andrea

On 18/04/17 11:45, Andrea Gazzarini wrote:

Hi,
I have a project, with SolrCloud, where I'm going to use the Suggester 
component (BlendedInfixLookupFactory with DocumentDictionaryFactory).

Some info:

  * I will have a suggest-only collection, with no NRT requirements
(indexes will be updated with a daily frequency)
  * I'm not yet sure about the replication factor (I have to do some
checks)
  * I'm using Solrj on the client side

After reading some documentation I have a couple of doubts:

  * how the *suggest.build* command is working? Can I issue this
command towards just one node, and have that node forward the
request to the other nodes (so each of them can build its own
suggester index portion)?
  * how things are working at query time? Can I use send a request
with only suggest.q=... to my /suggest request handler and get
back distributed suggestions?

Thanks in advance
Andrea

RE: DistributedUpdateProcessorFactory was explicitly disabled from this updateRequestProcessorChain

[Pratik Thaker]

Hi Ishan,

After making suggested changes to solrconfig.xml, I did upconfig on all 3 SOLR 
VMs and restarted SOLR engines.

But still I am facing same issue. Is it something I am missing ?

Regards,
Pratik Thaker

-Original Message-
From: Ishan Chattopadhyaya [mailto:ichattopadhy...@gmail.com]
Sent: 14 April 2017 02:12
To: solr-user@lucene.apache.org
Subject: Re: DistributedUpdateProcessorFactory was explicitly disabled from 
this updateRequestProcessorChain

Why are you adding these update processors (esp. the
AddSchemaFieldsUpdateProcessor) after DistributedUpdateProcessor? Try adding 
them before DUP, and it has a better chance to work.

On Wed, Apr 12, 2017 at 3:44 PM, Pratik Thaker < 
pratik.tha...@smartstreamrdu.com> wrote:

> Hi All,
>
> I am facing this issue since very long, can you please provide your
> suggestion on it ?
>
> Regards,
> Pratik Thaker
>
> -Original Message-
> From: Pratik Thaker [mailto:pratik.tha...@smartstreamrdu.com]
> Sent: 09 February 2017 21:24
> To: 'solr-user@lucene.apache.org'
> Subject: RE: DistributedUpdateProcessorFactory was explicitly disabled
> from this updateRequestProcessorChain
>
> Hi Friends,
>
> Can you please try to give me some details about below issue ?
>
> Regards,
> Pratik Thaker
>
> From: Pratik Thaker
> Sent: 07 February 2017 17:12
> To: 'solr-user@lucene.apache.org'
> Subject: DistributedUpdateProcessorFactory was explicitly disabled
> from this updateRequestProcessorChain
>
> Hi All,
>
> I am using SOLR Cloud 6.0
>
> I am receiving below exception very frequently in solr logs,
>
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
> RunUpdateProcessor has received an AddUpdateCommand containing a
> document that appears to still contain Atomic document update
> operations, most likely because DistributedUpdateProcessorFactory was
> explicitly disabled from this updateRequestProcessorChain
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(
> RunUpdateProcessorFactory.java:63)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at
> org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessor
> Factory$AddSchemaFieldsUpdateProcessor.processAdd(
> AddSchemaFieldsUpdateProcessorFactory.java:335)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.
> processAdd(FieldMutatingUpdateProcessor.java:117)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.
> processAdd(FieldMutatingUpdateProcessor.java:117)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.
> processAdd(FieldMutatingUpdateProcessor.java:117)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.
> processAdd(FieldMutatingUpdateProcessor.java:117)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at
> org.apache.solr.update.processor.FieldNameMutatingUpdateProcess
> orFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.
> processAdd(FieldMutatingUpdateProcessor.java:117)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> doLocalAdd(DistributedUpdateProcessor.java:936)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> versionAdd(DistributedUpdateProcessor.java:1091)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.
> processAdd(DistributedUpdateProcessor.java:714)
> at org.apache.solr.update.processor.UpdateRequestProcessor.
> processAdd(UpdateRequestProcessor.java:48)
> at
> org.apache.solr.update.processor.AbstractDefaultValueUpdateProc
> essorFactory$DefaultValueUpdateProcessor.processAdd(
> AbstractDefaultValueUpdateProcessorFactory.java:93)
> at org.apache.solr.handler.loader.JavabinLoader$1.update(
> JavabinLoader.java:97)
>
> Can you please help me with the root cause ? Below is the snapshot of
> solrconfig,
>
> 
> 
> 
>
> 
>
> 
> 
>   [^\w-\.]
>   _
> 
> 
> 
> 
> 
>   
> -MM-dd'T'HH:mm:ss.SSSZ
> -MM-dd'T'HH:mm:ss,SSSZ
> 

log4j configuration reload

[Vincenzo D'Amore]

Hi all,

I'm investigating around latest version of Solr (6.4.1) if it is possible
reload the log4j.properties configuration file without Solr restart.

As far as I understood it isn't possible, or at least, it's impossible
without develop an ad hoc component.

Do you know if there are workarounds or best practices about?

Best regards,
Vincenzo


-- 
Vincenzo D'Amore
email: v.dam...@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: Upgrading cluster from 4 to 5. Slow replication detected.

[Himanshu Sachdeva]

I am guessing that the index has got corrupted somehow and deleted the data
directory on the slave. It has started copying the index. I'll report here
once that gets completed. If there is any other suggestion you might have
please reply back in the meantime. Thanks.

On Wed, Apr 19, 2017 at 12:21 PM, Himanshu Sachdeva 
wrote:

> Hello Shawn,
>
> Thanks for taking the time out to help me. I had assigned 45GB to the heap
> as starting memory and maximum memory it can use. The logs show the
> following two warnings repeatedly :
>
>- IndexFetcher : Cannot complete replication attempt because file
>already exists.
>- IndexFetcher : Replication attempt was not successful - trying a
>full index replication reloadCore=false.
>
>
>
> On Tue, Apr 18, 2017 at 6:58 PM, Shawn Heisey  wrote:
>
>> On 4/14/2017 2:10 AM, Himanshu Sachdeva wrote:
>> > We're starting to upgrade our solr cluster to version 5.5. So we
>> > removed one slave node from the cluster and installed solr 5.5.4 on it
>> > and started solr. So it started copying the index from the master.
>> > However, we noticed a drop in the replication speed compared to the
>> > other nodes which were still running solr 4. To do a fair comparison,
>> > I removed another slave node from the cluster and disabled replication
>> > on it till the new node has caught up with it. When both these nodes
>> > were at the same index generation, I turned replication on for both
>> > the nodes. Now, it has been over 15 hours since this exercise and the
>> > new node has again started lagging behind. Currently, the node with
>> > solr 5.5 is seven generations behind the other node.
>>
>> Version 5 is capable of replication bandwidth throttling, but unless you
>> actually configure the maxWriteMBPerSec attribute in the replication
>> handler definition, this should not happen by default.
>>
>> One problem that I think might be possible is that the heap has been
>> left at the default 512MB on the new 5.5.4 install and therefore the
>> machine is doing constant full garbage collections to free up memory for
>> normal operation, which would make Solr run EXTREMELY slowly.
>> Eventually a machine in this state would most likely encounter an
>> OutOfMemoryError.  On non-windows systems, OOME will cause a forced halt
>> of the entire Solr instance.
>>
>> The heap might not be the problem ... if it's not, then I do not know
>> what is going on.  Are there any errors or warnings in solr.log?
>>
>> Thanks,
>> Shawn
>>
>>
>
>
> --
> Himanshu Sachdeva
>
>


-- 
Himanshu Sachdeva

Re: Upgrading cluster from 4 to 5. Slow replication detected.

[Himanshu Sachdeva]

Hello Shawn,

Thanks for taking the time out to help me. I had assigned 45GB to the heap
as starting memory and maximum memory it can use. The logs show the
following two warnings repeatedly :

   - IndexFetcher : Cannot complete replication attempt because file
   already exists.
   - IndexFetcher : Replication attempt was not successful - trying a full
   index replication reloadCore=false.



On Tue, Apr 18, 2017 at 6:58 PM, Shawn Heisey  wrote:

> On 4/14/2017 2:10 AM, Himanshu Sachdeva wrote:
> > We're starting to upgrade our solr cluster to version 5.5. So we
> > removed one slave node from the cluster and installed solr 5.5.4 on it
> > and started solr. So it started copying the index from the master.
> > However, we noticed a drop in the replication speed compared to the
> > other nodes which were still running solr 4. To do a fair comparison,
> > I removed another slave node from the cluster and disabled replication
> > on it till the new node has caught up with it. When both these nodes
> > were at the same index generation, I turned replication on for both
> > the nodes. Now, it has been over 15 hours since this exercise and the
> > new node has again started lagging behind. Currently, the node with
> > solr 5.5 is seven generations behind the other node.
>
> Version 5 is capable of replication bandwidth throttling, but unless you
> actually configure the maxWriteMBPerSec attribute in the replication
> handler definition, this should not happen by default.
>
> One problem that I think might be possible is that the heap has been
> left at the default 512MB on the new 5.5.4 install and therefore the
> machine is doing constant full garbage collections to free up memory for
> normal operation, which would make Solr run EXTREMELY slowly.
> Eventually a machine in this state would most likely encounter an
> OutOfMemoryError.  On non-windows systems, OOME will cause a forced halt
> of the entire Solr instance.
>
> The heap might not be the problem ... if it's not, then I do not know
> what is going on.  Are there any errors or warnings in solr.log?
>
> Thanks,
> Shawn
>
>


-- 
Himanshu Sachdeva