Re: URI is too long

[Shawn Heisey]

On 1/31/2016 7:20 AM, Salman Ansari wrote:
> I am building a long query containing multiple ORs between query terms. I
> started to receive the following exception:
>
> The remote server returned an error: (414) Request-URI Too Long. Any idea
> what is the limit of the URL in Solr? Moreover, as a solution I was
> thinking of chunking the query into multiple requests but I was wondering
> if anyone has a better approach?

The default HTTP header size limit on most webservers and containers
(including the Jetty that ships with Solr) is 8192 bytes.  A typical
request like this will start with "GET " and end with " HTTP/1.1", which
count against that 8192 bytes.  The max header size can be increased.

If you place the parameters into a POST request instead of on the URL,
then the default size limit of that POST request in Solr is 2MB.  This
can also be increased.

Thanks,
Shawn

Re: MLT Component only returns ID and score

[Robert Brown]

Thanks for the info, does this replace the mlt component?  ie, can I 
remove that component from being loaded/used?


These are my parameters, the results look okay, just want to ensure I'm 
using it right...



  fq => [
  'market:uk'
],
  q => '{!mlt 
qf=name,description,brand,category,ean,upc,asin}az-uk-b017u8thna',

  rows => 10,
  start => 0,
  wt => 'json'






On 31/01/16 20:17, Upayavira wrote:

Try the MLT query parser, which is a much newer way of doing this.
Perhaps it will work better for you.

Upayavira

On Sun, Jan 31, 2016, at 06:31 PM, Robert Brown wrote:

Hi,

I've had to switch to using the MLT component, rather than the handler,
since I'm running on Solrcloud (5.4) and if I hit a node without the
starting document, I get nothing back.

When I perform a MLT query, I only get back the ID and score for the
similar documents, yet my fl=*,score.

"moreLikeThis" : [
"tg-uk-6336277-5820875618687434337",
{
   "maxScore" : 16.857872,
   "numFound" : 49559,
   "docs" : [
  {
 "score" : 16.857872,
 "id" : "tg-uk-6336277-6676971947462687384"
  },
  {
 "score" : 16.857872,
 "id" : "tg-uk-6336277-1922478172471276129"
  },



Here's my config...

  

  

  edismax

  explicit

  0.1

  *,score

  
  name^5
  brand^2
  category^3
  

  name

  100%

  100

  *:*

  
  name^5 description^2 brand^3 category^3
  

  name,description,ean,upc,asin,brand,category

  

  
  query
  facet
  mlt
  

  

Re: Determine if Merge is triggered in SOLR

[abhi Abhishek]

Hi All,
any suggestions/ ideas?

Thanks,
Abhishek

On Tue, Jan 26, 2016 at 9:16 PM, abhi Abhishek  wrote:

> Hi All,
> is there a way in SOLR to determine if a merge has been triggered in
> SOLR? is there a API exposed to query this?
>
> if its not available is there a way to do the same using lucene jar files
> available in the SOLR libs?
>
> Appreciate your help.
>
> Best Regards,
> Abhishek
>

Re: Determine if Merge is triggered in SOLR

[Jack Krupansky]

You would have to implement your own MergeScheduler that wrapped an
existing merge scheduler and then save the merge info and then write a
custom request handler to retrieve that saved info.

See:
https://lucene.apache.org/core/5_4_1/core/org/apache/lucene/index/MergeScheduler.html
https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig


-- Jack Krupansky

On Sun, Jan 31, 2016 at 1:59 PM, abhi Abhishek  wrote:

> Hi All,
> any suggestions/ ideas?
>
> Thanks,
> Abhishek
>
> On Tue, Jan 26, 2016 at 9:16 PM, abhi Abhishek 
> wrote:
>
> > Hi All,
> > is there a way in SOLR to determine if a merge has been triggered in
> > SOLR? is there a API exposed to query this?
> >
> > if its not available is there a way to do the same using lucene jar files
> > available in the SOLR libs?
> >
> > Appreciate your help.
> >
> > Best Regards,
> > Abhishek
> >
>

Re: Executing Collector's Collect method on more than one thread

[Joel Bernstein]

Before thinking at all about threads you might try to speeding things up
with your implementation. In particular your call to the top level
docValues is going to be very slow. The way to speed this up is to switch
to the segment level doc value at each segment switch. That way you avoid
the rather large overhead involved with top level String docValues. Then I
would change your scorer to work directly with BytesRef rather then
converting to the utf8 String.

Joel Bernstein
http://joelsolr.blogspot.com/

On Sun, Jan 31, 2016 at 9:13 AM, adfel70  wrote:

> I am using RankQuery to implement my applicative scorer that returns a
> score
> based on the value of specific field (lets call it 'score_field') that is
> stored for every document.
> The RankQuery creates a collector, and for every collected docId I retrieve
> the value of score_field, calculate the score and add the doc id into
> priority queue:
>
> public class MyScorerrankQuery extends RankQuery {
> ...
>
> @Override
> public TopDocsCollector getTopDocsCollector(int i,
> SolrIndexerSearcher.QueryCommand cmd, IndexSearcher searcher) {
> ...
> return new MyCollector(...)
> }
> }
>
> public class MyCollector  extends TopDocsCollector{
> MyScorer scorer;
> SortedDocValues scoreFieldValues;
>
>
> @Override
> public void collect(int id){
> int docID = docBase + id;
> //1. get specific field from the doc using
> DocValues and calculate score
> using my scorer
> String value =
> scoreFieldValues.get(docID).utf8ToString();
> scorer.calcScore(value);
> //2. add docId and score (ScoreDoc object) into
> PriorityQueue.
> }
> }
>
> Problem is that the calcScore may take ~20 ms per call, so if query returns
> 100,000 docs, which is not unusual, query execution time will be become 16
> minutes. Is there a way to parallelize collector's logic, so more than one
> thread would call calcScore simultaneously?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Executing-Collector-s-Collect-method-on-more-than-one-thread-tp4254269.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Full Indexing is geeting failed with hybris application

[Erik Hatcher]

Manoj - the log attachments don’t come through on the listserv.  Did you 
upgrade all nodes in your system?   What version of Solr is running?


—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com 



> On Jan 31, 2016, at 1:12 PM, Manoj Agrawal  wrote:
> 
> Hi,
> 
> One more thing i observed that i have multiple  cores and after running the
> indexing for a particualr core but all the core are indexednign as per log.
> 
> -Manoj
> 
> On Sun, Jan 31, 2016 at 10:32 AM, Manoj Agrawal 
> wrote:
> 
>> Hi Erik,
>> 
>> Thanks for helping on this! i checked that all the nodes are already
>> started. attaching the solr.log files for your reference.
>> 
>> 
>> 
>> On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher 
>> wrote:
>> 
>>> Manoj - looks like the master isn’t available?   Did you start up all
>>> nodes?
>>> 
>>> As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion
>>> work, so if you need any assistance let us know.
>>> 
>>> 
>>> 
>>> —
>>> Erik Hatcher, Senior Solutions Architect
>>> http://www.lucidworks.com 
>>> 
>>> 
>>> 
 On Jan 31, 2016, at 10:33 AM, Manoj Agrawal 
>>> wrote:
 
 Hi Team,
 
 
 
 We have migrated our application from hybris version 5.2 to version
>>> 5.5.1.4
 and we are using solr version 4.6.
 
 
 
 After running the full indexing , below error is coming in the
>>> application
 log. Could anyone help me on this . I can give also give you more
 information on this as well.
 
 
 
 Error:
 
 
 
 /--
 
 ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller;
>>> Master
 at: http://solr-master:8983/solr/Austria_aus_Product is not available.
 Index fetch failed. Exception: Timeout occured while waiting response
>>> from
 server at: http://solr-master:8983/solr/Austria_aus_Product
 
 ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller;
>>> Master
 at: http://solr-master:8983/solr/Regro_aur_Product_1 is not available.
 Index fetch failed. Exception: Timeout occured while waiting response
>>> from
 server at: http://solr-master:8983/solr/Regro_aur_Product_1
 
 ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller;
>>> Master
 at: http://solr-master:8983/solr/Regro_aur_Product is not available.
>>> Index
 fetch failed. Exception: Timeout occured while waiting response from
>>> server
 at: http://solr-master:8983/solr/Regro_aur_Product
 
 ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller;
>>> Master
 at: http://solr-master:8983/solr/Austria_aus_Product_1 is not
>>> available.
 Index fetch failed. Exception: Timeout occured while waiting response
>>> from
 server at: http://solr-master:8983/solr/Austria_aus_Product_1
 
 INFO  - 2016-01-29 20:14:18.210;
 org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
 path=/admin/info/logging
 params={_=1454094992955=1454094846369=json} status=0 QTime=0
 
 INFO  - 2016-01-29 20:14:29.206;
 org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
 path=/admin/info/logging
 params={_=1454095003954=1454094856824=json} status=0 QTime=0
 
 INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:14:40.214;
 org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
 path=/admin/info/logging
 params={_=1454095014957=1454094856824=json} status=0 QTime=0
 
 INFO  - 2016-01-29 20:14:51.223;
 org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
 path=/admin/info/logging
 params={_=1454095025954=1454094856824=json} status=0 QTime=0
 
 INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:15:02.198;
 org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
 path=/admin/info/logging
 params={_=1454095036958=1454094856824=json} status=0 QTime=0
 
 INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.
 
 INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller;
>>> Slave
 in sync with master.

RE: Which version of java is preferrable to install on a Red Hat Enterprise Linux 7?

[Gian Maria Ricci - aka Alkampfer]

Thanks a lot, Oracle Java 8 and is the version that I usually use, and I 
install a standard JRE.

Thanks again.

--
Gian Maria Ricci
Cell: +39 320 0136949


-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: mercoledì 27 gennaio 2016 16:07
To: solr-user@lucene.apache.org
Subject: Re: Which version of java is preferrable to install on a Red Hat 
Enterprise Linux 7?

On 1/27/2016 5:34 AM, Gian Maria Ricci - aka Alkampfer wrote:
>
> As for the subject, which version of java is preferrable to install to 
> run Solr 5.3.1 on RHEL 7?
>
>  
>
> I remember in old doc (https://wiki.apache.org/solr/SolrInstall) that 
> for full international charset support there is the need to install 
> full IDK, it is still true or is preferable to install standard JRE as 
> for the official documentation 
> (https://cwiki.apache.org/confluence/display/solr/Installing+Solr )
>

The full JDK is not required, the JRE is sufficient.

The java version I would personally recommend the most is the latest version of 
Oracle Java 8.  Next would be the latest version of OpenJDK 8, followed by the 
latest version of Oracle Java 7, then the latest OpenJDK 7.  I am pretty sure 
that you can easily install openjdk8 with yum on RHEL7, without adding any 
extra yum repositories or downloading anything from Oracle.

Whatever major Java version you choose, you should obtain the latest update 
version.  IBM Java should be avoided, because there are known bugs running 
under that implementation.

I'm not sure there is an actual official recommendation, but the
*minimum* requirement would be Java 7 Update 1.

Thanks,
Shawn

Re: URI is too long

[Jack Krupansky]

Or try the terms query parser that lets you eliminate all the OR operators:
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-TermsQueryParser


-- Jack Krupansky

On Sun, Jan 31, 2016 at 9:23 AM, Paul Libbrecht  wrote:

> How about using POST?
>
> paul
>
> > Salman Ansari 
> > 31 January 2016 at 15:20
> > Hi,
> >
> > I am building a long query containing multiple ORs between query terms. I
> > started to receive the following exception:
> >
> > The remote server returned an error: (414) Request-URI Too Long. Any idea
> > what is the limit of the URL in Solr? Moreover, as a solution I was
> > thinking of chunking the query into multiple requests but I was wondering
> > if anyone has a better approach?
> >
> > Regards,
> > Salman
> >
>
>

MLT Component only returns ID and score

[Robert Brown]

Hi,

I've had to switch to using the MLT component, rather than the handler, 
since I'm running on Solrcloud (5.4) and if I hit a node without the 
starting document, I get nothing back.


When I perform a MLT query, I only get back the ID and score for the 
similar documents, yet my fl=*,score.


"moreLikeThis" : [
  "tg-uk-6336277-5820875618687434337",
  {
 "maxScore" : 16.857872,
 "numFound" : 49559,
 "docs" : [
{
   "score" : 16.857872,
   "id" : "tg-uk-6336277-6676971947462687384"
},
{
   "score" : 16.857872,
   "id" : "tg-uk-6336277-1922478172471276129"
},



Here's my config...





edismax

explicit

0.1

*,score


name^5
brand^2
category^3


name

100%

100

*:*


name^5 description^2 brand^3 category^3


name="mlt.fl">name,description,ean,upc,asin,brand,category





query
facet
mlt

Re: Memory leak defect or misssuse of SolrJ API?

[Walter Underwood]

I already answered this.

Move the creation of the HttpSolrClient outside the loop. Your code will run 
much fast, because it will be able to reuse the connections.

Put another way, your program should have exactly as many HttpSolrClient 
objects as there are servers it talks to. If there is one Solr server, you have 
one object.

There is no leak in HttpSolrClient, you are misusing the class, massively.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jan 31, 2016, at 2:10 PM, Steven White  wrote:
> 
> Thank you all for your feedback.
> 
> This is code that I inherited and the example i gave is intended to
> demonstrate the memory leak which based on YourKit is
> on java/util/LinkedHashMap$Entry.  In short, I'm getting core dumps with
> "Detail "java/lang/OutOfMemoryError" "Java heap space" received "
> 
> Here is a more detailed layout of the code.  This is a crawler that runs
> 24x7 without any recycle logic in place:
> 
>init_data()
> 
>while (true)
>{
>HttpSolrClient client = new HttpSolrClient("
> http://localhost:8983/solr/core1 /");
>  this is real code
> 
>see_if_we_have_new_data();
> 
>send_new_data_to_solr();
> 
>client.close(); this is real code
> 
>sleep_for_a_bit(N); 'N' can be any positive int
>}
> 
> By default, our Java program is given 4gb of ram "-Xmx4g" and N is set for
> 5 min.  We had a customer set N to 10 second and we started seeing core
> dumps with OOM.  As I started to debug, I narrowed the OOM to
> HttpSolrClient per my original email.
> 
> The follow up answers I got suggest that I move the construction of
> HttpSolrClient object outside the while loop which I did (but I also had to
> move "client.close()" outside the loop) and the leak is gone.
> 
> Give this, is this how HttpSolrClient is suppose to be used?  If so, what's
> the point of HttpSolrClient.close()?
> 
> Another side question.  I noticed HttpSolrClient has a setBaseUrl().  Now,
> if I call it and give it "http://localhost:8983/solr/core1
> /" (ntoice the "/" at the end) next
> time I use HttpSolrClient to send Solr data, I get back 404. The fix is to
> remove the ending "/".  This is not how the constructor of HttpSolrClient
> behaves; HttpSolrClient will take the URL with or without "/".
> 
> In summary, it would be good if someone can confirm f we have a memory leak
> in HttpSolrClient if used per my example; if so this is a defect.  Also,
> can someone confirm the fix I used for this issue: move the constructor of
> HttpSolrClient outside the loop and reuse the existing object "client".
> 
> Again, thank you all for the quick response it is much appreciated.
> 
> Steve
> 
> 
> 
> On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson 
> wrote:
> 
>> Assuming you're not really using code like above and it's a test case
>> 
>> What's your evidence that memory consumption goes up? Are you sure
>> you're not just seeing uncollected garbage?
>> 
>> When I attached Java Mission Control to this program it looked pretty
>> scary at first, but the heap allocated after old generation garbage
>> collections leveled out to a steady state.
>> 
>> 
>> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood 
>> wrote:
>>> Create one HttpSolrClient object for each Solr server you are talking
>> to. Reuse it for all requests to that Solr server.
>>> 
>>> It will manage a pool of connections and keep them alive for faster
>> communication.
>>> 
>>> I took a look at the JavaDoc and the wiki doc, neither one explains this
>> well. I don’t think they even point out what is thread safe.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> http://observer.wunderwood.org/  (my blog)
>>> 
>>> 
 On Jan 30, 2016, at 7:42 AM, Susheel Kumar 
>> wrote:
 
 Hi Steve,
 
 Can you please elaborate what error you are getting and i didn't
>> understand
 your code above, that why initiating Solr client object  is in loop.  In
 general  creating client instance should be outside the loop and a one
>> time
 activity during the complete execution of program.
 
 Thanks,
 Susheel
 
 On Sat, Jan 30, 2016 at 8:15 AM, Steven White 
>> wrote:
 
> Hi folks,
> 
> I'm getting memory leak in my code.  I narrowed the code to the
>> following
> minimal to cause the leak.
> 
>   while (true) {
>   HttpSolrClient client = new HttpSolrClient("
> http://192.168.202.129:8983/solr/core1;);
>   client.close();
>   }
> 
> Is this a defect or an issue in the way I'm using HttpSolrClient?
> 
> I'm on Solr 5.2.1
> 
> Thanks.
> 
> Steve
> 
>>> 
>> 

RE: Monitor backup progress when location parameter is used.

[Gian Maria Ricci - aka Alkampfer]

Thanks for the link, I'll have a look as soon as possible.

--
Gian Maria Ricci
Cell: +39 320 0136949



-Original Message-
From: KNitin [mailto:nitin.t...@gmail.com] 
Sent: domenica 31 gennaio 2016 19:35
To: solr-user@lucene.apache.org
Subject: Re: Monitor backup progress when location parameter is used.

You can also checkout : https://github.com/bloomreach/solrcloud-haft for doing 
backup and restore of your solrcloud collections.

On Fri, Jan 15, 2016 at 12:23 AM,
 Gian Maria Ricci - aka Alkampfer <
alkamp...@nablasoft.com> wrote:

> Ok thanks, I also think that it's worth a jira, because for restore 
> operation we have a convenient restorestatus command that tells 
> exactly the status of the restore operation, I think that a 
> backupstatus command could be useful.
>
> --
> Gian Maria Ricci
> Cell: +39 320 0136949
>
>
> -Original Message-
> From: Jack Krupansky [mailto:jack.krupan...@gmail.com]
> Sent: giovedì 14 gennaio 2016 17:00
> To: solr-user@lucene.apache.org
> Subject: Re: Monitor backup progress when location parameter is used.
>
> I think the doc is wrong or at least misleading:
>
> https://cwiki.apache.org/confluence/display/solr/Making+and+Restoring+
> Backups+of+SolrCores
>
> "The backup operation can be monitored to see if it has completed by 
> sending the details command to the /replication handler..."
>
> From reading the code, it looks like the snapshot details are only 
> stored and returned after the snapsh
ot completes, either successfully or fails,
> but there is nothing set or reported if a snapshot is in progress. So, 
> if you don't see a "backup" section in the response, that means the 
> snapshot is in progress.
>
> I think it's worth a Jira - either to improve the doc or improve the 
> code to report backup as "inProgress... StartedAt...".
>
> You can also look at the log... "Creating backup snapshot" indicates 
> the backup has started and "Done creating backup snapshot" indicates 
> success or "Exception while creating snapshot" indicates failure. If 
> only that first message appeals, it means the backup is still in progress.
>
>
> -- Jack Krupansky
>
> On Thu, Jan 14, 2016 at 9:23 AM, Gian Maria Ricci - aka Alkampfer < 
> alkamp...@nablasoft.com> wrote:
>
> > If I start a backup operation using the location parameter
> >
> >
> >
> > *http://localhost:8983/solr/mycore/replication?command=backup=m
> > yc
> > ore&
> >  > ore&>location=z:\temp\backupmycore*
> >
> >
> >
> > How can I monitor when the backup operation is finished? Issuing a 
> > standard *details* operation
> >
> >
> >
> > *http://localhost:8983/solr/  mycore
> > /replication?command=details*
> >
> >
> >
> > does not gives me useful information, because there are no 
> > information on backup on returning data.
> >
> >
> >
> >
> >
> > 
> >
> >
> >
> > 
> >
> > 0
> >
> > 1
> >
> > 
> >
> > 
> >
> > 57.62 GB
> >
> >  > name="indexPath">X:\NoSql\Solr\solr-5.3.1\server\solr\mycore\data\in
> > de
> > x/
> >
> > 
> >
> > 
> >
> > 1452534703494
> >
> > <
long name="generation">1509
> >
> > 
> >
> > _2cw.fdt
> >
> > _2cw.fdx
> >
> > _2cw.fnm
> >
> > _2cw.nvd
> >
> > _2cw.nvm
> >
> > _2cw.si
> >
> > _2cw_Lucene50_0.doc
> >
> > _2cw_Lucene50_0.dvd
> >
> > _2cw_Lucene50_0.dvm
> >
> > _2cw_Lucene50_0.pos
> >
> > _2cw_Lucene50_0.tim
> >
> > _2cw_Lucene50_0.tip
> >
> > segments_15x
> >
> > 
> >
> > 
> >
> > 
> >
> > true
> >
> > false
> >
> > 1452534703494
> >
> > 1509
> >
> > 
> >
> >  > name="confFiles">schema.xml,stopwords.txt,elevate.xml
> >
> > 
> >
> > optimize
> >
> > 
> >
> > true
> >
> > 
> >
> > 
> >
> >
> >
> > 
> >
> > --
> > Gian Maria Ricci
> > Cell: +39 320 0136949
> >
> > [image:
> > https://ci5.googleusercontent.com/proxy/5oNMOYAeFXZ_LDKanNfoLRHC37mA
> > Zk 
> > VVhkPN7QxMdA0K5JW2m0bm8azJe7oWZMNt8fKHNX1bzrUTd-kIyE40CmwT2Mlf8OI=s0
> > -d -e1-ft#http://www.codewrecks.com/files/signature/mvp.png]
> > 
> [image:
> > https://ci3.googleusercontent.com/proxy/f-unQbmk6NtkHFspO5Y6x4jlIf_x
> > rm GLUT3fU9y_7VUHSFUjLs7aUIMdZQYTh3eWIA0sBnvNX3WGXCU59chKXLuAHi
2ArWdAcBcl
> > KA=s0-d-e1-ft#http://www.codewrecks.com/files/signature/linkedin.jpg
> > ]  [image:
> > https://ci3.googleusercontent.com/proxy/gjapMzu3KEakBQUstx_-cN7gHJ_G
> > pc 
> > 

Re: Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

Hi,

One more thing i observed that i have multiple  cores and after running the
indexing for a particualr core but all the core are indexednign as per log.

-Manoj

On Sun, Jan 31, 2016 at 10:32 AM, Manoj Agrawal 
wrote:

> Hi Erik,
>
> Thanks for helping on this! i checked that all the nodes are already
> started. attaching the solr.log files for your reference.
>
>
>
> On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher 
> wrote:
>
>> Manoj - looks like the master isn’t available?   Did you start up all
>> nodes?
>>
>> As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion
>> work, so if you need any assistance let us know.
>>
>>
>>
>> —
>> Erik Hatcher, Senior Solutions Architect
>> http://www.lucidworks.com 
>>
>>
>>
>> > On Jan 31, 2016, at 10:33 AM, Manoj Agrawal 
>> wrote:
>> >
>> > Hi Team,
>> >
>> >
>> >
>> > We have migrated our application from hybris version 5.2 to version
>> 5.5.1.4
>> > and we are using solr version 4.6.
>> >
>> >
>> >
>> > After running the full indexing , below error is coming in the
>> application
>> > log. Could anyone help me on this . I can give also give you more
>> > information on this as well.
>> >
>> >
>> >
>> > Error:
>> >
>> >
>> >
>> > /--
>> >
>> > ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller;
>> Master
>> > at: http://solr-master:8983/solr/Austria_aus_Product is not available.
>> > Index fetch failed. Exception: Timeout occured while waiting response
>> from
>> > server at: http://solr-master:8983/solr/Austria_aus_Product
>> >
>> > ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller;
>> Master
>> > at: http://solr-master:8983/solr/Regro_aur_Product_1 is not available.
>> > Index fetch failed. Exception: Timeout occured while waiting response
>> from
>> > server at: http://solr-master:8983/solr/Regro_aur_Product_1
>> >
>> > ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller;
>> Master
>> > at: http://solr-master:8983/solr/Regro_aur_Product is not available.
>> Index
>> > fetch failed. Exception: Timeout occured while waiting response from
>> server
>> > at: http://solr-master:8983/solr/Regro_aur_Product
>> >
>> > ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller;
>> Master
>> > at: http://solr-master:8983/solr/Austria_aus_Product_1 is not
>> available.
>> > Index fetch failed. Exception: Timeout occured while waiting response
>> from
>> > server at: http://solr-master:8983/solr/Austria_aus_Product_1
>> >
>> > INFO  - 2016-01-29 20:14:18.210;
>> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>> > path=/admin/info/logging
>> > params={_=1454094992955=1454094846369=json} status=0 QTime=0
>> >
>> > INFO  - 2016-01-29 20:14:29.206;
>> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>> > path=/admin/info/logging
>> > params={_=1454095003954=1454094856824=json} status=0 QTime=0
>> >
>> > INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:14:40.214;
>> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>> > path=/admin/info/logging
>> > params={_=1454095014957=1454094856824=json} status=0 QTime=0
>> >
>> > INFO  - 2016-01-29 20:14:51.223;
>> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>> > path=/admin/info/logging
>> > params={_=1454095025954=1454094856824=json} status=0 QTime=0
>> >
>> > INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:15:02.198;
>> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>> > path=/admin/info/logging
>> > params={_=1454095036958=1454094856824=json} status=0 QTime=0
>> >
>> > INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:15:04.000; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:15:04.001; org.apache.solr.handler.SnapPuller;
>> Slave
>> > in sync with master.
>> >
>> > INFO  - 2016-01-29 20:15:13.206;
>> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>> > path=/admin/info/logging
>> > params={_=1454095047963=1454094856824=json} status=0 QTime=0

Re: Memory leak defect or misssuse of SolrJ API?

[Steven White]

Thanks Walter.  Yes, I saw your answer and fixed the issue per your
suggestion.

The JavaDoc need to make this clear.  The fact there is a close() on this
class and the JavaDoc does not say "your program should have exactly as
many HttpSolrClient objects as there are servers it talks to" is a prime
candidate for missuses of the class.

Steve


On Sun, Jan 31, 2016 at 5:20 PM, Walter Underwood 
wrote:

> I already answered this.
>
> Move the creation of the HttpSolrClient outside the loop. Your code will
> run much fast, because it will be able to reuse the connections.
>
> Put another way, your program should have exactly as many HttpSolrClient
> objects as there are servers it talks to. If there is one Solr server, you
> have one object.
>
> There is no leak in HttpSolrClient, you are misusing the class, massively.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>
> > On Jan 31, 2016, at 2:10 PM, Steven White  wrote:
> >
> > Thank you all for your feedback.
> >
> > This is code that I inherited and the example i gave is intended to
> > demonstrate the memory leak which based on YourKit is
> > on java/util/LinkedHashMap$Entry.  In short, I'm getting core dumps with
> > "Detail "java/lang/OutOfMemoryError" "Java heap space" received "
> >
> > Here is a more detailed layout of the code.  This is a crawler that runs
> > 24x7 without any recycle logic in place:
> >
> >init_data()
> >
> >while (true)
> >{
> >HttpSolrClient client = new HttpSolrClient("
> > http://localhost:8983/solr/core1  >/");
> >  this is real code
> >
> >see_if_we_have_new_data();
> >
> >send_new_data_to_solr();
> >
> >client.close(); this is real code
> >
> >sleep_for_a_bit(N); 'N' can be any positive int
> >}
> >
> > By default, our Java program is given 4gb of ram "-Xmx4g" and N is set
> for
> > 5 min.  We had a customer set N to 10 second and we started seeing core
> > dumps with OOM.  As I started to debug, I narrowed the OOM to
> > HttpSolrClient per my original email.
> >
> > The follow up answers I got suggest that I move the construction of
> > HttpSolrClient object outside the while loop which I did (but I also had
> to
> > move "client.close()" outside the loop) and the leak is gone.
> >
> > Give this, is this how HttpSolrClient is suppose to be used?  If so,
> what's
> > the point of HttpSolrClient.close()?
> >
> > Another side question.  I noticed HttpSolrClient has a setBaseUrl().
> Now,
> > if I call it and give it "http://localhost:8983/solr/core1
> > /" (ntoice the "/" at the end)
> next
> > time I use HttpSolrClient to send Solr data, I get back 404. The fix is
> to
> > remove the ending "/".  This is not how the constructor of HttpSolrClient
> > behaves; HttpSolrClient will take the URL with or without "/".
> >
> > In summary, it would be good if someone can confirm f we have a memory
> leak
> > in HttpSolrClient if used per my example; if so this is a defect.  Also,
> > can someone confirm the fix I used for this issue: move the constructor
> of
> > HttpSolrClient outside the loop and reuse the existing object "client".
> >
> > Again, thank you all for the quick response it is much appreciated.
> >
> > Steve
> >
> >
> >
> > On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson  >
> > wrote:
> >
> >> Assuming you're not really using code like above and it's a test
> case
> >>
> >> What's your evidence that memory consumption goes up? Are you sure
> >> you're not just seeing uncollected garbage?
> >>
> >> When I attached Java Mission Control to this program it looked pretty
> >> scary at first, but the heap allocated after old generation garbage
> >> collections leveled out to a steady state.
> >>
> >>
> >> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <
> wun...@wunderwood.org>
> >> wrote:
> >>> Create one HttpSolrClient object for each Solr server you are talking
> >> to. Reuse it for all requests to that Solr server.
> >>>
> >>> It will manage a pool of connections and keep them alive for faster
> >> communication.
> >>>
> >>> I took a look at the JavaDoc and the wiki doc, neither one explains
> this
> >> well. I don’t think they even point out what is thread safe.
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wun...@wunderwood.org
> >>> http://observer.wunderwood.org/  (my blog)
> >>>
> >>>
>  On Jan 30, 2016, at 7:42 AM, Susheel Kumar 
> >> wrote:
> 
>  Hi Steve,
> 
>  Can you please elaborate what error you are getting and i didn't
> >> understand
>  your code above, that why initiating Solr client object  is in loop.
> In
>  general  creating client instance should be outside the loop and a one
> >> time
>  activity during the complete execution of program.
> 
>  Thanks,
> 

Re: Increasing maxMergedSegmentMB value

[Jack Krupansky]

Make sure you fully digest Mike McCandless' blog post on segment merge
before trying to outguess his code:
http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

Generally, I don't think you would want to merge just two segments.
Generally, you should do a bunch at a time, typically 10. IOW, take all the
segments on a tier and merge them into one segment at the next tier.

There is no documented practical upper limit for how big to make a single
segment, but very large segments are not likely to be optimized well in
Lucene, hence the default max merge size of 5GB. If you want to get a lot
above that, you're in uncharted territory. Besides, if you start pushing
your index well above the amount of available system memory your query
performance will suffer. I'd watch for the latter before pushing on the
former.


-- Jack Krupansky

On Sun, Jan 31, 2016 at 10:43 AM, Zheng Lin Edwin Yeo 
wrote:

> Thanks for your reply Shawn and Jack.
>
> I wanted to increase the segment size to 15GB, so that there will be lesser
> segments to search for during the query, which should potentially improve
> the query speed.
>
> What if I set the segment size to 20GB? Will all the existing 10GB segments
> be merge to 20GB, as now merging two 10GB segments will results in a 20GB
> segment?
>
> Regards,
> Edwin
>
>
> On 31 January 2016 at 12:16, Jack Krupansky 
> wrote:
>
> > From the Lucene MergePolicy Javadoc:
> >
> > "Whenever the segments in an index have been altered by IndexWriter
> > <
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/IndexWriter.html
> > >,
> > either the addition of a newly flushed segment, addition of many segments
> > from addIndexes* calls, or a previous merge that may now need to cascade,
> > IndexWriter
> > <
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/IndexWriter.html
> > >
> >  invokes findMerges(org.apache.lucene.index.MergeTrigger,
> > org.apache.lucene.index.SegmentInfos,
> org.apache.lucene.index.IndexWriter)
> > <
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html#findMerges(org.apache.lucene.index.MergeTrigger
> > ,
> > org.apache.lucene.index.SegmentInfos,
> > org.apache.lucene.index.IndexWriter)> to
> > give the MergePolicy a chance to pick merges that are now required. This
> > method returns a MergePolicy.MergeSpecification
> > <
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.MergeSpecification.html
> > >
> > instance
> > describing the set of merges that should be done, or null if no merges
> are
> > necessary. When IndexWriter.forceMerge is called, it calls
> > findForcedMerges(SegmentInfos,int,Map,
> > IndexWriter)
> > <
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html#findForcedMerges(org.apache.lucene.index.SegmentInfos
> > ,
> > int, java.util.Map, org.apache.lucene.index.IndexWriter)> and the
> > MergePolicy should then return the necessary merges."
> >
> > See:
> >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html
> >
> > IOW, when the next commit occurs that closes and flushes the currently
> open
> > segment.
> >
> > Nothing will happen to any existing 10GB segments, now or ever in the
> > future since merging two 10GB segments would not be possible with a limit
> > of only 15GB.
> >
> > Maybe you could clue us in as to what effect you are trying to achieve. I
> > mean, why should any app care whether segments are 10GB or 15GB?
> >
> >
> > -- Jack Krupansky
> >
> > On Sat, Jan 30, 2016 at 6:28 PM, Shawn Heisey 
> wrote:
> >
> > > On 1/30/2016 7:31 AM, Zheng Lin Edwin Yeo wrote:
> > > > I would like to find out, when I increase the maxMergedSegmentMB from
> > > 10240
> > > > (10GB) to 15360 (15GB), will all the 10GB segments that were created
> > > > previously be automatically merge to 15GB?
> > >
> > > Not necessarily.  It will make those 10GB+ segments eligible for
> further
> > > merging, whereas they would have been ineligible before the change.
> > >
> > > This might mean that one or more of those large segments will be merged
> > > soon after the change and restart/reload, but I do not know when it
> > > might happen.  It would probably wait until at least one new segment
> was
> > > created, at which time the merge policy would be consulted.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>

Re: User-defined properties and configsets

[Erick Erickson]

Ah,OK. There's no mechanism that I know of for that.

I'd guess system properties wouldn't even work as two cores in the
same JVM would potentially want different values for the same embedded
system variable if they used different configsets.

So I think you're stuck at present with specifying the properties at
core creation time. I can imagine scripts that read your common
properties file and auto generate the CREATE call, but that would be
custom...

Best,
Erick

On Sun, Jan 31, 2016 at 3:36 AM, Georg Sorst  wrote:
> Erick Erickson  schrieb am Fr., 29. Jan. 2016 um
> 17:55 Uhr:
>
>> These are system properties, right? They go in the startup for all of
>> your Solr instances scattered about your cluster.
>>
>
> No, they will be used in the solrconfig.xml and schema.xml. I just
> mentioned the -Dmyproperty=... example because that's one way to get
> properties in there.
>
> What I'm looking for is one concise place to define properties that all
> cores using this configset will use as a default. I've tried several ways:
>
> * Using the ${property:default} syntax in the configs is no good, because
> the same property will occur several times in the configs
> * Setting them with the config API is no good, because then they live in my
> code, but I'd rather have them in a property file for visiblity and
> maintenance reasons
> * Setting them as system properties (-Dmyproperty=...) is no good, because
> that makes our deployment more complicated
>
> So, ideally I can just put a .properties file in the configset that will
> provide default values for all cores using this configset.
>
> Of course the properties may be changed from their default values on a
> per-core base, but that's what the config API is for.
>
> So, where do other people put their properties when they use configsets?
>
> Best regards,
> Georg
>
>
>>
>> The bin/solr script has a -a option for passing additional stuff to the
>> JVM...
>>
>> Best,
>> Erick
>>
>> On Thu, Jan 28, 2016 at 11:50 PM, Georg Sorst 
>> wrote:
>> > Any takers?
>> >
>> > Georg Sorst  schrieb am So., 24. Jän. 2016
>> 00:22:
>> >
>> >> Hi list!
>> >>
>> >> I've just started playing with Solr 5 (upgrading from Solr 4) and want
>> to
>> >> use configsets. I'm currently struggling with how to use user-defined
>> >> properties and configsets together.
>> >>
>> >> My solrconfig.xml contains a few properties. Previously these were in a
>> >> solrcore.properties and thus were properly loaded and substituted by
>> Solr.
>> >>
>> >> Now I've moved my configuration to a configset (as I may need to create
>> >> several cores with the same config). When I create a core with
>> >>
>> http://localhost:8983/solr/admin/cores?action=CREATE=mycore=myconfigset
>> Solr
>> >> tells me:
>> >>
>> >> Caused by: org.apache.solr.common.SolrException: Error loading solr
>> config
>> >> from //configsets/myconfigset/conf/solrconfig.xml
>> >> at
>> >>
>> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:186)
>> >> at
>> >>
>> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:94)
>> >> at
>> >>
>> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:74)
>> >> ... 30 more
>> >> Caused by: org.apache.solr.common.SolrException: No system property or
>> >> default value specified for  value:
>> >> 
>> >> at
>> >>
>> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:66)
>> >> ...
>> >>
>> >> Where should I put my properties so Solr can load them when I create a
>> new
>> >> core using this config set? From what I read I could specify them as
>> system
>> >> properties (-Dmyproperty=...) but I'd rather keep them in a file that I
>> can
>> >> check in.
>> >>
>> >> Thanks!
>> >> Georg
>> >>
>> >>
>> >>
>>

Re: Memory leak defect or misssuse of SolrJ API?

[Walter Underwood]

The JavaDoc needs a lot more information. As I remember it, SolrJ started as a 
thin layer over Apache HttpClient, so the authors may have assumed that 
programmers were familiar with that library. HttpClient makes a shared object 
that manages a pool of connections to the target server. HttpClient is 
seriously awesome—I first used it in the late 1990’s when I hit the limitations 
of the URL classes written by Sun.

I looked at the JavaDoc and various examples and none of them make this clear. 
Not your fault, we need a serious upgrade on those docs.

On the plus side, your program should be a lot faster after you reuse the 
client class.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jan 31, 2016, at 3:46 PM, Steven White  wrote:
> 
> Thanks Walter.  Yes, I saw your answer and fixed the issue per your
> suggestion.
> 
> The JavaDoc need to make this clear.  The fact there is a close() on this
> class and the JavaDoc does not say "your program should have exactly as
> many HttpSolrClient objects as there are servers it talks to" is a prime
> candidate for missuses of the class.
> 
> Steve
> 
> 
> On Sun, Jan 31, 2016 at 5:20 PM, Walter Underwood 
> wrote:
> 
>> I already answered this.
>> 
>> Move the creation of the HttpSolrClient outside the loop. Your code will
>> run much fast, because it will be able to reuse the connections.
>> 
>> Put another way, your program should have exactly as many HttpSolrClient
>> objects as there are servers it talks to. If there is one Solr server, you
>> have one object.
>> 
>> There is no leak in HttpSolrClient, you are misusing the class, massively.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 
>>> On Jan 31, 2016, at 2:10 PM, Steven White  wrote:
>>> 
>>> Thank you all for your feedback.
>>> 
>>> This is code that I inherited and the example i gave is intended to
>>> demonstrate the memory leak which based on YourKit is
>>> on java/util/LinkedHashMap$Entry.  In short, I'm getting core dumps with
>>> "Detail "java/lang/OutOfMemoryError" "Java heap space" received "
>>> 
>>> Here is a more detailed layout of the code.  This is a crawler that runs
>>> 24x7 without any recycle logic in place:
>>> 
>>>   init_data()
>>> 
>>>   while (true)
>>>   {
>>>   HttpSolrClient client = new HttpSolrClient("
>>> http://localhost:8983/solr/core1 >> /");
>>>  this is real code
>>> 
>>>   see_if_we_have_new_data();
>>> 
>>>   send_new_data_to_solr();
>>> 
>>>   client.close(); this is real code
>>> 
>>>   sleep_for_a_bit(N); 'N' can be any positive int
>>>   }
>>> 
>>> By default, our Java program is given 4gb of ram "-Xmx4g" and N is set
>> for
>>> 5 min.  We had a customer set N to 10 second and we started seeing core
>>> dumps with OOM.  As I started to debug, I narrowed the OOM to
>>> HttpSolrClient per my original email.
>>> 
>>> The follow up answers I got suggest that I move the construction of
>>> HttpSolrClient object outside the while loop which I did (but I also had
>> to
>>> move "client.close()" outside the loop) and the leak is gone.
>>> 
>>> Give this, is this how HttpSolrClient is suppose to be used?  If so,
>> what's
>>> the point of HttpSolrClient.close()?
>>> 
>>> Another side question.  I noticed HttpSolrClient has a setBaseUrl().
>> Now,
>>> if I call it and give it "http://localhost:8983/solr/core1
>>> /" (ntoice the "/" at the end)
>> next
>>> time I use HttpSolrClient to send Solr data, I get back 404. The fix is
>> to
>>> remove the ending "/".  This is not how the constructor of HttpSolrClient
>>> behaves; HttpSolrClient will take the URL with or without "/".
>>> 
>>> In summary, it would be good if someone can confirm f we have a memory
>> leak
>>> in HttpSolrClient if used per my example; if so this is a defect.  Also,
>>> can someone confirm the fix I used for this issue: move the constructor
>> of
>>> HttpSolrClient outside the loop and reuse the existing object "client".
>>> 
>>> Again, thank you all for the quick response it is much appreciated.
>>> 
>>> Steve
>>> 
>>> 
>>> 
>>> On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson >> 
>>> wrote:
>>> 
 Assuming you're not really using code like above and it's a test
>> case
 
 What's your evidence that memory consumption goes up? Are you sure
 you're not just seeing uncollected garbage?
 
 When I attached Java Mission Control to this program it looked pretty
 scary at first, but the heap allocated after old generation garbage
 collections leveled out to a steady state.
 
 
 On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <
>> wun...@wunderwood.org>
 wrote:
> Create one HttpSolrClient object for each Solr server you are talking

Re: Monitor backup progress when location parameter is used.

[KNitin]

You can also checkout : https://github.com/bloomreach/solrcloud-haft for
doing backup and restore of your solrcloud collections.

On Fri, Jan 15, 2016 at 12:23 AM, Gian Maria Ricci - aka Alkampfer <
alkamp...@nablasoft.com> wrote:

> Ok thanks, I also think that it's worth a jira, because for restore
> operation we have a convenient restorestatus command that tells exactly the
> status of the restore operation, I think that a backupstatus command could
> be useful.
>
> --
> Gian Maria Ricci
> Cell: +39 320 0136949
>
>
> -Original Message-
> From: Jack Krupansky [mailto:jack.krupan...@gmail.com]
> Sent: giovedì 14 gennaio 2016 17:00
> To: solr-user@lucene.apache.org
> Subject: Re: Monitor backup progress when location parameter is used.
>
> I think the doc is wrong or at least misleading:
>
> https://cwiki.apache.org/confluence/display/solr/Making+and+Restoring+Backups+of+SolrCores
>
> "The backup operation can be monitored to see if it has completed by
> sending the details command to the /replication handler..."
>
> From reading the code, it looks like the snapshot details are only stored
> and returned after the snapshot completes, either successfully or fails,
> but there is nothing set or reported if a snapshot is in progress. So, if
> you don't see a "backup" section in the response, that means the snapshot
> is in progress.
>
> I think it's worth a Jira - either to improve the doc or improve the code
> to report backup as "inProgress... StartedAt...".
>
> You can also look at the log... "Creating backup snapshot" indicates the
> backup has started and "Done creating backup snapshot" indicates success or
> "Exception while creating snapshot" indicates failure. If only that first
> message appeals, it means the backup is still in progress.
>
>
> -- Jack Krupansky
>
> On Thu, Jan 14, 2016 at 9:23 AM, Gian Maria Ricci - aka Alkampfer <
> alkamp...@nablasoft.com> wrote:
>
> > If I start a backup operation using the location parameter
> >
> >
> >
> > *http://localhost:8983/solr/mycore/replication?command=backup=myc
> > ore&
> >  > ore&>location=z:\temp\backupmycore*
> >
> >
> >
> > How can I monitor when the backup operation is finished? Issuing a
> > standard *details* operation
> >
> >
> >
> > *http://localhost:8983/solr/  mycore
> > /replication?command=details*
> >
> >
> >
> > does not gives me useful information, because there are no information
> > on backup on returning data.
> >
> >
> >
> >
> >
> > 
> >
> >
> >
> > 
> >
> > 0
> >
> > 1
> >
> > 
> >
> > 
> >
> > 57.62 GB
> >
> >  > name="indexPath">X:\NoSql\Solr\solr-5.3.1\server\solr\mycore\data\inde
> > x/
> >
> > 
> >
> > 
> >
> > 1452534703494
> >
> > 1509
> >
> > 
> >
> > _2cw.fdt
> >
> > _2cw.fdx
> >
> > _2cw.fnm
> >
> > _2cw.nvd
> >
> > _2cw.nvm
> >
> > _2cw.si
> >
> > _2cw_Lucene50_0.doc
> >
> > _2cw_Lucene50_0.dvd
> >
> > _2cw_Lucene50_0.dvm
> >
> > _2cw_Lucene50_0.pos
> >
> > _2cw_Lucene50_0.tim
> >
> > _2cw_Lucene50_0.tip
> >
> > segments_15x
> >
> > 
> >
> > 
> >
> > 
> >
> > true
> >
> > false
> >
> > 1452534703494
> >
> > 1509
> >
> > 
> >
> >  > name="confFiles">schema.xml,stopwords.txt,elevate.xml
> >
> > 
> >
> > optimize
> >
> > 
> >
> > true
> >
> > 
> >
> > 
> >
> >
> >
> > 
> >
> > --
> > Gian Maria Ricci
> > Cell: +39 320 0136949
> >
> > [image:
> > https://ci5.googleusercontent.com/proxy/5oNMOYAeFXZ_LDKanNfoLRHC37mAZk
> > VVhkPN7QxMdA0K5JW2m0bm8azJe7oWZMNt8fKHNX1bzrUTd-kIyE40CmwT2Mlf8OI=s0-d
> > -e1-ft#http://www.codewrecks.com/files/signature/mvp.png]
> > 
> [image:
> > https://ci3.googleusercontent.com/proxy/f-unQbmk6NtkHFspO5Y6x4jlIf_xrm
> > GLUT3fU9y_7VUHSFUjLs7aUIMdZQYTh3eWIA0sBnvNX3WGXCU59chKXLuAHi2ArWdAcBcl
> > KA=s0-d-e1-ft#http://www.codewrecks.com/files/signature/linkedin.jpg]
> >  [image:
> > https://ci3.googleusercontent.com/proxy/gjapMzu3KEakBQUstx_-cN7gHJ_Gpc
> > IZNEPjCzOYMrPl-r1DViPE378qNAQyEWbXMTj6mcduIAGaApe9qHG1KN_hyFxQAIkdNSVT
> > =s0-d-e1-ft#http://www.codewreck
> s.com/files/signature/twitter.jpg]
> >  [image:
> > https://ci5.googleusercontent.com/proxy/iuDOD2sdaxRDvTwS8MO7-CcXchpNJX
> > 96uaWuvagoVLcjpAPsJi88XeOonE4vHT6udVimo7yL9ZtdrYueEfH7jXnudmi_Vvw=s0-d
> > -e1-ft#http://www.codewrecks.com/files/signature/rss.jpg]
> > 

Re: MLT Component only returns ID and score

[Upayavira]

Try the MLT query parser, which is a much newer way of doing this.
Perhaps it will work better for you.

Upayavira

On Sun, Jan 31, 2016, at 06:31 PM, Robert Brown wrote:
> Hi,
> 
> I've had to switch to using the MLT component, rather than the handler, 
> since I'm running on Solrcloud (5.4) and if I hit a node without the 
> starting document, I get nothing back.
> 
> When I perform a MLT query, I only get back the ID and score for the 
> similar documents, yet my fl=*,score.
> 
> "moreLikeThis" : [
>"tg-uk-6336277-5820875618687434337",
>{
>   "maxScore" : 16.857872,
>   "numFound" : 49559,
>   "docs" : [
>  {
> "score" : 16.857872,
> "id" : "tg-uk-6336277-6676971947462687384"
>  },
>  {
> "score" : 16.857872,
> "id" : "tg-uk-6336277-1922478172471276129"
>  },
> 
> 
> 
> Here's my config...
> 
>  
> 
>  
> 
>  edismax
> 
>  explicit
> 
>  0.1
> 
>  *,score
> 
>  
>  name^5
>  brand^2
>  category^3
>  
> 
>  name
> 
>  100%
> 
>  100
> 
>  *:*
> 
>  
>  name^5 description^2 brand^3 category^3
>  
> 
>   name="mlt.fl">name,description,ean,upc,asin,brand,category
> 
>  
> 
>  
>  query
>  facet
>  mlt
>  
> 
>  
> 
> 
> 
> 

Re: Memory leak defect or misssuse of SolrJ API?

[Steven White]

Thank you all for your feedback.

This is code that I inherited and the example i gave is intended to
demonstrate the memory leak which based on YourKit is
on java/util/LinkedHashMap$Entry.  In short, I'm getting core dumps with
"Detail "java/lang/OutOfMemoryError" "Java heap space" received "

Here is a more detailed layout of the code.  This is a crawler that runs
24x7 without any recycle logic in place:

init_data()

while (true)
{
HttpSolrClient client = new HttpSolrClient("
http://localhost:8983/solr/core1 /");
  this is real code

see_if_we_have_new_data();

send_new_data_to_solr();

client.close(); this is real code

sleep_for_a_bit(N); 'N' can be any positive int
}

By default, our Java program is given 4gb of ram "-Xmx4g" and N is set for
5 min.  We had a customer set N to 10 second and we started seeing core
dumps with OOM.  As I started to debug, I narrowed the OOM to
HttpSolrClient per my original email.

The follow up answers I got suggest that I move the construction of
HttpSolrClient object outside the while loop which I did (but I also had to
move "client.close()" outside the loop) and the leak is gone.

Give this, is this how HttpSolrClient is suppose to be used?  If so, what's
the point of HttpSolrClient.close()?

Another side question.  I noticed HttpSolrClient has a setBaseUrl().  Now,
if I call it and give it "http://localhost:8983/solr/core1
/" (ntoice the "/" at the end) next
time I use HttpSolrClient to send Solr data, I get back 404. The fix is to
remove the ending "/".  This is not how the constructor of HttpSolrClient
behaves; HttpSolrClient will take the URL with or without "/".

In summary, it would be good if someone can confirm f we have a memory leak
in HttpSolrClient if used per my example; if so this is a defect.  Also,
can someone confirm the fix I used for this issue: move the constructor of
HttpSolrClient outside the loop and reuse the existing object "client".

Again, thank you all for the quick response it is much appreciated.

Steve



On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson 
wrote:

> Assuming you're not really using code like above and it's a test case
>
> What's your evidence that memory consumption goes up? Are you sure
> you're not just seeing uncollected garbage?
>
> When I attached Java Mission Control to this program it looked pretty
> scary at first, but the heap allocated after old generation garbage
> collections leveled out to a steady state.
>
>
> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood 
> wrote:
> > Create one HttpSolrClient object for each Solr server you are talking
> to. Reuse it for all requests to that Solr server.
> >
> > It will manage a pool of connections and keep them alive for faster
> communication.
> >
> > I took a look at the JavaDoc and the wiki doc, neither one explains this
> well. I don’t think they even point out what is thread safe.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >
> >> On Jan 30, 2016, at 7:42 AM, Susheel Kumar 
> wrote:
> >>
> >> Hi Steve,
> >>
> >> Can you please elaborate what error you are getting and i didn't
> understand
> >> your code above, that why initiating Solr client object  is in loop.  In
> >> general  creating client instance should be outside the loop and a one
> time
> >> activity during the complete execution of program.
> >>
> >> Thanks,
> >> Susheel
> >>
> >> On Sat, Jan 30, 2016 at 8:15 AM, Steven White 
> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> I'm getting memory leak in my code.  I narrowed the code to the
> following
> >>> minimal to cause the leak.
> >>>
> >>>while (true) {
> >>>HttpSolrClient client = new HttpSolrClient("
> >>> http://192.168.202.129:8983/solr/core1;);
> >>>client.close();
> >>>}
> >>>
> >>> Is this a defect or an issue in the way I'm using HttpSolrClient?
> >>>
> >>> I'm on Solr 5.2.1
> >>>
> >>> Thanks.
> >>>
> >>> Steve
> >>>
> >
>

Re: URI is too long

[Salman Ansari]

Cool. I would give POST a try. Any samples of using Post while passing the
query string values (such as ORing between Solr field values) using
Solr.NET?

Regards,
Salman

On Sun, Jan 31, 2016 at 10:21 PM, Shawn Heisey  wrote:

> On 1/31/2016 7:20 AM, Salman Ansari wrote:
> > I am building a long query containing multiple ORs between query terms. I
> > started to receive the following exception:
> >
> > The remote server returned an error: (414) Request-URI Too Long. Any idea
> > what is the limit of the URL in Solr? Moreover, as a solution I was
> > thinking of chunking the query into multiple requests but I was wondering
> > if anyone has a better approach?
>
> The default HTTP header size limit on most webservers and containers
> (including the Jetty that ships with Solr) is 8192 bytes.  A typical
> request like this will start with "GET " and end with " HTTP/1.1", which
> count against that 8192 bytes.  The max header size can be increased.
>
> If you place the parameters into a POST request instead of on the URL,
> then the default size limit of that POST request in Solr is 2MB.  This
> can also be increased.
>
> Thanks,
> Shawn
>
>

Re: implement exact match for one of the search fields only?

[Jan Høydahl]

> In our case we want to boost exact search ( not phrase search).
> Simply adding the copy field not tokenised  with the related boost to the
> qf list should be fine.
> Am I missing something ?

Please try it out and report back. You’ll find that the “qf” param will only 
ever produce matches for single tokens, split on whitespace by the query parser.

You are right that the pf solution is only good for boosting exact matches of a 
field which should also match partially, and Derek wants it to match phrase 
search but not partial search. This gets a bit trickier. You can fix it 
partially by rewriting the query into a normal dismax OR’ed with a phrase match 
on the exact field. But if you expect a sub-phrase to trigger the match, e.g. 
q=black dvd bracket, where black should match partially in the qfs and “dvd 
bracket” should match exact in the exact field, then it gets more tricky.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 29. jan. 2016 kl. 15.50 skrev Alessandro Benedetti :
> 
> Still not following.
> Why you would like to use pf for exact search ( which is different from
> phrase search) ?
> Let's summarise :
> 
> pf
> 
> 
> Phrase Fields: boosts the score of documents in cases where all of the
> terms in the q parameter appear in close proximity.
> 
> pf is for very specific use cases, when you want to boost phrase searches
> in a set of fields, to boost those documents up.
> In our case we want to boost exact search ( not phrase search).
> Simply adding the copy field not tokenised  with the related boost to the
> qf list should be fine.
> Am I missing something ?
> 
> On 29 January 2016 at 10:33, Jan Høydahl  wrote:
> 
>> Hi
>> 
>> Did you try? Sadly, the pf param will not apply to a non-analyzed field,
>> and is not added if query consists of one term only.
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>>> 29. jan. 2016 kl. 11.14 skrev Alessandro Benedetti <
>> abenede...@apache.org>:
>>> 
>>> Jan sorry If i insist, but I really don't see the benefit.
>>> If you use the edismax and a copy field not tokenised, you can apply the
>>> boost you want to the exact match, and the query will simply be q=foo.
>>> And you obtain exactly what you do without the markers.
>>> 
>>> But please, explain me how your solution add some benefit , because I am
>>> really curious and probably I missed some point ! :)
>>> If i misunderstood the problem, sorry about that !
>>> 
>>> Cheers
>>> 
>>> On 28 January 2016 at 21:55, Jan Høydahl  wrote:
>>> 
 Depends on what exactly you try to do. I think the Github README
>> explains
 in what situations my solution excels.
 Especially if you do not have control over the client application, you
 simply get a q=foo, then such a
 setup will allow you to boost exact matches very easily.
 
 --
 Jan Høydahl, search solution architect
 Cominvent AS - www.cominvent.com
 
> 28. jan. 2016 kl. 16.28 skrev Alessandro Benedetti <
 abenede...@apache.org>:
> 
> Jan,
> I admit I took a brief look, but what are the benefit of using your
> strategy instead of an additional not tokenised ( keywordTokenized)
>> copy
> field ?
> 
> Cheers
> 
> On 28 January 2016 at 15:22, Jan Høydahl 
>> wrote:
> 
>> Hi
>> 
>> Please look at my github repo with a template for a field type
>> allowing
>> exact match. Typical use is with disMax query parser and the “pf”
>> param.
>> See https://github.com/cominvent/exactmatch
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>>> 28. jan. 2016 kl. 10.52 skrev Derek Poh :
>>> 
>>> Hi
>>> 
>>> First of all, sorry for the long post.
>>> 
>>> How do I implement or structured the query such that one of the
>> search
>> fields is an exact phrase match while the rest of the search fields
>> can
 be
>> exact or partial matches? Is this possible?
>>> 
>>> I have the following search fields
>>> - P_VeryShortDescription
>>> - P_ShortDescription
>>> - P_CatConcatKeyword
>>> - spp_keyword_exact
>>> 
>>> For the spp_keyword_exact field, I want to apply an exact match to
>> it.
>>> 
>>> I have a document with the following information. If I search for
 'dvd',
>> this document should not match. However if I search for 'dvd bracket',
 this
>> document should match.
>>> Right now when I search for 'dvd', it is not return, which is
>> correct.
>>> I want it to be return when I search for 'dvd bracket' but it is not.
>>> I try enclosing it in double quotes "dvd bracket" but it is not
>> return.

Re: Nested documents and many-many relation

[Sathyakumar Seshachalam]

Thanks, query time joins are not an option for me, because of the size of
the index and hence the join performance,
I will look at Siren.
 

On 29/01/16, 10:16 PM, "Alessandro Benedetti" 
wrote:

>Probably if you are interested in a many-to-many relation, you could be
>interested in the query time join.
>it has been the first type of join integrated in Solr.
>It allow you to avoid redundancies.
>It's slower than block join, but it doesn't force you to any specific
>indexing approach.
>It became less and less popular but there are scenario could be useful !
>
>Take a look to Siren as well, it could be interesting, not sure it will
>help you as Siren will duplicate nested documents.
>Cheers
>
>On 29 January 2016 at 15:32, Jack Krupansky 
>wrote:
>
>> If you wish to change, add, or delete a child or change the parent you
>>must
>> do an add of the entire block again with both the parent and all
>>children.
>> This is because the efficiency of Block Join comes from the documents
>>being
>> adjacent in Lucene and segments are immutable in Lucene, so the entire
>> block must be written to a new segment.
>>
>>
>> -- Jack Krupansky
>>
>> On Fri, Jan 29, 2016 at 5:13 AM, Sathyakumar Seshachalam <
>> sathyakumar_seshacha...@trimble.com> wrote:
>>
>> > Hi,
>> >
>> > Am trying to investigate the possibility of using Block Join query
>>parser
>> > in a many-to-many relation scenario.
>> > Observation is that when a document is added as a child to more than
>>one
>> > parent document (I use Solrj to do this), I seem to get two copies of
>>the
>> > child document. Can this be avoided ? Is this per design ?
>> > Are there are articles talking about ways to model a many-to-many
>> > relationship (even if its a hacky solution).
>> >
>> >
>>
>
>
>
>-- 
>--
>
>Benedetti Alessandro
>Visiting card : http://about.me/alessandro_benedetti
>
>"Tyger, tyger burning bright
>In the forests of the night,
>What immortal hand or eye
>Could frame thy fearful symmetry?"
>
>William Blake - Songs of Experience -1794 England

CorruptIndexException during optimize.

[Modassar Ather]

Hi,

Got following error during optimize of index on 2 nodes of 12 node cluster.
Please let me know if the index can be recovered and how and what could be
the reason?
Total number of nodes: 12
No replica.
Solr version - 5.4.0
Java version - 1.7.0_91 (Open JDK 64 bit)
Ubuntu version : Ubuntu 14.04.3 LTS

2016-01-31 20:00:31.211 ERROR (qtp1698904557-9710) [c:core s:shard4
r:core_node3 x:core] o.a.s.h.RequestHandlerBase java.io.IOException:
Invalid vInt detected (too many bits)
at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141)
at
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.readNumericEntry(Lucene54DocValuesProducer.java:355)
at
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.readFields(Lucene54DocValuesProducer.java:243)
at
org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.(Lucene54DocValuesProducer.java:122)
at
org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113)
at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.(PerFieldDocValuesFormat.java:268)
at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:358)
at
org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51)
at
org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67)
at
org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:147)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:81)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
at
org.apache.lucene.index.BufferedUpdatesStream$SegmentState.(BufferedUpdatesStream.java:384)
at
org.apache.lucene.index.BufferedUpdatesStream.openSegmentStates(BufferedUpdatesStream.java:416)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:261)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3161)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3147)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3124)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3087)
at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1741)
at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1721)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:590)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:62)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1612)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1589)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:64)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at

Re: Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

one more issue i am getting in solr is as below. how can i fix that issue.

INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | ERROR [solr
indexer thread] [10.240.2.252] (Hagemeyerce) (3N68)
[RexelSolrDocumentFactory] Cannot resolve values for item with PK:
8814537801729 by resolver:
de.hybris.platform.solrfacetsearch.provider.impl.ClassificationPropertyValueProvider
for property: EF001632
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 |
de.hybris.platform.jalo.JaloInvalidParameterException: ambigous
attribute identifier 'EF001632' - found at least two maching attributes
CEC000830->EF001632(8796967961186) and
CEC000104->EF001632(8796686778978)[HY-0]
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.matchAttribute(FeatureContainer.java:932)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.getClassificationAttributeAssignment(FeatureContainer.java:871)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.hasFeature(FeatureContainer.java:1182)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.solrfacetsearch.provider.impl.ClassificationPropertyValueProvider.getFieldValues(ClassificationPropertyValueProvider.java:56)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
com.rexel.search.RexelSolrDocumentFactory.createInputDocument(RexelSolrDocumentFactory.java:248)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.solrfacetsearch.indexer.impl.DefaultIndexer.indexItems(DefaultIndexer.java:119)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.executeIndexerOperation(DefaultIndexerWorker.java:343)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.doRun(DefaultIndexerWorker.java:230)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.run(DefaultIndexerWorker.java:178)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
java.lang.Thread.run(Thread.java:745)


On Sun, Jan 31, 2016 at 6:44 PM, Manoj Agrawal 
wrote:

> yes, we updated all the nodes. we are using solr 4.6
>
> as part of upgrade , we have cloned the solr server of and their
> configuration.
>
> On Sun, Jan 31, 2016 at 3:21 PM, Erik Hatcher 
> wrote:
>
>> Manoj - the log attachments don’t come through on the listserv.  Did you
>> upgrade all nodes in your system?   What version of Solr is running?
>>
>>
>> —
>> Erik Hatcher, Senior Solutions Architect
>> http://www.lucidworks.com 
>>
>>
>>
>> > On Jan 31, 2016, at 1:12 PM, Manoj Agrawal 
>> wrote:
>> >
>> > Hi,
>> >
>> > One more thing i observed that i have multiple  cores and after running
>> the
>> > indexing for a particualr core but all the core are indexednign as per
>> log.
>> >
>> > -Manoj
>> >
>> > On Sun, Jan 31, 2016 at 10:32 AM, Manoj Agrawal <
>> manoj.agrawa...@gmail.com>
>> > wrote:
>> >
>> >> Hi Erik,
>> >>
>> >> Thanks for helping on this! i checked that all the nodes are already
>> >> started. attaching the solr.log files for your reference.
>> >>
>> >>
>> >>
>> >> On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher 
>> >> wrote:
>> >>
>> >>> Manoj - looks like the master isn’t available?   Did you start up all
>> >>> nodes?
>> >>>
>> >>> As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion
>> >>> work, so if you need any assistance let us know.
>> >>>
>> >>>
>> >>>
>> >>> —
>> >>> Erik Hatcher, Senior Solutions Architect
>> >>> http://www.lucidworks.com 
>> >>>
>> >>>
>> >>>
>>  On Jan 31, 2016, at 10:33 AM, Manoj Agrawal <
>> manoj.agrawa...@gmail.com>
>> >>> wrote:
>> 
>>  Hi Team,
>> 
>> 
>> 
>>  We have migrated our application from hybris version 5.2 to 

Re: Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

yes, we updated all the nodes. we are using solr 4.6

as part of upgrade , we have cloned the solr server of and their
configuration.

On Sun, Jan 31, 2016 at 3:21 PM, Erik Hatcher 
wrote:

> Manoj - the log attachments don’t come through on the listserv.  Did you
> upgrade all nodes in your system?   What version of Solr is running?
>
>
> —
> Erik Hatcher, Senior Solutions Architect
> http://www.lucidworks.com 
>
>
>
> > On Jan 31, 2016, at 1:12 PM, Manoj Agrawal 
> wrote:
> >
> > Hi,
> >
> > One more thing i observed that i have multiple  cores and after running
> the
> > indexing for a particualr core but all the core are indexednign as per
> log.
> >
> > -Manoj
> >
> > On Sun, Jan 31, 2016 at 10:32 AM, Manoj Agrawal <
> manoj.agrawa...@gmail.com>
> > wrote:
> >
> >> Hi Erik,
> >>
> >> Thanks for helping on this! i checked that all the nodes are already
> >> started. attaching the solr.log files for your reference.
> >>
> >>
> >>
> >> On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher 
> >> wrote:
> >>
> >>> Manoj - looks like the master isn’t available?   Did you start up all
> >>> nodes?
> >>>
> >>> As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion
> >>> work, so if you need any assistance let us know.
> >>>
> >>>
> >>>
> >>> —
> >>> Erik Hatcher, Senior Solutions Architect
> >>> http://www.lucidworks.com 
> >>>
> >>>
> >>>
>  On Jan 31, 2016, at 10:33 AM, Manoj Agrawal <
> manoj.agrawa...@gmail.com>
> >>> wrote:
> 
>  Hi Team,
> 
> 
> 
>  We have migrated our application from hybris version 5.2 to version
> >>> 5.5.1.4
>  and we are using solr version 4.6.
> 
> 
> 
>  After running the full indexing , below error is coming in the
> >>> application
>  log. Could anyone help me on this . I can give also give you more
>  information on this as well.
> 
> 
> 
>  Error:
> 
> 
> 
>  /--
> 
>  ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller;
> >>> Master
>  at: http://solr-master:8983/solr/Austria_aus_Product is not
> available.
>  Index fetch failed. Exception: Timeout occured while waiting response
> >>> from
>  server at: http://solr-master:8983/solr/Austria_aus_Product
> 
>  ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller;
> >>> Master
>  at: http://solr-master:8983/solr/Regro_aur_Product_1 is not
> available.
>  Index fetch failed. Exception: Timeout occured while waiting response
> >>> from
>  server at: http://solr-master:8983/solr/Regro_aur_Product_1
> 
>  ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller;
> >>> Master
>  at: http://solr-master:8983/solr/Regro_aur_Product is not available.
> >>> Index
>  fetch failed. Exception: Timeout occured while waiting response from
> >>> server
>  at: http://solr-master:8983/solr/Regro_aur_Product
> 
>  ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller;
> >>> Master
>  at: http://solr-master:8983/solr/Austria_aus_Product_1 is not
> >>> available.
>  Index fetch failed. Exception: Timeout occured while waiting response
> >>> from
>  server at: http://solr-master:8983/solr/Austria_aus_Product_1
> 
>  INFO  - 2016-01-29 20:14:18.210;
>  org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>  path=/admin/info/logging
>  params={_=1454094992955=1454094846369=json} status=0 QTime=0
> 
>  INFO  - 2016-01-29 20:14:29.206;
>  org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>  path=/admin/info/logging
>  params={_=1454095003954=1454094856824=json} status=0 QTime=0
> 
>  INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller;
> >>> Slave
>  in sync with master.
> 
>  INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller;
> >>> Slave
>  in sync with master.
> 
>  INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller;
> >>> Slave
>  in sync with master.
> 
>  INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller;
> >>> Slave
>  in sync with master.
> 
>  INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller;
> >>> Slave
>  in sync with master.
> 
>  INFO  - 2016-01-29 20:14:40.214;
>  org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>  path=/admin/info/logging
>  params={_=1454095014957=1454094856824=json} status=0 QTime=0
> 
>  INFO  - 2016-01-29 20:14:51.223;
>  org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
>  path=/admin/info/logging
>  params={_=1454095025954=1454094856824=json} status=0 QTime=0
> 
>  INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller;
> >>> Slave

Re: Increasing maxMergedSegmentMB value

[Zheng Lin Edwin Yeo]

Hi Jack,

Yes, I plan to merge all the 10GB segments to 20GB, and not just two of the
segments. Sorry for the confusion.

I have recently increased the system memory from 64GB to 192GB, but as our
index size grows (which means more segments are created), I found that the
query speed actually slow downs. So we have decided to increase the segment
size to see how it goes, as there will be fewer segments to search for.

Regards,
Edwin


On 1 February 2016 at 01:37, Jack Krupansky 
wrote:

> Make sure you fully digest Mike McCandless' blog post on segment merge
> before trying to outguess his code:
>
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>
> Generally, I don't think you would want to merge just two segments.
> Generally, you should do a bunch at a time, typically 10. IOW, take all the
> segments on a tier and merge them into one segment at the next tier.
>
> There is no documented practical upper limit for how big to make a single
> segment, but very large segments are not likely to be optimized well in
> Lucene, hence the default max merge size of 5GB. If you want to get a lot
> above that, you're in uncharted territory. Besides, if you start pushing
> your index well above the amount of available system memory your query
> performance will suffer. I'd watch for the latter before pushing on the
> former.
>
>
> -- Jack Krupansky
>
> On Sun, Jan 31, 2016 at 10:43 AM, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> wrote:
>
> > Thanks for your reply Shawn and Jack.
> >
> > I wanted to increase the segment size to 15GB, so that there will be
> lesser
> > segments to search for during the query, which should potentially improve
> > the query speed.
> >
> > What if I set the segment size to 20GB? Will all the existing 10GB
> segments
> > be merge to 20GB, as now merging two 10GB segments will results in a 20GB
> > segment?
> >
> > Regards,
> > Edwin
> >
> >
> > On 31 January 2016 at 12:16, Jack Krupansky 
> > wrote:
> >
> > > From the Lucene MergePolicy Javadoc:
> > >
> > > "Whenever the segments in an index have been altered by IndexWriter
> > > <
> > >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/IndexWriter.html
> > > >,
> > > either the addition of a newly flushed segment, addition of many
> segments
> > > from addIndexes* calls, or a previous merge that may now need to
> cascade,
> > > IndexWriter
> > > <
> > >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/IndexWriter.html
> > > >
> > >  invokes findMerges(org.apache.lucene.index.MergeTrigger,
> > > org.apache.lucene.index.SegmentInfos,
> > org.apache.lucene.index.IndexWriter)
> > > <
> > >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html#findMerges(org.apache.lucene.index.MergeTrigger
> > > ,
> > > org.apache.lucene.index.SegmentInfos,
> > > org.apache.lucene.index.IndexWriter)> to
> > > give the MergePolicy a chance to pick merges that are now required.
> This
> > > method returns a MergePolicy.MergeSpecification
> > > <
> > >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.MergeSpecification.html
> > > >
> > > instance
> > > describing the set of merges that should be done, or null if no merges
> > are
> > > necessary. When IndexWriter.forceMerge is called, it calls
> > > findForcedMerges(SegmentInfos,int,Map,
> > > IndexWriter)
> > > <
> > >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html#findForcedMerges(org.apache.lucene.index.SegmentInfos
> > > ,
> > > int, java.util.Map, org.apache.lucene.index.IndexWriter)> and the
> > > MergePolicy should then return the necessary merges."
> > >
> > > See:
> > >
> > >
> >
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html
> > >
> > > IOW, when the next commit occurs that closes and flushes the currently
> > open
> > > segment.
> > >
> > > Nothing will happen to any existing 10GB segments, now or ever in the
> > > future since merging two 10GB segments would not be possible with a
> limit
> > > of only 15GB.
> > >
> > > Maybe you could clue us in as to what effect you are trying to
> achieve. I
> > > mean, why should any app care whether segments are 10GB or 15GB?
> > >
> > >
> > > -- Jack Krupansky
> > >
> > > On Sat, Jan 30, 2016 at 6:28 PM, Shawn Heisey 
> > wrote:
> > >
> > > > On 1/30/2016 7:31 AM, Zheng Lin Edwin Yeo wrote:
> > > > > I would like to find out, when I increase the maxMergedSegmentMB
> from
> > > > 10240
> > > > > (10GB) to 15360 (15GB), will all the 10GB segments that were
> created
> > > > > previously be automatically merge to 15GB?
> > > >
> > > > Not necessarily.  It will make those 10GB+ segments eligible for
> > further
> > > > merging, whereas they would have been ineligible before the change.
> > > >
> > > > This might mean that one or more of those large segments will be
> 

Re: Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

thanks Erik!

On Sun, Jan 31, 2016 at 7:12 PM, Erik Hatcher 
wrote:

> This needs to be addressed by hybris support - it's in one of their custom
> components.
>
> > On Jan 31, 2016, at 19:53, Manoj Agrawal 
> wrote:
> >
> > one more issue i am getting in solr is as below. how can i fix that
> issue.
> >
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 |  [0;31mERROR
> [solr
> > indexer thread] [10.240.2.252] (Hagemeyerce) (3N68)
> > [RexelSolrDocumentFactory] Cannot resolve values for item with PK:
> > 8814537801729 by resolver:
> >
> de.hybris.platform.solrfacetsearch.provider.impl.ClassificationPropertyValueProvider
> > for property: EF001632
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 |
> >  [mde.hybris.platform.jalo.JaloInvalidParameterException: ambigous
> > attribute identifier 'EF001632' - found at least two maching attributes
> > CEC000830->EF001632(8796967961186) and
> > CEC000104->EF001632(8796686778978)[HY-0]
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.matchAttribute(FeatureContainer.java:932)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.getClassificationAttributeAssignment(FeatureContainer.java:871)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.hasFeature(FeatureContainer.java:1182)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.solrfacetsearch.provider.impl.ClassificationPropertyValueProvider.getFieldValues(ClassificationPropertyValueProvider.java:56)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> com.rexel.search.RexelSolrDocumentFactory.createInputDocument(RexelSolrDocumentFactory.java:248)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.solrfacetsearch.indexer.impl.DefaultIndexer.indexItems(DefaultIndexer.java:119)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.executeIndexerOperation(DefaultIndexerWorker.java:343)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.doRun(DefaultIndexerWorker.java:230)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.run(DefaultIndexerWorker.java:178)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> > java.lang.Thread.run(Thread.java:745)
> >
> >
> > On Sun, Jan 31, 2016 at 6:44 PM, Manoj Agrawal <
> manoj.agrawa...@gmail.com>
> > wrote:
> >
> >> yes, we updated all the nodes. we are using solr 4.6
> >>
> >> as part of upgrade , we have cloned the solr server of and their
> >> configuration.
> >>
> >> On Sun, Jan 31, 2016 at 3:21 PM, Erik Hatcher 
> >> wrote:
> >>
> >>> Manoj - the log attachments don’t come through on the listserv.  Did
> you
> >>> upgrade all nodes in your system?   What version of Solr is running?
> >>>
> >>>
> >>> —
> >>> Erik Hatcher, Senior Solutions Architect
> >>> http://www.lucidworks.com 
> >>>
> >>>
> >>>
>  On Jan 31, 2016, at 1:12 PM, Manoj Agrawal  >
> >>> wrote:
> 
>  Hi,
> 
>  One more thing i observed that i have multiple  cores and after
> running
> >>> the
>  indexing for a particualr core but all the core are indexednign as per
> >>> log.
> 
>  -Manoj
> 
>  On Sun, Jan 31, 2016 at 10:32 AM, Manoj Agrawal <
> >>> manoj.agrawa...@gmail.com>
>  wrote:
> 
> > Hi Erik,
> >
> > Thanks for helping on this! i checked that all the nodes are already
> > started. attaching the solr.log files for your reference.
> >
> >
> >
> > On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher <
> erik.hatc...@gmail.com>
> > wrote:

Solr-8496

[Jay Potharaju]

Hi
I am just starting off on a project (using solr 5.4) which uses multi
select faceting as one of the key features. Should I wait till solr 5.5 is
released and resolves the issue outlined in solr-8496? Is there a
recommended version I should be using?

Is there an ETA on when will 5.5 be released?
Thanks


On Wed, Jan 27, 2016 at 8:28 AM, Shawn Heisey  wrote:

> On 1/27/2016 8:59 AM, David Smith wrote:
> > So we definitely don’t have CP yet — our very first network outage
> resulted in multiple overlapped lost updates.  As a result, I can’t pick
> one replica and make it the new “master”.  I must rebuild this collection
> from scratch, which I can do, but that requires downtime which is a problem
> in our app (24/7 High Availability with few maintenance windows).
>
> I don't have anything to add regarding the initial problem or how you
> can prevent it from happening again.  You're already on the latest minor
> Solr version (5.4), though I think you could probably upgrade to 5.4.1.
> The list of bugfixes for 5.4.1 does not seem to include anything that
> could explain the problem you have described.  There is SOLR-8496, but
> that only affects multi-select faceting, not the numDoc counts on the
> core summary screen.
>
> For the specific issue mentioned above (rebuilding without downtime),
> what I would recommend that you do is build an entirely new collection,
> then delete the old one and create an alias to the new collection using
> the old collection's name.  Deleting the old collection and making the
> alias should happen very quickly.
>
> I don't think any documentation states this, but it seems like a good
> idea to me use an alias from day one, so that you always have the option
> of swapping the "real" collection that you are using without needing to
> change anything else.  I'll need to ask some people if they think this
> is a good documentation addition, and think of a good place to mention
> it in the reference guide.
>
> Thanks,
> Shawn
>
>


-- 
Thanks
Jay Potharaju

Handling fields used for both display & index

[Jay Potharaju]

Hi,
I am trying to decide if I should use text_en or string as my field type.
The fields have to be both indexed and stored for display. One solution is
to duplicate fields, one for indexing other for display.One of the field
happens to be a description field which I would like to avoid duplicating.
Solr should return results when someone searches for John or john.Is
storing a copy of the field the best way to go about this problem?


Thanks

Re: Handling fields used for both display & index

[Sameer Maggon]

Hi Jay,

You could use one field for both unless there is a specific requirement you
are looking for that is not being met by that one field (e.g. faceting,
etc). Typically, if you have a field that is marked as both "indexed" and
"stored", the value that is passed while indexing to that field is stored
as is. However, it's indexed based on the field type that you've specified
for that field.

e.g. a description field with the field type of "text_en" would be indexed
per the pipeline in the text_en fieldtype and the text as is will be stored
(which is what is returned in your response in the results).

Thanks,
-- 
*Sameer Maggon*
Measured Search | Solr-as-a-Service | Solr Monitoring | Search Analytics
www.measuredsearch.com


On Sun, Jan 31, 2016 at 5:56 PM, Jay Potharaju 
wrote:

> Hi,
> I am trying to decide if I should use text_en or string as my field type.
> The fields have to be both indexed and stored for display. One solution is
> to duplicate fields, one for indexing other for display.One of the field
> happens to be a description field which I would like to avoid duplicating.
> Solr should return results when someone searches for John or john.Is
> storing a copy of the field the best way to go about this problem?
>
>
> Thanks
>

Re: Full Indexing is geeting failed with hybris application

[Erik Hatcher]

This needs to be addressed by hybris support - it's in one of their custom 
components. 

> On Jan 31, 2016, at 19:53, Manoj Agrawal  wrote:
> 
> one more issue i am getting in solr is as below. how can i fix that issue.
> 
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | ERROR [solr
> indexer thread] [10.240.2.252] (Hagemeyerce) (3N68)
> [RexelSolrDocumentFactory] Cannot resolve values for item with PK:
> 8814537801729 by resolver:
> de.hybris.platform.solrfacetsearch.provider.impl.ClassificationPropertyValueProvider
> for property: EF001632
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 |
> de.hybris.platform.jalo.JaloInvalidParameterException: ambigous
> attribute identifier 'EF001632' - found at least two maching attributes
> CEC000830->EF001632(8796967961186) and
> CEC000104->EF001632(8796686778978)[HY-0]
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.matchAttribute(FeatureContainer.java:932)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.getClassificationAttributeAssignment(FeatureContainer.java:871)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.catalog.jalo.classification.util.FeatureContainer.hasFeature(FeatureContainer.java:1182)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.solrfacetsearch.provider.impl.ClassificationPropertyValueProvider.getFieldValues(ClassificationPropertyValueProvider.java:56)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> com.rexel.search.RexelSolrDocumentFactory.createInputDocument(RexelSolrDocumentFactory.java:248)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.solrfacetsearch.indexer.impl.DefaultIndexer.indexItems(DefaultIndexer.java:119)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.executeIndexerOperation(DefaultIndexerWorker.java:343)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.doRun(DefaultIndexerWorker.java:230)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> de.hybris.platform.solrfacetsearch.indexer.workers.impl.DefaultIndexerWorker.run(DefaultIndexerWorker.java:178)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> INFO   | jvm 1| main| 2016/01/31 22:04:33.471 | at
> java.lang.Thread.run(Thread.java:745)
> 
> 
> On Sun, Jan 31, 2016 at 6:44 PM, Manoj Agrawal 
> wrote:
> 
>> yes, we updated all the nodes. we are using solr 4.6
>> 
>> as part of upgrade , we have cloned the solr server of and their
>> configuration.
>> 
>> On Sun, Jan 31, 2016 at 3:21 PM, Erik Hatcher 
>> wrote:
>> 
>>> Manoj - the log attachments don’t come through on the listserv.  Did you
>>> upgrade all nodes in your system?   What version of Solr is running?
>>> 
>>> 
>>> —
>>> Erik Hatcher, Senior Solutions Architect
>>> http://www.lucidworks.com 
>>> 
>>> 
>>> 
 On Jan 31, 2016, at 1:12 PM, Manoj Agrawal 
>>> wrote:
 
 Hi,
 
 One more thing i observed that i have multiple  cores and after running
>>> the
 indexing for a particualr core but all the core are indexednign as per
>>> log.
 
 -Manoj
 
 On Sun, Jan 31, 2016 at 10:32 AM, Manoj Agrawal <
>>> manoj.agrawa...@gmail.com>
 wrote:
 
> Hi Erik,
> 
> Thanks for helping on this! i checked that all the nodes are already
> started. attaching the solr.log files for your reference.
> 
> 
> 
> On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher 
> wrote:
> 
>> Manoj - looks like the master isn’t available?   Did you start up all
>> nodes?
>> 
>> As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion
>> work, so if you need any assistance let us know.
>> 
>> 
>> 
>> —
>> Erik Hatcher, Senior 

Re: Handling fields used for both display & index

[Binoy Dalal]

Adding to sameer's answer, use string types when you want exact matches,
both in terms of query and case.
In case you want to perform additional operations on the input, like
tokenization and applying filters, then you're better off with one of the
other text types.
You should take a look at the field type definition in the schema file to
see if a predefined type fits your need, else create a custom type based on
your requirements.

On Mon, 1 Feb 2016, 07:36 Sameer Maggon  wrote:

> Hi Jay,
>
> You could use one field for both unless there is a specific requirement you
> are looking for that is not being met by that one field (e.g. faceting,
> etc). Typically, if you have a field that is marked as both "indexed" and
> "stored", the value that is passed while indexing to that field is stored
> as is. However, it's indexed based on the field type that you've specified
> for that field.
>
> e.g. a description field with the field type of "text_en" would be indexed
> per the pipeline in the text_en fieldtype and the text as is will be stored
> (which is what is returned in your response in the results).
>
> Thanks,
> --
> *Sameer Maggon*
> Measured Search | Solr-as-a-Service | Solr Monitoring | Search Analytics
> www.measuredsearch.com
> <
> https://mailtrack.io/trace/link/dca98638f8114f38d1ff30ed04feb547877c848e?url=http%3A%2F%2Fmeasuredsearch.com%2F=797ba5008ecc48b8
> >
>
> On Sun, Jan 31, 2016 at 5:56 PM, Jay Potharaju 
> wrote:
>
> > Hi,
> > I am trying to decide if I should use text_en or string as my field type.
> > The fields have to be both indexed and stored for display. One solution
> is
> > to duplicate fields, one for indexing other for display.One of the field
> > happens to be a description field which I would like to avoid
> duplicating.
> > Solr should return results when someone searches for John or john.Is
> > storing a copy of the field the best way to go about this problem?
> >
> >
> > Thanks
> >
>
-- 
Regards,
Binoy Dalal

Re: User-defined properties and configsets

[Georg Sorst]

Erick Erickson  schrieb am Fr., 29. Jan. 2016 um
17:55 Uhr:

> These are system properties, right? They go in the startup for all of
> your Solr instances scattered about your cluster.
>

No, they will be used in the solrconfig.xml and schema.xml. I just
mentioned the -Dmyproperty=... example because that's one way to get
properties in there.

What I'm looking for is one concise place to define properties that all
cores using this configset will use as a default. I've tried several ways:

* Using the ${property:default} syntax in the configs is no good, because
the same property will occur several times in the configs
* Setting them with the config API is no good, because then they live in my
code, but I'd rather have them in a property file for visiblity and
maintenance reasons
* Setting them as system properties (-Dmyproperty=...) is no good, because
that makes our deployment more complicated

So, ideally I can just put a .properties file in the configset that will
provide default values for all cores using this configset.

Of course the properties may be changed from their default values on a
per-core base, but that's what the config API is for.

So, where do other people put their properties when they use configsets?

Best regards,
Georg


>
> The bin/solr script has a -a option for passing additional stuff to the
> JVM...
>
> Best,
> Erick
>
> On Thu, Jan 28, 2016 at 11:50 PM, Georg Sorst 
> wrote:
> > Any takers?
> >
> > Georg Sorst  schrieb am So., 24. Jän. 2016
> 00:22:
> >
> >> Hi list!
> >>
> >> I've just started playing with Solr 5 (upgrading from Solr 4) and want
> to
> >> use configsets. I'm currently struggling with how to use user-defined
> >> properties and configsets together.
> >>
> >> My solrconfig.xml contains a few properties. Previously these were in a
> >> solrcore.properties and thus were properly loaded and substituted by
> Solr.
> >>
> >> Now I've moved my configuration to a configset (as I may need to create
> >> several cores with the same config). When I create a core with
> >>
> http://localhost:8983/solr/admin/cores?action=CREATE=mycore=myconfigset
> Solr
> >> tells me:
> >>
> >> Caused by: org.apache.solr.common.SolrException: Error loading solr
> config
> >> from //configsets/myconfigset/conf/solrconfig.xml
> >> at
> >>
> org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:186)
> >> at
> >>
> org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:94)
> >> at
> >>
> org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:74)
> >> ... 30 more
> >> Caused by: org.apache.solr.common.SolrException: No system property or
> >> default value specified for  value:
> >> 
> >> at
> >>
> org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:66)
> >> ...
> >>
> >> Where should I put my properties so Solr can load them when I create a
> new
> >> core using this config set? From what I read I could specify them as
> system
> >> properties (-Dmyproperty=...) but I'd rather keep them in a file that I
> can
> >> check in.
> >>
> >> Thanks!
> >> Georg
> >>
> >>
> >>
>

Re: Full Indexing is geeting failed with hybris application

[Erik Hatcher]

Manoj - looks like the master isn’t available?   Did you start up all nodes?

As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion work, so 
if you need any assistance let us know.



—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com 



> On Jan 31, 2016, at 10:33 AM, Manoj Agrawal  wrote:
> 
> Hi Team,
> 
> 
> 
> We have migrated our application from hybris version 5.2 to version 5.5.1.4
> and we are using solr version 4.6.
> 
> 
> 
> After running the full indexing , below error is coming in the application
> log. Could anyone help me on this . I can give also give you more
> information on this as well.
> 
> 
> 
> Error:
> 
> 
> 
> /--
> 
> ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller; Master
> at: http://solr-master:8983/solr/Austria_aus_Product is not available.
> Index fetch failed. Exception: Timeout occured while waiting response from
> server at: http://solr-master:8983/solr/Austria_aus_Product
> 
> ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller; Master
> at: http://solr-master:8983/solr/Regro_aur_Product_1 is not available.
> Index fetch failed. Exception: Timeout occured while waiting response from
> server at: http://solr-master:8983/solr/Regro_aur_Product_1
> 
> ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller; Master
> at: http://solr-master:8983/solr/Regro_aur_Product is not available. Index
> fetch failed. Exception: Timeout occured while waiting response from server
> at: http://solr-master:8983/solr/Regro_aur_Product
> 
> ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller; Master
> at: http://solr-master:8983/solr/Austria_aus_Product_1 is not available.
> Index fetch failed. Exception: Timeout occured while waiting response from
> server at: http://solr-master:8983/solr/Austria_aus_Product_1
> 
> INFO  - 2016-01-29 20:14:18.210;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454094992955=1454094846369=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:14:29.206;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095003954=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:14:40.214;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095014957=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:14:51.223;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095025954=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:15:02.198;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095036958=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:15:04.000; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:15:04.001; org.apache.solr.handler.SnapPuller; Slave
> in sync with master.
> 
> INFO  - 2016-01-29 20:15:13.206;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095047963=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:15:24.194;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095058956=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:15:35.190;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095069956=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:15:46.202;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095080959=1454094856824=json} status=0 QTime=0
> 
> INFO  - 2016-01-29 20:15:57.249;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1454095091957=1454094856824=json} status=0
> 
> 
> 
> 
> 
> 
> 
> Thanks,
> 
> Manoj

Re: Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

Hi Erik,

Thanks for helping on this. But i cheked all the nodes are alreday started.
but i am getting this error.


Please see the error coming in the solr loggin.

Master Solr-
[image: Inline image 1]

Slave Solr-

[image: Inline image 2]





On Sun, Jan 31, 2016 at 9:51 AM, Erik Hatcher 
wrote:

> Manoj - looks like the master isn’t available?   Did you start up all
> nodes?
>
> As a related aside - I/we/Lucidworks have done some hybris/Solr/Fusion
> work, so if you need any assistance let us know.
>
>
>
> —
> Erik Hatcher, Senior Solutions Architect
> http://www.lucidworks.com 
>
>
>
> > On Jan 31, 2016, at 10:33 AM, Manoj Agrawal 
> wrote:
> >
> > Hi Team,
> >
> >
> >
> > We have migrated our application from hybris version 5.2 to version
> 5.5.1.4
> > and we are using solr version 4.6.
> >
> >
> >
> > After running the full indexing , below error is coming in the
> application
> > log. Could anyone help me on this . I can give also give you more
> > information on this as well.
> >
> >
> >
> > Error:
> >
> >
> >
> > /--
> >
> > ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller;
> Master
> > at: http://solr-master:8983/solr/Austria_aus_Product is not available.
> > Index fetch failed. Exception: Timeout occured while waiting response
> from
> > server at: http://solr-master:8983/solr/Austria_aus_Product
> >
> > ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller;
> Master
> > at: http://solr-master:8983/solr/Regro_aur_Product_1 is not available.
> > Index fetch failed. Exception: Timeout occured while waiting response
> from
> > server at: http://solr-master:8983/solr/Regro_aur_Product_1
> >
> > ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller;
> Master
> > at: http://solr-master:8983/solr/Regro_aur_Product is not available.
> Index
> > fetch failed. Exception: Timeout occured while waiting response from
> server
> > at: http://solr-master:8983/solr/Regro_aur_Product
> >
> > ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller;
> Master
> > at: http://solr-master:8983/solr/Austria_aus_Product_1 is not available.
> > Index fetch failed. Exception: Timeout occured while waiting response
> from
> > server at: http://solr-master:8983/solr/Austria_aus_Product_1
> >
> > INFO  - 2016-01-29 20:14:18.210;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454094992955=1454094846369=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:14:29.206;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454095003954=1454094856824=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:14:40.214;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454095014957=1454094856824=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:14:51.223;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454095025954=1454094856824=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:15:02.198;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454095036958=1454094856824=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:15:04.000; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:15:04.001; org.apache.solr.handler.SnapPuller;
> Slave
> > in sync with master.
> >
> > INFO  - 2016-01-29 20:15:13.206;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454095047963=1454094856824=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:15:24.194;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> > path=/admin/info/logging
> > params={_=1454095058956=1454094856824=json} status=0 QTime=0
> >
> > INFO  - 2016-01-29 20:15:35.190;
> > org.apache.solr.servlet.SolrDispatchFilter; [admin] 

URI is too long

[Salman Ansari]

Hi,

I am building a long query containing multiple ORs between query terms. I
started to receive the following exception:

The remote server returned an error: (414) Request-URI Too Long. Any idea
what is the limit of the URL in Solr? Moreover, as a solution I was
thinking of chunking the query into multiple requests but I was wondering
if anyone has a better approach?

Regards,
Salman

Executing Collector's Collect method on more than one thread

[adfel70]

I am using RankQuery to implement my applicative scorer that returns a score
based on the value of specific field (lets call it 'score_field') that is
stored for every document. 
The RankQuery creates a collector, and for every collected docId I retrieve
the value of score_field, calculate the score and add the doc id into
priority queue: 

public class MyScorerrankQuery extends RankQuery { 
... 

@Override 
public TopDocsCollector getTopDocsCollector(int i,
SolrIndexerSearcher.QueryCommand cmd, IndexSearcher searcher) { 
... 
return new MyCollector(...) 
} 
} 

public class MyCollector  extends TopDocsCollector{ 
MyScorer scorer; 
SortedDocValues scoreFieldValues;


@Override 
public void collect(int id){ 
int docID = docBase + id; 
//1. get specific field from the doc using DocValues 
and calculate score
using my scorer 
String value = 
scoreFieldValues.get(docID).utf8ToString(); 
scorer.calcScore(value); 
//2. add docId and score (ScoreDoc object) into 
PriorityQueue. 
} 
} 

Problem is that the calcScore may take ~20 ms per call, so if query returns
100,000 docs, which is not unusual, query execution time will be become 16
minutes. Is there a way to parallelize collector's logic, so more than one
thread would call calcScore simultaneously?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Executing-Collector-s-Collect-method-on-more-than-one-thread-tp4254269.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: URI is too long

[Paul Libbrecht]

How about using POST?

paul

> Salman Ansari 
> 31 January 2016 at 15:20
> Hi,
>
> I am building a long query containing multiple ORs between query terms. I
> started to receive the following exception:
>
> The remote server returned an error: (414) Request-URI Too Long. Any idea
> what is the limit of the URL in Solr? Moreover, as a solution I was
> thinking of chunking the query into multiple requests but I was wondering
> if anyone has a better approach?
>
> Regards,
> Salman
>

Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

Hi Team,



We have migrated our application from hybris version 5.2 to version 5.5.1.4
and we are using solr version 4.6.



After running the full indexing , below error is coming in the application
log. Could anyone help me on this . I can give also give you more
information on this as well.



Error:



/--

ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Austria_aus_Product is not available.
Index fetch failed. Exception: Timeout occured while waiting response from
server at: http://solr-master:8983/solr/Austria_aus_Product

ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Regro_aur_Product_1 is not available.
Index fetch failed. Exception: Timeout occured while waiting response from
server at: http://solr-master:8983/solr/Regro_aur_Product_1

ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Regro_aur_Product is not available. Index
fetch failed. Exception: Timeout occured while waiting response from server
at: http://solr-master:8983/solr/Regro_aur_Product

ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Austria_aus_Product_1 is not available.
Index fetch failed. Exception: Timeout occured while waiting response from
server at: http://solr-master:8983/solr/Austria_aus_Product_1

INFO  - 2016-01-29 20:14:18.210;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454094992955=1454094846369=json} status=0 QTime=0

INFO  - 2016-01-29 20:14:29.206;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095003954=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:40.214;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095014957=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:14:51.223;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095025954=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:02.198;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095036958=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:04.000; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:04.001; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:13.206;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095047963=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:24.194;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095058956=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:35.190;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095069956=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:46.202;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095080959=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:57.249;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095091957=1454094856824=json} status=0







Thanks,

Manoj

Full Indexing is geeting failed with hybris application

[Manoj Agrawal]

Hi Team,



We have migrated our application from hybris version 5.2 to version 5.5.1.4
and we are using solr version 4.6.



After running the full indexing , below error is coming in the application
log. Could anyone help me on this . I can give also give you more
information on this as well.



Error:



/--

ERROR - 2016-01-29 20:14:09.673; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Austria_aus_Product is not available.
Index fetch failed. Exception: Timeout occured while waiting response from
server at: http://solr-master:8983/solr/Austria_aus_Product

ERROR - 2016-01-29 20:14:11.790; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Regro_aur_Product_1 is not available.
Index fetch failed. Exception: Timeout occured while waiting response from
server at: http://solr-master:8983/solr/Regro_aur_Product_1

ERROR - 2016-01-29 20:14:16.802; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Regro_aur_Product is not available. Index
fetch failed. Exception: Timeout occured while waiting response from server
at: http://solr-master:8983/solr/Regro_aur_Product

ERROR - 2016-01-29 20:14:16.824; org.apache.solr.handler.SnapPuller; Master
at: http://solr-master:8983/solr/Austria_aus_Product_1 is not available.
Index fetch failed. Exception: Timeout occured while waiting response from
server at: http://solr-master:8983/solr/Austria_aus_Product_1

INFO  - 2016-01-29 20:14:18.210;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454094992955=1454094846369=json} status=0 QTime=0

INFO  - 2016-01-29 20:14:29.206;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095003954=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:14:31.614; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:31.616; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:31.619; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:33.958; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:34.575; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:14:40.214;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095014957=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:14:51.223;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095025954=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:00.006; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:02.198;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095036958=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:03.998; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:04.000; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:04.001; org.apache.solr.handler.SnapPuller; Slave
in sync with master.

INFO  - 2016-01-29 20:15:13.206;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095047963=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:24.194;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095058956=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:35.190;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095069956=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:46.202;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095080959=1454094856824=json} status=0 QTime=0

INFO  - 2016-01-29 20:15:57.249;
org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
path=/admin/info/logging
params={_=1454095091957=1454094856824=json} status=0







Thanks,

Manoj

Re: Increasing maxMergedSegmentMB value

[Zheng Lin Edwin Yeo]

Thanks for your reply Shawn and Jack.

I wanted to increase the segment size to 15GB, so that there will be lesser
segments to search for during the query, which should potentially improve
the query speed.

What if I set the segment size to 20GB? Will all the existing 10GB segments
be merge to 20GB, as now merging two 10GB segments will results in a 20GB
segment?

Regards,
Edwin


On 31 January 2016 at 12:16, Jack Krupansky 
wrote:

> From the Lucene MergePolicy Javadoc:
>
> "Whenever the segments in an index have been altered by IndexWriter
> <
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/IndexWriter.html
> >,
> either the addition of a newly flushed segment, addition of many segments
> from addIndexes* calls, or a previous merge that may now need to cascade,
> IndexWriter
> <
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/IndexWriter.html
> >
>  invokes findMerges(org.apache.lucene.index.MergeTrigger,
> org.apache.lucene.index.SegmentInfos, org.apache.lucene.index.IndexWriter)
> <
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html#findMerges(org.apache.lucene.index.MergeTrigger
> ,
> org.apache.lucene.index.SegmentInfos,
> org.apache.lucene.index.IndexWriter)> to
> give the MergePolicy a chance to pick merges that are now required. This
> method returns a MergePolicy.MergeSpecification
> <
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.MergeSpecification.html
> >
> instance
> describing the set of merges that should be done, or null if no merges are
> necessary. When IndexWriter.forceMerge is called, it calls
> findForcedMerges(SegmentInfos,int,Map,
> IndexWriter)
> <
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html#findForcedMerges(org.apache.lucene.index.SegmentInfos
> ,
> int, java.util.Map, org.apache.lucene.index.IndexWriter)> and the
> MergePolicy should then return the necessary merges."
>
> See:
>
> https://lucene.apache.org/core/5_4_0/core/org/apache/lucene/index/MergePolicy.html
>
> IOW, when the next commit occurs that closes and flushes the currently open
> segment.
>
> Nothing will happen to any existing 10GB segments, now or ever in the
> future since merging two 10GB segments would not be possible with a limit
> of only 15GB.
>
> Maybe you could clue us in as to what effect you are trying to achieve. I
> mean, why should any app care whether segments are 10GB or 15GB?
>
>
> -- Jack Krupansky
>
> On Sat, Jan 30, 2016 at 6:28 PM, Shawn Heisey  wrote:
>
> > On 1/30/2016 7:31 AM, Zheng Lin Edwin Yeo wrote:
> > > I would like to find out, when I increase the maxMergedSegmentMB from
> > 10240
> > > (10GB) to 15360 (15GB), will all the 10GB segments that were created
> > > previously be automatically merge to 15GB?
> >
> > Not necessarily.  It will make those 10GB+ segments eligible for further
> > merging, whereas they would have been ineligible before the change.
> >
> > This might mean that one or more of those large segments will be merged
> > soon after the change and restart/reload, but I do not know when it
> > might happen.  It would probably wait until at least one new segment was
> > created, at which time the merge policy would be consulted.
> >
> > Thanks,
> > Shawn
> >
> >
>