Re: About solr restart

[Erick Erickson]

For quite some time, collection information has been stored in
individual state.json nodes, not clusterstate.json. See the
MIGRATESTATEFORMAT Collections API call.

How many collections and replicas do you have all told?

Best,
Erick

On Sun, Jun 24, 2018 at 6:43 PM, 苗海泉  wrote:
> We found a problem in the solr6.0 version. After solr restarts data
> recovery on the zookeeper, the solr collection configuration will be
> centralized in the clusterstate.json file. If the number of clusters and
> replicas is a bit more, it is very easy to exceed 1M. Has caused a series
> of problems such as the inability to read the data on Zookeepe. Did someone
> have a good solution, thank you!
>
> --
> ==
> 联创科技
> 知行如一
> ==
> [image: Mailtrack]
> 
> Sender
> notified by
> Mailtrack
> 
> 18/06/25
> 上午9:43:21

About solr restart

[苗海泉]

We found a problem in the solr6.0 version. After solr restarts data
recovery on the zookeeper, the solr collection configuration will be
centralized in the clusterstate.json file. If the number of clusters and
replicas is a bit more, it is very easy to exceed 1M. Has caused a series
of problems such as the inability to read the data on Zookeepe. Did someone
have a good solution, thank you!

-- 
==
联创科技
知行如一
==
[image: Mailtrack]

Sender
notified by
Mailtrack

18/06/25
上午9:43:21

Re: about solr reduce shard nums

[苗海泉]

We ended up using a simple method to copy the directory directly to the
node that was not deleted, then delete the solr data from zookeeper and
restart it to achieve the effect.
Thank you for your suggestion.



[image: Mailtrack]

Sender
notified by
Mailtrack

18/06/25
上午9:40:22

2018-05-20 20:35 GMT+08:00 Erick Erickson :

> Simplest would be to host multiple shards on the same machine. Use
> ADDREPLICA/DELETEREPLICA (collections API calls) to move the replicas
> hosted on the nodes you want to use for another purpose and, when all
> replicas are moved you can repurpose those machines.
>
> Another option would be to create a _new_ collection on the machines
> you'll have dedicated to Solr with fewer shards, re-index all the
> documents to the new machine and use collection aliasing (collections
> API CREATEALIAS) to point to the new collection, then delete the old
> collection.
>
> There are significantly more complex options to truly merge the
> shards, but I wouldn't consider them until the above was proven to be
> unsatisfactory.
>
> In any case, you need to be sure that the machines that remain are
> powerful enough to host all the documents you've put on them.
>
> Best,
> Erick
>
> On Sat, May 19, 2018 at 11:58 PM, 苗海泉  wrote:
> > Hello everyone, I encountered a shard reduction problem with solr. My
> > current solr cluster is deployed in solrcloud mode. Now I need to use
> > several solr machines for other purposes. The solr version I use is Solr
> > 6.0. What should I do? Do it, thank you for your help.--
> > ==
> > 联创科技
> > 知行如一
> > ==
>



-- 
==
联创科技
知行如一
==

SolrCloud Large Cluster Performance Issues

[苗海泉]

Hello, everyone, we encountered two solr problems and hoped to get help.
Our data volume is very large, 24.5TB a day, and the number of records is
110 billion. We originally used 49 solr nodes. Because of insufficient
storage, we expanded to 100. For a solr cluster composed of multiple
machines, we found that the performance of 60 solrclouds and the overall
performance of 49 solr clusters are the same. How do we optimize it? Now
the cluster speed is 1.5 million on average per second. Why is that?

The second problem solrhome can only specify a solrhome, but now the disk
is divided into two directories, another solr can be stored using hdfs, but
the overall indexing performance is not up to standard, how to do, thank
you for your attention.
[image: Mailtrack]

Sender
notified by
Mailtrack

18/06/25
上午9:38:13

Re: solr-ltr

[Shawn Heisey]

On 6/24/2018 8:27 AM, Vivek Singh wrote:

I am trying to ltr in solr but when i uploaded models and features and
restarted the server.  I am following steps as mentioned in documentation.I
am getting this error.

- *techproducts:*
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Failed to create new ManagedResource /schema/feature-store of type
org.apache.solr.ltr.store.rest.ManagedFeatureStore due to:
org.apache.solr.common.SolrException:


It's telling you what the problem is:


org.apache.solr.ltr.model.ModelException: model 'myEfiModel' already
exists. Please use a different name.


You've already uploaded a model with that name, so it already exists.  
You'll need to either upload it with a different name, or delete the 
existing model first.


The instructions for deleting a model are here:

https://lucene.apache.org/solr/guide/6_6/learning-to-rank.html#LearningToRank-Models

Thanks,
Shawn

Re: Error in solr Plugin

[Erick Erickson]

bq.  The lucene-core7 had some useful functions like  incrementToken which I
could not find in previous versions because of that I used this version.

Do not do this. You simply cannot mix jar versions because there was a
function in the old
version that you want to use. The support for that function has been
removed, which is why
you can't find it.

Worse, there's no guarantee that everything necessary to support that
function is
in a single jar file.

You must work with a single version of Solr jars and figure out how
you can get the same or
equivalent functionality with the new code.

Best,
Erick

On Sun, Jun 24, 2018 at 6:33 AM, Andrea Gazzarini  wrote:
> Hi,
> I mean you should use Maven which would pickup, starting from a number (e.g.
> 6.6.1), all the correct dependencies you need for developing the plugin.
> Yes, the "top" libraries (e.g. Solr and Lucene) should have the same version
> but on top of that, the plugin could require some other direct or indirect
> dependency, and here there's no a magic rule for determining which version
> of such dependency you would need. So again: please use Maven (or Gradle).
>
> About the second point: generally speaking, if you're targeting Solr 6.6.1
> your plugin will live in that context, with all the beauties coming with the
> 6.6.1 version. And here, you can imagine why you cannot use something that
> is not yet there, something that will be implemented in a later version.
>
> On top of that, if you meant the TokenStream.incrementToken method please
> check better: this is part of the core Lucene API and unless I didn't get
> you, you will find it also in past versions.
>
> Best,
> Andrea
>
>
> On 24/06/18 14:35, Zahra Aminolroaya wrote:
>>
>> Thanks Andrea. Do you mean all of my jar file versions should be 6.6.1?
>>
>> The lucene-core7 had some useful functions like  incrementToken which I
>> could not find in previous versions because of that I used this version.
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
>

solr-ltr

[Vivek Singh]

I am trying to ltr in solr but when i uploaded models and features and
restarted the server.  I am following steps as mentioned in documentation.I
am getting this error.


   - *techproducts:*
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
   Failed to create new ManagedResource /schema/feature-store of type
   org.apache.solr.ltr.store.rest.ManagedFeatureStore due to:
   org.apache.solr.common.SolrException:
   org.apache.solr.ltr.model.ModelException: model 'myEfiModel' already
   exists. Please use a different name.




-- 
Regards,
Vivek Singh.
9818214334

Re: Error in solr Plugin

[Andrea Gazzarini]

Hi,
I mean you should use Maven which would pickup, starting from a number 
(e.g. 6.6.1), all the correct dependencies you need for developing the 
plugin.
Yes, the "top" libraries (e.g. Solr and Lucene) should have the same 
version but on top of that, the plugin could require some other direct 
or indirect dependency, and here there's no a magic rule for determining 
which version of such dependency you would need. So again: please use 
Maven (or Gradle).


About the second point: generally speaking, if you're targeting Solr 
6.6.1 your plugin will live in that context, with all the beauties 
coming with the 6.6.1 version. And here, you can imagine why you cannot 
use something that is not yet there, something that will be implemented 
in a later version.


On top of that, if you meant the TokenStream.incrementToken method 
please check better: this is part of the core Lucene API and unless I 
didn't get you, you will find it also in past versions.


Best,
Andrea

On 24/06/18 14:35, Zahra Aminolroaya wrote:

Thanks Andrea. Do you mean all of my jar file versions should be 6.6.1?

The lucene-core7 had some useful functions like  incrementToken which I
could not find in previous versions because of that I used this version.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Delete By Query issue followed by Delete By Id Issues

[Emir Arnautović]

Hi Sujatha,
Did I get it right that you are deleting the same documents that will be 
updated afterward? If that’s the case, then you can simply skip deleting, and 
just send updated version of document. Solr (Lucene) does not have delete - 
it’ll just flag document as deleted. Updating document (assuming id is the 
same) will result in the same thing - old document will not be retrievable and 
will be removed from index when segments holding it is merged.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 21 Jun 2018, at 19:59, sujatha sankaran  wrote:
> 
> Thanks,Shawn.
> 
> Our use case is something like this in a batch load of  several 1000's of
> documents,we do a delete first followed by update.Example delete all 1000
> docs and send an update request for 1000.
> 
> What we see is that there are many missing docs due to DBQ re-ordering of
> the order of  deletes followed by updates.We also saw issue with nodes
> going down
> similar tot issue described here:
> http://lucene.472066.n3.nabble.com/SolrCloud-Nodes-going-to-recovery-state-during-indexing-td4369396.html
> 
> we see at the end of this batch process, many (several thousand ) missing
> docs.
> 
> Due to this and after reading above thread , we decided to move to DBI and
> now are facing issues due to custom routing or implicit routing which we
> have in place.So I don't think DBQ was working for us, but we did have
> several such process ( DBQ followed by updates) for different activities in
> the collection happening at the same time.
> 
> 
> Sujatha
> 
> On Thu, Jun 21, 2018 at 1:21 PM, Shawn Heisey  wrote:
> 
>> On 6/21/2018 9:59 AM, sujatha sankaran wrote:
>>> Currently from our business perspective we find that we are left with no
>>> options for deleting docs in a batch load as :
>>> 
>>> DBQ+ batch does not work well together
>>> DBI+ custom routing (batch load / normal)would not work as well.
>> 
>> I would expect DBQ to work, just with the caveat that if you are trying
>> to do other indexing operations at the same time, you may run into
>> significant delays, and if there are timeouts configured anywhere that
>> are shorter than those delays, requests may return failure responses or
>> log failures.
>> 
>> If you are using DBQ, you just need to be sure that there are no other
>> operations happening at the same time, or that your error handling is
>> bulletproof.  Making sure that no other operations are happening at the
>> same time as the DBQ is in my opinion a better option.
>> 
>> Thanks,
>> Shawn
>> 
>> 

Re: Error in solr Plugin

[Zahra Aminolroaya]

Thanks Andrea. Do you mean all of my jar file versions should be 6.6.1? 

The lucene-core7 had some useful functions like  incrementToken which I
could not find in previous versions because of that I used this version.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Error in solr Plugin

[Andrea Gazzarini]

Hi Zahra,
I think your guessing it's right: I see some mess in libraries versions. 
If I got you


 * the target platform is Solr 6.6.1
 * the compile classpath includes solr-core-4.1.0, 1.4.0 (!) and lucene
   7.4.0?

If that is correct, with a ClassCastException you're just scraping the 
surface of all errors you can get :) As a rule of thumb you should 
always work with the same version of the target Solr (because that 
version will be the runtime container of your plugin).


I suggest you to avoid putting libraries manually in the classpath; you 
should use a dependency management tool like Maven or Gradle, set the 
target Solr version once and have the required dependencies 
automatically set up.


Let me know if you need some help with the last point.

Best,
Andrea

On 24/06/18 14:07, Zahra Aminolroaya wrote:

I am using solr 6.6.1. I want to write my own analyzer for the field type
"text_general" in schema. the field type in schema is as follows:
   
 
   
   
   
  
 
When I test the filter in Java, everything is alright; However, when I start
my solr I get the following error:

Caused by: java.lang.ClassCastException: class
normalizing.myNormalizerFilterFactory
at java.lang.Class.asSubclass(Unknown Source)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:539)
at
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:624)
at
org.apache.solr.schema.FieldTypePluginLoader$3.create(FieldTypePluginLoader.java:397)
at
org.apache.solr.schema.FieldTypePluginLoader$3.create(FieldTypePluginLoader.java:390)
at
org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:152)
... 20 more

I used the jar files in my lib as follows: solr-core-4.1.0,
slf4j-api-1.6.6, lucene-core-7.4.0, apache-solr-core-1.4.0, apache-lucene.

Why do I get this error? Is it becuase of the wrong jar file versions
especially the lucene-core-7.4.0 as my lucene version is 6.6.1?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Error in solr Plugin

[Zahra Aminolroaya]

I am using solr 6.6.1. I want to write my own analyzer for the field type
"text_general" in schema. the field type in schema is as follows:
  

  
  
  
  

When I test the filter in Java, everything is alright; However, when I start
my solr I get the following error:

Caused by: java.lang.ClassCastException: class
normalizing.myNormalizerFilterFactory
at java.lang.Class.asSubclass(Unknown Source)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:539)
at
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:624)
at
org.apache.solr.schema.FieldTypePluginLoader$3.create(FieldTypePluginLoader.java:397)
at
org.apache.solr.schema.FieldTypePluginLoader$3.create(FieldTypePluginLoader.java:390)
at
org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:152)
... 20 more

I used the jar files in my lib as follows: solr-core-4.1.0, 
slf4j-api-1.6.6, lucene-core-7.4.0, apache-solr-core-1.4.0, apache-lucene.

Why do I get this error? Is it becuase of the wrong jar file versions
especially the lucene-core-7.4.0 as my lucene version is 6.6.1?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html