Regarding document routing

[manish tanger]

Hello Solr-dev folks,

I am having a doubt in implicit routing and didn't find much info about
this over the internet, so Please help me out on this.

*About environment:*
M/c 1: Zookeeper 1 and Solr 1
M/c 2: Zookeeper 2 and Solr 2

I am using clustered zookeeper and using "CloudSolrClient" from solrJ
API in java.

*this.solrCloudClient = new
CloudSolrClient.Builder().withZkHost(zkHostList).build();*

*Requirement:*

My requirement is to store lots of data on solr using a single collection.
so my idea is that i am going to create a new shard for every hour so that
indexing doesn't take much time.

I choose for the implicit document routing, but I am unable to redirect the
docs on the particular shard. Zookeeper is still distributing it on all
nodes and shards.


*What I have tried:*
1. I have created a collection with implicit routing and put customer
routing field "*dateandhour*" and add it as a filed in my collection.

While adding solr input doc I am setting this filed with shard name.


2. I have also tried to add shard name to id filed like:
 id="*shardName!*uniquedocumentId"


If you guys have some example or doc Please share with me.

Thanks for all your help.


Best Regards,

Manish

Spatial search

[Leila Deljkovic]

Hi,

I’m quite new to Solr and am interested in using spatial search for geospatial 
data (Solr 7.1).

One problem is addressing feature density over a layer and using this to 
determine if a layer would be a relevant result over a search extent. I’d like 
to know is it feasible/possible to “split” a data layer into nested documents 
and index them, then at query time, count the number of nested documents that 
coincide with the search extent. Or maybe make use of overlapRatio or similar.

Thanks

Regarding document routing

[manish tanger]

Hello All,

I am having a doubt in implicit routing and didn't find much info about
this over the internet, so Please help me out on this.

*About environment:*
M/c 1: Zookeeper 1 and Solr 1
M/c 2: Zookeeper 2 and Solr 2

I am using clustered zookeeper and using "CloudSolrClient" from solrJ
API in java.

*this.solrCloudClient = new
CloudSolrClient.Builder().withZkHost(zkHostList).build();*

*Requirement:*

My requirement is to store lots of data on solr using a single collection.
so my idea is that i am going to create a new shard for every hour so that
indexing doesn't take much time.

I choose for the implicit document routing, but I am unable to redirect the
docs on the particular shard. Zookeeper is still distributing it on all
nodes and shards.


*What I have tried:*
1. I have created a collection with implicit routing and put customer
routing field "*dateandhour*" and add it as a filed in my collection.

While adding solr input doc I am setting this filed with shard name.


2. I have also tried to add shard name to id filed like:
 id="*shardName!*uniquedocumentId"


If you guys have some example or doc Please share with me.

Thanks for all your help.


Best Regards,

Manish

Re: ClassicTokenizer

[Shawn Heisey]

On 1/9/2018 9:36 AM, Rick Leir wrote:
> A while ago the default was changed to StandardTokenizer from 
> ClassicTokenizer. The biggest difference seems to be that Classic does not 
> break on hyphens. There is also a different character pr(mumble). I prefer 
> the Classic's non-break on hyphens.

To have any ability to research changes, we're going to need to know
precisely what you mean by "default" in that statement.

Are you talking about the example schemas, or some kind of inherent
default when an analysis chain is not specified?

Probably the reason for the change is an attempt to move into the modern
era, become more standardized, and stop using old/legacy
implementations.  The name of the new default contains the word
"Standard" which would fit in with that goal.

I can't locate any changes in the last couple of years that change the
classic tokenizer to standard.  Maybe I just don't know the right place
to look.

> What was the reason for changing this default? If I understand this better I 
> can avoid some pitfalls, perhaps.

If you are talking about example schemas, then the following may apply:

Because you understand how analysis components work well enough to even
ask your question, I think you're probably the kind of admin who is
going to thoroughly customize the schema and not rely on the defaults
for TextField types that come with Solr.  You're free to continue using
the classic tokenizer in your schema if that meets your needs better
than whatever changes are made to the examples by the devs.  The
examples are only starting points, virtually all Solr installs require
customizing the schema.

Thanks,
Shawn

Re: SolrException undefined field *

[Chris Hostetter]

: Might be the case as you mentioned Shawn. But there are no search requests
: triggered and might be that somehow a search query is getting fired to Solr
: end while indexing. Given the complete log information(solr.log) while the
: indexing is triggered.

the search request is triggered by a "newSearcher" cache warming query -- 
solr explicitly adds an "event=newSearcher" param so it shows up right 
there in the log for the query that's failing...

: 
(searcherExecutor-46-thread-1-processing-x:master_backoffice_backoffice_product_default)
: [   x:master_backoffice_backoffice_product_default] o.a.s.c.S.Request
: [master_backoffice_backoffice_product_default]  webapp=null path=null
: 
params={q=*:*%26facet%3Dtrue%26facet.field%3DcatalogVersion%26facet.field%3DcatalogId%26facet.field%3DapprovalStatus_string%26facet.field%3Dcategory_string_mv=false=newSearcher}
: status=400 QTime=0

You probably have something like this in your solrconfig.xml...


  
   
 *:*facet=true...
  ...

When what you should have is sometihng like...


  
   
 *:*
 true
 ...

https://lucene.apache.org/solr/guide/7_2/query-settings-in-solrconfig.html#QuerySettingsinSolrConfig-Query-RelatedListeners



-Hoss
http://www.lucidworks.com/

Re: SolrException undefined field *

[padmanabhan]

Might be the case as you mentioned Shawn. But there are no search requests
triggered and might be that somehow a search query is getting fired to Solr
end while indexing. Given the complete log information(solr.log) while the
indexing is triggered.

2018-01-09 15:42:25.292 INFO  (qtp834133664-15) [   ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={core=master_backoffice_backoffice_product_default=STATUS=true=javabin=2}
status=0 QTime=0
2018-01-09 15:42:25.296 INFO  (qtp834133664-14) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager@ab61d3a]
for /schema/analysis/synonyms/de
2018-01-09 15:42:25.297 INFO  (qtp834133664-14) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/synonyms/de params={} status=0 QTime=1
2018-01-09 15:42:25.307 INFO  (qtp834133664-18) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager@ab61d3a]
for /schema/analysis/synonyms/de
2018-01-09 15:42:25.307 INFO  (qtp834133664-18) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.ManagedResource
Processing update to /schema/analysis/synonyms/de: {} is a
java.util.LinkedHashMap
2018-01-09 15:42:25.308 INFO  (qtp834133664-18) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/synonyms/de params={} status=0 QTime=3
2018-01-09 15:42:25.312 INFO  (qtp834133664-19) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager@214c1af1]
for /schema/analysis/synonyms/en
2018-01-09 15:42:25.312 INFO  (qtp834133664-19) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/synonyms/en params={} status=0 QTime=0
2018-01-09 15:42:25.316 INFO  (qtp834133664-11) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager@214c1af1]
for /schema/analysis/synonyms/en
2018-01-09 15:42:25.316 INFO  (qtp834133664-11) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.ManagedResource
Processing update to /schema/analysis/synonyms/en: {} is a
java.util.LinkedHashMap
2018-01-09 15:42:25.316 INFO  (qtp834133664-11) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/synonyms/en params={} status=0 QTime=0
2018-01-09 15:42:25.327 INFO  (qtp834133664-16) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedWordSetResource@182ac811] for
/schema/analysis/stopwords/de
2018-01-09 15:42:25.327 INFO  (qtp834133664-16) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/stopwords/de params={} status=0 QTime=0
2018-01-09 15:42:25.334 INFO  (qtp834133664-15) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedWordSetResource@182ac811] for
/schema/analysis/stopwords/de
2018-01-09 15:42:25.334 INFO  (qtp834133664-15) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.ManagedResource
Processing update to /schema/analysis/stopwords/de: [] is a
java.util.ArrayList
2018-01-09 15:42:25.334 INFO  (qtp834133664-15) [  
x:master_backoffice_backoffice_product_default]
o.a.s.r.s.a.ManagedWordSetResource Applying updates: []
2018-01-09 15:42:25.334 INFO  (qtp834133664-15) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/stopwords/de params={} status=0 QTime=0
2018-01-09 15:42:25.338 INFO  (qtp834133664-13) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedWordSetResource@70f76fc2] for
/schema/analysis/stopwords/en
2018-01-09 15:42:25.338 INFO  (qtp834133664-13) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager
[master_backoffice_backoffice_product_default]  webapp=/solr
path=/schema/analysis/stopwords/en params={} status=0 QTime=1
2018-01-09 15:42:25.342 INFO  (qtp834133664-20) [  
x:master_backoffice_backoffice_product_default] o.a.s.r.RestManager Found
ManagedResource
[org.apache.solr.rest.schema.analysis.ManagedWordSetResource@70f76fc2] for
/schema/analysis/stopwords/en
2018-01-09 

Re: regarding exposing merge metrics

[suresh pendap]

Thanks Shalin for sharing the link. However if I follow the thread then it
seems like there was no conclusive evidence found that the performance
degradation was due to the merge or index related metrics.
If that is the case then can we just get rid of the config and publish
these metrics by default?


Regards
suresh



On Mon, Jan 8, 2018 at 10:25 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> The merge metrics were enabled by default in 6.4 but they were turned
> off in 6.4.2 because of large performance degradations. For more
> details, see https://issues.apache.org/jira/browse/SOLR-10130
>
> On Tue, Jan 9, 2018 at 9:11 AM, S G  wrote:
> > Yes, this is actually confusing and the documentation (
> > https://lucene.apache.org/solr/guide/7_2/metrics-reporting.html) does
> not
> > help either:
> >
> > *Index Merge Metrics* : These metrics are collected in respective
> > registries for each core (e.g., solr.core.collection1….), under the INDEX
> > category.
> > Basic metrics are always collected - collection of additional metrics can
> > be turned on using boolean parameters in the /config/indexConfig/metrics.
> >
> > However, we do not see the merge-metrics being collected if the above
> > config is absent. So what basic metrics are always collected for merge?
> > And why do the merge metrics require an additional config while most of
> the
> > others are reported directly?
> >
> > Thanks
> > SG
> >
> >
> >
> > On Mon, Jan 8, 2018 at 2:02 PM, suresh pendap 
> > wrote:
> >
> >> Hi,
> >> I am following the instructions from
> >> https://lucene.apache.org/solr/guide/7_1/metrics-reporting.html
> >>  in order to expose the Index merge related metrics.
> >>
> >> The document says that we have to add the below snippet in order to
> expose
> >> the merge metrics
> >>
> >> 
> >>   ...
> >>   
> >> 
> >>   524288
> >>   true
> >> 
> >> ...
> >>   
> >> ...
> >>
> >>
> >>
> >> I would like to know why is this metrics not exposed by default just
> like
> >> all the other metrics?
> >>
> >> Is there any performance overhead that we should be concerned about it?
> >>
> >> If there was no particular reason, then can we expose it by default?
> >>
> >>
> >>
> >> Regards
> >> Suresh
> >>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Always use leader for searching queries

[Novin Novin]

Thank you very much for all your help.

On Tue 9 Jan 2018, 16:32 Erick Erickson,  wrote:

> One thing to be aware of is that the commit points on the replicas in a
> replica may (will) fire at different times. So when you're comparing the
> number of docs on the replicas in a shard you have to compare before the
> last commit interval. So say you have a soft commit of 1 minute. When
> comparing the docs on each shard you need to restrict the query to things
> older than 1 minute or stop indexing and wait for 1 minute (i.e. until
> after the autocommit fires).
>
> Glad things worked out!
> Erick
>
> On Tue, Jan 9, 2018 at 4:08 AM, Novin Novin  wrote:
>
> > Hi Erick,
> >
> > Apology for delay.
> >
> > [This isn't what I meant. I meant to query each replica directly
> > _within_ the same shard. Your problem statement is that the leader and
> > replicas (I use "followers") have different document counts. How are
> > you verifying this? Through the admin UI? Using =false is
> > useful when you want to query each core directly (and you have to use
> > the core name) in some automated fashion.]
> >
> > I might be wrong here because now I can't produce it with distrib=false
> >
> > I also did as you said
> > [OK, I'm assuming then that you issue a manual commit sometime, right?
> > Here's what I'd do:
> > 1> turn off indexing
> > 2> issue a commit (soft or hard-with-opensearcher-true)
> > 3> now look at your doc counts on each replica.]
> >
> > Everything is seems ok now, I must have doing something wrong before.
> >
> > Thanks for all yours and walter's  help
> > Best,
> > Navin
> >
> >
> > On Wed, 3 Jan 2018 at 17:09 Walter Underwood 
> > wrote:
> >
> > > If you have a field for the indexed datetime, you can use a filter
> query
> > > to get rid of recent updates that might be in transit. I’d use double
> the
> > > autocommit time, to leave time for the followers to index.
> > >
> > > If the autocommit interval is one minute:
> > >
> > > fq=indexed_datetime:[* TO NOW-2MIN]
> > >
> > > wunder
> > > Walter Underwood
> > > wun...@wunderwood.org
> > > http://observer.wunderwood.org/  (my blog)
> > >
> > >
> > > > On Jan 3, 2018, at 8:58 AM, Erick Erickson 
> > > wrote:
> > > >
> > > > [I probably not need to do this because I have only one shard but I
> did
> > > > anyway count was different.]
> > > >
> > > > This isn't what I meant. I meant to query each replica directly
> > > > _within_ the same shard. Your problem statement is that the leader
> and
> > > > replicas (I use "followers") have different document counts. How are
> > > > you verifying this? Through the admin UI? Using =false is
> > > > useful when you want to query each core directly (and you have to use
> > > > the core name) in some automated fashion.
> > > >
> > > > [I have actually turned off auto soft commit for a time being but
> > > > nothing changed]
> > > >
> > > > OK, I'm assuming then that you issue a manual commit sometime, right?
> > > > Here's what I'd do:
> > > > 1> turn off indexing
> > > > 2> issue a commit (soft or hard-with-opensearcher-true)
> > > > 3> now look at your doc counts on each replica.
> > > >
> > > > If the counts are different then something's not right, Solr tries
> > > > very hard to not lose data, it's concerning if the leader and
> replicas
> > > > have different counts.
> > > >
> > > > Best,
> > > > Erick
> > > >
> > > > On Wed, Jan 3, 2018 at 1:51 AM, Novin Novin 
> > wrote:
> > > >> Hi Erick,
> > > >>
> > > >> Thanks for your reply.
> > > >>
> > > >> [ First of all, replicas can be off in terms of counts for the soft
> > > >> commit interval. The commits don't all happen on the replicas at the
> > > >> same wall-clock time. Solr promises eventual consistency, in this
> case
> > > >> NOW-autocommit time.]
> > > >>
> > > >> I realized that, to stop it. I have actually turned off auto soft
> > commit
> > > >> for a time being but nothing changed. Non leader replica still had
> > extra
> > > >> documents.
> > > >>
> > > >> [ So my first question is whether the replicas in the shard are
> > > >> inconsistent as of, say, NOW-your_soft_commit_time. I'd add a fudge
> > > >> factor of 10 seconds earlier just to be sure I was past autowarming.
> > > >> This does require that there be a time stamp. Absent a timestamp,
> you
> > > >> could suspend indexing for a few minutes and run the test like
> below.]
> > > >>
> > > >> When data was indexing at that time I was checking how the counts
> are
> > in
> > > >> both replica. What I found leader replica has 3 doc less than other
> > > replica
> > > >> always. I don't think so they were of by NOW-soft_commit_time,
> > > CloudSolrClient
> > > >> add some thing like this "_stateVer_=main:114" in query which I
> assume
> > > is
> > > >> for results to be consistent between both replica search.
> > > >>
> > > >> [Adding =false to your command and directing it at a
> 

ClassicTokenizer

[Rick Leir]

Hi all
A while ago the default was changed to StandardTokenizer from ClassicTokenizer. 
The biggest difference seems to be that Classic does not break on hyphens. 
There is also a different character pr(mumble). I prefer the Classic's 
non-break on hyphens. 

What was the reason for changing this default? If I understand this better I 
can avoid some pitfalls, perhaps.
Thanks -- Rick
-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: Always use leader for searching queries

[Erick Erickson]

One thing to be aware of is that the commit points on the replicas in a
replica may (will) fire at different times. So when you're comparing the
number of docs on the replicas in a shard you have to compare before the
last commit interval. So say you have a soft commit of 1 minute. When
comparing the docs on each shard you need to restrict the query to things
older than 1 minute or stop indexing and wait for 1 minute (i.e. until
after the autocommit fires).

Glad things worked out!
Erick

On Tue, Jan 9, 2018 at 4:08 AM, Novin Novin  wrote:

> Hi Erick,
>
> Apology for delay.
>
> [This isn't what I meant. I meant to query each replica directly
> _within_ the same shard. Your problem statement is that the leader and
> replicas (I use "followers") have different document counts. How are
> you verifying this? Through the admin UI? Using =false is
> useful when you want to query each core directly (and you have to use
> the core name) in some automated fashion.]
>
> I might be wrong here because now I can't produce it with distrib=false
>
> I also did as you said
> [OK, I'm assuming then that you issue a manual commit sometime, right?
> Here's what I'd do:
> 1> turn off indexing
> 2> issue a commit (soft or hard-with-opensearcher-true)
> 3> now look at your doc counts on each replica.]
>
> Everything is seems ok now, I must have doing something wrong before.
>
> Thanks for all yours and walter's  help
> Best,
> Navin
>
>
> On Wed, 3 Jan 2018 at 17:09 Walter Underwood 
> wrote:
>
> > If you have a field for the indexed datetime, you can use a filter query
> > to get rid of recent updates that might be in transit. I’d use double the
> > autocommit time, to leave time for the followers to index.
> >
> > If the autocommit interval is one minute:
> >
> > fq=indexed_datetime:[* TO NOW-2MIN]
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >
> > > On Jan 3, 2018, at 8:58 AM, Erick Erickson 
> > wrote:
> > >
> > > [I probably not need to do this because I have only one shard but I did
> > > anyway count was different.]
> > >
> > > This isn't what I meant. I meant to query each replica directly
> > > _within_ the same shard. Your problem statement is that the leader and
> > > replicas (I use "followers") have different document counts. How are
> > > you verifying this? Through the admin UI? Using =false is
> > > useful when you want to query each core directly (and you have to use
> > > the core name) in some automated fashion.
> > >
> > > [I have actually turned off auto soft commit for a time being but
> > > nothing changed]
> > >
> > > OK, I'm assuming then that you issue a manual commit sometime, right?
> > > Here's what I'd do:
> > > 1> turn off indexing
> > > 2> issue a commit (soft or hard-with-opensearcher-true)
> > > 3> now look at your doc counts on each replica.
> > >
> > > If the counts are different then something's not right, Solr tries
> > > very hard to not lose data, it's concerning if the leader and replicas
> > > have different counts.
> > >
> > > Best,
> > > Erick
> > >
> > > On Wed, Jan 3, 2018 at 1:51 AM, Novin Novin 
> wrote:
> > >> Hi Erick,
> > >>
> > >> Thanks for your reply.
> > >>
> > >> [ First of all, replicas can be off in terms of counts for the soft
> > >> commit interval. The commits don't all happen on the replicas at the
> > >> same wall-clock time. Solr promises eventual consistency, in this case
> > >> NOW-autocommit time.]
> > >>
> > >> I realized that, to stop it. I have actually turned off auto soft
> commit
> > >> for a time being but nothing changed. Non leader replica still had
> extra
> > >> documents.
> > >>
> > >> [ So my first question is whether the replicas in the shard are
> > >> inconsistent as of, say, NOW-your_soft_commit_time. I'd add a fudge
> > >> factor of 10 seconds earlier just to be sure I was past autowarming.
> > >> This does require that there be a time stamp. Absent a timestamp, you
> > >> could suspend indexing for a few minutes and run the test like below.]
> > >>
> > >> When data was indexing at that time I was checking how the counts are
> in
> > >> both replica. What I found leader replica has 3 doc less than other
> > replica
> > >> always. I don't think so they were of by NOW-soft_commit_time,
> > CloudSolrClient
> > >> add some thing like this "_stateVer_=main:114" in query which I assume
> > is
> > >> for results to be consistent between both replica search.
> > >>
> > >> [Adding =false to your command and directing it at a specific
> > >> _core_ (something like collection1_shard1_replica1) will only return
> > >> data from that core.]
> > >> I probably not need to do this because I have only one shard but I did
> > >> anyway count was different.
> > >>
> > >> [When you say you index every minute, I'm guessing you only index for
> > >> part of that minute, is that true? In that case you might 

Re: SolrException undefined field *

[Shawn Heisey]

On 1/9/2018 8:31 AM, padmanabhan wrote:
> I get the below error whenever an indexing is executed.. I didn't find enough
> clue on to where this field is coming from and how could i debug on to it..
> any help would be appreciated



> 2018-01-09 16:03:11.705 INFO 
> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
> [   x:master_backoffice_backoffice_product_default] o.a.s.c.S.Request
> [master_backoffice_backoffice_product_default]  webapp=null path=null
> params={q=*:*%26facet%3Dtrue%26facet.field%3DcatalogVersion%26facet.field%3DcatalogId%26facet.field%3DapprovalStatus_string%26facet.field%3Dcategory_string_mv=false=newSearcher}
> status=400 QTime=0

As Michael told you, this is a query, not an indexing request.

The source of the problem here is aggressive URL encoding.  Because of
this, what you have is the following text as the query string -- looks
like 130 characters:

*:*=true=catalogVersion=catalogId=approvalStatus_string=category_string_mv

This is happening because ampersand characters separating the URL
parameters have been URL encoded as %26 in the HTTP request, so the
decoded ampersands and the parameters around them are part of the query
string and are not being interpreted by the web server as separate URL
parameters.  Other characters are also being encoded, such as the equal
sign as %3D.  Therefore the first asterisk is being interpreted as a
field name, instead of being part of the three-character special string
"*:*" which is shorthand for all documents.

You're going to need to change whatever code you have that is generating
this query so that it doesn't do this aggressive URL encoding.

Thanks,
Shawn

Re: Newbie Question

[Deepak Goel]

*Hello*

*The code which worked for me:*

SolrClient client = new HttpSolrClient.Builder("
http://localhost:8983/solr/shakespeare;).build();

SolrQuery query = new SolrQuery();
query.setRequestHandler("/select");
query.setQuery("text_entry:henry");
query.setFields("text_entry");

QueryResponse queryResponse = null;
try
{
queryResponse = client.query(query);
}
catch (Exception e)
{

}

System.out.println("Query Response: " +queryResponse.toString());

if (queryResponse!=null && queryResponse.getResponse().size()>0)
{
SolrDocumentList results = queryResponse.getResults();
for (int i = 0; i < results.size(); ++i) {
SolrDocument document = results.get(i);
System.out.println("The result is: " +results.get(i));
System.out.println("The Document field names are: "
+document.getFieldNames());
}
}

*The data:*

{"index":{"_index":"shakespeare","_id":0}}
{"type":"act","line_id":1,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}
{"index":{"_index":"shakespeare","_id":1}}
{"type":"scene","line_id":2,"play_name":"Henry
IV","speech_number":"","line_number":"","speaker":"","text_entry":"SCENE I.
London. The palace."}

*Deepak*



Virus-free.
www.avg.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Tue, Jan 9, 2018 at 8:09 PM, Shawn Heisey  wrote:

> On 1/8/2018 10:23 AM, Deepak Goel wrote:
> > *I am trying to search for documents in my collection (Shakespeare). The
> > code is as follows:*
> >
> > SolrClient client = new HttpSolrClient.Builder("
> > http://localhost:8983/solr/shakespeare;).build();
> >
> > SolrDocument doc = client.getById("2");
> > *However this does not return any document. What mistake am I making?*
>
> The getById method accesses the handler named "/get", normally defined
> with the RealTimeGetHandler class.  In recent Solr versions, the /get
> handler is defined implicitly and does not have to be configured, but in
> older versions (not sure which ones) you do need to have it in
> solrconfig.xml.
>
> I didn't expect your code to work because getById method returns a
> SolrDocumentList and you have SolrDocument, but apparently this actually
> does work.  I have tried code very similar to yours against the
> techproducts example in version 7.1, and it works perfectly.  I will
> share the exact code I tried and what results I got below.
>
> What code have you tried after the code you've shared?  How are you
> determining that no document is returned?  Are there any error messages
> logged by the client code or Solr?  If there are, can you share them?
>
> Do you have a document in the shakespeare index that has the value "2"
> in whatever field is the uniqueKey?  Does the schema have a uniqueKey
> defined?
>
> Can you find the entry in solr.log that logs the query and share that
> entire log entry?
>
> Code:
>
> public static void main(String[] args) throws SolrServerException,
> IOException
> {
>   String baseUrl = "http://localhost:8983/solr/techproducts;;
>   SolrClient client = new HttpSolrClient.Builder(baseUrl).build();
>   SolrDocument doc = client.getById("SP2514N");
>   System.out.println(doc.getFieldValue("name"));
> }
>
> Console log from that code:
>
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further details.
> Samsung SpinPoint P120 SP2514N - hard drive - 250 GB - ATA-133
>
>
> Including the collection/core name in the URL is an older way of writing
> SolrJ code.  It works well, but multiple collections can be accessed
> through one client object if you change it and your SolrJ version is new
> enough.
>
> Thanks,
> Shawn
>
>

Re: SolrException undefined field *

[Michael Kuhlmann]

To correct myself, querying "*" is allowed in the sense that asking for
all fields is done by assigning "*" to the fl parameter.

So the problem is possibly not that "*" is requested, but that the star
is used somewhere else, probably in the q parameter.

We can help you better when you pass the full query string (if you're
able to fetch it).

-Michael


Am 09.01.2018 um 16:38 schrieb Michael Kuhlmann:
> First, you might want to index, but what Solr is executing here is a
> search request.
> 
> Second, you're querying for a dynamic field "*" which is not defined in
> your schema. This is quite obvious, the exception says right this.
> 
> So whatever is sending the query (some client, it seems) is doing the
> wrong thing. Or your schema definition is not matching what the client
> expects.
> 
> Since we don't know what client code you're using, we can't tell more.
> 
> -Michael
> 
> 
> Am 09.01.2018 um 16:31 schrieb padmanabhan:
>> I get the below error whenever an indexing is executed.. I didn't find enough
>> clue on to where this field is coming from and how could i debug on to it..
>> any help would be appreciated
>>
>> 2018-01-09 16:03:11.705 INFO 
>> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
>> [   x:master_backoffice_backoffice_product_default]
>> o.a.s.c.QuerySenderListener QuerySenderListener sending requests to
>> Searcher@232ae42b[master_backoffice_backoffice_product_default]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1p(6.4.1):C56)))}
>> 2018-01-09 16:03:11.705 ERROR
>> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
>> [   x:master_backoffice_backoffice_product_default]
>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined
>> field *
>>  at
>> org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1308)
>>  at 
>> org.apache.solr.schema.IndexSchema.getFieldType(IndexSchema.java:1260)
>>  at
>> org.apache.solr.parser.SolrQueryParserBase.getWildcardQuery(SolrQueryParserBase.java:932)
>>  at
>> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:616)
>>  at org.apache.solr.parser.QueryParser.Term(QueryParser.java:312)
>>  at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:182)
>>  at org.apache.solr.parser.QueryParser.Query(QueryParser.java:102)
>>  at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:91)
>>  at
>> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:194)
>>  at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
>>  at org.apache.solr.search.QParser.getQuery(QParser.java:168)
>>  at
>> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
>>  at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
>>  at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
>>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2306)
>>  at
>> org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:72)
>>  at 
>> org.apache.solr.core.SolrCore.lambda$getSearcher$4(SolrCore.java:2094)
>>  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>  at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>  at java.lang.Thread.run(Thread.java:748)
>>
>> 2018-01-09 16:03:11.705 INFO 
>> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
>> [   x:master_backoffice_backoffice_product_default] o.a.s.c.S.Request
>> [master_backoffice_backoffice_product_default]  webapp=null path=null
>> params={q=*:*%26facet%3Dtrue%26facet.field%3DcatalogVersion%26facet.field%3DcatalogId%26facet.field%3DapprovalStatus_string%26facet.field%3Dcategory_string_mv=false=newSearcher}
>> status=400 QTime=0
>>
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
> 

Re: SolrException undefined field *

[Michael Kuhlmann]

First, you might want to index, but what Solr is executing here is a
search request.

Second, you're querying for a dynamic field "*" which is not defined in
your schema. This is quite obvious, the exception says right this.

So whatever is sending the query (some client, it seems) is doing the
wrong thing. Or your schema definition is not matching what the client
expects.

Since we don't know what client code you're using, we can't tell more.

-Michael


Am 09.01.2018 um 16:31 schrieb padmanabhan:
> I get the below error whenever an indexing is executed.. I didn't find enough
> clue on to where this field is coming from and how could i debug on to it..
> any help would be appreciated
> 
> 2018-01-09 16:03:11.705 INFO 
> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
> [   x:master_backoffice_backoffice_product_default]
> o.a.s.c.QuerySenderListener QuerySenderListener sending requests to
> Searcher@232ae42b[master_backoffice_backoffice_product_default]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1p(6.4.1):C56)))}
> 2018-01-09 16:03:11.705 ERROR
> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
> [   x:master_backoffice_backoffice_product_default]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined
> field *
>   at
> org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1308)
>   at 
> org.apache.solr.schema.IndexSchema.getFieldType(IndexSchema.java:1260)
>   at
> org.apache.solr.parser.SolrQueryParserBase.getWildcardQuery(SolrQueryParserBase.java:932)
>   at
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:616)
>   at org.apache.solr.parser.QueryParser.Term(QueryParser.java:312)
>   at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:182)
>   at org.apache.solr.parser.QueryParser.Query(QueryParser.java:102)
>   at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:91)
>   at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:194)
>   at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
>   at org.apache.solr.search.QParser.getQuery(QParser.java:168)
>   at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
>   at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
>   at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2306)
>   at
> org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:72)
>   at 
> org.apache.solr.core.SolrCore.lambda$getSearcher$4(SolrCore.java:2094)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 
> 2018-01-09 16:03:11.705 INFO 
> (searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
> [   x:master_backoffice_backoffice_product_default] o.a.s.c.S.Request
> [master_backoffice_backoffice_product_default]  webapp=null path=null
> params={q=*:*%26facet%3Dtrue%26facet.field%3DcatalogVersion%26facet.field%3DcatalogId%26facet.field%3DapprovalStatus_string%26facet.field%3Dcategory_string_mv=false=newSearcher}
> status=400 QTime=0
> 
> 
> 
> 
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> 

Re: Custom Sort option to apply at SOLR index

[padmanabhan]

Thank you Erick, it worked.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SolrException undefined field *

[padmanabhan]

I get the below error whenever an indexing is executed.. I didn't find enough
clue on to where this field is coming from and how could i debug on to it..
any help would be appreciated

2018-01-09 16:03:11.705 INFO 
(searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
[   x:master_backoffice_backoffice_product_default]
o.a.s.c.QuerySenderListener QuerySenderListener sending requests to
Searcher@232ae42b[master_backoffice_backoffice_product_default]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1p(6.4.1):C56)))}
2018-01-09 16:03:11.705 ERROR
(searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
[   x:master_backoffice_backoffice_product_default]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined
field *
at
org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1308)
at 
org.apache.solr.schema.IndexSchema.getFieldType(IndexSchema.java:1260)
at
org.apache.solr.parser.SolrQueryParserBase.getWildcardQuery(SolrQueryParserBase.java:932)
at
org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:616)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:312)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:182)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:102)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:91)
at
org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:194)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
at org.apache.solr.search.QParser.getQuery(QParser.java:168)
at
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2306)
at
org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:72)
at 
org.apache.solr.core.SolrCore.lambda$getSearcher$4(SolrCore.java:2094)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

2018-01-09 16:03:11.705 INFO 
(searcherExecutor-51-thread-1-processing-x:master_backoffice_backoffice_product_default)
[   x:master_backoffice_backoffice_product_default] o.a.s.c.S.Request
[master_backoffice_backoffice_product_default]  webapp=null path=null
params={q=*:*%26facet%3Dtrue%26facet.field%3DcatalogVersion%26facet.field%3DcatalogId%26facet.field%3DapprovalStatus_string%26facet.field%3Dcategory_string_mv=false=newSearcher}
status=400 QTime=0




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: SSL configuration with Master/Slave

[Sundaram, Dinesh]

FYI, This has been resolved.

Dinesh Sundaram
MBS Platform Engineering

Mastercard
[cid:image001.png@01D3892A.668B6700]

From: Sundaram, Dinesh
Sent: Monday, January 8, 2018 1:58 PM
To: solr-user 
Subject: SSL configuration with Master/Slave

Team,

I'm facing an SSL issue while configuring Master/Slave. Master runs fine lone 
with SSL and Slave runs fine lone with SSL but getting SSL exception during the 
synch up. It gives the below error. I believe we need to trust the target 
server at source. Can you give me the steps to allow inbound calls at source 
jvm. FYI, the same synch up works fine via http.

2018-01-08 13:57:06.735 WARN  (qtp33524623-16) [c:dm-global s:shard1 
r:core_node2 x:dm-global_shard1_replica_n1] o.a.s.h.ReplicationHandler 
Exception while invoking 'details' method for replication on master
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: 
https://test21.mastercard.int:8983/solr/dm-global_shard1_replica_n1
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:640)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.handler.IndexFetcher.getDetails(IndexFetcher.java:1823)
at 
org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:954)
at 
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:332)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2484)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: 
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
   

Re: Newbie Question

[Shawn Heisey]

On 1/8/2018 10:23 AM, Deepak Goel wrote:
> *I am trying to search for documents in my collection (Shakespeare). The
> code is as follows:*
>
> SolrClient client = new HttpSolrClient.Builder("
> http://localhost:8983/solr/shakespeare;).build();
>
> SolrDocument doc = client.getById("2");
> *However this does not return any document. What mistake am I making?*

The getById method accesses the handler named "/get", normally defined
with the RealTimeGetHandler class.  In recent Solr versions, the /get
handler is defined implicitly and does not have to be configured, but in
older versions (not sure which ones) you do need to have it in
solrconfig.xml.

I didn't expect your code to work because getById method returns a
SolrDocumentList and you have SolrDocument, but apparently this actually
does work.  I have tried code very similar to yours against the
techproducts example in version 7.1, and it works perfectly.  I will
share the exact code I tried and what results I got below.

What code have you tried after the code you've shared?  How are you
determining that no document is returned?  Are there any error messages
logged by the client code or Solr?  If there are, can you share them?

Do you have a document in the shakespeare index that has the value "2"
in whatever field is the uniqueKey?  Does the schema have a uniqueKey
defined?

Can you find the entry in solr.log that logs the query and share that
entire log entry?

Code:

public static void main(String[] args) throws SolrServerException,
IOException
{
  String baseUrl = "http://localhost:8983/solr/techproducts;;
  SolrClient client = new HttpSolrClient.Builder(baseUrl).build();
  SolrDocument doc = client.getById("SP2514N");
  System.out.println(doc.getFieldValue("name"));
}

Console log from that code:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
further details.
Samsung SpinPoint P120 SP2514N - hard drive - 250 GB - ATA-133


Including the collection/core name in the URL is an older way of writing
SolrJ code.  It works well, but multiple collections can be accessed
through one client object if you change it and your SolrJ version is new
enough.

Thanks,
Shawn

solr cluster: solr auto suggestion with requestHandler

[Venkata MR]

Hi All,

Problem: Not able to build suggest data on all solr cluster nodes

Configured three solr using external zookeeper
Configured the requestHandler for auto-suggestion as below



  true
  5
  Name


  suggest




 
  Name
  name
  name
  AnalyzingInfixLookupFactory
  name_suggester_infix_dir
  DocumentDictionaryFactory
  key
  lowercase
  name_suggestor_dictionary
  string
 


When we manually issue request with suggest.build=true on one of the node for 
building suggest data, suggest data is built for that particular node only, 
other nodes of cluster are not getting build the suggest data.
Any configuration mismatch?

Thanks a lot.

Thanks & Regards
Venkata MR
+91 98455 77125

::DISCLAIMER::
--
The contents of this e-mail and any attachment(s) are confidential and intended 
for the named recipient(s) only. E-mail transmission is not guaranteed to be 
secure or error-free as information could be intercepted, corrupted, lost, 
destroyed, arrive late or incomplete, or may contain viruses in transmission. 
The e mail and its contents (with or without referred errors) shall therefore 
not attach any liability on the originator or HCL or its affiliates. Views or 
opinions, if any, presented in this email are solely those of the author and 
may not necessarily reflect the views or opinions of HCL or its affiliates. Any 
form of reproduction, dissemination, copying, disclosure, modification, 
distribution and / or publication of this message without the prior written 
consent of authorized representative of HCL is strictly prohibited. If you have 
received this email in error please delete it and notify the sender 
immediately. Before opening any email and/or attachments, please check them for 
viruses and other defects.
--

Re: Always use leader for searching queries

[Novin Novin]

Hi Erick,

Apology for delay.

[This isn't what I meant. I meant to query each replica directly
_within_ the same shard. Your problem statement is that the leader and
replicas (I use "followers") have different document counts. How are
you verifying this? Through the admin UI? Using =false is
useful when you want to query each core directly (and you have to use
the core name) in some automated fashion.]

I might be wrong here because now I can't produce it with distrib=false

I also did as you said
[OK, I'm assuming then that you issue a manual commit sometime, right?
Here's what I'd do:
1> turn off indexing
2> issue a commit (soft or hard-with-opensearcher-true)
3> now look at your doc counts on each replica.]

Everything is seems ok now, I must have doing something wrong before.

Thanks for all yours and walter's  help
Best,
Navin


On Wed, 3 Jan 2018 at 17:09 Walter Underwood  wrote:

> If you have a field for the indexed datetime, you can use a filter query
> to get rid of recent updates that might be in transit. I’d use double the
> autocommit time, to leave time for the followers to index.
>
> If the autocommit interval is one minute:
>
> fq=indexed_datetime:[* TO NOW-2MIN]
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>
> > On Jan 3, 2018, at 8:58 AM, Erick Erickson 
> wrote:
> >
> > [I probably not need to do this because I have only one shard but I did
> > anyway count was different.]
> >
> > This isn't what I meant. I meant to query each replica directly
> > _within_ the same shard. Your problem statement is that the leader and
> > replicas (I use "followers") have different document counts. How are
> > you verifying this? Through the admin UI? Using =false is
> > useful when you want to query each core directly (and you have to use
> > the core name) in some automated fashion.
> >
> > [I have actually turned off auto soft commit for a time being but
> > nothing changed]
> >
> > OK, I'm assuming then that you issue a manual commit sometime, right?
> > Here's what I'd do:
> > 1> turn off indexing
> > 2> issue a commit (soft or hard-with-opensearcher-true)
> > 3> now look at your doc counts on each replica.
> >
> > If the counts are different then something's not right, Solr tries
> > very hard to not lose data, it's concerning if the leader and replicas
> > have different counts.
> >
> > Best,
> > Erick
> >
> > On Wed, Jan 3, 2018 at 1:51 AM, Novin Novin  wrote:
> >> Hi Erick,
> >>
> >> Thanks for your reply.
> >>
> >> [ First of all, replicas can be off in terms of counts for the soft
> >> commit interval. The commits don't all happen on the replicas at the
> >> same wall-clock time. Solr promises eventual consistency, in this case
> >> NOW-autocommit time.]
> >>
> >> I realized that, to stop it. I have actually turned off auto soft commit
> >> for a time being but nothing changed. Non leader replica still had extra
> >> documents.
> >>
> >> [ So my first question is whether the replicas in the shard are
> >> inconsistent as of, say, NOW-your_soft_commit_time. I'd add a fudge
> >> factor of 10 seconds earlier just to be sure I was past autowarming.
> >> This does require that there be a time stamp. Absent a timestamp, you
> >> could suspend indexing for a few minutes and run the test like below.]
> >>
> >> When data was indexing at that time I was checking how the counts are in
> >> both replica. What I found leader replica has 3 doc less than other
> replica
> >> always. I don't think so they were of by NOW-soft_commit_time,
> CloudSolrClient
> >> add some thing like this "_stateVer_=main:114" in query which I assume
> is
> >> for results to be consistent between both replica search.
> >>
> >> [Adding =false to your command and directing it at a specific
> >> _core_ (something like collection1_shard1_replica1) will only return
> >> data from that core.]
> >> I probably not need to do this because I have only one shard but I did
> >> anyway count was different.
> >>
> >> [When you say you index every minute, I'm guessing you only index for
> >> part of that minute, is that true? In that case you might get more
> >> consistency if, instead of relying totally on your autoconfig
> >> settings, specify commitWithin on your update command. That should
> >> force the commits to happen more closely in-sync, although still not
> >> perfect.]
> >>
> >> We receive data every minute, so whenever we have new data we send it to
> >> Solr cloud using queue. You said don't rely on auto config. Do you mean
> I
> >> should turn off autocommit and use commitWithin using solrj or leave
> >> autoCommit as it is and also use commitWithin from solrj client.
> >>
> >> I apologize If I am not clear, thanks for your help again.
> >>
> >> Thanks in advance,
> >> Navin
> >>
> >>
> >>
> >>
> >>
> >> On Tue, 2 Jan 2018 at 18:05 Erick Erickson 
> wrote:
> >>
> >>> First of all, 

Re: Deliver static html content via solr

[Shawn Heisey]

On 1/7/2018 1:30 PM, Rick Leir wrote:

The easy solution is to put something like solr-security-proxy [1] in front of 
a Solr/Velocity app, and this is working for me. However, this has a blacklist 
for Solr parms and I think it should have a whitelist instead. Also, it does 
not check ranges or filter chars. Is this proxy adequate for use on the open 
internet? In particular, what character filtering should I add to it?


I don't have information like that readily available.  I would be 
worried with any proxy software that something important had been 
forgotten and would open the door to either changing the index or not 
blocking denial of service requests.


My recommendation is to never expose Solr to the Internet, or to anybody 
who is not responsible for its care.  There should always be some kind 
of front end server-side software that handles searching on behalf of 
the user.


Even with those precautions, clever users will probably be able to 
figure out how to send denial of service queries, but without direct 
access to Solr's API, it would not be as vulnerable.


Thanks,
Shawn

Re: In-place update vs Atomic updates

[Shawn Heisey]

On 1/8/2018 10:17 PM, kshitij tyagi wrote:

1. Does in place updates opens a new searcher by itself or not?
2. As the entire segment is rewriten, it means that frequent in place
updates are expensive as each in place update will rewrite the entire
segment again? Correct me here if my understanding is not correct.


Opening a new searcher is not related to the update.  It's something 
that happens at commit time, if the commit has openSearcher=true (which 
is the default setting).


In-place updates don't rewrite the entire segment, they only rewrite 
part of the docValues information for the segment -- only the portion 
for the fields that got updated.  The information is written into a new 
file, and the original file is untouched.


If there are multiple fields with docValues and not all of them are 
updated, then it would not be possible to delete the old file until the 
segment gets merged.  I am not sure about what happens if *every* field 
with docValues is eligible for in-place updates and all of them get 
updated.  If that were the case, then it would be possible to have an 
optimization that removes the old docValues file, but I have no idea 
whether Lucene actually has that as an optimization.  I would not expect 
most indexes to be eligible for the optimization even if Lucene can do it.


Yes, frequent in-place updates can be expensive, and can make the index 
larger, because the values in the updated field for every document in 
the segment will be written to a new file.  If you never optimize the 
index and mostly update recently added documents, then the segments 
involved will probably be small, and performance would be pretty good.


Thanks,
Shawn

Re: EmbeddedSolrServer removed quietly in 7.0?

[Robert Krüger]

On Tue, Jan 9, 2018 at 11:37 AM, Shawn Heisey  wrote:

> On 1/9/2018 2:42 AM, Robert Krüger wrote:
>
>> I am looking to upgrade an application that still uses version 4.6.1 to
>> latest solr (7.2) but realized that support for EmbeddedSolrServer seems
>> to
>> have vanished with 7.0 but I could find no mention of it in the release
>> notes, which strikes me as odd for such a disruptive change. Is it
>> somewhere else or has it just gone silently? The class wasn't deprecated
>> in
>> 6.x as far as I can see. What am I missing?
>>
>> I have no problem updating to 6.6.2 for now to delay having to worry about
>> a bigger migration but just don't want to do it, because I am missing
>> something obvious.
>>
>> Btw. I do not want to start yet another thread about the merits of using
>> an
>> embedded solr server ;-).
>>
>
> It's still there.  Here's the 7.2 javadoc:
>
> https://lucene.apache.org/solr/7_2_0/solr-core/org/apache/
> solr/client/solrj/embedded/EmbeddedSolrServer.html


Duh, thanks so much. I was looking in the solrj javadoc, not solr-core! My
bad.


>
>
> Since 5.0, it is a descendant of SolrClient, and the SolrServer abstract
> class was removed in 6.0.
>
> Regarding whether you should use the embedded server or not:  For
> production usage, I would not recommend it, because redundancy is not
> possible.  But there are certain situations (mostly dev/testing) where it's
> absolutely the right solution.
>

We're using it in a classic embedded situation in a desktop app where
redundancy is not an issue and are super-happy with it.


>
> I am not expecting EmbeddedSolrServer to be removed.  It is used
> extensively in Solr tests and by many users in the wild.
>
>
That is very good to know. Thanks again!

Re: EmbeddedSolrServer removed quietly in 7.0?

[Shawn Heisey]

On 1/9/2018 2:42 AM, Robert Krüger wrote:

I am looking to upgrade an application that still uses version 4.6.1 to
latest solr (7.2) but realized that support for EmbeddedSolrServer seems to
have vanished with 7.0 but I could find no mention of it in the release
notes, which strikes me as odd for such a disruptive change. Is it
somewhere else or has it just gone silently? The class wasn't deprecated in
6.x as far as I can see. What am I missing?

I have no problem updating to 6.6.2 for now to delay having to worry about
a bigger migration but just don't want to do it, because I am missing
something obvious.

Btw. I do not want to start yet another thread about the merits of using an
embedded solr server ;-).


It's still there.  Here's the 7.2 javadoc:

https://lucene.apache.org/solr/7_2_0/solr-core/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.html

Since 5.0, it is a descendant of SolrClient, and the SolrServer abstract 
class was removed in 6.0.


Regarding whether you should use the embedded server or not:  For 
production usage, I would not recommend it, because redundancy is not 
possible.  But there are certain situations (mostly dev/testing) where 
it's absolutely the right solution.


I am not expecting EmbeddedSolrServer to be removed.  It is used 
extensively in Solr tests and by many users in the wild.


Thanks,
Shawn