Re: maxBooleanClauses change in solr.xml not reflecting in solr 8.4.1
Thanks Hoss,
Yes, i was making the change in solr.xml in wrong directory earlier.
Also as you said:
: You need to update EVERY solrconfig.xml that the JVM is loading for this
to
: actually work.
that has not been true for a while, see SOLR-13336 / SOLR-10921 ...
I validated this and it's working as expected. We don't need to update
every solrconfig.xml.
The value mentioned in solr.xml is global and if maxBooleanClauses for any
collection in solrconfig.xml exceeds the limit specified in solr.xml then
we get the exception.
Thanks for replying.
On Wed, Jan 6, 2021 at 10:57 PM dinesh naik
wrote:
> Thanks Shawn,
>
> This entry ${sol
> r.max.booleanClauses:2048} in solr.xml was introduced only in solr
> 8.x version and were not present in 7.6 version.
>
> We have this in solrconfig.xml in 8.4.1 version.
> ${solr.max.booleanClauses:2048} maxBooleanClauses>
> i was updating the solr.xml in the installation directory and not the
> installed data directory, hence the change was not reflecting.
> After updating the correct solr.xml and restarting the Solr nodes the new
> value is working as expected.
>
> On Wed, Jan 6, 2021 at 10:34 PM Chris Hostetter
> wrote:
>
>>
>> : You need to update EVERY solrconfig.xml that the JVM is loading for
>> this to
>> : actually work.
>>
>> that has not been true for a while, see SOLR-13336 / SOLR-10921 ...
>>
>> : > 2. updated solr.xml :
>> : > ${solr.max.booleanClauses:2048}
>> :
>> : I don't think it's currently possible to set the value with solr.xml.
>>
>> Not only is it possible, it's neccessary -- the value in solr.xml acts as
>> a hard upper limit (and affects all queries, even internally expanded
>> queries) on the "soft limit" in solrconfig.xml (that only affects
>> explicitly supplied boolean queries from users)
>>
>> As to the original question...
>>
>> > 2021-01-05 14:03:59.603 WARN (qtp1545077099-27)
>> x:col1_shard1_replica_n3
>> > o.a.s.c.SolrConfig solrconfig.xml: of 2048 is
>> greater
>> > than global limit of 1024 and will have no effect
>>
>> I attempted to reproduce this with 8.4.1 and did not see the probem you
>> are describing.
>>
>> Are you 100% certain you are updating the correct solr.xml file? If you
>> add some non-xml giberish to the solr.xml you are editing does the solr
>> node fail to start up?
>>
>> Remember that when using SolrCloud, solr will try to load solr.xml from
>> zk
>> first, and only look on local disk if it can't be found in ZK ... look
>> for
>> log messages like "solr.xml found in ZooKeeper. Loading..." vs "Loading
>> solr.xml from SolrHome (not found in ZooKeeper)"
>>
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
>
> --
> Best Regards,
> Dinesh Naik
>
--
Best Regards,
Dinesh Naik
Re: maxBooleanClauses change in solr.xml not reflecting in solr 8.4.1
Thanks Shawn,
This entry ${solr.max.booleanClauses:2048} in solr.xml was introduced only in solr 8.x version and were not
present in 7.6 version.
We have this in solrconfig.xml in 8.4.1 version.
${solr.max.booleanClauses:2048}
i was updating the solr.xml in the installation directory and not the
installed data directory, hence the change was not reflecting.
After updating the correct solr.xml and restarting the Solr nodes the new
value is working as expected.
On Wed, Jan 6, 2021 at 10:34 PM Chris Hostetter
wrote:
>
> : You need to update EVERY solrconfig.xml that the JVM is loading for this
> to
> : actually work.
>
> that has not been true for a while, see SOLR-13336 / SOLR-10921 ...
>
> : > 2. updated solr.xml :
> : > ${solr.max.booleanClauses:2048}
> :
> : I don't think it's currently possible to set the value with solr.xml.
>
> Not only is it possible, it's neccessary -- the value in solr.xml acts as
> a hard upper limit (and affects all queries, even internally expanded
> queries) on the "soft limit" in solrconfig.xml (that only affects
> explicitly supplied boolean queries from users)
>
> As to the original question...
>
> > 2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
> > o.a.s.c.SolrConfig solrconfig.xml: of 2048 is greater
> > than global limit of 1024 and will have no effect
>
> I attempted to reproduce this with 8.4.1 and did not see the probem you
> are describing.
>
> Are you 100% certain you are updating the correct solr.xml file? If you
> add some non-xml giberish to the solr.xml you are editing does the solr
> node fail to start up?
>
> Remember that when using SolrCloud, solr will try to load solr.xml from zk
> first, and only look on local disk if it can't be found in ZK ... look for
> log messages like "solr.xml found in ZooKeeper. Loading..." vs "Loading
> solr.xml from SolrHome (not found in ZooKeeper)"
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
--
Best Regards,
Dinesh Naik
maxBooleanClauses change in solr.xml not reflecting in solr 8.4.1
Hi all,
I want to update the maxBooleanClauses to 2048 (from default value 1024).
Below are the steps tried:
1. updated solrconfig.xml :
${solr.max.booleanClauses:2048}
2. updated solr.xml :
${solr.max.booleanClauses:2048}
3. Restarted the solr nodes.
4. Tried query with more than 2000 OR clauses and getting below waring
message in solr logs:
2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
o.a.s.c.SolrConfig solrconfig.xml: of 2048 is greater
than global limit of 1024 and will have no effect
2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
o.a.s.c.SolrConfig set 'maxBooleanClauses' in solr.xml to increase global
limit
Note: In 7.6.1 version we just need to change the solrconfig.xml and it
works.
Kindly let me know if i am missing something for making it work in 8.4.1
version.
--
Best Regards,
Dinesh Naik
Queries on adding headers to solrj Request
Hi all, We are planning to add security to Solr using . For this we are adding few information in the headers of each SolrJ Request. These request will be intercepted by some application (proxy) in the Solr VM and then route it to Solr ( Considering Solr port as 8983 ) . Could you please answer below queries: 1. Are there any API ( Path ) that Solr Client cannot access and only Solr uses for Intra node communication? 2. As the SolrJ client will add headers, Intra communication from Solr also needs to add these headers ( like ping request from Solr1 Node to Solr2 Node ). Could Solr add custom headers for intra node communication? 3. Apart from 8983 node, are there any other ports Solr is using for intra node communication? 4. how to add headers to CloudSolrClient ? -- Best Regards, Dinesh Naik
Re: Solr 7.6.0: PingRequestHandler - Changing the default query (*:*)
Hi Erick, Each vm has 128GB of physical memory. On Mon, Aug 5, 2019, 8:38 PM Erick Erickson wrote: > How much total physical memory on your machine? Lucene holds a lot of the > index in MMapDirectory space. My starting point is to allocate no more than > 50% of my physical memory to the Java heap. You’re allocating 31G, if you > don’t > have at _least_ 64G on these machines you’re probably swapping. > > See: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Best, > Erick > > > > On Aug 5, 2019, at 10:58 AM, dinesh naik > wrote: > > > > Hi Shawn, > > yes i am running solr in cloud mode and Even after adding the params > row=0 > > and distrib=false, the query response is more than 15 sec due to more > than > > a billion doc set. > > Also the soft commit setting can not be changed to a higher no. due to > > requirement from business team. > > > > > http://hostname:8983/solr/parts/select?indent=on=*:*=0=json=false > > takes more than 10 sec always. > > > > Here are the java heap and G1GC setting i have , > > > > /usr/java/default/bin/java -server -Xmx31g -Xms31g -XX:+UseG1GC > > -XX:MaxGCPauseMillis=250 -XX:ConcGCThreads=5 > > -XX:ParallelGCThreads=10 -XX:+UseLargePages -XX:+AggressiveOpts > > -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled > > -XX:InitiatingHeapOccupancyPercent=50 -XX:G1ReservePercent=18 > > -XX:MaxNewSize=6G -XX:PrintFLSStatistics=1 > > -XX:+PrintPromotionFailure -XX:+HeapDumpOnOutOfMemoryError > > -XX:HeapDumpPath=/solr7/logs/heapdump > > -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps > > -XX:+PrintGCTimeStamps > > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime > > > > JVM heap has never crossed 20GB in my setup , also Young G1GC timing is > > well within milli seconds (in range of 25-200 ms). > > > > On Mon, Aug 5, 2019 at 6:37 PM Shawn Heisey wrote: > > > >> On 8/4/2019 10:15 PM, dinesh naik wrote: > >>> My question is regarding the custom query being used. Here i am > querying > >>> for field _root_ which is available in all of my cluster and defined > as a > >>> string field. The result for _root_:abc might not get me any match as > >>> well(i am ok with not finding any matches, the query should not be > taking > >>> 10-15 seconds for getting the response). > >> > >> Typically the *:* query is the fastest option. It is special syntax > >> that means "all documents" and it usually executes very quickly. It > >> will be faster than querying for a value in a specific field, which is > >> what you have defined currently. > >> > >> I will typically add a "rows" parameter to the ping handler with a value > >> of 1, so Solr will not be retrieving a large amount of data. If you are > >> running Solr in cloud mode, you should experiment with setting the > >> distrib parameter to false, which will hopefully limit the query to the > >> receiving node only. > >> > >> Erick has already mentioned GC pauses as a potential problem. With a > >> 10-15 second response time, I think that has high potential to be the > >> underlying cause. > >> > >> The response you included at the beginning of the thread indicates there > >> are 1.3 billion documents, which is going to require a fair amount of > >> heap memory. If seeing such long ping times with a *:* query is > >> something that happens frequently, your heap may be too small, which > >> will cause frequent full garbage collections. > >> > >> The very low autoSoftCommit time can contribute to system load. I think > >> it's very likely, especially with such a large index, that in many cases > >> those automatic commits are taking far longer than 5 seconds to > >> complete. If that's the case, you're not achieving a 5 second > >> visibility interval and you are putting a lot of load on Solr, so I > >> would consider increasing it. > >> > >> Thanks, > >> Shawn > >> > > > > > > -- > > Best Regards, > > Dinesh Naik > >
Re: Solr 7.6.0: PingRequestHandler - Changing the default query (*:*)
Hi Shawn, yes i am running solr in cloud mode and Even after adding the params row=0 and distrib=false, the query response is more than 15 sec due to more than a billion doc set. Also the soft commit setting can not be changed to a higher no. due to requirement from business team. http://hostname:8983/solr/parts/select?indent=on=*:*=0=json=false takes more than 10 sec always. Here are the java heap and G1GC setting i have , /usr/java/default/bin/java -server -Xmx31g -Xms31g -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:ConcGCThreads=5 -XX:ParallelGCThreads=10 -XX:+UseLargePages -XX:+AggressiveOpts -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:InitiatingHeapOccupancyPercent=50 -XX:G1ReservePercent=18 -XX:MaxNewSize=6G -XX:PrintFLSStatistics=1 -XX:+PrintPromotionFailure -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/solr7/logs/heapdump -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime JVM heap has never crossed 20GB in my setup , also Young G1GC timing is well within milli seconds (in range of 25-200 ms). On Mon, Aug 5, 2019 at 6:37 PM Shawn Heisey wrote: > On 8/4/2019 10:15 PM, dinesh naik wrote: > > My question is regarding the custom query being used. Here i am querying > > for field _root_ which is available in all of my cluster and defined as a > > string field. The result for _root_:abc might not get me any match as > > well(i am ok with not finding any matches, the query should not be taking > > 10-15 seconds for getting the response). > > Typically the *:* query is the fastest option. It is special syntax > that means "all documents" and it usually executes very quickly. It > will be faster than querying for a value in a specific field, which is > what you have defined currently. > > I will typically add a "rows" parameter to the ping handler with a value > of 1, so Solr will not be retrieving a large amount of data. If you are > running Solr in cloud mode, you should experiment with setting the > distrib parameter to false, which will hopefully limit the query to the > receiving node only. > > Erick has already mentioned GC pauses as a potential problem. With a > 10-15 second response time, I think that has high potential to be the > underlying cause. > > The response you included at the beginning of the thread indicates there > are 1.3 billion documents, which is going to require a fair amount of > heap memory. If seeing such long ping times with a *:* query is > something that happens frequently, your heap may be too small, which > will cause frequent full garbage collections. > > The very low autoSoftCommit time can contribute to system load. I think > it's very likely, especially with such a large index, that in many cases > those automatic commits are taking far longer than 5 seconds to > complete. If that's the case, you're not achieving a 5 second > visibility interval and you are putting a lot of load on Solr, so I > would consider increasing it. > > Thanks, > Shawn > -- Best Regards, Dinesh Naik
Re: Solr 7.6.0: PingRequestHandler - Changing the default query (*:*)
Hi Nikolas,
The restart of node is not helping , the node keeps trying to recover and
always fails:
here is the log :
2019-07-31 06:10:08.049 INFO
(coreZkRegister-1-thread-1-processing-n:replica_host:8983_solr
x:parts_shard30_replica_n2697 c:parts s:shard30 r:core_node2698)
x:parts_shard30_replica_n2697 o.a.s.c.ZkController Core needs to
recover:parts_shard30_replica_n2697
2019-07-31 06:10:08.050 INFO
(updateExecutor-3-thread-1-processing-n:replica_host:8983_solr
x:parts_shard30_replica_n2697 c:parts s:shard30 r:core_node2698)
x:parts_shard30_replica_n2697 o.a.s.u.DefaultSolrCoreState Running recovery
2019-07-31 06:10:08.056 INFO
(recoveryExecutor-4-thread-1-processing-n:replica_host:8983_solr
x:parts_shard30_replica_n2697 c:parts s:shard30 r:core_node2698)
x:parts_shard30_replica_n2697 o.a.s.c.RecoveryStrategy Starting recovery
process. recoveringAfterStartup=true
2019-07-31 06:10:08.261 INFO
(recoveryExecutor-4-thread-1-processing-n:replica_host:8983_solr
x:parts_shard30_replica_n2697 c:parts s:shard30 r:core_node2698)
x:parts_shard30_replica_n2697 o.a.s.c.RecoveryStrategy startupVersions
size=49956 range=[1640550593276674048 to 1640542396328443904]
2019-07-31 06:10:08.328 INFO (qtp689401025-58) o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/key params={omitHeader=true=json}
status=0 QTime=0
2019-07-31 06:10:09.276 INFO
(recoveryExecutor-4-thread-1-processing-n:replica_host:8983_solr
x:parts_shard30_replica_n2697 c:parts s:shard30 r:core_node2698)
x:parts_shard30_replica_n2697 o.a.s.c.RecoveryStrategy Failed to connect
leader http://hostname:8983/solr on recovery, try again
The ping request query is being called from solr itself and not via some
script,so there is no way to stop it .
code where the time is hardcoded to 1 sec:
try (HttpSolrClient httpSolrClient = new
HttpSolrClient.Builder(leaderReplica.getCoreUrl())
.withSocketTimeout(1000)
.withConnectionTimeout(1000)
.withHttpClient(cc.getUpdateShardHandler().getRecoveryOnlyHttpClient())
.build()) {
SolrPingResponse resp = httpSolrClient.ping();
return leaderReplica;
} catch (IOException e) {
log.info("Failed to connect leader {} on recovery, try again",
leaderReplica.getBaseUrl());
Thread.sleep(500);
} catch (Exception e) {
if (e.getCause() instanceof IOException) {
log.info("Failed to connect leader {} on recovery, try again",
leaderReplica.getBaseUrl());
Thread.sleep(500);
} else {
return leaderReplica;
}
}
On Mon, Aug 5, 2019 at 1:19 PM Nicolas Franck
wrote:
> If the ping request handler is taking too long,
> and the server is not recovering automatically,
> there is not much you can do automatically on that server.
> You have to intervene manually, and restart Solr on that node.
>
> First of all: the ping is just an internal check. If it takes too long
> to respond, the requester (i.e. the script calling it), should stop
> the request, and mark that node as problematic. If there are
> for example memory problems every subsequent request will only enhance
> the problem, and Solr cannot recover from that.
>
> > On 5 Aug 2019, at 06:15, dinesh naik wrote:
> >
> > Thanks john,Erick and Furknan.
> >
> > I have already defined the ping request handler in solrconfig.xml as
> below:
> > > name="invariants"> /select _root_:abc
> >
> > My question is regarding the custom query being used. Here i am querying
> > for field _root_ which is available in all of my cluster and defined as a
> > string field. The result for _root_:abc might not get me any match as
> > well(i am ok with not finding any matches, the query should not be taking
> > 10-15 seconds for getting the response).
> >
> > If the response comes within 1 second , then the core recovery issue is
> > solved, hence need your suggestion if using _root_ field in custom query
> is
> > fine?
> >
> >
> > On Mon, Aug 5, 2019 at 2:49 AM Furkan KAMACI
> wrote:
> >
> >> Hi,
> >>
> >> You can change invariants i.e. *qt* and *q* of a *PingRequestHandler*:
> >>
> >>
> >>
> >> /search
> >> some test query
> >>
> >>
> >>
> >> Check documentation fore more info:
> >>
> >>
> https://lucene.apache.org/solr/7_6_0//solr-core/org/apache/solr/handler/PingRequestHandler.html
> >>
> >> Kind Regards,
> >> Furkan KAMACI
> >>
> >> On Sat, Aug 3, 2019 at 4:17 PM Erick Erickson
> >> wrote:
> >>
> >>> You can also (I think) explicitly define the ping request handler in
> >>> solrconfig.xml to do something else.
> >&g
Re: Solr 7.6.0: PingRequestHandler - Changing the default query (*:*)
Thanks john,Erick and Furknan.
I have already defined the ping request handler in solrconfig.xml as below:
/select _root_:abc
My question is regarding the custom query being used. Here i am querying
for field _root_ which is available in all of my cluster and defined as a
string field. The result for _root_:abc might not get me any match as
well(i am ok with not finding any matches, the query should not be taking
10-15 seconds for getting the response).
If the response comes within 1 second , then the core recovery issue is
solved, hence need your suggestion if using _root_ field in custom query is
fine?
On Mon, Aug 5, 2019 at 2:49 AM Furkan KAMACI wrote:
> Hi,
>
> You can change invariants i.e. *qt* and *q* of a *PingRequestHandler*:
>
>
>
> /search
> some test query
>
>
>
> Check documentation fore more info:
>
> https://lucene.apache.org/solr/7_6_0//solr-core/org/apache/solr/handler/PingRequestHandler.html
>
> Kind Regards,
> Furkan KAMACI
>
> On Sat, Aug 3, 2019 at 4:17 PM Erick Erickson
> wrote:
>
> > You can also (I think) explicitly define the ping request handler in
> > solrconfig.xml to do something else.
> >
> > > On Aug 2, 2019, at 9:50 AM, Jörn Franke wrote:
> > >
> > > Not sure if this is possible, but why not create a query handler in
> Solr
> > with any custom query and you use that as ping replacement ?
> > >
> > >> Am 02.08.2019 um 15:48 schrieb dinesh naik >:
> > >>
> > >> Hi all,
> > >> I have few clusters with huge data set and whenever a node goes down
> its
> > >> not able to recover due to below reasons:
> > >>
> > >> 1. ping request handler is taking more than 10-15 seconds to respond.
> > The
> > >> ping requesthandler however, expects it will return in less than 1
> > second
> > >> and fails a requestrecovery if it is not responded to in this time.
> > >> Therefore recoveries never would start.
> > >>
> > >> 2. soft commit is very low ie. 5 sec. This is a business requirement
> so
> > >> not much can be done here.
> > >>
> > >> As the standard/default admin/ping request handler is using *:*
> queries
> > ,
> > >> the response time is much higher, and i am looking for an option to
> > change
> > >> the same so that the ping handler returns the results within few
> > >> miliseconds.
> > >>
> > >> here is an example for standard query time:
> > >>
> > >> snip---
> > >> curl "
> > >>
> >
> http://hostname:8983/solr/parts/select?indent=on=*:*=0=json=false=timing
> > >> "
> > >> {
> > >> "responseHeader":{
> > >> "zkConnected":true,
> > >> "status":0,
> > >> "QTime":16620,
> > >> "params":{
> > >> "q":"*:*",
> > >> "distrib":"false",
> > >> "debug":"timing",
> > >> "indent":"on",
> > >> "rows":"0",
> > >> "wt":"json"}},
> > >> "response":{"numFound":1329638799,"start":0,"docs":[]
> > >> },
> > >> "debug":{
> > >> "timing":{
> > >> "time":16620.0,
> > >> "prepare":{
> > >> "time":0.0,
> > >> "query":{
> > >> "time":0.0},
> > >> "facet":{
> > >> "time":0.0},
> > >> "facet_module":{
> > >> "time":0.0},
> > >> "mlt":{
> > >> "time":0.0},
> > >> "highlight":{
> > >> "time":0.0},
> > >> "stats":{
> > >> "time":0.0},
> > >> "expand":{
> > >> "time":0.0},
> > >> "terms":{
> > >> "time":0.0},
> > >> "block-expensive-queries":{
> > >> "time":0.0},
> > >> "slow-query-logger":{
> > >> "time":0.0},
> > >> "debug":{
> > >> "tim
Solr 7.6.0: PingRequestHandler - Changing the default query (*:*)
Hi all,
I have few clusters with huge data set and whenever a node goes down its
not able to recover due to below reasons:
1. ping request handler is taking more than 10-15 seconds to respond. The
ping requesthandler however, expects it will return in less than 1 second
and fails a requestrecovery if it is not responded to in this time.
Therefore recoveries never would start.
2. soft commit is very low ie. 5 sec. This is a business requirement so
not much can be done here.
As the standard/default admin/ping request handler is using *:* queries ,
the response time is much higher, and i am looking for an option to change
the same so that the ping handler returns the results within few
miliseconds.
here is an example for standard query time:
snip---
curl "
http://hostname:8983/solr/parts/select?indent=on=*:*=0=json=false=timing
"
{
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":16620,
"params":{
"q":"*:*",
"distrib":"false",
"debug":"timing",
"indent":"on",
"rows":"0",
"wt":"json"}},
"response":{"numFound":1329638799,"start":0,"docs":[]
},
"debug":{
"timing":{
"time":16620.0,
"prepare":{
"time":0.0,
"query":{
"time":0.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"terms":{
"time":0.0},
"block-expensive-queries":{
"time":0.0},
"slow-query-logger":{
"time":0.0},
"debug":{
"time":0.0}},
"process":{
"time":16619.0,
"query":{
"time":16619.0},
"facet":{
"time":0.0},
"facet_module":{
"time":0.0},
"mlt":{
"time":0.0},
"highlight":{
"time":0.0},
"stats":{
"time":0.0},
"expand":{
"time":0.0},
"terms":{
"time":0.0},
"block-expensive-queries":{
"time":0.0},
"slow-query-logger":{
"time":0.0},
"debug":{
"time":0.0}
snap
can we use query: _root_:abc in the ping request handler ? Tried this query
and its returning the results within few miliseconds and also the nodes are
able to recover without any issue.
we want to use _root_ field for querying as this field is available in all
our clusters with below definition:
Could you please let me know if using _root_ for querying in
pingRequestHandler will cause any problem?
/select _root_:abc
--
Best Regards,
Dinesh Naik
SSL in Solr 7.6.0
Hi all, I am working on securing Solr and Client communication by implementing SSL for a multi node cluster(100+). The client are connecting to Solr via CloudSolrClient through Zoo keeper and i am looking for best way to create the certificate for making the connection secured. for a cluster of size 100 and plus, it becomes hard to have all the hostnames/ip's while generating the certificate and wildcard option is ruled out due tp security concerns, so what is the best way to handle this scenario. Also if you give some light on usage of SOLR_SSL_CHECK_PEER_NAME param and if that will help in any way ? -- Best Regards, Dinesh Naik
Re: how to achieve mulitple wild card searches in solr 5.2.1
Thanks Erick, I tried making it to String, but i need to compress the part first and then look for wild card search? With string i can not do that. How do i achieve this? On Wed, Jan 4, 2017 at 2:52 AM, Erick Erickson <erickerick...@gmail.com> wrote: > My guess is that you're searching on a _tokenized_ field and that > you'd get the results you expect on a string field.. > > Add =query to the URL and you'll see what the parsed query is > and that'll give you a very good idea of what's acaully happening. > > Best, > Erick > > On Tue, Jan 3, 2017 at 7:16 AM, dinesh naik <dineshkumarn...@gmail.com> > wrote: > > Hi all, > > How can we achieve multiple wild card searches in solr? > > > > For example: I am searching for AB TEST1.EC*TEST2* > > But I get also results for AB TEST1.EC*TEST3*, AB TEST1.EC*TEST4*,? > instead > > of AB TEST1.EC*TEST2* > > > > It seems only the first * is being considered, second * is not considered > > for wildcard match. > > -- > > Best Regards, > > Dinesh Naik > -- Best Regards, Dinesh Naik
how to achieve mulitple wild card searches in solr 5.2.1
Hi all, How can we achieve multiple wild card searches in solr? For example: I am searching for AB TEST1.EC*TEST2* But I get also results for AB TEST1.EC*TEST3*, AB TEST1.EC*TEST4*,? instead of AB TEST1.EC*TEST2* It seems only the first * is being considered, second * is not considered for wildcard match. -- Best Regards, Dinesh Naik
Re: solr-5.2.1: All folders in solr box(Linux) are sitting in RAM
Hi Eric, Thanks a lot. Got the point. On Sep 21, 2016 10:18 PM, "Erick Erickson" <erickerick...@gmail.com> wrote: > Why do you want to avoid this? Having the index in RAM (either the JVM or > OS) > is essential to fast querying. Perhaps you're being mislead by the > MMapDirectory's > consumption of the OS memory? See Uwe's excellent article here: > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Best, > Erick > > On Wed, Sep 21, 2016 at 1:45 AM, dinesh naik <dineshkumarn...@gmail.com> > wrote: > > Hi all, > > > > i have a linux box with 48GB RAM . > > > > In this box i have solr and jdk installed. i have few other folders as > > well. > > > > [solruser@server1 ~]$ du -sh * > > 4.0Kclusterstate.json > > 1.5MConf > > 15G jdk1.8.0_25 > > 151Mjdk_old > > 262Mjvm_1.7 > > 538Mscripts > > 11G solrhome > > > > My actual index size is 9GB (inside solr installation directory solrhome) > > .In solr admin UI the physical memory shows 32GB. > > > > It seems all the folders are sitting in RAM . Kindly suggest how can i > > avoid this? > > > > -- > > Best Regards, > > Dinesh Naik >
solr-5.2.1: All folders in solr box(Linux) are sitting in RAM
Hi all, i have a linux box with 48GB RAM . In this box i have solr and jdk installed. i have few other folders as well. [solruser@server1 ~]$ du -sh * 4.0Kclusterstate.json 1.5MConf 15G jdk1.8.0_25 151Mjdk_old 262Mjvm_1.7 538Mscripts 11G solrhome My actual index size is 9GB (inside solr installation directory solrhome) .In solr admin UI the physical memory shows 32GB. It seems all the folders are sitting in RAM . Kindly suggest how can i avoid this? -- Best Regards, Dinesh Naik
Re: Different boost values for multiple parsers in Solr 5.2.1
Hi Upayavira,
We have an issue here.
The boosting work as expected when we run the query from Admin console:
where we pass q and bq param as below.
q=(((_query_:"{!synonym_edismax qf='itemname OR itemnumber OR itemdesc'
v='HTC' bq='' mm=100 synonyms=true synonyms.constructPhrases=true
synonyms.ignoreQueryOperators=true}") OR (itemname:"HTC" OR
itemnamecomp:HTC* OR itemnumber:"HTC" OR itemnumbercomp:HTC* OR
itemdesc:"HTC"~500)) AND (warehouse:Ind02 OR warehouse:Ind03 OR
warehouse:Ind04 ))
bq=warehouse:Ind02^1000
This works absolutely fine when tried from Admin cosnole.
But when we use SolrJ API , we are not geting the expected boost value
being returned in score field.
We are using SolrQuery class for adding the bq parameter.
queryEngine.set("bq", boostQuery);
where boostQuery is : warehouse:Ind02^1000
How can we handle this. Is this becuase of bq='' being used for
synonym_edismax parser?
On Tue, Sep 8, 2015 at 5:49 PM, dinesh naik <dineshkumarn...@gmail.com>
wrote:
> Thanks Alot Upayavira. It worked as expected.
>
>
> On Tue, Sep 8, 2015 at 2:09 PM, Upayavira <u...@odoko.co.uk> wrote:
>
>> you can add bq= inside your {!synonym_edismax} section, if you wish and
>> it will apply to that query parser only.
>>
>> Upayavira
>>
>> On Mon, Sep 7, 2015, at 03:05 PM, dinesh naik wrote:
>> > Please find below the detail:
>> >
>> > My main query is like this:
>> >
>> > q=(((_query_:"{!synonym_edismax qf='itemname OR itemnumber OR itemdesc'
>> > v='HTC' mm=100 synonyms=true synonyms.constructPhrases=true
>> > synonyms.ignoreQueryOperators=true}") OR (itemname:"HTC" OR
>> > itemnamecomp:HTC* OR itemnumber:"HTC" OR itemnumbercomp:HTC* OR
>> > itemdesc:"HTC"~500)) AND (warehouse:Ind02 OR warehouse:Ind03 OR
>> > warehouse:Ind04 ))
>> >
>> > Giving Boost of 1000 for warehouse Ind02
>> > using below parameter:
>> >
>> > bq=warehouse:Ind02^1000
>> >
>> >
>> > Here i am expecting a boost of 1004 but , somehow 1000 is added extra
>> may
>> > be because of my additional parser. How can i avoid this?
>> >
>> >
>> > Debug information for the boost :
>> >
>> >
>> > 2004.0 = sum of:
>> > 1004.0 = sum of:
>> > 1003.0 = sum of:
>> > 1001.0 = sum of:
>> > 1.0 = max of:
>> > 1.0 = weight(itemname:HTC in 235500) [CustomSimilarity],
>> result
>> > of:
>> > 1.0 = fieldWeight in 235500, product of:
>> > 1.0 = tf(freq=1.0), with freq of:
>> > 1.0 = termFreq=1.0
>> > 1.0 = idf(docFreq=26, maxDocs=1738053)
>> > 1.0 = fieldNorm(doc=235500)
>> > 1000.0 = weight(warehouse:e02^1000.0 in 235500)
>> > [CustomSimilarity],
>> > result of:
>> > 1000.0 = score(doc=235500,freq=1.0), product of:
>> > 1000.0 = queryWeight, product of:
>> > 1000.0 = boost
>> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
>> > 1.0 = queryNorm
>> > 1.0 = fieldWeight in 235500, product of:
>> > 1.0 = tf(freq=1.0), with freq of:
>> > 1.0 = termFreq=1.0
>> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
>> > 1.0 = fieldNorm(doc=235500)
>> > 2.0 = sum of:
>> > 1.0 = weight(itemname:HTC in 235500) [CustomSimilarity], result
>> > of:
>> > 1.0 = fieldWeight in 235500, product of:
>> > 1.0 = tf(freq=1.0), with freq of:
>> > 1.0 = termFreq=1.0
>> > 1.0 = idf(docFreq=26, maxDocs=1738053)
>> > 1.0 = fieldNorm(doc=235500)
>> > 1.0 = itemnamecomp:HTC*, product of:
>> > 1.0 = boost
>> > 1.0 = queryNorm
>> > 1.0 = sum of:
>> > 1.0 = weight(warehouse:e02 in 235500) [CustomSimilarity], result
>> > of:
>> > 1.0 = fieldWeight in 235500, product of:
>> > 1.0 = tf(freq=1.0), with freq of:
>> > 1.0 = termFreq=1.0
>> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
>> > 1.0 = fieldNorm(doc=235500)
>> > 1000.0 = weight(warehouse:e02^1000.0 in 235500) [CustomSimilarity],
>> > result of:
>> > 1000.0 = score(doc=235500,freq=1.0), product of:
>> > 1000.0
Re: Different boost values for multiple parsers in Solr 5.2.1
Thanks Alot Upayavira. It worked as expected.
On Tue, Sep 8, 2015 at 2:09 PM, Upayavira <u...@odoko.co.uk> wrote:
> you can add bq= inside your {!synonym_edismax} section, if you wish and
> it will apply to that query parser only.
>
> Upayavira
>
> On Mon, Sep 7, 2015, at 03:05 PM, dinesh naik wrote:
> > Please find below the detail:
> >
> > My main query is like this:
> >
> > q=(((_query_:"{!synonym_edismax qf='itemname OR itemnumber OR itemdesc'
> > v='HTC' mm=100 synonyms=true synonyms.constructPhrases=true
> > synonyms.ignoreQueryOperators=true}") OR (itemname:"HTC" OR
> > itemnamecomp:HTC* OR itemnumber:"HTC" OR itemnumbercomp:HTC* OR
> > itemdesc:"HTC"~500)) AND (warehouse:Ind02 OR warehouse:Ind03 OR
> > warehouse:Ind04 ))
> >
> > Giving Boost of 1000 for warehouse Ind02
> > using below parameter:
> >
> > bq=warehouse:Ind02^1000
> >
> >
> > Here i am expecting a boost of 1004 but , somehow 1000 is added extra may
> > be because of my additional parser. How can i avoid this?
> >
> >
> > Debug information for the boost :
> >
> >
> > 2004.0 = sum of:
> > 1004.0 = sum of:
> > 1003.0 = sum of:
> > 1001.0 = sum of:
> > 1.0 = max of:
> > 1.0 = weight(itemname:HTC in 235500) [CustomSimilarity], result
> > of:
> > 1.0 = fieldWeight in 235500, product of:
> > 1.0 = tf(freq=1.0), with freq of:
> > 1.0 = termFreq=1.0
> > 1.0 = idf(docFreq=26, maxDocs=1738053)
> > 1.0 = fieldNorm(doc=235500)
> > 1000.0 = weight(warehouse:e02^1000.0 in 235500)
> > [CustomSimilarity],
> > result of:
> > 1000.0 = score(doc=235500,freq=1.0), product of:
> > 1000.0 = queryWeight, product of:
> > 1000.0 = boost
> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
> > 1.0 = queryNorm
> > 1.0 = fieldWeight in 235500, product of:
> > 1.0 = tf(freq=1.0), with freq of:
> > 1.0 = termFreq=1.0
> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
> > 1.0 = fieldNorm(doc=235500)
> > 2.0 = sum of:
> > 1.0 = weight(itemname:HTC in 235500) [CustomSimilarity], result
> > of:
> > 1.0 = fieldWeight in 235500, product of:
> > 1.0 = tf(freq=1.0), with freq of:
> > 1.0 = termFreq=1.0
> > 1.0 = idf(docFreq=26, maxDocs=1738053)
> > 1.0 = fieldNorm(doc=235500)
> > 1.0 = itemnamecomp:HTC*, product of:
> > 1.0 = boost
> > 1.0 = queryNorm
> > 1.0 = sum of:
> > 1.0 = weight(warehouse:e02 in 235500) [CustomSimilarity], result
> > of:
> > 1.0 = fieldWeight in 235500, product of:
> > 1.0 = tf(freq=1.0), with freq of:
> > 1.0 = termFreq=1.0
> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
> > 1.0 = fieldNorm(doc=235500)
> > 1000.0 = weight(warehouse:e02^1000.0 in 235500) [CustomSimilarity],
> > result of:
> > 1000.0 = score(doc=235500,freq=1.0), product of:
> > 1000.0 = queryWeight, product of:
> > 1000.0 = boost
> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
> > 1.0 = queryNorm
> > 1.0 = fieldWeight in 235500, product of:
> > 1.0 = tf(freq=1.0), with freq of:
> > 1.0 = termFreq=1.0
> > 1.0 = idf(docFreq=416190, maxDocs=1738053)
> > 1.0 = fieldNorm(doc=235500)
> >
> >
> > On Mon, Sep 7, 2015 at 7:21 PM, dinesh naik <dineshkumarn...@gmail.com>
> > wrote:
> > Hi all,
> >
> > Is there a way to apply different boost , using bq parameter for
> > different
> > parser.
> >
> > for example if i am using a synonym parser and edismax parser in a single
> > query, my bq param value is getting applied for both the parser making
> > the
> > boost value double.
> >
> > --
> > Best Regards,
> > Dinesh Naik
> >
> >
> >
> >
> >
> > On Mon, Sep 7, 2015 at 7:21 PM, dinesh naik <dineshkumarn...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > Is there a way to apply different boost , using bq parameter for
> different
> > > parser.
> > >
> > > for example if i am using a synonym parser and edismax parser in a
> single
> > > query, my bq param value is getting applied for both the parser making
> the
> > > boost value double.
> > >
> > > --
> > > Best Regards,
> > > Dinesh Naik
> > >
> >
> >
> >
> > --
> > Best Regards,
> > Dinesh Naik
>
--
Best Regards,
Dinesh Naik
Different boost values for multiple parsers in Solr 5.2.1
Hi all, Is there a way to apply different boost , using bq parameter for different parser. for example if i am using a synonym parser and edismax parser in a single query, my bq param value is getting applied for both the parser making the boost value double. -- Best Regards, Dinesh Naik
Re: Different boost values for multiple parsers in Solr 5.2.1
Please find below the detail:
My main query is like this:
q=(((_query_:"{!synonym_edismax qf='itemname OR itemnumber OR itemdesc'
v='HTC' mm=100 synonyms=true synonyms.constructPhrases=true
synonyms.ignoreQueryOperators=true}") OR (itemname:"HTC" OR
itemnamecomp:HTC* OR itemnumber:"HTC" OR itemnumbercomp:HTC* OR
itemdesc:"HTC"~500)) AND (warehouse:Ind02 OR warehouse:Ind03 OR
warehouse:Ind04 ))
Giving Boost of 1000 for warehouse Ind02
using below parameter:
bq=warehouse:Ind02^1000
Here i am expecting a boost of 1004 but , somehow 1000 is added extra may
be because of my additional parser. How can i avoid this?
Debug information for the boost :
2004.0 = sum of:
1004.0 = sum of:
1003.0 = sum of:
1001.0 = sum of:
1.0 = max of:
1.0 = weight(itemname:HTC in 235500) [CustomSimilarity], result
of:
1.0 = fieldWeight in 235500, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
1.0 = idf(docFreq=26, maxDocs=1738053)
1.0 = fieldNorm(doc=235500)
1000.0 = weight(warehouse:e02^1000.0 in 235500) [CustomSimilarity],
result of:
1000.0 = score(doc=235500,freq=1.0), product of:
1000.0 = queryWeight, product of:
1000.0 = boost
1.0 = idf(docFreq=416190, maxDocs=1738053)
1.0 = queryNorm
1.0 = fieldWeight in 235500, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
1.0 = idf(docFreq=416190, maxDocs=1738053)
1.0 = fieldNorm(doc=235500)
2.0 = sum of:
1.0 = weight(itemname:HTC in 235500) [CustomSimilarity], result of:
1.0 = fieldWeight in 235500, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
1.0 = idf(docFreq=26, maxDocs=1738053)
1.0 = fieldNorm(doc=235500)
1.0 = itemnamecomp:HTC*, product of:
1.0 = boost
1.0 = queryNorm
1.0 = sum of:
1.0 = weight(warehouse:e02 in 235500) [CustomSimilarity], result of:
1.0 = fieldWeight in 235500, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
1.0 = idf(docFreq=416190, maxDocs=1738053)
1.0 = fieldNorm(doc=235500)
1000.0 = weight(warehouse:e02^1000.0 in 235500) [CustomSimilarity],
result of:
1000.0 = score(doc=235500,freq=1.0), product of:
1000.0 = queryWeight, product of:
1000.0 = boost
1.0 = idf(docFreq=416190, maxDocs=1738053)
1.0 = queryNorm
1.0 = fieldWeight in 235500, product of:
1.0 = tf(freq=1.0), with freq of:
1.0 = termFreq=1.0
1.0 = idf(docFreq=416190, maxDocs=1738053)
1.0 = fieldNorm(doc=235500)
On Mon, Sep 7, 2015 at 7:21 PM, dinesh naik <dineshkumarn...@gmail.com>
wrote:
Hi all,
Is there a way to apply different boost , using bq parameter for different
parser.
for example if i am using a synonym parser and edismax parser in a single
query, my bq param value is getting applied for both the parser making the
boost value double.
--
Best Regards,
Dinesh Naik
On Mon, Sep 7, 2015 at 7:21 PM, dinesh naik <dineshkumarn...@gmail.com>
wrote:
> Hi all,
>
> Is there a way to apply different boost , using bq parameter for different
> parser.
>
> for example if i am using a synonym parser and edismax parser in a single
> query, my bq param value is getting applied for both the parser making the
> boost value double.
>
> --
> Best Regards,
> Dinesh Naik
>
--
Best Regards,
Dinesh Naik
Solr 5.2 index time field boost not working as expected
Hi all, We need to boost a field in a document if field matches certain criteria. For example: if title contains Secrete , then we want to boost the field to 100 . For this we have the below code in solrj api while indexing the document: CollectionSolrInputDocument docs = new ArrayListSolrInputDocument(); SolrInputDocument doc = new SolrInputDocument(); doc.addField(title, Secrete , 100.0f); // Field Boost doc.addField(id, 11); doc.addField(modelnumber, AK10005); doc.addField(name, XX5); docs.add(doc); Also , we made omitNorms=false for this field in schema.xml field name=title type=string indexed=true stored=true required=true omitNorms=false / But still we do not see this document coming at the top. Is there any other setting which has to be done for index time boosting? Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik
Re: Restore index API does not work in solr 5.1.0 ?
Hi all, How can we restore index in Solr 5.1.0 ? Best Regards, Dinesh Naik On Thu, Jul 9, 2015 at 6:54 PM, dinesh naik dineshkumarn...@gmail.com wrote: Hi all, How can we restore the index in Solr 5.1.0 ? We did following: 1:- Started Solr Cloud from: bin/solr start -e cloud -noprompt 2:- posted some documents to solr from examples folder using : java -Dc=gettingstarted -jar post.jar *.xml 3:- Backed up the Index using: http://localhost:8983/solr/gettingstarted/replication?command=backup 4:- Deleted 1 document using: http://localhost:8983/solr/gettingstarted/update?stream.body=deletequeryid:IW-02/query/deletecommit=true 5:- restored the index using: http://localhost:8983/solr/gettingstarted/replication?command=restore The Restore works fine with same steps for 5.2 versions but not 5.1 Is there any other way to restore index in Solr 5.1.0? -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik
Restore index API does not work in solr 5.1.0 ?
Hi all, How can we restore the index in Solr 5.1.0 ? We did following: 1:- Started Solr Cloud from: bin/solr start -e cloud -noprompt 2:- posted some documents to solr from examples folder using : java -Dc=gettingstarted -jar post.jar *.xml 3:- Backed up the Index using: http://localhost:8983/solr/gettingstarted/replication?command=backup 4:- Deleted 1 document using: http://localhost:8983/solr/gettingstarted/update?stream.body=deletequeryid:IW-02/query/deletecommit=true 5:- restored the index using: http://localhost:8983/solr/gettingstarted/replication?command=restore The Restore works fine with same steps for 5.2 versions but not 5.1 Is there any other way to restore index in Solr 5.1.0? -- Best Regards, Dinesh Naik
Synonym with Proximity search in solr 5.1.0
Hi, We have a synonym file with below content: 1 2 cell phone ,nokia mobile And we have 3 documents: doc1: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 doc field name=id1001/field field name=nameDoc 1/field field name=textI like nokia mobile /field /doc doc2: doc field name=id1002/field field name=nameDoc 2/field field name=textI cant leave without cell phone /field /doc doc3: doc field name=id1003/field field name=nameDoc 3/field field name=textI work with Nokia inc/field /doc when i search for cell phone, I should get doc1 and doc2 returned but not doc3. The search syntax is : text: cell phone~500 How could i achieve this? Best Regards, Dinesh Naik
Re: Synonym with Proximity search in solr 5.1.0
Hi Alessandro, I have gone through the above suggested links, but i am not able to achieve the above expected result. The issue here is , my searched text is a part of field 'text' . field name=textI like nokia mobile /field searched text: nokia mobile~500. Best Regards, Dinesh Naik On Wed, Jul 8, 2015 at 8:36 PM, Alessandro Benedetti benedetti.ale...@gmail.com wrote: Showing your debug query would clarify the situation, but I assume you got into a classic multi-word synonym problem[1] . Hope the documents I pointed out are good for you. Cheers [1] http://nolanlawson.com/2012/10/31/better-synonym-handling-in-solr/ [2] http://lucidworks.com/blog/solution-for-multi-term-synonyms-in-lucenesolr-using-the-auto-phrasing-tokenfilter/ 2015-07-08 15:47 GMT+01:00 dinesh naik dineshkumarn...@gmail.com: Hi, We have a synonym file with below content: 1 2 cell phone ,nokia mobile And we have 3 documents: doc1: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 doc field name=id1001/field field name=nameDoc 1/field field name=textI like nokia mobile /field /doc doc2: doc field name=id1002/field field name=nameDoc 2/field field name=textI cant leave without cell phone /field /doc doc3: doc field name=id1003/field field name=nameDoc 3/field field name=textI work with Nokia inc/field /doc when i search for cell phone, I should get doc1 and doc2 returned but not doc3. The search syntax is : text: cell phone~500 How could i achieve this? Best Regards, Dinesh Naik -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry? William Blake - Songs of Experience -1794 England -- Best Regards, Dinesh Naik
Location of config files in Zoo Keeper
Hi all, For solr version 5.1.0, Where does Zoo keeper keep all the config files ?How do we access them ? From Admin console , Cloud--Tree--config , we are able to see them but where does Zoo Keeper store them(location)? -- Best Regards, Dinesh Naik
Re: Reading indexed data from solr 5.1.0 using admin/luke?
Thanks Eric and Upayavira for your inputs. Is there a way i can associate this to a unique id of document, either using schema browser or TermsComponent? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 2:55 AM, Upayavira u...@odoko.co.uk wrote: Use the schema browser on the admin UI, and click the load term info button. It'll show you the terms in your index. You can also use the analysis tab which will show you how it would tokenise stuff for a specific field. Upayavira On Mon, Jun 29, 2015, at 06:53 PM, Dinesh Naik wrote: Hi Eric, By compressed value I meant value of a field after removing special characters . In my example its -. Compressed form of red-apple is redapple . I wanted to know if we can see the analyzed version of fields . For example if I use ngram on a field , how do I see the analyzed values in index ? -Original Message- From: Erick Erickson erickerick...@gmail.com Sent: 29-06-2015 18:12 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data from solr 5.1.0 using admin/luke? Not quite sure what you mean by compressed values. admin/luke doesn't show the results of the compression of the stored values, there's no way I know of to do that. Best, Erick On Mon, Jun 29, 2015 at 8:20 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi all, Is there a way to read the indexed data for field on which the analysis/processing has been done ? I know using admin GUI we can see field wise analysis But how can i get hold on the complete document using admin/luke? or any other way? For example, if i have 2 fields called name and compressedname. name has values like apple, green-apple,red-apple compressedname has values like apple,greenapple,redapple Even though i make both these field indexed=true and stored=true I am not able to see the compressed values using admin/luke?id=mydocid in response i see something like this- lst name=name str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst lst name=compressedname str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik
Re: Reading indexed data from solr 5.1.0 using admin/luke?
Hi Erick, This is mainly for debugging purpose. If i have 20M records and few fields in some of the documents are not indexed as expected or something went wrong during indexing then how do we pin point the exact issue and fix the problem? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 5:56 PM, Erick Erickson erickerick...@gmail.com wrote: In short, not unless you want to get into low-level Lucene coding. Inverted indexes are, well, inverted so their very structure makes this difficult. It looks like this: But I'm not convinced yet that this isn't an XY problem. What is the high-level problem you're trying to solve here? Maybe there's another way to go about it. Best, Erick On Tue, Jun 30, 2015 at 3:32 AM, dinesh naik dineshkumarn...@gmail.com wrote: Thanks Eric and Upayavira for your inputs. Is there a way i can associate this to a unique id of document, either using schema browser or TermsComponent? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 2:55 AM, Upayavira u...@odoko.co.uk wrote: Use the schema browser on the admin UI, and click the load term info button. It'll show you the terms in your index. You can also use the analysis tab which will show you how it would tokenise stuff for a specific field. Upayavira On Mon, Jun 29, 2015, at 06:53 PM, Dinesh Naik wrote: Hi Eric, By compressed value I meant value of a field after removing special characters . In my example its -. Compressed form of red-apple is redapple . I wanted to know if we can see the analyzed version of fields . For example if I use ngram on a field , how do I see the analyzed values in index ? -Original Message- From: Erick Erickson erickerick...@gmail.com Sent: 29-06-2015 18:12 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data from solr 5.1.0 using admin/luke? Not quite sure what you mean by compressed values. admin/luke doesn't show the results of the compression of the stored values, there's no way I know of to do that. Best, Erick On Mon, Jun 29, 2015 at 8:20 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi all, Is there a way to read the indexed data for field on which the analysis/processing has been done ? I know using admin GUI we can see field wise analysis But how can i get hold on the complete document using admin/luke? or any other way? For example, if i have 2 fields called name and compressedname. name has values like apple, green-apple,red-apple compressedname has values like apple,greenapple,redapple Even though i make both these field indexed=true and stored=true I am not able to see the compressed values using admin/luke?id=mydocid in response i see something like this- lst name=name str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst lst name=compressedname str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik
Re: Reading indexed data from solr 5.1.0 using admin/luke?
Hi Erick, I agree with you. But i was checking if we could get hold on the whole document (to see all analyzed field values) . There might be chances that field value is common for multiple documents . In such cases it will be difficult to backtrack which document has the issue . Because admin/analysis can be used to see for field level analysis only. Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 7:08 PM, Erick Erickson erickerick...@gmail.com wrote: Dinesh: This is what the admin/analysis page is for. It shows you exactly what tokens are produced by what steps in the analysis chain. That would be far better than trying to analyze the indexed terms. Best, Erick On Tue, Jun 30, 2015 at 8:35 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi Erick, This is mainly for debugging purpose. If i have 20M records and few fields in some of the documents are not indexed as expected or something went wrong during indexing then how do we pin point the exact issue and fix the problem? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 5:56 PM, Erick Erickson erickerick...@gmail.com wrote: In short, not unless you want to get into low-level Lucene coding. Inverted indexes are, well, inverted so their very structure makes this difficult. It looks like this: But I'm not convinced yet that this isn't an XY problem. What is the high-level problem you're trying to solve here? Maybe there's another way to go about it. Best, Erick On Tue, Jun 30, 2015 at 3:32 AM, dinesh naik dineshkumarn...@gmail.com wrote: Thanks Eric and Upayavira for your inputs. Is there a way i can associate this to a unique id of document, either using schema browser or TermsComponent? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 2:55 AM, Upayavira u...@odoko.co.uk wrote: Use the schema browser on the admin UI, and click the load term info button. It'll show you the terms in your index. You can also use the analysis tab which will show you how it would tokenise stuff for a specific field. Upayavira On Mon, Jun 29, 2015, at 06:53 PM, Dinesh Naik wrote: Hi Eric, By compressed value I meant value of a field after removing special characters . In my example its -. Compressed form of red-apple is redapple . I wanted to know if we can see the analyzed version of fields . For example if I use ngram on a field , how do I see the analyzed values in index ? -Original Message- From: Erick Erickson erickerick...@gmail.com Sent: 29-06-2015 18:12 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data from solr 5.1.0 using admin/luke? Not quite sure what you mean by compressed values. admin/luke doesn't show the results of the compression of the stored values, there's no way I know of to do that. Best, Erick On Mon, Jun 29, 2015 at 8:20 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi all, Is there a way to read the indexed data for field on which the analysis/processing has been done ? I know using admin GUI we can see field wise analysis But how can i get hold on the complete document using admin/luke? or any other way? For example, if i have 2 fields called name and compressedname. name has values like apple, green-apple,red-apple compressedname has values like apple,greenapple,redapple Even though i make both these field indexed=true and stored=true I am not able to see the compressed values using admin/luke?id=mydocid in response i see something like this- lst name=name str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst lst name=compressedname str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik -- Best Regards, Dinesh Naik
Re: Reading indexed data from solr 5.1.0 using admin/luke?
Hi Alessandro, I am able to check the field wise analyzed results. I was interested in getting the complete document. As Erick mentioned - Reconstructing the doc from the postings lists isactually quite tedious. The Luke program (not request handler) has a function that does this, it's not fast though, more for troubleshooting than trying to do anything in a production environment. I ll try looking into the Luke program if i can get this done. Thanks and Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 7:42 PM, Alessandro Benedetti benedetti.ale...@gmail.com wrote: Do you have the original document available ? Or stored in the field of interest ? Should be quite an easy test to reproduce the Analysis simply using the analysis tool Upaya and Erick suggested. Just use your real document content and you will see how it is exactly analysed. Cheers 2015-06-30 15:03 GMT+01:00 dinesh naik dineshkumarn...@gmail.com: Hi Erick, I agree with you. But i was checking if we could get hold on the whole document (to see all analyzed field values) . There might be chances that field value is common for multiple documents . In such cases it will be difficult to backtrack which document has the issue . Because admin/analysis can be used to see for field level analysis only. Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 7:08 PM, Erick Erickson erickerick...@gmail.com wrote: Dinesh: This is what the admin/analysis page is for. It shows you exactly what tokens are produced by what steps in the analysis chain. That would be far better than trying to analyze the indexed terms. Best, Erick On Tue, Jun 30, 2015 at 8:35 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi Erick, This is mainly for debugging purpose. If i have 20M records and few fields in some of the documents are not indexed as expected or something went wrong during indexing then how do we pin point the exact issue and fix the problem? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 5:56 PM, Erick Erickson erickerick...@gmail.com wrote: In short, not unless you want to get into low-level Lucene coding. Inverted indexes are, well, inverted so their very structure makes this difficult. It looks like this: But I'm not convinced yet that this isn't an XY problem. What is the high-level problem you're trying to solve here? Maybe there's another way to go about it. Best, Erick On Tue, Jun 30, 2015 at 3:32 AM, dinesh naik dineshkumarn...@gmail.com wrote: Thanks Eric and Upayavira for your inputs. Is there a way i can associate this to a unique id of document, either using schema browser or TermsComponent? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 2:55 AM, Upayavira u...@odoko.co.uk wrote: Use the schema browser on the admin UI, and click the load term info button. It'll show you the terms in your index. You can also use the analysis tab which will show you how it would tokenise stuff for a specific field. Upayavira On Mon, Jun 29, 2015, at 06:53 PM, Dinesh Naik wrote: Hi Eric, By compressed value I meant value of a field after removing special characters . In my example its -. Compressed form of red-apple is redapple . I wanted to know if we can see the analyzed version of fields . For example if I use ngram on a field , how do I see the analyzed values in index ? -Original Message- From: Erick Erickson erickerick...@gmail.com Sent: 29-06-2015 18:12 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data from solr 5.1.0 using admin/luke? Not quite sure what you mean by compressed values. admin/luke doesn't show the results of the compression of the stored values, there's no way I know of to do that. Best, Erick On Mon, Jun 29, 2015 at 8:20 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi all, Is there a way to read the indexed data for field on which the analysis/processing has been done ? I know using admin GUI we can see field wise analysis But how can i get hold on the complete document using admin/luke? or any other way? For example, if i have 2 fields called name and compressedname. name has values like apple, green-apple,red-apple compressedname has values like apple,greenapple,redapple Even though i make both these field indexed=true and stored=true I am not able to see the compressed values using admin/luke?id=mydocid in response i see
RE: Reading indexed data from solr 5.1.0 using admin/luke?
Hi Alessandro, Lets say I have 20M documents with 50 fields in each. I have applied text analysis like compression,ngram,synonym expansion on these fields. Checking individually field level analysis can be easily done via admin/analysis . But I need to do 50 times analysis check for these 50 fields . I wanted to know if solr provides a way to see all these analyzed fields at once (for ex. By using unique id ). Best Regards, Dinesh Naik -Original Message- From: Alessandro Benedetti benedetti.ale...@gmail.com Sent: 30-06-2015 21:43 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data from solr 5.1.0 using admin/luke? But what do you mean with the complete document ? Is it not available anymore ? So you have lost your original document and you want to try to reconstruct from the index ? 2015-06-30 16:05 GMT+01:00 dinesh naik dineshkumarn...@gmail.com: Hi Alessandro, I am able to check the field wise analyzed results. I was interested in getting the complete document. As Erick mentioned - Reconstructing the doc from the postings lists isactually quite tedious. The Luke program (not request handler) has a function that does this, it's not fast though, more for troubleshooting than trying to do anything in a production environment. I ll try looking into the Luke program if i can get this done. Thanks and Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 7:42 PM, Alessandro Benedetti benedetti.ale...@gmail.com wrote: Do you have the original document available ? Or stored in the field of interest ? Should be quite an easy test to reproduce the Analysis simply using the analysis tool Upaya and Erick suggested. Just use your real document content and you will see how it is exactly analysed. Cheers 2015-06-30 15:03 GMT+01:00 dinesh naik dineshkumarn...@gmail.com: Hi Erick, I agree with you. But i was checking if we could get hold on the whole document (to see all analyzed field values) . There might be chances that field value is common for multiple documents . In such cases it will be difficult to backtrack which document has the issue . Because admin/analysis can be used to see for field level analysis only. Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 7:08 PM, Erick Erickson erickerick...@gmail.com wrote: Dinesh: This is what the admin/analysis page is for. It shows you exactly what tokens are produced by what steps in the analysis chain. That would be far better than trying to analyze the indexed terms. Best, Erick On Tue, Jun 30, 2015 at 8:35 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi Erick, This is mainly for debugging purpose. If i have 20M records and few fields in some of the documents are not indexed as expected or something went wrong during indexing then how do we pin point the exact issue and fix the problem? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 5:56 PM, Erick Erickson erickerick...@gmail.com wrote: In short, not unless you want to get into low-level Lucene coding. Inverted indexes are, well, inverted so their very structure makes this difficult. It looks like this: But I'm not convinced yet that this isn't an XY problem. What is the high-level problem you're trying to solve here? Maybe there's another way to go about it. Best, Erick On Tue, Jun 30, 2015 at 3:32 AM, dinesh naik dineshkumarn...@gmail.com wrote: Thanks Eric and Upayavira for your inputs. Is there a way i can associate this to a unique id of document, either using schema browser or TermsComponent? Best Regards, Dinesh Naik On Tue, Jun 30, 2015 at 2:55 AM, Upayavira u...@odoko.co.uk wrote: Use the schema browser on the admin UI, and click the load term info button. It'll show you the terms in your index. You can also use the analysis tab which will show you how it would tokenise stuff for a specific field. Upayavira On Mon, Jun 29, 2015, at 06:53 PM, Dinesh Naik wrote: Hi Eric, By compressed value I meant value of a field after removing special characters . In my example its -. Compressed form of red-apple is redapple . I wanted to know if we can see the analyzed version of fields . For example if I use ngram on a field , how do I see the analyzed values in index ? -Original Message- From: Erick Erickson erickerick...@gmail.com Sent: 29-06-2015 18:12 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data
Reading indexed data from solr 5.1.0 using admin/luke?
Hi all, Is there a way to read the indexed data for field on which the analysis/processing has been done ? I know using admin GUI we can see field wise analysis But how can i get hold on the complete document using admin/luke? or any other way? For example, if i have 2 fields called name and compressedname. name has values like apple, green-apple,red-apple compressedname has values like apple,greenapple,redapple Even though i make both these field indexed=true and stored=true I am not able to see the compressed values using admin/luke?id=mydocid in response i see something like this- lst name=name str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst lst name=compressedname str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst -- Best Regards, Dinesh Naik
RE: Reading indexed data from solr 5.1.0 using admin/luke?
Hi Eric, By compressed value I meant value of a field after removing special characters . In my example its -. Compressed form of red-apple is redapple . I wanted to know if we can see the analyzed version of fields . For example if I use ngram on a field , how do I see the analyzed values in index ? -Original Message- From: Erick Erickson erickerick...@gmail.com Sent: 29-06-2015 18:12 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: Reading indexed data from solr 5.1.0 using admin/luke? Not quite sure what you mean by compressed values. admin/luke doesn't show the results of the compression of the stored values, there's no way I know of to do that. Best, Erick On Mon, Jun 29, 2015 at 8:20 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi all, Is there a way to read the indexed data for field on which the analysis/processing has been done ? I know using admin GUI we can see field wise analysis But how can i get hold on the complete document using admin/luke? or any other way? For example, if i have 2 fields called name and compressedname. name has values like apple, green-apple,red-apple compressedname has values like apple,greenapple,redapple Even though i make both these field indexed=true and stored=true I am not able to see the compressed values using admin/luke?id=mydocid in response i see something like this- lst name=name str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst lst name=compressedname str name=typestring/str str name=schemaITS--/str str name=flagsITS--/str str name=valueGREEN-APPLE/str str name=internalGREEN-APPLE/str float name=boost1.0/float int name=docFreq0/int /lst -- Best Regards, Dinesh Naik
Dynamic boosting on a document for Solr4.10.2
Hi , We are looking for an option to boost a document while indexing based on the values of certain field. For example : lets say we have 10 documents with fields say- name,acc no, status, age address etc. Now for documents with status 'Active' we want to boost by value 1000 and if status is 'Closed' we want to do negative boost say -100 . Also if age is between '20-50' we want to boost by 2000 etc. Please let us know how can we achieve this ? -- Best Regards, Dinesh Naik
Internal document format for Solr 4.10.2
Hi, Is there a way to read the internal document once solr does the indexing ? Also is there a possibility to store this internal document in xml format ? -- Best Regards, Dinesh Naik
RE: How to achieve lemmatization for english words in Solr 4.10.2
Hi Jack, We are looking for something like this- For example if you search for a text -go We should also get other forms of this text like going,gone,goes etc. This is not being achieved via stemming. -Original Message- From: Jack Krupansky jack.krupan...@gmail.com Sent: 18-02-2015 21:50 To: solr-user@lucene.apache.org solr-user@lucene.apache.org Subject: Re: How to achieve lemmatization for english words in Solr 4.10.2 Please provide a few examples that illustrate your requirements. Specifically, requirements that are not met by the existing Solr stemming filters. What is your specific goal? -- Jack Krupansky On Wed, Feb 18, 2015 at 10:50 AM, dinesh naik dineshkumarn...@gmail.com wrote: Hi, IS there a way to achieve lemmatization in Solr? Stemming option is not meeting the requirement. -- Best Regards, Dinesh Naik
How to achieve lemmatization for english words in Solr 4.10.2
Hi, IS there a way to achieve lemmatization in Solr? Stemming option is not meeting the requirement. -- Best Regards, Dinesh Naik
Better way of copying/backup of index in Solr 4.10.2
What is the best way for copying/backup of index in Solr 4.10.2? -- Best Regards, Dinesh Naik
Possibility of Indexing without feeding again in Solr 4.10.2
Hi all, How to can do re-indexing in Solr without importing the data again? Is there a way to do re-indexing only for few documents ? -- Best Regards, Dinesh Naik
American /British Dictionary for solr-4.10.2
Hi, What are the dictionaries available for Solr 4.10.2? We are looking for a dictionary to support American/British English synonym. -- Best Regards, Dinesh Naik
American British Dictionary for Solr
Hi , We are looking for a dictionary to support American/British English synonym. Could you please let us know what all dictionaries are available ? -- Best Regards, Dinesh Naik
