Re: ***UNCHECKED*** Limit Solr search to number of character/words (without changing index)

[Muhammad Zahid Iqbal]

Hi Alessandro,

Thanks for making it more clear. As I mentioned I do not want to change my
index (mentioned in subject) for the feature I requested.


search query will have to look for first 100 characters indexed in same XYZ
field. "
How can I achieve this without changing index? I want at searching side.


On Mon, Jan 29, 2018 at 4:13 PM, alessandro.benedetti 
wrote:

> This seems different from what you initially asked ( and Diego responded)
> "One is simple, search query will look for whole content indexed in XYZ
> field
> Other one is, search query will have to look for first 100 characters
> indexed in same XYZ field. "
>
> This is still doable at Indexing time using a copy field.
> You can have your "originalField" and your "truncatedField" with no problem
> at all.
> Just use a combination of copyFields[1] and what Erick suggested.
>
> Cheers
>
> [1] https://lucene.apache.org/solr/guide/6_6/copying-fields.html
>
>
>
> -
> ---
> Alessandro Benedetti
> Search Consultant, R Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: ***UNCHECKED*** Limit Solr search to number of character/words (without changing index)

[Muhammad Zahid Iqbal]

Thanks Erick.

This is fine but I do not want to update my indexes as this configuration
will get applied to indexing as well. I have a requirement where one field
(XYZ) of type (text) requires two types of searches.

One is simple, search query will look for whole content indexed in XYZ field
Other one is, search query will have to look for first 100 characters
indexed in same XYZ field.

So I just want to do this at query time only.

Any idea? Would be much appreciated!


On Sat, Jan 27, 2018 at 10:27 PM, Erick Erickson <erickerick...@gmail.com>
wrote:

> Sure, use TruncateFieldUpdateProcessorFactory in your update chain,
> here's the base definition:
>
>   
> 
>   trunc
>   5
> 
>   
>
> This _can_ be configured to operate on "all StrField", or "all
> TextFields" as well, see the Javadocs.
>
> This is static, that is the field is truncated at index time so you
> can't change the values per-request.
>
> Best,
> Erick
>
>
>
> On Sat, Jan 27, 2018 at 6:32 AM, Muhammad Zahid Iqbal
> <zahid.iq...@northbaysolutions.net> wrote:
> > Thanks.
> >
> > I do not want to search if the query is shorter than a certain number of
> > terms/characters.
> >
> > For example, I have a 10MB document indexed in Solr what I want is to
> > search query in first 1MB content of that indexed document.
> >
> > Any workaround e.g .can I send query to Solr to look for only 1MB from
> > start of document.?
> >
> >
> >
> > On Fri, Jan 26, 2018 at 10:46 PM, Diego Ceccarelli (BLOOMBERG/ LONDON) <
> > dceccarel...@bloomberg.net> wrote:
> >
> >> Hi Zahid, if you want to allow searching only if the query is shorter
> than
> >> a certain number of terms / characters, I would do it before calling
> solr
> >> probably, otherwise you could write a QueryParserPlugin (see [1]) and
> check
> >> that the query is sound before processing it.
> >> See also: http://coding-art.blogspot.co.uk/2016/05/writing-custom-
> >> solr-query-parser-for.html
> >>
> >> Cheers,
> >> Diego
> >>
> >> [1] https://wiki.apache.org/solr/SolrPlugins
> >>
> >>
> >> From: solr-user@lucene.apache.org At: 01/26/18 13:24:36To:
> >> solr-user@lucene.apache.org
> >> Cc:  apa...@elyograg.org
> >> Subject: ***UNCHECKED*** Limit Solr search to number of character/words
> >> (without changing index)
> >>
> >> Hi All,
> >>
> >> Is there any way I can restrict Solr search query to look for specified
> >> number of characters/words (for only searching purposes not for
> >> highlighting)
> >>
> >> *For example:*
> >>
> >> *Indexed content:*
> >> *I am a man of my words I am a lazy man...*
> >>
> >> Search to consider only below mentioned (words=7 or characters=16)
> >> *I am a man of my words*
> >>
> >> If I search for *lazy *no record should find.
> >> If I search for *a *1 record should find.
> >>
> >>
> >> Thanks
> >> Zahid Iqbal
> >>
> >>
> >>
>

Re: ***UNCHECKED*** Limit Solr search to number of character/words (without changing index)

[Muhammad Zahid Iqbal]

Thanks.

I do not want to search if the query is shorter than a certain number of
terms/characters.

For example, I have a 10MB document indexed in Solr what I want is to
search query in first 1MB content of that indexed document.

Any workaround e.g .can I send query to Solr to look for only 1MB from
start of document.?



On Fri, Jan 26, 2018 at 10:46 PM, Diego Ceccarelli (BLOOMBERG/ LONDON) <
dceccarel...@bloomberg.net> wrote:

> Hi Zahid, if you want to allow searching only if the query is shorter than
> a certain number of terms / characters, I would do it before calling solr
> probably, otherwise you could write a QueryParserPlugin (see [1]) and check
> that the query is sound before processing it.
> See also: http://coding-art.blogspot.co.uk/2016/05/writing-custom-
> solr-query-parser-for.html
>
> Cheers,
> Diego
>
> [1] https://wiki.apache.org/solr/SolrPlugins
>
>
> From: solr-user@lucene.apache.org At: 01/26/18 13:24:36To:
> solr-user@lucene.apache.org
> Cc:  apa...@elyograg.org
> Subject: ***UNCHECKED*** Limit Solr search to number of character/words
> (without changing index)
>
> Hi All,
>
> Is there any way I can restrict Solr search query to look for specified
> number of characters/words (for only searching purposes not for
> highlighting)
>
> *For example:*
>
> *Indexed content:*
> *I am a man of my words I am a lazy man...*
>
> Search to consider only below mentioned (words=7 or characters=16)
> *I am a man of my words*
>
> If I search for *lazy *no record should find.
> If I search for *a *1 record should find.
>
>
> Thanks
> Zahid Iqbal
>
>
>

***UNCHECKED*** Limit Solr search to number of character/words (without changing index)

[Muhammad Zahid Iqbal]

Hi All,

Is there any way I can restrict Solr search query to look for specified
number of characters/words (for only searching purposes not for
highlighting)

*For example:*

*Indexed content:*
*I am a man of my words I am a lazy man...*

Search to consider only below mentioned (words=7 or characters=16)
*I am a man of my words*

If I search for *lazy *no record should find.
If I search for *a *1 record should find.


Thanks
Zahid Iqbal

Re: Indexing word with plus sign

[Muhammad Zahid Iqbal]

Hi,


Before applying tokenizer, you can replace your special symbols with some
phrase to preserve it and after tokenized you can replace it back.

For example:



Thanks,
Zahid iqbal

On Mon, May 22, 2017 at 12:57 AM, Fundera Developer <
funderadevelo...@outlook.com> wrote:

> Hi all,
>
> I am a bit stuck at a problem that I feel must be easy to solve. In
> Spanish it is usual to find the term 'i+d'. We are working with Solr 5.5,
> and StandardTokenizer splits 'i' and 'd' and sometimes, as we have in the
> index documents both in Spanish and Catalan, and in Catalan it is frequent
> to find 'i' as a word, when a user searches for 'i+d' it gets Catalan
> documents as results.
>
> I have tried to use the SynonymFilter, with something like:
>
> i+d => investigacionYdesarrollo
>
> But it does not seem to change anything.
>
> Is there a way I could set an exception to the Tokenizer so that it does
> not split this word?
>
> Thanks in advance!
>
>

Solr JOIN - Self join within core not working

[Muhammad Zahid Iqbal]

Hi,

I have a Solr instance setup with one core named as Book (contains total
100 records and a field isprimary:1 has 50 records)

When I querying, bring me accurate results (50), its simple I know:

http://localhost:7161/solr/Book/select?q=%3A=IsPrimary%3A1=10=xml=true

But When I query like this why its showing all records(100), I know I am in
the same core but why its behaving strange?:

http://localhost:7161/solr/Book/select?q=%3A=%7B!join+fromIndex%3DBook+from%3DGroupID+to%3DGroupID%7D%2BIsPrimary%3A1=10=xml=true

If Solr JOIN works like INNER QUERY this should not be the case.

Any valuable feedback will be appreciated!

Re: slow updates/searches

[Muhammad Zahid Iqbal]

Rallavagu,

Did you find any solution to slow searches? As far as I know jetty
container default configuration is bit slow for large production
environment.

On Tue, Sep 20, 2016 at 8:05 AM, Erick Erickson 
wrote:

> If both queries _and_ updates are slow, it's hard to see how upping
> the number of
> threads would help overall. Hmmm, you also reported that the CPUs
> didn't seem to be
> stressed so its worth a try, perhaps there's some kind of blocking going
> on
>
> Best,
> Erick
>
> On Mon, Sep 19, 2016 at 5:33 PM, Rallavagu  wrote:
> > Hi Erick,
> >
> > Would increasing (or adjusting) update threads help as per this JIRA
> ((Allow
> > the number of threads ConcurrentUpdateSolrClient StreamingSolrClients
> > configurable by a system property) here?
> >
> > https://issues.apache.org/jira/browse/SOLR-8500
> >
> > Thanks
> >
> >
> > On 9/19/16 8:30 AM, Erick Erickson wrote:
> >>
> >> Hmmm, not sure, and also not sure what to suggest next. QTimes
> >> measure only the search time, not, say, time waiting for the request to
> >> get
> >> serviced.
> >>
> >> I'm afraid the next suggestion is to throw a profiler at it 'cause
> nothing
> >> jumps
> >> out at me..'
> >>
> >> Best,
> >> Erick
> >>
> >> On Fri, Sep 16, 2016 at 10:23 AM, Rallavagu 
> wrote:
> >>>
> >>> Comments in line...
> >>>
> >>> On 9/16/16 10:15 AM, Erick Erickson wrote:
> 
> 
>  Well, the next thing I'd look at is CPU activity. If you're flooding
> the
>  system
>  with updates there'll be CPU contention.
> >>>
> >>>
> >>>
> >>> Monitoring does not suggest any high CPU but as you can see from vmstat
> >>> output "user" cpu is a bit high during updates that are taking time (34
> >>> user, 65 idle).
> >>>
> 
>  And there are a number of things you can do that make updates in
>  particular
>  much less efficient, from committing very frequently (sometimes
> combined
>  with excessive autowarm parameters) and the like.
> >>>
> >>>
> >>>
> >>> softCommit is set to 10 minutes, autowarm count is set to 0 and commit
> is
> >>> set to 15 sec for NRT.
> >>>
> 
>  There are a series of ideas that might trigger an "aha" moment:
>  https://wiki.apache.org/solr/SolrPerformanceFactors
> >>>
> >>>
> >>>
> >>> Reviewed this document and made few changes accordingly a while ago.
> 
> 
> 
>  But the crude measure is just to look at CPU usage when updates
> happen,
>  or
>  just before. Are you running hot with queries alone then add an update
>  burden?
> >>>
> >>>
> >>>
> >>> Essentially, it is high QTimes for queries got me looking into logs,
> >>> system
> >>> etc and I could correlate updates slowness and searching slowness. Some
> >>> other time QTimes go high is right after softCommit which is expected.
> >>>
> >>> Wondering what causes update threads wait and if it has any impact on
> >>> search
> >>> at all. I had couple of more CPUs added but I still see similar
> behavior.
> >>>
> >>> Thanks.
> >>>
> >>>
> 
>  Best,
>  Erick
> 
>  On Fri, Sep 16, 2016 at 9:19 AM, Rallavagu 
> wrote:
> >
> >
> > Erick,
> >
> > Was monitoring GC activity and couldn't align GC pauses to this
> > behavior.
> > Also, the vmstat shows no swapping or cpu I/O wait. However,
> whenever I
> > see
> > high update response times (corresponding high QTimes for searches)
> > vmstat
> > shows as series of number of "waiting to runnable" processes in "r"
> > column
> > of "procs" section.
> >
> >
> >
> > https://dl.dropboxusercontent.com/u/39813705/Screen%20Shot%
> 202016-09-16%20at%209.05.51%20AM.png
> >
> > procs ---memory-- ---swap--
> > -io -system-- cpu -timestamp-
> >  r  b swpd freeinact   active   si   so
> bi
> > bo
> > in   cs  us  sy  id  wa  st CDT
> >  2  071068 18688496  2526604 2420444000 0
> > 0
> > 1433  462  27   1  73   0   0 2016-09-16 11:02:32
> >  1  071068 18688180  2526600 2420456800 0
> > 0
> > 1388  404  26   1  74   0   0 2016-09-16 11:02:33
> >  1  071068 18687928  2526600 2420456800 0
> > 0
> > 1354  401  25   0  75   0   0 2016-09-16 11:02:34
> >  1  071068 18687800  2526600 2420457200 0
> > 0
> > 1311  397  25   0  74   0   0 2016-09-16 11:02:35
> >  1  071068 18687164  2527116 2420484400 0
> > 0
> > 1770  702  31   1  69   0   0 2016-09-16 11:02:36
> >  1  071068 18686944  2527108 2420490800 0
> > 52
> > 1266  421  26   0  74   0   0 2016-09-16 11:02:37
> > 12  171068 18682676  2528560 242071160   

Re: How to set NOT clause on Date range query in Solr

[Muhammad Zahid Iqbal]

Intend your question properly so that someone can understand.

I am out!

On Tue, Sep 20, 2016 at 12:23 PM, Sandeep Khanzode <
sandeep_khanz...@yahoo.com.invalid> wrote:

> Have been trying to understand this for a while ...How can I specify NOT
> clause in the following query?{!field f=schedule
> op=Intersects}[2016-08-26T12:30:00Z TO 2016-08-26T18:30:00Z]{!field
> f=schedule op=Contains}[2016-08-26T12:30:00Z TO
> 2016-08-26T18:30:00Z]Like, without LocalParams, we can specify
> -DateField:[2016-08-26T12:30:00Z TO 2016-08-26T18:30:00Z] to get an
> equivalent NOT clause. But, I need a NOT Contains Date Range query.I have
> tried a few options but I end up getting parsing errors. Surely there must
> be some obvious way I am missing. SRK

Re: Solr Special Character Search

[Muhammad Zahid Iqbal]

Hi,

To handled special characters, either you need to create your own custom
filter factory or need to replace already specified filter factory with
some other, if you are using StandardFilterFactory.



On Tue, Sep 20, 2016 at 5:16 PM, Alexandre Rafalovitch 
wrote:

> What's your field definition? What happens when the text goes through the
> analysis chain as you can test in Admin UI?
>
> Regards,
>Alex
>
> On 20 Sep 2016 6:49 PM, "Cheatham, Kevin" 
> wrote:
>
> > Hello - Has anyone out there had success with anything similar to our
> > issue below and be kind enough to share?
> >
> > We posted several files as text and we're able to search for alphanumeric
> > characters, but not able to search for special characters such as @ or ©
> > through Solrcloud Admin 5.2 UI.
> > We've searched through lots of documentation but haven't had success yet.
> >
> > We also tried posting files not as text but seems we're not able to
> search
> > for any special characters below hexadecimal 20.
> >
> > Any assistance would be greatly appreciated!
> >
> > Thanks!
> >
> > Kevin Cheatham | Office (314) 573-5534 | kevin.cheat...@graybar.com
> > www.graybar.com - Graybar Works to Your Advantage
> >
> >
>

Re: Stopping Solr JVM on OOM

[Muhammad Zahid Iqbal]

You can use ping functionality by setting time-out that suits for your
container/web-apps. If its not working then you can restart your container.
Cheers!

If any other solution I am interested too.

On Fri, Feb 26, 2016 at 2:19 AM, CP Mishra  wrote:

> Solr & Lucene dev folks must be catching Throwable for a reason. Anyway, I
> am asking for solutions that I can use.
>
> On Thu, Feb 25, 2016 at 3:06 PM, Fuad Efendi  wrote:
>
> > The best practice: do not ever try to catch Throwable or its descendants
> > Error, VirtualMachineError, OutOfMemoryError, and etc.
> >
> > Never ever.
> >
> > Also, do not swallow InterruptedException in a loop.
> >
> > Few simple rules to avoid hanging application. If we follow these, there
> > will be no question "what is the best way to stop Solr when it gets in
> OOM”
> > (or just becomes irresponsive because of swallowed exceptions)
> >
> >
> > --
> > Fuad Efendi
> > 416-993-2060(cell)
> >
> > On February 25, 2016 at 2:37:45 PM, CP Mishra (mishr...@gmail.com)
> wrote:
> >
> > Looking at the previous threads (and in our tests), oom script specified
> > at
> > command line does not work as OOM exception is trapped and converted to
> > RuntimeException. So, what is the best way to stop Solr when it gets in
> > OOM
> > state? The only way I see is to override multiple handlers and do
> > System.exit() from there. Is there a better way?
> >
> > We are using Solr with default Jetty container.
> >
> > Thanks,
> > CP Mishra
> >
> >
>

Re: Is it possible to sort on a BooleanField?

[Muhammad Zahid Iqbal]

Please share your schema.

On Thu, Dec 3, 2015 at 11:28 AM, Clemens Wyss DEV 
wrote:

> Looks like not. I get to see
> 'can not sort on a field which is neither indexed nor has doc values:
> '
>
> - Clemens
>

Re: Upgrading from 4.x to 5.x

[Muhammad Zahid Iqbal]

Hi daniel

You need to update your config/scehma file on the path like
'...\solr-dir\server\solr' . When you are done then you can update your
index path in solrconfig.xml.

I hope you got it.

Best,
Zahid


On Thu, Nov 19, 2015 at 1:58 PM, Daniel Miller  wrote:

> Thank you - but I still don't understand where to install/copy/modify
> config files or schema to point at my current index. My 4.x schema.xml was
> fairly well optimized, and I believe I removed any deprecated usage, so I
> assume it would be compatible with the 5.x server.
>
> Daniel
>
>
>
>
> On November 18, 2015 4:55:40 AM Jan Høydahl  wrote:
>
> Hi
>>
>> You could try this
>>
>> Instead of example/, use the server/ folder (it has Jetty in it)
>> Start Solr using bin/solr start script instead of java -jar start.jar …
>> Leave your solrconfig and schema as is to keep back-compat with 4.x.
>> You may need to remove use of 3.x classes that were deprecated in 4.x
>>
>>
>> https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> 18. nov. 2015 kl. 10.10 skrev Daniel Miller :
>>>
>>> Hi!
>>>
>>> I'm a very inexperienced user with Solr.  I've been using Solr to
>>> provide indexes for my Dovecot IMAP server.  Using version 3.x, and later
>>> 4.x, I have been able to do so without too much of a challenge.  However,
>>> version 5.x has certainly changed quite a bit and I'm very uncertain how to
>>> proceed.
>>>
>>> I currently have a working 4.10.3 installation, using the "example"
>>> server provided with the Solr distribution package, and a schema.xml
>>> optimized for Dovecot.  I haven't found anything on migrating from 4 to 5 -
>>> at least anything I actually understood.  Can you point me in the right
>>> direction?
>>>
>>> --
>>> Daniel
>>>
>>
>>
>
>

Re: Upgrading from 4.x to 5.x

[Muhammad Zahid Iqbal]

Daniel,

You are close, delete those *configsets* folder and paste you
*collection1 *folder
and run the server. It will do the trick.

On Thu, Nov 19, 2015 at 2:54 PM, Daniel Miller <dmil...@amfes.com> wrote:

> Not quite but I'm improving. Or something...
>
> Looking under solr5/server/solr I see configsets with the three default
> choices. What "feels" right is to make a new folder in there for my app
> (dovecot) and then copy my solr4/example/solr/collection1/conf folder. I'm
> hoping I'm on the right track - maybe working too hard.
>
> If that was correct, then I tried "solr create -n dovecot -c dovecot"
> (after stopping my old server and starting a new one) and it did create an
> entry. I then stopped the server, copied my old data folder over to the new
> location, and started the server.
>
> I then tried searching, which may have worked...I'm not certain if the
> search results came from solr or my imap server manually searching.
>
> I'm sure I'm overcomplicating things - just not seeing the obvious.
>
> Daniel
>
>
>
>
> On November 19, 2015 1:09:07 AM Muhammad Zahid Iqbal <
> zahid.iq...@northbaysolutions.net> wrote:
>
> Hi daniel
>>
>> You need to update your config/scehma file on the path like
>> '...\solr-dir\server\solr' . When you are done then you can update your
>> index path in solrconfig.xml.
>>
>> I hope you got it.
>>
>> Best,
>> Zahid
>>
>>
>> On Thu, Nov 19, 2015 at 1:58 PM, Daniel Miller <dmil...@amfes.com> wrote:
>>
>> Thank you - but I still don't understand where to install/copy/modify
>>> config files or schema to point at my current index. My 4.x schema.xml
>>> was
>>> fairly well optimized, and I believe I removed any deprecated usage, so I
>>> assume it would be compatible with the 5.x server.
>>>
>>> Daniel
>>>
>>>
>>>
>>>
>>> On November 18, 2015 4:55:40 AM Jan Høydahl <jan@cominvent.com>
>>> wrote:
>>>
>>> Hi
>>>
>>>>
>>>> You could try this
>>>>
>>>> Instead of example/, use the server/ folder (it has Jetty in it)
>>>> Start Solr using bin/solr start script instead of java -jar start.jar …
>>>> Leave your solrconfig and schema as is to keep back-compat with 4.x.
>>>> You may need to remove use of 3.x classes that were deprecated in 4.x
>>>>
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5
>>>>
>>>> --
>>>> Jan Høydahl, search solution architect
>>>> Cominvent AS - www.cominvent.com
>>>>
>>>> 18. nov. 2015 kl. 10.10 skrev Daniel Miller <dmil...@amfes.com>:
>>>>
>>>>>
>>>>> Hi!
>>>>>
>>>>> I'm a very inexperienced user with Solr.  I've been using Solr to
>>>>> provide indexes for my Dovecot IMAP server.  Using version 3.x, and
>>>>> later
>>>>> 4.x, I have been able to do so without too much of a challenge.
>>>>> However,
>>>>> version 5.x has certainly changed quite a bit and I'm very uncertain
>>>>> how to
>>>>> proceed.
>>>>>
>>>>> I currently have a working 4.10.3 installation, using the "example"
>>>>> server provided with the Solr distribution package, and a schema.xml
>>>>> optimized for Dovecot.  I haven't found anything on migrating from 4
>>>>> to 5 -
>>>>> at least anything I actually understood.  Can you point me in the right
>>>>> direction?
>>>>>
>>>>> --
>>>>> Daniel
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>
>