Is it right to index multi-valued fields with hash value?

Hi All During search, i find it difficult to handle every multi-valued field with different analyzers. So i believe indexing multi-valued field with hash value may solve the problem of searching with different analyzers. Any of you tried hash value in lucene index? if so, please share

Re: Is housekeeping of Lucene indexes block index update but allow search ?

Hi Gaurav Thanks for the clarification. If possible, please share your NRT manager API related code example. i believe, it will help me to understand little better. - Kumaran R On Tue, Aug 5, 2014 at 12:39 PM, Gaurav gupta gupta.gaurav0...@gmail.com wrote: Thanks Kumaran and Erik for

Re: How does Lucene decides which fields have termvectors stored and which not?

Hi Sachin Kulkarni, If possible, Please share your code. - Kumaran R On Tue, Aug 19, 2014 at 9:07 AM, Sachin Kulkarni kulk...@hawk.iit.edu wrote: Hi, I am using Lucene 4.6.0. I have been storing 5 fields for my documents in the index, namely body, title, docname, docdate and

Re: How does Lucene decides which fields have termvectors stored and which not?

()) // System.out.println(c); } // *88 // On Tue, Aug 19, 2014 at 2:04 AM, Kumaran Ramasubramanian kums@gmail.com wrote: Hi Sachin Kulkarni, If possible, Please share your code. - Kumaran R On Tue, Aug 19, 2014 at 9:07 AM, Sachin Kulkarni

Two different types of values in same field name in single index

Hi All, i have indexed module wise data in same index. In this case, we index two types of field in same name in two different document like this. *document1:* > module:1 > status:4 ( as LongField ) > *code:* > long longValue=Long.parseLong(value); > LongField field=new LongField(f > ield >

Re: Two different types of values in same field name in single index

an - > > Aren't you creating an unworkable scenario for sorting? > > -will > > On 10/27/15 5:49 AM, Kumaran Ramasubramanian wrote: > >> Hi All, >> >> i have indexed module wise data in same index. In this case, we index >> two >> types of fi

Re: Two different types of values in same field name in single index

ixing numeric > and text data. Since you're not sorting you should be > OK with the caveat that searching for > "02" won't match an indexed value of "2" unless you > remove all leading zeros at both index and query > time. > > Best, > Erick > > On Tue, Oct

Need change one field type from IntField to String including indexOptions to store positions & Norms

Hi All, Previous Post - http://www.gossamer-threads.com/lists/lucene/java-user/289159 i have indexed one field "STATUS" as both IntField & String field in same index. Now i want to take IntField containing documents and change the value of field "STATUS" to string with norms & positions (

Re: Need change one field type from IntField to String including indexOptions to store positions & Norms

upansky <jack.krupan...@gmail.com> > > wrote: > > > > > > The standard answer is that you need to reindex all of your data. > > > > > > -- Jack Krupansky > > > > > > On Thu, Dec 17, 2015 at 6:10 AM, Kumaran Ramasubramanian < > > ku

Re: Need change one field type from IntField to String including indexOptions to store positions & Norms

ent index and > then write new documents to a new index and then use the new index. That's > if all of the indexed field values are stored. > > -- Jack Krupansky > > On Thu, Dec 17, 2015 at 2:10 PM, Kumaran Ramasubramanian < > kums@gmail.com > > wrote: > > >

IntField is not indexing properly - field properties changed to "just stored"

Hi All, i am using lucene 4.10.4. i am using below code to index IntField. All properties of IntField is vanished except stored. Any help is much appreciated. IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_4_10_4, new > ClassicAnalyzer(Version.LUCENE_30, new StringReader(""))); >

Re: Need change one field type from IntField to String including indexOptions to store positions & Norms

Dear All i am using lucene 4.10.4. Is there any more information i missed to provide? Please let me know. -- Kumaran R*​* On Wed, Dec 16, 2015 at 10:35 PM, Kumaran Ramasubramanian < kums@gmail.com> wrote: > > Hi All, > > Previous Post - > http://www.gossamer-thr

Re: Using ​phrase query in Termfilters

iltering clause in a BooleanQuery > (using Occur.FILTER). > > Uwe > > - > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Kumaran Ramasubramanian [mailto:kums

Using ​phrase query in Termfilters

Hi All, Am using lucene 4.10.4. Is it right to add analyzed multi valued fields & phrase query for the same field in boolean filter. i believe we could not apply analyzers to values in filters. So am not getting results for those filters' match. String phraseTerm = "hello world" > Term term =

Lucene term modifiers during search time - importance based on listed order

Hi All, i am trying to retrieve most related results using user given search queries. If there is no results for exact queries, i want to use term modifiers during search time. And so, i want to give more importance to term modifiers in the defined listed order given below 1. exact

Re: Lucene term modifiers during search time - importance based on listed order

Kumaran R On Tue, Jun 7, 2016 at 10:16 PM, Kumaran Ramasubramanian <kums@gmail.com > wrote: > > > Hi All, > > i am trying to retrieve most related results using user given search > queries. If there is no results for exact queries, i want to use term > modifier

Re: Difference in Retrieving values of a docvalue field using atomicreader over duplicate StoredField

doc values + some stored fields you're looking at > multiple seeks per docID you retrieve. > > Really you should only do this for a few hits, e.g. one page worth. > > Mike McCandless > > http://blog.mikemccandless.com > > On Wed, Jun 1, 2016 at 3:15 PM, Kumaran Ra

Difference in Retrieving values of a docvalue field using atomicreader over duplicate StoredField

Hi All, In javadoc of every docvalue field , there is line like ​ > If you also need to store the value, you should add a separate StoredField >

query norm understanding when we use term boosting during search time

Hi All, i have read some discussions on impact of querynorm values in lucene score when we specify more boosting. Impact is, if boosting increases, querynorm decreases a lot, and so more boosting does not results in final lucene score Consider an example like, i need to order results based on

Re: Indexing and storing Long fields

many more anomalies. At best you need to > index both forms as text rather than mixing numeric > and text data. Thanks in advance... -- Kumaran R On Sun, Jul 24, 2016 at 1:54 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > On Sat, Jul 23, 2016 at 4:48 AM, Kumar

Re: Indexing and storing Long fields

Hi Mike, For your information, am using lucene 4.10.4.. am i missing anything? ​-- Kumaran R​ On Wed, Jul 27, 2016 at 1:52 AM, Kumaran Ramasubramanian <kums@gmail.com > wrote: > > Hi Mike, > > 1.if we index one field as analyzed and not analyzed using same name

Re: Indexing and storing Long fields

candless.com > > On Thu, Jul 28, 2016 at 9:31 AM, Kumaran Ramasubramanian < > kums@gmail.com > > wrote: > > > Hi Mike, > > > > For your information, am using lucene 4.10.4.. am i missing anything? > > > > > > > > ​-- > >

Re: Indexing and storing Long fields

Hi Mike, *Two different fields can be the same name* Is it so? You mean we can index one field as docvaluefield and also stored field, Using same name? And AFAIK, We cannot index one field as analyzed and not analyzed using the same name. Am i right? Kumaran R On Jul 21, 2016 11:50 PM,

How to store and retrieve latest utf8mb4 emoji / smiley characters in lucene index

Hi All, Am using lucene 4.10.4. Using lucene index, Is there any way to store and retrieve latest utf8 and utf8mb4 emoji / smiley characters?? In any latest lucene version?? Thanks in advance. -- Kumaran R

Re: How to store and retrieve latest utf8mb4 emoji / smiley characters in lucene index

Hi All, Is there any pointers on storing smileys in lucene index?? Any help is much appreciated. Thank you. -- Kumaran R On Jul 30, 2016 12:24 PM, "Kumaran Ramasubramanian" <kums@gmail.com> wrote: > > Hi All, > > Am using lucene 4.10.4. Using lucene inde

Re: How to store and retrieve latest utf8mb4 emoji / smiley characters in lucene index

fgh​ Any further pointers on the same ?? -- Kumaran R On Mon, Aug 1, 2016 at 12:07 AM, Kumaran Ramasubramanian <kums@gmail.com> wrote: > > Hi All, > > Is there any pointers on storing smileys in lucene index?? Any help is much appreciated. > > Thank you. >

In lucene faceting, advantages of taxonomy index over ​SortedSetDocValuesFacetField

Hi All, As per documentation of SortedSetDocValuesFacetCounts, Compute facets counts from previously indexed > ​​ > SortedSetDocValuesFacetField, without require a separate taxonomy index. > Faceting is a bit slower (~25%), and there is added cost on every > IndexReader open to create a new

Re: Recommended number of fields in one lucene index

o index > arbitrary documents and provide search for them, which cannot scale, or to > the fact that the index contained many unrelated documents that should have > been put into different indices. This limit has been very useful to catch > such design problems early instead of waiting for t

Recommended number of fields in one lucene index

Hi All, Elasticsearch allows 1000 fields by default. In lucene, What are the indexing and searching performance impacts of having 10 fields vs 3000 fields in a lucene index? In my case, while indexing, i index and store all fields and so i can provide update on one field where we use to take out

Re: parent-child relationship in lucene - to avoid reindexing if parent information changes

dex. > > === > Ralph > > > > On 30.08.2016 08:38, Kumaran Ramasubramanian wrote: > >> Hi All, >> >> >> Am building a sample application, where a group of members can interact as >> a chat room. i am trying to enable search for message level search

parent-child relationship in lucene - to avoid reindexing if parent information changes

Hi All, Am building a sample application, where a group of members can interact as a chat room. i am trying to enable search for message level search... If i denormalize group_name & group_members in every lucene document, then below cases will reindex more number of lucene documents... 1.

Clarification on LUCENE 4795 discussions ( Add FacetsCollector based on SortedSetDocValues )

​ ​Hi All,​ ​i want to know the list of features ​which can be used by applications using facet module of lucene. https://issues.apache.org/jira/browse/LUCENE-4795?focusedCommentId=13599687 I ask because it seems that the only thing that we get from this SortedSet > approach is not having to

Re: Clarification on LUCENE 4795 discussions ( Add FacetsCollector based on SortedSetDocValues )

c8f02d12e4fdb76f97891c8593ef5fbc;hb=HEAD > > Hope this helps! > > Shai > > On Tue, Sep 27, 2016 at 7:20 AM Kumaran Ramasubramanian < kums@gmail.com> > wrote: > > > Hi mike, > > > > Thanks for the clarification. Any example about difference in using

Faceting : what are the limitations of Taxonomy (Separate index and hierarchical facets) and SortedSetDocValuesFacetField ( flat facets and no sidecar index) ?

Hi All, We all know that Lucene supports faceting by providing Taxonomy(Separate index and hierarchical facets) and SortedSetDocValuesFacetField ( flat facets and no sidecar index). Then why did solr and elastic search go for its own implementation ? ( that is, solr uses block

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Hi Adrien How to do this? Any Pointers? ​ > If it is fine to add the ascii folding filter at the end of the analysis chain, then you could use AnalyzerWrapper. ​ > ​- Kumaran R​ On Tue, Oct 11, 2016 at 9:59 PM, Kumaran Ramasubramanian <kums@gmail.com > wrote: >

indexing analyzed and not_analyzed values in same field

Hi All, i have indexed 4 documents in an index where BANKNAME field is analyzed in two documents and it is not_analyzed in another two documents. i have mentioned search cases below where i am able to search using both analyzed ( using classic analyzer ) and not_analyzed ( using keyword analyzer

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

mar. 18 oct. 2016 à 18:46, Kumaran Ramasubramanian <kums@gmail.com> > a écrit : > > > Hi Adrien > > > > How to do this? Any Pointers? > > > > ​ > > > If it is fine to add the ascii folding filter at the en

how do lucene read large index files?

Hi All, how do lucene read large index files? for example, if one file (for eg: .dat file) is 4GB. lucene read only part of file to RAM? or is it different approach for different lucene file formats? Related Link: How do applications (and OS) handle very big files? http://superuser.com/a/361201

Re: how do lucene read large index files?

Sorry I can't be more help there > > > > I can help! > > > >> On Thu, Nov 24, 2016 at 7:29 AM, Kumaran Ramasubramanian > >> <kums@gmail.com> wrote: > >> > Erick, Thanks a lot for sharing an excellent post... > >&

Re: how do lucene read large index files?

Uwe's blog: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Short form: files are read into the OS's memory as needed. the whole > file isn't read at once. > > Best, > Erick > > On Wed, Nov 23, 2016 at 12:04 AM, Kumaran Ramasubramanian > <kums@

Re: Indexing values of different datatype under same field

Hi Rajnish It is not advisable to index values with two data types in a field. Features like phrase query, sorting may break in those indexes. related previous discussion : http://www.gossamer-threads.com/lists/lucene/java-user/289159?do=post_view_flat#289159 - Kumaran R On Fri, Nov 4,

How to add ASCIIFoldingFilter in ClassicAnalyzer

Hi All, Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without writing a new custom analyzer ? should i extend StopwordAnalyzerBase again? I know that ClassicAnalyzer is final. any special purpose for making it as final? Because, StandardAnalyzer was not final before ?

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

lyzer > > > > Hi Kumaran, > > > > If it is fine to add the ascii folding filter at the end of the analysis > > chain, then you could use AnalyzerWrapper. Otherwise, you need to create > a > > new analyzer that has the same analysis chain as ClassicAn

Re: Heavy usage of final in Lucene classes

Hi I want to know the purpose of having final in analyzers. For eg: classicanalyzer. It will be easy to add asciifolding filter over classicanalyzer. - Kumaran R On Jan 12, 2017 5:41 AM, "Michael McCandless" wrote: I don't think it's about efficiency but rather

Re: Replacement for Filter-as-abstract-class in Lucene 5.4?

I always use filter when i need to add more than 1024 ( for no scoring cases ). If filter is removed in lucene 6, what will happen to maxbooleanclauses limit? Am i missing anything? - Kumaran R On Jan 12, 2017 5:01 AM, "Trejkaz" wrote: On Thu, Jan 21, 2016 at 4:25 AM,

Re: how do lucene read large index files?

eading > so perhaps NIOFSDirectory won't be much slower. > > Why not use MMapDirectory for both indexing and searching? > Mike McCandless > > http://blog.mikemccandless.com > > > On Mon, Nov 28, 2016 at 7:20 AM, Kumaran Ramasubramanian > <kums@gmail.com> wrote

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

Thank you Adrien. "NumericDocValuesField is the one that supports sorting." Does this mean LongField/IntField just supports lexicographic order in sorting? - Kumaran R On Dec 22, 2016 11:28 PM, "Adrien Grand" <jpou...@gmail.com> wrote: Le jeu. 22 déc. 2016 à 18:5

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

t; > and sorting (numeric). > > > > A field with docValues="true" just supports faceting, grouping and > > sorting without having to "uninvert" the field in the Java heap, the > > data is out in OS cache. See Uwe's excellent blog here: > >

Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

Hi All, I want to provide sorting, range search and faceting in numeric fields. AFAIK, Purpose of different numeric field types are, NumericDocValuesField supports sorting and faceting LongField/IntField supports range query and sorting 1. Should i duplicate one field in above mentioned

Re: Encryption at lucene index

is proprietary, but I'm not sure. Maybe more googling might help > find the exact page where his solution is described. > > On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian < > kums@gmail.com> > wrote: > > > Hi Erick, i want to encrypt some fields of an

Re: Encryption at lucene index

much been abandoned. > > Best, > Erick > > On Thu, Aug 10, 2017 at 11:17 PM, Kumaran Ramasubramanian > <kums@gmail.com> wrote: > > Hi Ishan, thank you :-) > > > > - > > - > > Kumaran R > > > > > > > > On Mon, Aug 7,

Filters Vs queries - for terms more than 1024

Hi All, i am using lucene 4.10.4 In lucene search, i know we have 1024 limitation in number of boolean query clauses. i know we can increase this limit.. but i want to understand queries vs filter in lucene 4.10.4... i want to make queries larger than 1024.. Relevance is not needed for me. What

Re: Filters Vs queries - for terms more than 1024

rary to large boolean > queries. > > Otherwise you would probably need to rethink how you design your documents > in order to be able to run simpler queries. > > Le lun. 17 juil. 2017 à 16:28, Kumaran Ramasubramanian <kums@gmail.com > > > a écrit : > > > Hi All

Re: Filters Vs queries - for terms more than 1024

en things should be mostly ok if you have many clauses. But like TermsQuery, BooleanFilter always consume all matching documents from all its clauses. So if you intersect it with a selective query, it is wasteful. Le mar. 18 juil. 2017 à 11:42, Kumaran Ramasubramanian <kums@gmail.com> a

Re: Filters Vs queries - for terms more than 1024

eredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504) > at > org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150) If i use BooleanQuery and wrap it with ConstantScoreQuery, shall i use 1 lakh boolean clauses in booleanquery ? - ​- Kumaran R ​ On Wed, Jul 19, 2017 at 8:

Encryption at lucene index

Hi All, After looking at all below discussions, i have one doubt which may be silly or novice but i want to throw this to lucene user list. if we have encryption layer included in our analyzer's flow of filters like EncryptionFilter to control field-level encryption. what are the consequences ?

Re: Encryption at lucene index

experts. > > Best, > Erick > > On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian > <kums@gmail.com> wrote: > > Hi All, > > > > > > After looking at all below discussions, i have one doubt which may be > silly > > or novice but i want

Re: Encryption at lucene index

is > quite large, much too large to conduct a seminar through the user's > list. > > Best, > Erick > > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian > <kums@gmail.com> wrote: > > Hi Erick, > > > > Thanks for the information. Any pointers abo

Re: email field - analyzed and not analyzed in single field using custom analyzer

} > }; > assertAnalyzesTo(a, "will.sm...@yahoo.com", > new String[] { "will.sm...@yahoo.com", "will", "smith", "yahoo", > "com" }, > null, null, null, > new int[] { 1, 0, 1, 1, 1 }, > nu

email field - analyzed and not analyzed in single field using custom analyzer

Hi All, i want to index email fields as both analyzed and not analyzed using custom analyzer. for example, sm...@yahoo.com will.sm...@yahoo.com that is, indexing sm...@yahoo.com as single token as well as analyzed tokens in same email field... My existing custom analyzer, public class

Index fields configuration - suggestions

Hi all, Regarding configurations about every fields( stored? analyzed? sort needed? numeric ? ), elastic search designed cluster state to hold these configurations index wise.. solr have those configurations in xml format. If we have data center in multiple locations, is there any better way of

Autocompletion based on one field in index

Hi All, I would like to compute autocompletion based on one field's data. For example, title field of a list of webpages. Is there anyway to achieve this? Regards Kumaran R

Re: Autocompletion based on one field in index

Hi All, Any input would be appreciated.. thanks in advance. Kumaran R On Tue, Mar 10, 2020, 11:44 PM Kumaran Ramasubramanian wrote: > > > Hi Mikhail > > Thanks for the input. But i would like to suggest title of the available > documents (for the query typed in search b

Re: Autocompletion based on one field in index

its help widget ) [image: Screenshot from 2020-03-10 23-40-14.png] -- *K*umaran *R* On Wed, Mar 4, 2020 at 2:44 AM Mikhail Khludnev wrote: > Hi, > > org.apache.lucene.search.spell.DirectSpellChecker > > > On Tue, Mar 3, 2020 at 8:14 AM Kumaran Ramasubramanian > > wr

currency based search using query time calculated field match with expression

I am having one use case regarding currency based search. I want to get any suggestions or pointers.. For example, Assume, 1USD = 75 INR 1USD = 42190 IRR similarly, we have support for 100 currencies as of now. Record1 created with PRICE 150 INR & EXCHANGE_RATE 75 for USD Record2 created with

Re: currency based search using query time calculated field match with expression

> suggestions, you need to be much more explicit about your use case, > what you've tried, why it didn't work, etc. > > On Fri, Sep 3, 2021 at 6:08 AM Kumaran Ramasubramanian > wrote: > > > > Hi Michael, Thanks for the response. > > > > Based on my u

Re: currency based search using query time calculated field match with expression

simple > expression language. It might prove useful to you if the exchange rate > needs to be tracked very dynamically. > > On Thu, Sep 2, 2021 at 2:15 PM Kumaran Ramasubramanian > wrote: > > > > I am having one use case regarding currency based search. I want to get

Handling Indian regional languages

For handling Indian regional languages, what is the advisable approach? 1. Indexing each language data(Tamil, Hindi etc) in specific fields like content_tamil, content_hindi with specific per field Analyzer like Tamil for content_tamil, HindiAnalyzer for content_hindi? 2. Indexing all language

Re: Handling Indian regional languages

Hi Robert Muir, we will check on this. Thanks a lot for the pointers. -- *K*umaran *R* On Mon, Jan 16, 2023 at 11:16 PM Robert Muir wrote: > On Tue, Jan 10, 2023 at 2:04 AM Kumaran Ramasubramanian > wrote: > > > > For handling Indian regional languages, what is the