Boost a dynamic field

[Kelv]

Hello,

I'm trying to boost a document score based on the existence of a dynamic 
field. I can't seem to get the syntax right and get either Solr server 
errors or it just doesn't do anything to the Solr response.


In solrconfig.xml the dynamic fields are defined as...

stored="true" multiValued="true"/>


The field I want to check for is called DYNAMIC_rank. If it exists I 
want to boost the score so the document shows up first.


Hoping someone can help!

Thanks,

Kelv

Re: Dynamic field schema

[Shawn Heisey]

On 7/10/2019 6:52 AM, ericstein wrote:

the documents are in both cores. the official title field data exists in
both. However, it only gives me the official_title_s field in the second
core when I query. When I look at the schema in the admin it only shows the
official_title_s field


The schema browser examines the Lucene index.  Lucene doesn't know about 
the schema, so only fields that have actually been used at the Lucene 
layer will be shown.


If the schema browser does not have a field mentioned, it means that 
when the indexed documents made it down to the Lucene layer, that field 
was not present in any of them.


Thanks,
Shawn

Re: Dynamic field schema

[ericstein]

Hi Shawn,

the documents are in both cores. the official title field data exists in
both. However, it only gives me the official_title_s field in the second
core when I query. When I look at the schema in the admin it only shows the
official_title_s field



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Dynamic field schema

[Shawn Heisey]

On 7/9/2019 5:42 PM, ericstein wrote:

I am expecting both cores to have the following fields:
official_title_s
official_title_t

However, the second core only recognizes:
official_title_s

It seems that the schema doesn't recognize my field the same in both cores.


What do you mean by "recognize"?  Be very specific about what you are 
examining that has led to this conclusion.


Search results will only contain fields that were actually included in 
the documents when they were indexed.  Maybe documents in the second 
core do not have the official_title_t field.


Thanks,
Shawn

Dynamic field schema

[ericstein]

Hi all, 
I am new to the SOLR world, so bear with me. I have currently have 2 cores
that share the same schema, or so I think? I have noticed that certain
fields don't exist in both cores even though they are set to look at the
same config sets (S:\solr-6.1.0\server\solr\configsets\sitecore_configs).

I am expecting both cores to have the following fields: 
official_title_s
official_title_t

However, the second core only recognizes:
official_title_s

It seems that the schema doesn't recognize my field the same in both cores.

My dynamic fields are:





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Add dynamic field to existing index slow

[Erick Erickson]

OK, then let’s see the indexing code. Make sure you don’t
1> commit after every batch
2> never, never, never optimize.

BTW, you do not want to turn off commits entirely, there are some internal data 
structures that grow between commits. So I might do something like specify 
commitWithin on my adds for something like 5 minutes.

Best,
Erick

> On Jul 2, 2019, at 6:24 AM, derrick cui  
> wrote:
> 
> I have tested the query desperately, actually executing query is pretty fast, 
> it only took a few minutes to go through all results including converting 
> solr document to java object. So I believe the slowness is in persistence 
> end.  BTW,  I am using linux system.
> 
> 
> Sent from Yahoo Mail for iPhone
> 
> 
> On Sunday, June 30, 2019, 4:52 PM, Shawn Heisey  wrote:
> 
> On 6/30/2019 2:08 PM, derrick cui wrote:
>> Good point Erick, I will try it today, but I have already use cursorMark in 
>> my query for deep pagination.
>> Also I noticed that my cpu usage is pretty high, 8 cores, usage is over 
>> 700%. I am not sure it will help if I use ssd disk
> 
> That depends on whether the load is caused by iowait or by actual CPU usage.
> 
> If it's caused by iowait, then SSD would help, but additional memory 
> would help more.  Retrieving data from the OS disk cache (which exists 
> in main memory) is faster than SSD.
> 
> If it is actual CPU load, then it will take some additional poking 
> around to figure out which part of your activities causes the load, as 
> Erick mentioned.
> 
> It's normally a little bit easier to learn these things from Unix-like 
> operating systems than from Windows.  What OS are you running Solr on?
> 
> Thanks,
> Shawn
> 
> 
> 

Re: Add dynamic field to existing index slow

[derrick cui]

I have tested the query desperately, actually executing query is pretty fast, 
it only took a few minutes to go through all results including converting solr 
document to java object. So I believe the slowness is in persistence end.  BTW, 
 I am using linux system.


Sent from Yahoo Mail for iPhone


On Sunday, June 30, 2019, 4:52 PM, Shawn Heisey  wrote:

On 6/30/2019 2:08 PM, derrick cui wrote:
> Good point Erick, I will try it today, but I have already use cursorMark in 
> my query for deep pagination.
> Also I noticed that my cpu usage is pretty high, 8 cores, usage is over 700%. 
> I am not sure it will help if I use ssd disk

That depends on whether the load is caused by iowait or by actual CPU usage.

If it's caused by iowait, then SSD would help, but additional memory 
would help more.  Retrieving data from the OS disk cache (which exists 
in main memory) is faster than SSD.

If it is actual CPU load, then it will take some additional poking 
around to figure out which part of your activities causes the load, as 
Erick mentioned.

It's normally a little bit easier to learn these things from Unix-like 
operating systems than from Windows.  What OS are you running Solr on?

Thanks,
Shawn

Re: Add dynamic field to existing index slow

[Shawn Heisey]

On 6/30/2019 2:08 PM, derrick cui wrote:

Good point Erick, I will try it today, but I have already use cursorMark in my 
query for deep pagination.
Also I noticed that my cpu usage is pretty high, 8 cores, usage is over 700%. I 
am not sure it will help if I use ssd disk


That depends on whether the load is caused by iowait or by actual CPU usage.

If it's caused by iowait, then SSD would help, but additional memory 
would help more.  Retrieving data from the OS disk cache (which exists 
in main memory) is faster than SSD.


If it is actual CPU load, then it will take some additional poking 
around to figure out which part of your activities causes the load, as 
Erick mentioned.


It's normally a little bit easier to learn these things from Unix-like 
operating systems than from Windows.  What OS are you running Solr on?


Thanks,
Shawn

Re: Add dynamic field to existing index slow

[derrick cui]

Good point Erick, I will try it today, but I have already use cursorMark in my 
query for deep pagination.
Also I noticed that my cpu usage is pretty high, 8 cores, usage is over 700%. I 
am not sure it will help if I use ssd disk 


Sent from Yahoo Mail for iPhone


On Sunday, June 30, 2019, 2:57 PM, Erick Erickson  
wrote:

Well, the first thing I’d do is see what’s taking the time, querying or 
updating? Should be easy enough to comment out whatever it is that sends docs 
to Solr.

If it’s querying, it sounds like you’re paging through your entire data set and 
may be hitting the “deep paging” problem. Use cursorMark in that case.

Best,
Erick

> On Jun 30, 2019, at 9:12 AM, Alexandre Rafalovitch  wrote:
> 
> Only thing I can think of is to check whether you can do in-place
> rather than atomic updates:
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates
> But the conditions are quite restrictive: non-indexed
> (indexed="false"), non-stored (stored="false"), single valued
> (multiValued="false") numeric docValues (docValues="true") field
> 
> The other option may be to use an external value field and not update
> Solr documents at all:
> https://lucene.apache.org/solr/guide/8_1/working-with-external-files-and-processes.html
> 
> Regards,
>  Alex.
> 
> On Sun, 30 Jun 2019 at 10:53, derrick cui
>  wrote:
>> 
>> Thanks Alex,
>> My usage is that
>> 1. I execute query and get result, return id only 2. Add a value to a 
>> dynamic field3. Save to solr with batch size1000
>> I have define 50 queries and run them parallel Also I disable hard commit 
>> and soft commit per 1000 docs
>> 
>> I am wondering whether any configuration can speed it
>> 
>> 
>> 
>> 
>> Sent from Yahoo Mail for iPhone
>> 
>> 
>> On Sunday, June 30, 2019, 10:39 AM, Alexandre Rafalovitch 
>>  wrote:
>> 
>> Indexing new documents is just adding additional segments.
>> 
>> Adding new field to a document means:
>> 1) Reading existing document (may not always be possible, depending on
>> field configuration)
>> 2) Marking existing document as deleted
>> 3) Creating new document with reconstructed+plus new fields
>> 4) Possibly triggering a marge if a lot of documents have been updated
>> 
>> Perhaps the above is a contributing factor. But I also feel that maybe
>> there is some detail in your question I did not fully understand.
>> 
>> Regards,
>>  Alex.
>> 
>> On Sun, 30 Jun 2019 at 10:33, derrick cui
>>  wrote:
>>> 
>>> I have 400k data, indexing is pretty fast, only take 10 minutes, but add 
>>> dynamic field to all documents according to query results is very slow, 
>>> take about 1.5 hours.
>>> Anyone knows what could be the reason?
>>> Thanks
>>> 
>>> 
>>> 
>>> Sent from Yahoo Mail for iPhone
>> 
>> 
>> 

Re: Add dynamic field to existing index slow

[Erick Erickson]

Well, the first thing I’d do is see what’s taking the time, querying or 
updating? Should be easy enough to comment out whatever it is that sends docs 
to Solr.

If it’s querying, it sounds like you’re paging through your entire data set and 
may be hitting the “deep paging” problem. Use cursorMark in that case.

Best,
Erick

> On Jun 30, 2019, at 9:12 AM, Alexandre Rafalovitch  wrote:
> 
> Only thing I can think of is to check whether you can do in-place
> rather than atomic updates:
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates
> But the conditions are quite restrictive: non-indexed
> (indexed="false"), non-stored (stored="false"), single valued
> (multiValued="false") numeric docValues (docValues="true") field
> 
> The other option may be to use an external value field and not update
> Solr documents at all:
> https://lucene.apache.org/solr/guide/8_1/working-with-external-files-and-processes.html
> 
> Regards,
>   Alex.
> 
> On Sun, 30 Jun 2019 at 10:53, derrick cui
>  wrote:
>> 
>> Thanks Alex,
>> My usage is that
>> 1. I execute query and get result, return id only 2. Add a value to a 
>> dynamic field3. Save to solr with batch size1000
>> I have define 50 queries and run them parallel Also I disable hard commit 
>> and soft commit per 1000 docs
>> 
>> I am wondering whether any configuration can speed it
>> 
>> 
>> 
>> 
>> Sent from Yahoo Mail for iPhone
>> 
>> 
>> On Sunday, June 30, 2019, 10:39 AM, Alexandre Rafalovitch 
>>  wrote:
>> 
>> Indexing new documents is just adding additional segments.
>> 
>> Adding new field to a document means:
>> 1) Reading existing document (may not always be possible, depending on
>> field configuration)
>> 2) Marking existing document as deleted
>> 3) Creating new document with reconstructed+plus new fields
>> 4) Possibly triggering a marge if a lot of documents have been updated
>> 
>> Perhaps the above is a contributing factor. But I also feel that maybe
>> there is some detail in your question I did not fully understand.
>> 
>> Regards,
>>  Alex.
>> 
>> On Sun, 30 Jun 2019 at 10:33, derrick cui
>>  wrote:
>>> 
>>> I have 400k data, indexing is pretty fast, only take 10 minutes, but add 
>>> dynamic field to all documents according to query results is very slow, 
>>> take about 1.5 hours.
>>> Anyone knows what could be the reason?
>>> Thanks
>>> 
>>> 
>>> 
>>> Sent from Yahoo Mail for iPhone
>> 
>> 
>> 

Re: Add dynamic field to existing index slow

[Alexandre Rafalovitch]

Only thing I can think of is to check whether you can do in-place
rather than atomic updates:
https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates
But the conditions are quite restrictive: non-indexed
(indexed="false"), non-stored (stored="false"), single valued
(multiValued="false") numeric docValues (docValues="true") field

The other option may be to use an external value field and not update
Solr documents at all:
https://lucene.apache.org/solr/guide/8_1/working-with-external-files-and-processes.html

Regards,
   Alex.

On Sun, 30 Jun 2019 at 10:53, derrick cui
 wrote:
>
> Thanks Alex,
> My usage is that
> 1. I execute query and get result, return id only 2. Add a value to a dynamic 
> field3. Save to solr with batch size1000
> I have define 50 queries and run them parallel Also I disable hard commit and 
> soft commit per 1000 docs
>
> I am wondering whether any configuration can speed it
>
>
>
>
> Sent from Yahoo Mail for iPhone
>
>
> On Sunday, June 30, 2019, 10:39 AM, Alexandre Rafalovitch 
>  wrote:
>
> Indexing new documents is just adding additional segments.
>
> Adding new field to a document means:
> 1) Reading existing document (may not always be possible, depending on
> field configuration)
> 2) Marking existing document as deleted
> 3) Creating new document with reconstructed+plus new fields
> 4) Possibly triggering a marge if a lot of documents have been updated
>
> Perhaps the above is a contributing factor. But I also feel that maybe
> there is some detail in your question I did not fully understand.
>
> Regards,
>   Alex.
>
> On Sun, 30 Jun 2019 at 10:33, derrick cui
>  wrote:
> >
> > I have 400k data, indexing is pretty fast, only take 10 minutes, but add 
> > dynamic field to all documents according to query results is very slow, 
> > take about 1.5 hours.
> > Anyone knows what could be the reason?
> > Thanks
> >
> >
> >
> > Sent from Yahoo Mail for iPhone
>
>
>

Re: Add dynamic field to existing index slow

[derrick cui]

Thanks Alex,
My usage is that 
1. I execute query and get result, return id only 2. Add a value to a dynamic 
field3. Save to solr with batch size1000
I have define 50 queries and run them parallel Also I disable hard commit and 
soft commit per 1000 docs 

I am wondering whether any configuration can speed it




Sent from Yahoo Mail for iPhone


On Sunday, June 30, 2019, 10:39 AM, Alexandre Rafalovitch  
wrote:

Indexing new documents is just adding additional segments.

Adding new field to a document means:
1) Reading existing document (may not always be possible, depending on
field configuration)
2) Marking existing document as deleted
3) Creating new document with reconstructed+plus new fields
4) Possibly triggering a marge if a lot of documents have been updated

Perhaps the above is a contributing factor. But I also feel that maybe
there is some detail in your question I did not fully understand.

Regards,
  Alex.

On Sun, 30 Jun 2019 at 10:33, derrick cui
 wrote:
>
> I have 400k data, indexing is pretty fast, only take 10 minutes, but add 
> dynamic field to all documents according to query results is very slow, take 
> about 1.5 hours.
> Anyone knows what could be the reason?
> Thanks
>
>
>
> Sent from Yahoo Mail for iPhone

Re: Add dynamic field to existing index slow

[Alexandre Rafalovitch]

Indexing new documents is just adding additional segments.

Adding new field to a document means:
1) Reading existing document (may not always be possible, depending on
field configuration)
2) Marking existing document as deleted
3) Creating new document with reconstructed+plus new fields
4) Possibly triggering a marge if a lot of documents have been updated

Perhaps the above is a contributing factor. But I also feel that maybe
there is some detail in your question I did not fully understand.

Regards,
   Alex.

On Sun, 30 Jun 2019 at 10:33, derrick cui
 wrote:
>
> I have 400k data, indexing is pretty fast, only take 10 minutes, but add 
> dynamic field to all documents according to query results is very slow, take 
> about 1.5 hours.
> Anyone knows what could be the reason?
> Thanks
>
>
>
> Sent from Yahoo Mail for iPhone

Add dynamic field to existing index slow

[derrick cui]

I have 400k data, indexing is pretty fast, only take 10 minutes, but add 
dynamic field to all documents according to query results is very slow, take 
about 1.5 hours.
Anyone knows what could be the reason?
Thanks



Sent from Yahoo Mail for iPhone

Re: dynamic field issue

[Erick Erickson]

Dynamic fields are exactly the same as statically defined fields at the Lucene 
level, so this is exactly equivalent to “would defining N fields statically 
increase heap”.

IOW, no more than defining that many fields manually.

Best,
Erick

> On Feb 21, 2019, at 8:40 AM, Midas A  wrote:
> 
> Here we are indexing dynamic fields and  we are using one of this field in*
> bf *.
> Would only indexing dynamic field will increase heap and load of master -
> slave solr servers ?
> 
> 
> Regards,
> Midas
> 
> On Thu, Feb 21, 2019 at 10:03 PM Erick Erickson 
> wrote:
> 
>> 300 is still not excessive. As far as memory goes, sure. If you’re
>> faceting, grouping, or sorting docValues would _certainly_ help with memory
>> consumption.
>> 
>>> On Feb 21, 2019, at 8:31 AM, Midas A  wrote:
>>> 
>>> Hi ,
>>> Plelase help me here we have crossed 100+ fields per dyanmic fields and
>> we
>>> have three dynamic fields.
>>> using docValues in dynamic fields will help in improving heap and query
>>> time ?
>>> 
>>> Regards,
>>> Abhishek Tiwari
>>> 
>>> 
>>> On Thu, Feb 21, 2019 at 9:38 PM Midas A  wrote:
>>> 
>>>> Yes . We have crossed  100 fields .
>>>> 
>>>> Would docValues help here ?
>>>> 
>>>> What kind of information you want from my side ?
>>>> 
>>>> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, 
>>>> wrote:
>>>> 
>>>>> There’s no way to answer this given you’ve provided almost no
>>>>> information.
>>>>> 
>>>>> Do note that once you get to more than a few hundred fields,
>>>>> Solr still functions, but I’ve seen performance degrade and
>>>>> memory increase.
>>>>> 
>>>>> Best,
>>>>> Erick
>>>>> 
>>>>>> On Feb 21, 2019, at 7:54 AM, Midas A  wrote:
>>>>>> 
>>>>>> Thanks for quick reply .
>>>>>> 
>>>>>> we are creating  search *query(keyword)*  for dynamic field creation
>> to
>>>>>> use click ,cart  and order data  in search.
>>>>>> 
>>>>>> But we are experiencing  more heap and increase in query time .
>>>>>> What could be the problem? can be anything related to it ?
>>>>>> 
>>>>>> 
>>>>>> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey 
>>>>> wrote:
>>>>>> 
>>>>>>> On 2/21/2019 8:01 AM, Midas A wrote:
>>>>>>>> How many dynamic field we can create in solr ?. is there any
>>>>> limitation ?
>>>>>>>> Is indexing dynamic field can increase heap memory on server .
>>>>>>> 
>>>>>>> At the Lucene level, there is absolutely no difference between a
>>>>>>> standard field and a dynamic field.  The difference in Solr is how
>> the
>>>>>>> field is defined, nothing more.
>>>>>>> 
>>>>>>> Lucene has no hard limitations on the number of fields you can
>> create,
>>>>>>> but the more you have the larger your index will probably be.  Larger
>>>>>>> indexes perform slower than smaller ones and require more resources
>>>>> like
>>>>>>> memory.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Shawn
>>>>>>> 
>>>>> 
>>>>> 
>> 
>> 

Re: dynamic field issue

[Midas A]

Here we are indexing dynamic fields and  we are using one of this field in*
bf *.
Would only indexing dynamic field will increase heap and load of master -
slave solr servers ?


Regards,
Midas

On Thu, Feb 21, 2019 at 10:03 PM Erick Erickson 
wrote:

> 300 is still not excessive. As far as memory goes, sure. If you’re
> faceting, grouping, or sorting docValues would _certainly_ help with memory
> consumption.
>
> > On Feb 21, 2019, at 8:31 AM, Midas A  wrote:
> >
> > Hi ,
> > Plelase help me here we have crossed 100+ fields per dyanmic fields and
> we
> > have three dynamic fields.
> > using docValues in dynamic fields will help in improving heap and query
> > time ?
> >
> > Regards,
> > Abhishek Tiwari
> >
> >
> > On Thu, Feb 21, 2019 at 9:38 PM Midas A  wrote:
> >
> >> Yes . We have crossed  100 fields .
> >>
> >> Would docValues help here ?
> >>
> >> What kind of information you want from my side ?
> >>
> >> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, 
> >> wrote:
> >>
> >>> There’s no way to answer this given you’ve provided almost no
> >>> information.
> >>>
> >>> Do note that once you get to more than a few hundred fields,
> >>> Solr still functions, but I’ve seen performance degrade and
> >>> memory increase.
> >>>
> >>> Best,
> >>> Erick
> >>>
> >>>> On Feb 21, 2019, at 7:54 AM, Midas A  wrote:
> >>>>
> >>>> Thanks for quick reply .
> >>>>
> >>>> we are creating  search *query(keyword)*  for dynamic field creation
> to
> >>>> use click ,cart  and order data  in search.
> >>>>
> >>>> But we are experiencing  more heap and increase in query time .
> >>>> What could be the problem? can be anything related to it ?
> >>>>
> >>>>
> >>>> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey 
> >>> wrote:
> >>>>
> >>>>> On 2/21/2019 8:01 AM, Midas A wrote:
> >>>>>> How many dynamic field we can create in solr ?. is there any
> >>> limitation ?
> >>>>>> Is indexing dynamic field can increase heap memory on server .
> >>>>>
> >>>>> At the Lucene level, there is absolutely no difference between a
> >>>>> standard field and a dynamic field.  The difference in Solr is how
> the
> >>>>> field is defined, nothing more.
> >>>>>
> >>>>> Lucene has no hard limitations on the number of fields you can
> create,
> >>>>> but the more you have the larger your index will probably be.  Larger
> >>>>> indexes perform slower than smaller ones and require more resources
> >>> like
> >>>>> memory.
> >>>>>
> >>>>> Thanks,
> >>>>> Shawn
> >>>>>
> >>>
> >>>
>
>

Re: dynamic field issue

[Erick Erickson]

300 is still not excessive. As far as memory goes, sure. If you’re faceting, 
grouping, or sorting docValues would _certainly_ help with memory consumption.

> On Feb 21, 2019, at 8:31 AM, Midas A  wrote:
> 
> Hi ,
> Plelase help me here we have crossed 100+ fields per dyanmic fields and we
> have three dynamic fields.
> using docValues in dynamic fields will help in improving heap and query
> time ?
> 
> Regards,
> Abhishek Tiwari
> 
> 
> On Thu, Feb 21, 2019 at 9:38 PM Midas A  wrote:
> 
>> Yes . We have crossed  100 fields .
>> 
>> Would docValues help here ?
>> 
>> What kind of information you want from my side ?
>> 
>> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, 
>> wrote:
>> 
>>> There’s no way to answer this given you’ve provided almost no
>>> information.
>>> 
>>> Do note that once you get to more than a few hundred fields,
>>> Solr still functions, but I’ve seen performance degrade and
>>> memory increase.
>>> 
>>> Best,
>>> Erick
>>> 
>>>> On Feb 21, 2019, at 7:54 AM, Midas A  wrote:
>>>> 
>>>> Thanks for quick reply .
>>>> 
>>>> we are creating  search *query(keyword)*  for dynamic field creation  to
>>>> use click ,cart  and order data  in search.
>>>> 
>>>> But we are experiencing  more heap and increase in query time .
>>>> What could be the problem? can be anything related to it ?
>>>> 
>>>> 
>>>> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey 
>>> wrote:
>>>> 
>>>>> On 2/21/2019 8:01 AM, Midas A wrote:
>>>>>> How many dynamic field we can create in solr ?. is there any
>>> limitation ?
>>>>>> Is indexing dynamic field can increase heap memory on server .
>>>>> 
>>>>> At the Lucene level, there is absolutely no difference between a
>>>>> standard field and a dynamic field.  The difference in Solr is how the
>>>>> field is defined, nothing more.
>>>>> 
>>>>> Lucene has no hard limitations on the number of fields you can create,
>>>>> but the more you have the larger your index will probably be.  Larger
>>>>> indexes perform slower than smaller ones and require more resources
>>> like
>>>>> memory.
>>>>> 
>>>>> Thanks,
>>>>> Shawn
>>>>> 
>>> 
>>> 

Re: dynamic field issue

[Midas A]

Hi ,
Plelase help me here we have crossed 100+ fields per dyanmic fields and we
have three dynamic fields.
using docValues in dynamic fields will help in improving heap and query
time ?

Regards,
Abhishek Tiwari


On Thu, Feb 21, 2019 at 9:38 PM Midas A  wrote:

> Yes . We have crossed  100 fields .
>
> Would docValues help here ?
>
> What kind of information you want from my side ?
>
> On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, 
> wrote:
>
>> There’s no way to answer this given you’ve provided almost no
>> information.
>>
>> Do note that once you get to more than a few hundred fields,
>> Solr still functions, but I’ve seen performance degrade and
>> memory increase.
>>
>> Best,
>> Erick
>>
>> > On Feb 21, 2019, at 7:54 AM, Midas A  wrote:
>> >
>> > Thanks for quick reply .
>> >
>> > we are creating  search *query(keyword)*  for dynamic field creation  to
>> > use click ,cart  and order data  in search.
>> >
>> > But we are experiencing  more heap and increase in query time .
>> > What could be the problem? can be anything related to it ?
>> >
>> >
>> > On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey 
>> wrote:
>> >
>> >> On 2/21/2019 8:01 AM, Midas A wrote:
>> >>> How many dynamic field we can create in solr ?. is there any
>> limitation ?
>> >>> Is indexing dynamic field can increase heap memory on server .
>> >>
>> >> At the Lucene level, there is absolutely no difference between a
>> >> standard field and a dynamic field.  The difference in Solr is how the
>> >> field is defined, nothing more.
>> >>
>> >> Lucene has no hard limitations on the number of fields you can create,
>> >> but the more you have the larger your index will probably be.  Larger
>> >> indexes perform slower than smaller ones and require more resources
>> like
>> >> memory.
>> >>
>> >> Thanks,
>> >> Shawn
>> >>
>>
>>

Re: dynamic field issue

[Midas A]

Yes . We have crossed  100 fields .

Would docValues help here ?

What kind of information you want from my side ?

On Thu, 21 Feb 2019, 9:31 pm Erick Erickson, 
wrote:

> There’s no way to answer this given you’ve provided almost no
> information.
>
> Do note that once you get to more than a few hundred fields,
> Solr still functions, but I’ve seen performance degrade and
> memory increase.
>
> Best,
> Erick
>
> > On Feb 21, 2019, at 7:54 AM, Midas A  wrote:
> >
> > Thanks for quick reply .
> >
> > we are creating  search *query(keyword)*  for dynamic field creation  to
> > use click ,cart  and order data  in search.
> >
> > But we are experiencing  more heap and increase in query time .
> > What could be the problem? can be anything related to it ?
> >
> >
> > On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey 
> wrote:
> >
> >> On 2/21/2019 8:01 AM, Midas A wrote:
> >>> How many dynamic field we can create in solr ?. is there any
> limitation ?
> >>> Is indexing dynamic field can increase heap memory on server .
> >>
> >> At the Lucene level, there is absolutely no difference between a
> >> standard field and a dynamic field.  The difference in Solr is how the
> >> field is defined, nothing more.
> >>
> >> Lucene has no hard limitations on the number of fields you can create,
> >> but the more you have the larger your index will probably be.  Larger
> >> indexes perform slower than smaller ones and require more resources like
> >> memory.
> >>
> >> Thanks,
> >> Shawn
> >>
>
>

Re: dynamic field issue

[Erick Erickson]

There’s no way to answer this given you’ve provided almost no
information.

Do note that once you get to more than a few hundred fields,
Solr still functions, but I’ve seen performance degrade and
memory increase.

Best,
Erick

> On Feb 21, 2019, at 7:54 AM, Midas A  wrote:
> 
> Thanks for quick reply .
> 
> we are creating  search *query(keyword)*  for dynamic field creation  to
> use click ,cart  and order data  in search.
> 
> But we are experiencing  more heap and increase in query time .
> What could be the problem? can be anything related to it ?
> 
> 
> On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey  wrote:
> 
>> On 2/21/2019 8:01 AM, Midas A wrote:
>>> How many dynamic field we can create in solr ?. is there any limitation ?
>>> Is indexing dynamic field can increase heap memory on server .
>> 
>> At the Lucene level, there is absolutely no difference between a
>> standard field and a dynamic field.  The difference in Solr is how the
>> field is defined, nothing more.
>> 
>> Lucene has no hard limitations on the number of fields you can create,
>> but the more you have the larger your index will probably be.  Larger
>> indexes perform slower than smaller ones and require more resources like
>> memory.
>> 
>> Thanks,
>> Shawn
>> 

Re: dynamic field issue

[Midas A]

Thanks for quick reply .

we are creating  search *query(keyword)*  for dynamic field creation  to
use click ,cart  and order data  in search.

But we are experiencing  more heap and increase in query time .
What could be the problem? can be anything related to it ?


On Thu, Feb 21, 2019 at 8:43 PM Shawn Heisey  wrote:

> On 2/21/2019 8:01 AM, Midas A wrote:
> > How many dynamic field we can create in solr ?. is there any limitation ?
> > Is indexing dynamic field can increase heap memory on server .
>
> At the Lucene level, there is absolutely no difference between a
> standard field and a dynamic field.  The difference in Solr is how the
> field is defined, nothing more.
>
> Lucene has no hard limitations on the number of fields you can create,
> but the more you have the larger your index will probably be.  Larger
> indexes perform slower than smaller ones and require more resources like
> memory.
>
> Thanks,
> Shawn
>

dynamic field issue

[Midas A]

Hi All,

How many dynamic field we can create in solr ?. is there any limitation ?
Is indexing dynamic field can increase heap memory on server .


Regards,
Midas

Re: dynamic field issue

[Shawn Heisey]

On 2/21/2019 8:01 AM, Midas A wrote:

How many dynamic field we can create in solr ?. is there any limitation ?
Is indexing dynamic field can increase heap memory on server .


At the Lucene level, there is absolutely no difference between a 
standard field and a dynamic field.  The difference in Solr is how the 
field is defined, nothing more.


Lucene has no hard limitations on the number of fields you can create, 
but the more you have the larger your index will probably be.  Larger 
indexes perform slower than smaller ones and require more resources like 
memory.


Thanks,
Shawn

7.4.0 Newbie Question about bin/post and parsing/extracting html document parts into solr dynamic field.

[Bell, Bob]

Hi, 

New to solr, so forgive any missing info on my part.   

1. I am trying figure out how to get an html document html element 
parsed into a solr dynamic field.  Is it possible ?   So let's say I have some 
specific html tag or xml tags within the html 
document, that I created a  Dynamic field for, how do I specify that kind of 
parsing/extraction ?I can successfully
index the documents, and search finds the documents based on the indexing, but 
I would extract a specific set
of data from page, could be embedded xml tag, structured data, hidden fields, 
or specific html tag.   I am just not
sure how to accomplish this.Thank you for any ideas !! 

Thanks,
Bob
Austin Public Library

Re: Error when indexing against a specific dynamic field type

[Erick Erickson]

Steve's comment is much more germane. KeywordTokenizer,
used in alphaOnlySort last I knew is not appropriate at all.
Do you really want single tokens that consist of the entire
document for sorting purposes? Wouldn't the first 1K be enough?

It looks like this was put in in 4.0, so I'm guessing your analysis chain
is different now between the two versions.

It doesn't really matter though, this is not going to be changed.
You'll have to do something about your long fields or your
analysis chain. And/or revisit what you hope to accomplish
with using that field type on such a field, I'm almost certain
your use case is flawed.

Best,
Erick




On Tue, May 1, 2018 at 10:35 AM, THADC
 wrote:
> Erick, thanks for the response. I have a number of documents in our database
> where solr is throwing the same exception against *_tsing types.
>
> However, when I index against the same document with our solr 4.7, it is
> successfully indexed. So, I assume something is different between 4.7 and
> 7.3. I was assuming I could adjust the dynamic field somehow so that it
> indexes against these documents without errors when using 7.3.
>
> I can't remove the offending documents. Its my customer's data
>
> Is there some adjustment I can make to the dynamic field?
>
> Thanks again.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Error when indexing against a specific dynamic field type

[THADC]

Erick, thanks for the response. I have a number of documents in our database
where solr is throwing the same exception against *_tsing types.

However, when I index against the same document with our solr 4.7, it is
successfully indexed. So, I assume something is different between 4.7 and
7.3. I was assuming I could adjust the dynamic field somehow so that it
indexes against these documents without errors when using 7.3.

I can't remove the offending documents. Its my customer's data 

Is there some adjustment I can make to the dynamic field?

Thanks again.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Error when indexing against a specific dynamic field type

[Shawn Heisey]

On 5/1/2018 8:40 AM, THADC wrote:
> I get the following exception:
>
> *Exception writing document id FULL_36265 to the index; possible analysis
> error: Document contains at least one immense term in
> field="gridFacts_tsing" (whose UTF8 encoding is longer than the max length
> 32766), all of which were skipped.  Please correct the analyzer to not
> produce such terms.  The prefix of the first immense term is: '[108, 111,
> 114, 101, 109, 32, 105, 112, 115, 117, 109, 32, 100, 111, 108, 111, 114, 32,
> 115, 105, 116, 32, 97, 109, 101, 116, 44, 32, 99, 111]...', original
> message: bytes can be at most 32766 in length; got 68144.*
>
> Any ideas are greatly appreciated. Thank you.

The error is not ambiguous.  It tells you precisely what the problem
is.  A single term in a Lucene index cannot be longer than about 32K,
that one has a term that's more than twice that size.

I'm guessing that the fieldType named alphaOnlySort is one of two
things:  Either the StrField class, or the TextField class with the
keyword tokenizer factory.

To fix this problem you will need to either reduce the size of the input
on the field, or use an analysis chain that splits the input into
smaller tokens.  It appears that the input string is comma separated
numbers, which probably should be tokenized, not treated as a single term.

Thanks,
Shawn

Re: Error when indexing against a specific dynamic field type

[Steve Rowe]

The input in the error message starts “lorem ipsum”, so it contains spaces, but 
the alphaOnlySort field type (in Solr’s example schemas anyway) uses 
KeywordTokenizer, which tokenizes the entire input as a single token.

As Erick implied, you maybe should not be doing that with this kind of data - 
perhaps the analyzer used by this dynamic field should change?

Alternatively, you could:

a) truncate long values so that a prefix makes it through the indexing process, 
e.g. by adding TruncateTokenFilterFactory[1] to alphaOnlySort’s analyzer, or by 
adding TruncateFieldUpdateProcessorFactory[2] to your update request processor 
chain; or

b) entirely eliminate overly long values, e.g. using LengthFilterFactory[3].

[1] 
https://lucene.apache.org/core/7_3_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/TruncateTokenFilterFactory.html
[2] 
https://lucene.apache.org/solr/7_3_0/solr-core/org/apache/solr/update/processor/TruncateFieldUpdateProcessorFactory.html
[3] 
https://lucene.apache.org/solr/guide/7_3/filter-descriptions.html#length-filter

--
Steve
www.lucidworks.com

> On May 1, 2018, at 11:28 AM, Erick Erickson  wrote:
> 
> You're sending it a huge term. My guess is you're sending something
> like base64-encoded data or perhaps just a single unbroken string in
> your field.
> 
> Examine your document, it should jump out at you.
> 
> Best,
> Erick
> 
> On Tue, May 1, 2018 at 7:40 AM, THADC  
> wrote:
>> Hello,
>> 
>> We are migrating from solr 4.7 to 7.3. When I encounter a data item that
>> matches a custom dynamic field from our 4.7 schema:
>> 
>> *> stored="true" multiValued="false"/>*
>> 
>> , I get the following exception:
>> 
>> *Exception writing document id FULL_36265 to the index; possible analysis
>> error: Document contains at least one immense term in
>> field="gridFacts_tsing" (whose UTF8 encoding is longer than the max length
>> 32766), all of which were skipped.  Please correct the analyzer to not
>> produce such terms.  The prefix of the first immense term is: '[108, 111,
>> 114, 101, 109, 32, 105, 112, 115, 117, 109, 32, 100, 111, 108, 111, 114, 32,
>> 115, 105, 116, 32, 97, 109, 101, 116, 44, 32, 99, 111]...', original
>> message: bytes can be at most 32766 in length; got 68144.*
>> 
>> Any ideas are greatly appreciated. Thank you.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Error when indexing against a specific dynamic field type

[Erick Erickson]

You're sending it a huge term. My guess is you're sending something
like base64-encoded data or perhaps just a single unbroken string in
your field.

Examine your document, it should jump out at you.

Best,
Erick

On Tue, May 1, 2018 at 7:40 AM, THADC  wrote:
> Hello,
>
> We are migrating from solr 4.7 to 7.3. When I encounter a data item that
> matches a custom dynamic field from our 4.7 schema:
>
> * stored="true" multiValued="false"/>*
>
> , I get the following exception:
>
> *Exception writing document id FULL_36265 to the index; possible analysis
> error: Document contains at least one immense term in
> field="gridFacts_tsing" (whose UTF8 encoding is longer than the max length
> 32766), all of which were skipped.  Please correct the analyzer to not
> produce such terms.  The prefix of the first immense term is: '[108, 111,
> 114, 101, 109, 32, 105, 112, 115, 117, 109, 32, 100, 111, 108, 111, 114, 32,
> 115, 105, 116, 32, 97, 109, 101, 116, 44, 32, 99, 111]...', original
> message: bytes can be at most 32766 in length; got 68144.*
>
> Any ideas are greatly appreciated. Thank you.
>
>
>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Error when indexing against a specific dynamic field type

[THADC]

Hello,

We are migrating from solr 4.7 to 7.3. When I encounter a data item that
matches a custom dynamic field from our 4.7 schema:

**

, I get the following exception:

*Exception writing document id FULL_36265 to the index; possible analysis
error: Document contains at least one immense term in
field="gridFacts_tsing" (whose UTF8 encoding is longer than the max length
32766), all of which were skipped.  Please correct the analyzer to not
produce such terms.  The prefix of the first immense term is: '[108, 111,
114, 101, 109, 32, 105, 112, 115, 117, 109, 32, 100, 111, 108, 111, 114, 32,
115, 105, 116, 32, 97, 109, 101, 116, 44, 32, 99, 111]...', original
message: bytes can be at most 32766 in length; got 68144.*

Any ideas are greatly appreciated. Thank you.







--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: How to get field names of dynamic field

[Alexandre Rafalovitch]

Ugh.

Ok, so if the list is different per-request, you should probably look
at writing a custom search component if you want to keep it all in
Solr layer.

But really, you have middle-ware, right? Why don't you formulate your
query with all the business requirement specifics in there.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 17 April 2017 at 11:19, Midas A  wrote:
> Here if i am saying dynamic say *by_* *is the  field then by_color, by_size
> .. etc . would be the dynamic fields .
>
> Regards,
> Abhishek Tiwari
>
> On Mon, Apr 17, 2017 at 1:47 PM, Midas A  wrote:
>
>> Thanks alex,
>>
>> I want all dynamic fields related to a query( i.e. category_id : 199
>>  where 199 category has around 1 docs) .
>>
>> How we can get this with help of luke Request handler.
>>
>> On Mon, Apr 17, 2017 at 1:39 PM, Alexandre Rafalovitch > > wrote:
>>
>>> You could externalize that by periodically:
>>> 1) Running a luke query or Schema API to get the list of fields
>>> 2) Running Request Parameter API to update the list of field to return
>>> (this does not cause core reload)
>>> 3) If you have permanent field list on top of dynamic ones, you could
>>> use parameter substitution as well to combine them
>>> 4) if you really, really need to be in sync, you could use commit hook
>>> to trigger this
>>>
>>> Regards,
>>>Alex.
>>> 
>>> http://www.solr-start.com/ - Resources for Solr users, new and
>>> experienced
>>>
>>>
>>> On 17 April 2017 at 11:00, Midas A  wrote:
>>> > Can we do faceting for all fields created by dynamically created fields
>>> > rather sending explicitly sending it by query
>>> > will facet.fields=*by_* *
>>> > work or any other alternate ??
>>> >
>>> > I was thinking that i will get all dynamic fields with help of Luke
>>> handler
>>> > . But probably it is not possible through luke for a query
>>> >
>>> >
>>> >
>>> > On Fri, Apr 14, 2017 at 5:32 PM, Ahmet Arslan >> >
>>> > wrote:
>>> >
>>> >> Hi Midas,
>>> >>
>>> >> LukeRequestHandler shows that information.
>>> >>
>>> >> Ahmet
>>> >>
>>> >> On Friday, April 14, 2017, 1:16:09 PM GMT+3, Midas A <
>>> test.mi...@gmail.com>
>>> >> wrote:
>>> >> Actually , i am looking for APi
>>> >>
>>> >> On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini 
>>> >> wrote:
>>> >>
>>> >> > I can see those names in the "Schema  browser" of the admin UI, so I
>>> >> guess
>>> >> > using the (lucene?) API it shouldn't be hard to get this info.
>>> >> >
>>> >> > I don' know if the schema api (or some other service) offer this
>>> service
>>> >> >
>>> >> > Andrea
>>> >> >
>>> >> > On 14 Apr 2017 10:03, "Midas A"  wrote:
>>> >> >
>>> >> > > Hi,
>>> >> > >
>>> >> > >
>>> >> > > Can i get all the field created for dynamic field in solr .
>>> >> > >
>>> >> > > Like
>>> >> > > my dynamic field is by_*
>>> >> > >
>>> >> > > and i have index
>>> >> > > by_color
>>> >> > > by_size ..
>>> >> > > etc
>>> >> > >
>>> >> > > I want to retrieve all these field name .
>>> >> > > Is there any way to do this  based on some query
>>> >> > >
>>> >> >
>>> >>
>>>
>>
>>

Re: How to get field names of dynamic field

[Midas A]

Here if i am saying dynamic say *by_* *is the  field then by_color, by_size
.. etc . would be the dynamic fields .

Regards,
Abhishek Tiwari

On Mon, Apr 17, 2017 at 1:47 PM, Midas A  wrote:

> Thanks alex,
>
> I want all dynamic fields related to a query( i.e. category_id : 199
>  where 199 category has around 1 docs) .
>
> How we can get this with help of luke Request handler.
>
> On Mon, Apr 17, 2017 at 1:39 PM, Alexandre Rafalovitch  > wrote:
>
>> You could externalize that by periodically:
>> 1) Running a luke query or Schema API to get the list of fields
>> 2) Running Request Parameter API to update the list of field to return
>> (this does not cause core reload)
>> 3) If you have permanent field list on top of dynamic ones, you could
>> use parameter substitution as well to combine them
>> 4) if you really, really need to be in sync, you could use commit hook
>> to trigger this
>>
>> Regards,
>>Alex.
>> 
>> http://www.solr-start.com/ - Resources for Solr users, new and
>> experienced
>>
>>
>> On 17 April 2017 at 11:00, Midas A  wrote:
>> > Can we do faceting for all fields created by dynamically created fields
>> > rather sending explicitly sending it by query
>> > will facet.fields=*by_* *
>> > work or any other alternate ??
>> >
>> > I was thinking that i will get all dynamic fields with help of Luke
>> handler
>> > . But probably it is not possible through luke for a query
>> >
>> >
>> >
>> > On Fri, Apr 14, 2017 at 5:32 PM, Ahmet Arslan > >
>> > wrote:
>> >
>> >> Hi Midas,
>> >>
>> >> LukeRequestHandler shows that information.
>> >>
>> >> Ahmet
>> >>
>> >> On Friday, April 14, 2017, 1:16:09 PM GMT+3, Midas A <
>> test.mi...@gmail.com>
>> >> wrote:
>> >> Actually , i am looking for APi
>> >>
>> >> On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini 
>> >> wrote:
>> >>
>> >> > I can see those names in the "Schema  browser" of the admin UI, so I
>> >> guess
>> >> > using the (lucene?) API it shouldn't be hard to get this info.
>> >> >
>> >> > I don' know if the schema api (or some other service) offer this
>> service
>> >> >
>> >> > Andrea
>> >> >
>> >> > On 14 Apr 2017 10:03, "Midas A"  wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > >
>> >> > > Can i get all the field created for dynamic field in solr .
>> >> > >
>> >> > > Like
>> >> > > my dynamic field is by_*
>> >> > >
>> >> > > and i have index
>> >> > > by_color
>> >> > > by_size ..
>> >> > > etc
>> >> > >
>> >> > > I want to retrieve all these field name .
>> >> > > Is there any way to do this  based on some query
>> >> > >
>> >> >
>> >>
>>
>
>

Re: How to get field names of dynamic field

[Midas A]

Thanks alex,

I want all dynamic fields related to a query( i.e. category_id : 199  where
199 category has around 1 docs) .

How we can get this with help of luke Request handler.

On Mon, Apr 17, 2017 at 1:39 PM, Alexandre Rafalovitch 
wrote:

> You could externalize that by periodically:
> 1) Running a luke query or Schema API to get the list of fields
> 2) Running Request Parameter API to update the list of field to return
> (this does not cause core reload)
> 3) If you have permanent field list on top of dynamic ones, you could
> use parameter substitution as well to combine them
> 4) if you really, really need to be in sync, you could use commit hook
> to trigger this
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 17 April 2017 at 11:00, Midas A  wrote:
> > Can we do faceting for all fields created by dynamically created fields
> > rather sending explicitly sending it by query
> > will facet.fields=*by_* *
> > work or any other alternate ??
> >
> > I was thinking that i will get all dynamic fields with help of Luke
> handler
> > . But probably it is not possible through luke for a query
> >
> >
> >
> > On Fri, Apr 14, 2017 at 5:32 PM, Ahmet Arslan  >
> > wrote:
> >
> >> Hi Midas,
> >>
> >> LukeRequestHandler shows that information.
> >>
> >> Ahmet
> >>
> >> On Friday, April 14, 2017, 1:16:09 PM GMT+3, Midas A <
> test.mi...@gmail.com>
> >> wrote:
> >> Actually , i am looking for APi
> >>
> >> On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini 
> >> wrote:
> >>
> >> > I can see those names in the "Schema  browser" of the admin UI, so I
> >> guess
> >> > using the (lucene?) API it shouldn't be hard to get this info.
> >> >
> >> > I don' know if the schema api (or some other service) offer this
> service
> >> >
> >> > Andrea
> >> >
> >> > On 14 Apr 2017 10:03, "Midas A"  wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > >
> >> > > Can i get all the field created for dynamic field in solr .
> >> > >
> >> > > Like
> >> > > my dynamic field is by_*
> >> > >
> >> > > and i have index
> >> > > by_color
> >> > > by_size ..
> >> > > etc
> >> > >
> >> > > I want to retrieve all these field name .
> >> > > Is there any way to do this  based on some query
> >> > >
> >> >
> >>
>

Re: How to get field names of dynamic field

[Alexandre Rafalovitch]

You could externalize that by periodically:
1) Running a luke query or Schema API to get the list of fields
2) Running Request Parameter API to update the list of field to return
(this does not cause core reload)
3) If you have permanent field list on top of dynamic ones, you could
use parameter substitution as well to combine them
4) if you really, really need to be in sync, you could use commit hook
to trigger this

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 17 April 2017 at 11:00, Midas A  wrote:
> Can we do faceting for all fields created by dynamically created fields
> rather sending explicitly sending it by query
> will facet.fields=*by_* *
> work or any other alternate ??
>
> I was thinking that i will get all dynamic fields with help of Luke handler
> . But probably it is not possible through luke for a query
>
>
>
> On Fri, Apr 14, 2017 at 5:32 PM, Ahmet Arslan 
> wrote:
>
>> Hi Midas,
>>
>> LukeRequestHandler shows that information.
>>
>> Ahmet
>>
>> On Friday, April 14, 2017, 1:16:09 PM GMT+3, Midas A 
>> wrote:
>> Actually , i am looking for APi
>>
>> On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini 
>> wrote:
>>
>> > I can see those names in the "Schema  browser" of the admin UI, so I
>> guess
>> > using the (lucene?) API it shouldn't be hard to get this info.
>> >
>> > I don' know if the schema api (or some other service) offer this service
>> >
>> > Andrea
>> >
>> > On 14 Apr 2017 10:03, "Midas A"  wrote:
>> >
>> > > Hi,
>> > >
>> > >
>> > > Can i get all the field created for dynamic field in solr .
>> > >
>> > > Like
>> > > my dynamic field is by_*
>> > >
>> > > and i have index
>> > > by_color
>> > > by_size ..
>> > > etc
>> > >
>> > > I want to retrieve all these field name .
>> > > Is there any way to do this  based on some query
>> > >
>> >
>>

Re: How to get field names of dynamic field

[Midas A]

Can we do faceting for all fields created by dynamically created fields
rather sending explicitly sending it by query
will facet.fields=*by_* *
work or any other alternate ??

I was thinking that i will get all dynamic fields with help of Luke handler
. But probably it is not possible through luke for a query



On Fri, Apr 14, 2017 at 5:32 PM, Ahmet Arslan 
wrote:

> Hi Midas,
>
> LukeRequestHandler shows that information.
>
> Ahmet
>
> On Friday, April 14, 2017, 1:16:09 PM GMT+3, Midas A 
> wrote:
> Actually , i am looking for APi
>
> On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini 
> wrote:
>
> > I can see those names in the "Schema  browser" of the admin UI, so I
> guess
> > using the (lucene?) API it shouldn't be hard to get this info.
> >
> > I don' know if the schema api (or some other service) offer this service
> >
> > Andrea
> >
> > On 14 Apr 2017 10:03, "Midas A"  wrote:
> >
> > > Hi,
> > >
> > >
> > > Can i get all the field created for dynamic field in solr .
> > >
> > > Like
> > > my dynamic field is by_*
> > >
> > > and i have index
> > > by_color
> > > by_size ..
> > > etc
> > >
> > > I want to retrieve all these field name .
> > > Is there any way to do this  based on some query
> > >
> >
>

Re: How to get field names of dynamic field

[Ahmet Arslan]

Hi Midas,

LukeRequestHandler shows that information.

Ahmet

On Friday, April 14, 2017, 1:16:09 PM GMT+3, Midas A  
wrote:
Actually , i am looking for APi

On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini  wrote:

> I can see those names in the "Schema  browser" of the admin UI, so I guess
> using the (lucene?) API it shouldn't be hard to get this info.
>
> I don' know if the schema api (or some other service) offer this service
>
> Andrea
>
> On 14 Apr 2017 10:03, "Midas A"  wrote:
>
> > Hi,
> >
> >
> > Can i get all the field created for dynamic field in solr .
> >
> > Like
> > my dynamic field is by_*
> >
> > and i have index
> > by_color
> > by_size ..
> > etc
> >
> > I want to retrieve all these field name .
> > Is there any way to do this  based on some query
> >
>

Re: How to get field names of dynamic field

[Midas A]

Actually , i am looking for APi

On Fri, Apr 14, 2017 at 3:36 PM, Andrea Gazzarini  wrote:

> I can see those names in the "Schema  browser" of the admin UI, so I guess
> using the (lucene?) API it shouldn't be hard to get this info.
>
> I don' know if the schema api (or some other service) offer this service
>
> Andrea
>
> On 14 Apr 2017 10:03, "Midas A"  wrote:
>
> > Hi,
> >
> >
> > Can i get all the field created for dynamic field in solr .
> >
> > Like
> > my dynamic field is by_*
> >
> > and i have index
> > by_color
> > by_size ..
> > etc
> >
> > I want to retrieve all these field name .
> > Is there any way to do this  based on some query
> >
>

Re: How to get field names of dynamic field

[Andrea Gazzarini]

I can see those names in the "Schema  browser" of the admin UI, so I guess
using the (lucene?) API it shouldn't be hard to get this info.

I don' know if the schema api (or some other service) offer this service

Andrea

On 14 Apr 2017 10:03, "Midas A"  wrote:

> Hi,
>
>
> Can i get all the field created for dynamic field in solr .
>
> Like
> my dynamic field is by_*
>
> and i have index
> by_color
> by_size ..
> etc
>
> I want to retrieve all these field name .
> Is there any way to do this  based on some query
>

How to get field names of dynamic field

[Midas A]

Hi,


Can i get all the field created for dynamic field in solr .

Like
my dynamic field is by_*

and i have index
by_color
by_size ..
etc

I want to retrieve all these field name .
Is there any way to do this  based on some query

Re: dynamic field sorting

[Mikhail Khludnev]

Since it hits heap, moving to docValues might make sense.

On Wed, Mar 22, 2017 at 7:47 AM, Midas A  wrote:

> waiting for reply . Actually Heap utilization increases when we sort with
> dynamic fields
>
> On Tue, Mar 21, 2017 at 10:37 AM, Midas A  wrote:
>
> > Hi ,
> >
> > How can i improve the performance of dynamic field sorting .
> >
> > index size is : 20 GB
> >
> > Regards,
> > Midas
> >
>



-- 
Sincerely yours
Mikhail Khludnev

Re: dynamic field sorting

[Midas A]

waiting for reply . Actually Heap utilization increases when we sort with
dynamic fields

On Tue, Mar 21, 2017 at 10:37 AM, Midas A  wrote:

> Hi ,
>
> How can i improve the performance of dynamic field sorting .
>
> index size is : 20 GB
>
> Regards,
> Midas
>

dynamic field sorting

[Midas A]

Hi ,

How can i improve the performance of dynamic field sorting .

index size is : 20 GB

Regards,
Midas

Re: Fwd: Solr dynamic field blowing up the index size

[Pratik Patel]

I think I have found something concrete. Reading up more on nvd file
extension, I found that it is being used to store length and boost factors
for documents and fields. These are normalization files. Normalization on a
field is controlled by omitNorms attribute. If omitNorms=true then the
field will not be normalized. I explicitly added omitNorms=true for the
field type text_general and re-indexed the data. Now, my index size is much
smaller. I haven't yet verified this with complete data set yet but I can
see that index size is reduced. We have a large data set and it takes about
5-6 hours to index it completely so I'll index the whole data set overnight
to confirm the fix.

But now I am curious about omitNorms attribute. What would be the default
value for omitNorms for field type "text_general". The documentation says
that omitNorms=true for primitive field types like string, int etc. but I
don't know what is the default value for "text_general"?

I never had omitNorms set explicitly on text_general field type or any of
the fields having type text_general. Has the default value of omitNorms
been changed from solr 5.0.0 to 6.4.1?

Any clarification on this would be really helpful.

I am posting some relevant links here for someone who might face similar
issue in future.

http://apprize.info/php/solr_4/2.html
http://stackoverflow.com/questions/18694242/what-is-omitnorms-and-version-field-in-solr-schema
https://lucidworks.com/2009/09/02/scaling-lucene-and-solr/#d0e71

Thanks,
Pratik

On Tue, Feb 21, 2017 at 12:03 PM, Pratik Patel  wrote:

> I am using the schema from solr 5 which does not have any field with
> docValues enabled.In fact to ensure that everything is same as solr 5
> (except the breaking changes) I am using the solrconfig.xml also from solr
> 5 with schemaFactory set as classicSchemaFactory to use schema.xml from
> solr 5.
>
>
> On Tue, Feb 21, 2017 at 11:33 AM, Alexandre Rafalovitch <
> arafa...@gmail.com> wrote:
>
>> Did you reuse the schema or rebuilt it on top of the latest examples?
>> Because the latest example schema enabled docValues for strings on the
>> fieldType level.
>>
>> I would do a diff of the schemas to see what changed. If they look
>> very different and you are looking for tools to normalize/extract
>> elements from schemas, you may find my latest Revolution presentation
>> useful for that:
>> https://www.slideshare.net/arafalov/rebuilding-solr-6-exampl
>> es-layer-by-layer-lucenesolrrevolution-2016
>> (e.g. slide 20). There is also the video there at the end.
>>
>> Regards,
>>Alex.
>> 
>> http://www.solr-start.com/ - Resources for Solr users, new and
>> experienced
>>
>>
>> On 21 February 2017 at 11:18, Mike Thomsen 
>> wrote:
>> > Correct me if I'm wrong, but heavy use of doc values should actually
>> blow
>> > up the size of your index considerably if they are in fields that get
>> sent
>> > a lot of data.
>> >
>> > On Tue, Feb 21, 2017 at 10:50 AM, Pratik Patel 
>> wrote:
>> >
>> >> Thanks for the reply. I can see that in solr 6, more than 50% of the
>> index
>> >> directory is occupied by ".nvd" file extension. It is something
>> related to
>> >> norms and doc values.
>> >>
>> >> On Tue, Feb 21, 2017 at 10:27 AM, Alexandre Rafalovitch <
>> >> arafa...@gmail.com>
>> >> wrote:
>> >>
>> >> > Did you look in the data directories to check what index file
>> extensions
>> >> > contribute most to the difference? That could give a hint.
>> >> >
>> >> > Regards,
>> >> > Alex
>> >> >
>> >> > On 21 Feb 2017 9:47 AM, "Pratik Patel"  wrote:
>> >> >
>> >> > > Here is the same question in stackOverflow for better format.
>> >> > >
>> >> > > http://stackoverflow.com/questions/42370231/solr-
>> >> > > dynamic-field-blowing-up-the-index-size
>> >> > >
>> >> > > Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app
>> fine
>> >> > but
>> >> > > the problem is that index size with solr 6 is way too large. In
>> solr 5,
>> >> > > index size was about 15GB and in solr 6, for the same data, the
>> index
>> >> > size
>> >> > > is 300GB! I am not able to understand what contributes to such huge
>> >> > > difference in solr 6.
>> >> > >
>> >> > > I have been able to identify

Re: Fwd: Solr dynamic field blowing up the index size

[Pratik Patel]

I am using the schema from solr 5 which does not have any field with
docValues enabled.In fact to ensure that everything is same as solr 5
(except the breaking changes) I am using the solrconfig.xml also from solr
5 with schemaFactory set as classicSchemaFactory to use schema.xml from
solr 5.

On Tue, Feb 21, 2017 at 11:33 AM, Alexandre Rafalovitch 
wrote:

> Did you reuse the schema or rebuilt it on top of the latest examples?
> Because the latest example schema enabled docValues for strings on the
> fieldType level.
>
> I would do a diff of the schemas to see what changed. If they look
> very different and you are looking for tools to normalize/extract
> elements from schemas, you may find my latest Revolution presentation
> useful for that:
> https://www.slideshare.net/arafalov/rebuilding-solr-6-
> examples-layer-by-layer-lucenesolrrevolution-2016
> (e.g. slide 20). There is also the video there at the end.
>
> Regards,
>Alex.
> 
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 21 February 2017 at 11:18, Mike Thomsen  wrote:
> > Correct me if I'm wrong, but heavy use of doc values should actually blow
> > up the size of your index considerably if they are in fields that get
> sent
> > a lot of data.
> >
> > On Tue, Feb 21, 2017 at 10:50 AM, Pratik Patel 
> wrote:
> >
> >> Thanks for the reply. I can see that in solr 6, more than 50% of the
> index
> >> directory is occupied by ".nvd" file extension. It is something related
> to
> >> norms and doc values.
> >>
> >> On Tue, Feb 21, 2017 at 10:27 AM, Alexandre Rafalovitch <
> >> arafa...@gmail.com>
> >> wrote:
> >>
> >> > Did you look in the data directories to check what index file
> extensions
> >> > contribute most to the difference? That could give a hint.
> >> >
> >> > Regards,
> >> > Alex
> >> >
> >> > On 21 Feb 2017 9:47 AM, "Pratik Patel"  wrote:
> >> >
> >> > > Here is the same question in stackOverflow for better format.
> >> > >
> >> > > http://stackoverflow.com/questions/42370231/solr-
> >> > > dynamic-field-blowing-up-the-index-size
> >> > >
> >> > > Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app
> fine
> >> > but
> >> > > the problem is that index size with solr 6 is way too large. In
> solr 5,
> >> > > index size was about 15GB and in solr 6, for the same data, the
> index
> >> > size
> >> > > is 300GB! I am not able to understand what contributes to such huge
> >> > > difference in solr 6.
> >> > >
> >> > > I have been able to identify a field which is blowing up the size of
> >> > index.
> >> > > It is as follows.
> >> > >
> >> > >  >> > > stored="true" multiValued="true"  />
> >> > >
> >> > >  >> > > stored="false" multiValued="true"  />
> >> > > 
> >> > >
> >> > > When this field is commented out, the index size reduces to less
> than
> >> > 10GB.
> >> > >
> >> > > This field is of type text_general. Following is the definition of
> this
> >> > > type.
> >> > >
> >> > >  >> > > positionIncrementGap="100">
> >> > >   
> >> > > 
> >> > > 
> >> > > 
> >> > >  >> > > pattern="((?m)[a-z]+)'s" replacement="$1s" />
> >> > >  >> > > protected="protwords.txt" generateWordParts="1"
> >> > > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> >> > > catenateAll="0" splitOnCaseChange="0"/>
> >> > > 
> >> > >  >> > > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> >> > > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> >> > > />
> >> > >   
> >> > >   
> >> > > 
> >> > > 
> >> > > 
> >> > >  >> > > pattern="((?m)[a-z]+)'s" replacement="$1s" />
> >> > >   

Re: Fwd: Solr dynamic field blowing up the index size

[Alexandre Rafalovitch]

Did you reuse the schema or rebuilt it on top of the latest examples?
Because the latest example schema enabled docValues for strings on the
fieldType level.

I would do a diff of the schemas to see what changed. If they look
very different and you are looking for tools to normalize/extract
elements from schemas, you may find my latest Revolution presentation
useful for that:
https://www.slideshare.net/arafalov/rebuilding-solr-6-examples-layer-by-layer-lucenesolrrevolution-2016
(e.g. slide 20). There is also the video there at the end.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 21 February 2017 at 11:18, Mike Thomsen  wrote:
> Correct me if I'm wrong, but heavy use of doc values should actually blow
> up the size of your index considerably if they are in fields that get sent
> a lot of data.
>
> On Tue, Feb 21, 2017 at 10:50 AM, Pratik Patel  wrote:
>
>> Thanks for the reply. I can see that in solr 6, more than 50% of the index
>> directory is occupied by ".nvd" file extension. It is something related to
>> norms and doc values.
>>
>> On Tue, Feb 21, 2017 at 10:27 AM, Alexandre Rafalovitch <
>> arafa...@gmail.com>
>> wrote:
>>
>> > Did you look in the data directories to check what index file extensions
>> > contribute most to the difference? That could give a hint.
>> >
>> > Regards,
>> > Alex
>> >
>> > On 21 Feb 2017 9:47 AM, "Pratik Patel"  wrote:
>> >
>> > > Here is the same question in stackOverflow for better format.
>> > >
>> > > http://stackoverflow.com/questions/42370231/solr-
>> > > dynamic-field-blowing-up-the-index-size
>> > >
>> > > Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app fine
>> > but
>> > > the problem is that index size with solr 6 is way too large. In solr 5,
>> > > index size was about 15GB and in solr 6, for the same data, the index
>> > size
>> > > is 300GB! I am not able to understand what contributes to such huge
>> > > difference in solr 6.
>> > >
>> > > I have been able to identify a field which is blowing up the size of
>> > index.
>> > > It is as follows.
>> > >
>> > > > > > stored="true" multiValued="true"  />
>> > >
>> > > > > > stored="false" multiValued="true"  />
>> > > 
>> > >
>> > > When this field is commented out, the index size reduces to less than
>> > 10GB.
>> > >
>> > > This field is of type text_general. Following is the definition of this
>> > > type.
>> > >
>> > > > > > positionIncrementGap="100">
>> > >   
>> > > 
>> > > 
>> > > 
>> > > > > > pattern="((?m)[a-z]+)'s" replacement="$1s" />
>> > > > > > protected="protwords.txt" generateWordParts="1"
>> > > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>> > > catenateAll="0" splitOnCaseChange="0"/>
>> > > 
>> > > > > > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
>> > > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
>> > > />
>> > >   
>> > >   
>> > > 
>> > > 
>> > > 
>> > > > > > pattern="((?m)[a-z]+)'s" replacement="$1s" />
>> > > > > > protected="protwords.txt" generateWordParts="1"
>> > > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>> > > catenateAll="0" splitOnCaseChange="0"/>
>> > > 
>> > > > > > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
>> > > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
>> > > />
>> > >   
>> > >   
>> > >
>> > > Few things which I did to debug this issue:
>> > >
>> > >- I have ensured that field type definition is same as what I was
>> > using
>> > >in solr 5 and it is also valid in version 6. This field type
>> > considers a
>> > >list of "stopwords" to be ignored during indexing. I have supplied
>> the
>> > > same
>> > >list of stopwords which we were using in solr 5. I have verified
>> that
>> > > path
>> > >of this file is correct and it is being loaded fine in solr admin
>> UI.
>> > > When
>> > >I analyse these fields using "Analysis" tab of the solr admin UI, I
>> > can
>> > > see
>> > >that stopwords are being filtered out. However, when I query with
>> some
>> > > of
>> > >these stopwords, I do get the results back which makes me think that
>> > >probably stopwords are being indexed.
>> > >
>> > > Any idea what could increase the size of index by so much in solr 6?
>> > >
>> >
>>

Re: Fwd: Solr dynamic field blowing up the index size

[Mike Thomsen]

Correct me if I'm wrong, but heavy use of doc values should actually blow
up the size of your index considerably if they are in fields that get sent
a lot of data.

On Tue, Feb 21, 2017 at 10:50 AM, Pratik Patel  wrote:

> Thanks for the reply. I can see that in solr 6, more than 50% of the index
> directory is occupied by ".nvd" file extension. It is something related to
> norms and doc values.
>
> On Tue, Feb 21, 2017 at 10:27 AM, Alexandre Rafalovitch <
> arafa...@gmail.com>
> wrote:
>
> > Did you look in the data directories to check what index file extensions
> > contribute most to the difference? That could give a hint.
> >
> > Regards,
> > Alex
> >
> > On 21 Feb 2017 9:47 AM, "Pratik Patel"  wrote:
> >
> > > Here is the same question in stackOverflow for better format.
> > >
> > > http://stackoverflow.com/questions/42370231/solr-
> > > dynamic-field-blowing-up-the-index-size
> > >
> > > Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app fine
> > but
> > > the problem is that index size with solr 6 is way too large. In solr 5,
> > > index size was about 15GB and in solr 6, for the same data, the index
> > size
> > > is 300GB! I am not able to understand what contributes to such huge
> > > difference in solr 6.
> > >
> > > I have been able to identify a field which is blowing up the size of
> > index.
> > > It is as follows.
> > >
> > >  > > stored="true" multiValued="true"  />
> > >
> > >  > > stored="false" multiValued="true"  />
> > > 
> > >
> > > When this field is commented out, the index size reduces to less than
> > 10GB.
> > >
> > > This field is of type text_general. Following is the definition of this
> > > type.
> > >
> > >  > > positionIncrementGap="100">
> > >   
> > > 
> > > 
> > > 
> > >  > > pattern="((?m)[a-z]+)'s" replacement="$1s" />
> > >  > > protected="protwords.txt" generateWordParts="1"
> > > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> > > catenateAll="0" splitOnCaseChange="0"/>
> > > 
> > >  > > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> > > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> > > />
> > >   
> > >   
> > > 
> > > 
> > > 
> > >  > > pattern="((?m)[a-z]+)'s" replacement="$1s" />
> > >  > > protected="protwords.txt" generateWordParts="1"
> > > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> > > catenateAll="0" splitOnCaseChange="0"/>
> > > 
> > >  > > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> > > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> > > />
> > >   
> > >   
> > >
> > > Few things which I did to debug this issue:
> > >
> > >- I have ensured that field type definition is same as what I was
> > using
> > >in solr 5 and it is also valid in version 6. This field type
> > considers a
> > >list of "stopwords" to be ignored during indexing. I have supplied
> the
> > > same
> > >list of stopwords which we were using in solr 5. I have verified
> that
> > > path
> > >of this file is correct and it is being loaded fine in solr admin
> UI.
> > > When
> > >I analyse these fields using "Analysis" tab of the solr admin UI, I
> > can
> > > see
> > >that stopwords are being filtered out. However, when I query with
> some
> > > of
> > >these stopwords, I do get the results back which makes me think that
> > >probably stopwords are being indexed.
> > >
> > > Any idea what could increase the size of index by so much in solr 6?
> > >
> >
>

Re: Fwd: Solr dynamic field blowing up the index size

[Pratik Patel]

Thanks for the reply. I can see that in solr 6, more than 50% of the index
directory is occupied by ".nvd" file extension. It is something related to
norms and doc values.

On Tue, Feb 21, 2017 at 10:27 AM, Alexandre Rafalovitch 
wrote:

> Did you look in the data directories to check what index file extensions
> contribute most to the difference? That could give a hint.
>
> Regards,
> Alex
>
> On 21 Feb 2017 9:47 AM, "Pratik Patel"  wrote:
>
> > Here is the same question in stackOverflow for better format.
> >
> > http://stackoverflow.com/questions/42370231/solr-
> > dynamic-field-blowing-up-the-index-size
> >
> > Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app fine
> but
> > the problem is that index size with solr 6 is way too large. In solr 5,
> > index size was about 15GB and in solr 6, for the same data, the index
> size
> > is 300GB! I am not able to understand what contributes to such huge
> > difference in solr 6.
> >
> > I have been able to identify a field which is blowing up the size of
> index.
> > It is as follows.
> >
> >  > stored="true" multiValued="true"  />
> >
> >  > stored="false" multiValued="true"  />
> > 
> >
> > When this field is commented out, the index size reduces to less than
> 10GB.
> >
> > This field is of type text_general. Following is the definition of this
> > type.
> >
> >  > positionIncrementGap="100">
> >   
> > 
> > 
> > 
> >  > pattern="((?m)[a-z]+)'s" replacement="$1s" />
> >  > protected="protwords.txt" generateWordParts="1"
> > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> > catenateAll="0" splitOnCaseChange="0"/>
> > 
> >  > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> > />
> >   
> >   
> > 
> > 
> > 
> >  > pattern="((?m)[a-z]+)'s" replacement="$1s" />
> >  > protected="protwords.txt" generateWordParts="1"
> > generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> > catenateAll="0" splitOnCaseChange="0"/>
> > 
> >  > words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> > solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> > />
> >   
> >   
> >
> > Few things which I did to debug this issue:
> >
> >- I have ensured that field type definition is same as what I was
> using
> >in solr 5 and it is also valid in version 6. This field type
> considers a
> >list of "stopwords" to be ignored during indexing. I have supplied the
> > same
> >list of stopwords which we were using in solr 5. I have verified that
> > path
> >of this file is correct and it is being loaded fine in solr admin UI.
> > When
> >I analyse these fields using "Analysis" tab of the solr admin UI, I
> can
> > see
> >that stopwords are being filtered out. However, when I query with some
> > of
> >these stopwords, I do get the results back which makes me think that
> >probably stopwords are being indexed.
> >
> > Any idea what could increase the size of index by so much in solr 6?
> >
>

Re: Fwd: Solr dynamic field blowing up the index size

[Alexandre Rafalovitch]

Did you look in the data directories to check what index file extensions
contribute most to the difference? That could give a hint.

Regards,
Alex

On 21 Feb 2017 9:47 AM, "Pratik Patel"  wrote:

> Here is the same question in stackOverflow for better format.
>
> http://stackoverflow.com/questions/42370231/solr-
> dynamic-field-blowing-up-the-index-size
>
> Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app fine but
> the problem is that index size with solr 6 is way too large. In solr 5,
> index size was about 15GB and in solr 6, for the same data, the index size
> is 300GB! I am not able to understand what contributes to such huge
> difference in solr 6.
>
> I have been able to identify a field which is blowing up the size of index.
> It is as follows.
>
>  stored="true" multiValued="true"  />
>
>  stored="false" multiValued="true"  />
> 
>
> When this field is commented out, the index size reduces to less than 10GB.
>
> This field is of type text_general. Following is the definition of this
> type.
>
>  positionIncrementGap="100">
>   
> 
> 
> 
>  pattern="((?m)[a-z]+)'s" replacement="$1s" />
>  protected="protwords.txt" generateWordParts="1"
> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="0"/>
> 
>  words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> />
>   
>   
> 
> 
> 
>  pattern="((?m)[a-z]+)'s" replacement="$1s" />
>  protected="protwords.txt" generateWordParts="1"
> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="0"/>
> 
>  words="C:/Users/pratik/Desktop/solr-6.4.1_playground/
> solr-6.4.1/server/solr/collection1/conf/stopwords.txt"
> />
>   
>   
>
> Few things which I did to debug this issue:
>
>- I have ensured that field type definition is same as what I was using
>in solr 5 and it is also valid in version 6. This field type considers a
>list of "stopwords" to be ignored during indexing. I have supplied the
> same
>list of stopwords which we were using in solr 5. I have verified that
> path
>of this file is correct and it is being loaded fine in solr admin UI.
> When
>I analyse these fields using "Analysis" tab of the solr admin UI, I can
> see
>that stopwords are being filtered out. However, when I query with some
> of
>these stopwords, I do get the results back which makes me think that
>probably stopwords are being indexed.
>
> Any idea what could increase the size of index by so much in solr 6?
>

Fwd: Solr dynamic field blowing up the index size

[Pratik Patel]

Here is the same question in stackOverflow for better format.

http://stackoverflow.com/questions/42370231/solr-
dynamic-field-blowing-up-the-index-size

Recently, I upgraded from solr 5.0 to solr 6.4.1. I can run my app fine but
the problem is that index size with solr 6 is way too large. In solr 5,
index size was about 15GB and in solr 6, for the same data, the index size
is 300GB! I am not able to understand what contributes to such huge
difference in solr 6.

I have been able to identify a field which is blowing up the size of index.
It is as follows.






When this field is commented out, the index size reduces to less than 10GB.

This field is of type text_general. Following is the definition of this
type.


  







  
  







  
  

Few things which I did to debug this issue:

   - I have ensured that field type definition is same as what I was using
   in solr 5 and it is also valid in version 6. This field type considers a
   list of "stopwords" to be ignored during indexing. I have supplied the same
   list of stopwords which we were using in solr 5. I have verified that path
   of this file is correct and it is being loaded fine in solr admin UI. When
   I analyse these fields using "Analysis" tab of the solr admin UI, I can see
   that stopwords are being filtered out. However, when I query with some of
   these stopwords, I do get the results back which makes me think that
   probably stopwords are being indexed.

Any idea what could increase the size of index by so much in solr 6?

Re: facet on dynamic field

[Erik Hatcher]

You'll have to enumerate them (see the Luke request handler) and specify them 
explicitly. 

> On Nov 4, 2016, at 03:40, Midas A  wrote:
> 
> i want to create facet on all dynamic field (by_*) . what should be the
> query ?

facet on dynamic field

[Midas A]

i want to create facet on all dynamic field (by_*) . what should be the
query ?

Re: Specifying dynamic field type without polluting actual field names with type indicators

[Abdel Belkasri]

That would be a welcomed feature for sure!

On Mon, May 23, 2016 at 6:11 AM, Horváth Péter Gergely <
peter.gergely.horv...@gmail.com> wrote:

> Hi Steve,
>
> Thank you very much for your inputs. Yes, I do know the aliasing mechanism
> offered in Solr. I think the whole question boils down to one thing: how
> much do you know about the data being stored -- and sometimes you know
> nothing about that.
>
> In some cases, you have to provide a generic solution for users to store
> and query their own data. With Solr / Lucene backing your storage, you can
> easily expose a restricted (but still powerful) subset of the Solr / Lucene
> query syntax for querying user-defined data. Things however would start
> getting complicated if you have to tell your customers, that the field you
> loaded as "foo" must be referred as "foo_s" and the field you loaded "bar"
> must be referred  as "bar_i", since it contains a number and so on...
> Implementing the mapping in your application would be overly complex, as
> you would have to maintain a mapping between the internal representation
> ("foo_s") and the query interface ("foo") and alias results from the
> internal format to the format visible to the user ("foo_s" --> "foo"). I
> think you get the idea.
>
> I like the way Solr can use the name for specifying type: having a
> configuration option (either global or at collection level), which tells
> Solr to handle type postfixes slightly differently and strip the type
> prefix automatically would be perfectly enough for this use-case.
>
> Imagine the following approach: if configured so, Solr would still create
> the field based on the type postfix, but would strip it from the name: for
> example, if a document is inserted with the field "foo_s" and "bar_i", Solr
> could create a string field named "foo" and a numeric field "bar".
>
> I think this solution would be both backwards compatible (has to be
> explicitly enabled) and relatively simple to implement in the Solr code
> base. I have created a Jira issue for the feature request:
> https://issues.apache.org/jira/browse/SOLR-9150
>
> What do you think?
>
> Thanks,
> Peter
>
>
>
> 2016-05-19 15:30 GMT+02:00 Steve Rowe :
>
> > Peter,
> >
> > It’s an interesting idea.  Could you make a Solr JIRA?
> >
> > I don’t know where the field type specification would go, but providing a
> > mechanism to specify field type for previously non-existent fields,
> outside
> > of the field names themselves, seems useful.
> >
> > In the meantime, do you know about field aliasing?
> >
> > 1. You can get results back that rename fields to whatever you want: see
> > the section “Field Name Aliases” here: <
> > https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters
> >.
> >
> > 2. On the query side, eDisMax can perform aliasing so that user-specified
> > field names in queries get mapped to one or more indexed fields: look for
> > “alias” in <
> >
> https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
> > >.
> >
> > --
> > Steve
> > www.lucidworks.com
> >
> > > On May 19, 2016, at 4:43 AM, Horváth Péter Gergely <
> > peter.gergely.horv...@gmail.com> wrote:
> > >
> > > Hi Steve,
> > >
> > > Yes, I know the schema API, however I do not want to specify the field
> > type
> > > problematically for every single field.
> > >
> > > I would like to be able to specify the field type when it is being
> added
> > > (similar to the name postfixes, but without affecting the field names).
> > >
> > > Thanks,
> > > Peter
> > >
> > >
> > > 2016-05-17 17:08 GMT+02:00 Steve Rowe :
> > >
> > >> Hi Peter,
> > >>
> > >> Are you familiar with the Schema API?: <
> > >> https://cwiki.apache.org/confluence/display/solr/Schema+API>
> > >>
> > >> You can use it to create fields, field types, etc. prior to ingesting
> > your
> > >> data.
> > >>
> > >> --
> > >> Steve
> > >> www.lucidworks.com
> > >>
> > >>> On May 17, 2016, at 11:05 AM, Horváth Péter Gergely <
> > >> peter.gergely.horv...@gmail.com> wrote:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> By default Solr allows you to define the type of a dynamic field by
> > >>> appending 

Re: Specifying dynamic field type without polluting actual field names with type indicators

[Horváth Péter Gergely]

Hi Steve,

Thank you very much for your inputs. Yes, I do know the aliasing mechanism
offered in Solr. I think the whole question boils down to one thing: how
much do you know about the data being stored -- and sometimes you know
nothing about that.

In some cases, you have to provide a generic solution for users to store
and query their own data. With Solr / Lucene backing your storage, you can
easily expose a restricted (but still powerful) subset of the Solr / Lucene
query syntax for querying user-defined data. Things however would start
getting complicated if you have to tell your customers, that the field you
loaded as "foo" must be referred as "foo_s" and the field you loaded "bar"
must be referred  as "bar_i", since it contains a number and so on...
Implementing the mapping in your application would be overly complex, as
you would have to maintain a mapping between the internal representation
("foo_s") and the query interface ("foo") and alias results from the
internal format to the format visible to the user ("foo_s" --> "foo"). I
think you get the idea.

I like the way Solr can use the name for specifying type: having a
configuration option (either global or at collection level), which tells
Solr to handle type postfixes slightly differently and strip the type
prefix automatically would be perfectly enough for this use-case.

Imagine the following approach: if configured so, Solr would still create
the field based on the type postfix, but would strip it from the name: for
example, if a document is inserted with the field "foo_s" and "bar_i", Solr
could create a string field named "foo" and a numeric field "bar".

I think this solution would be both backwards compatible (has to be
explicitly enabled) and relatively simple to implement in the Solr code
base. I have created a Jira issue for the feature request:
https://issues.apache.org/jira/browse/SOLR-9150

What do you think?

Thanks,
Peter



2016-05-19 15:30 GMT+02:00 Steve Rowe :

> Peter,
>
> It’s an interesting idea.  Could you make a Solr JIRA?
>
> I don’t know where the field type specification would go, but providing a
> mechanism to specify field type for previously non-existent fields, outside
> of the field names themselves, seems useful.
>
> In the meantime, do you know about field aliasing?
>
> 1. You can get results back that rename fields to whatever you want: see
> the section “Field Name Aliases” here: <
> https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters>.
>
> 2. On the query side, eDisMax can perform aliasing so that user-specified
> field names in queries get mapped to one or more indexed fields: look for
> “alias” in <
> https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
> >.
>
> --
> Steve
> www.lucidworks.com
>
> > On May 19, 2016, at 4:43 AM, Horváth Péter Gergely <
> peter.gergely.horv...@gmail.com> wrote:
> >
> > Hi Steve,
> >
> > Yes, I know the schema API, however I do not want to specify the field
> type
> > problematically for every single field.
> >
> > I would like to be able to specify the field type when it is being added
> > (similar to the name postfixes, but without affecting the field names).
> >
> > Thanks,
> > Peter
> >
> >
> > 2016-05-17 17:08 GMT+02:00 Steve Rowe :
> >
> >> Hi Peter,
> >>
> >> Are you familiar with the Schema API?: <
> >> https://cwiki.apache.org/confluence/display/solr/Schema+API>
> >>
> >> You can use it to create fields, field types, etc. prior to ingesting
> your
> >> data.
> >>
> >> --
> >> Steve
> >> www.lucidworks.com
> >>
> >>> On May 17, 2016, at 11:05 AM, Horváth Péter Gergely <
> >> peter.gergely.horv...@gmail.com> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> By default Solr allows you to define the type of a dynamic field by
> >>> appending a post-fix to the name itself. E.g. creating a color_s field
> >>> instructs Solr to create a string field. My understanding is that if we
> >> do
> >>> this, all queries must refer the post-fixed field name as well. So
> >>> instead of a query like color:"red", we will have to write something
> like
> >>> color_s:"red" -- and so on for other field types as well.
> >>>
> >>> I am wondering if it is possible to specify the data type used for a
> >> field
> >>> in Solr 6.0.0, without having to modify the field name. (Or at least
> in a
> >>> way that would allow us to use the original field name) Do you have any
> >>> idea, how to achieve this? I am fine, if we have to specify the field
> >> type
> >>> during the insertion of a document, however, I do not want to keep
> using
> >>> post-fixes while running queries...
> >>>
> >>> Thanks,
> >>> Peter
> >>
> >>
>
>

Re: Creating dynamic field, but strip the type indicator postfix from the name

[Erick Erickson]

Why is this important? I'm failing to see how this helps in
"making the queries simpler to form". What's the difference
between making a query
q=field:value
and
q=field_s:value?

If it's a display issue, it seems like the display app could simply strip
anything after the terminal underscore from the display...

Best,
Erick

On Thu, May 19, 2016 at 5:59 AM, Horváth Péter Gergely
 wrote:
> Hi Everyone,
>
> I am wondering if it is possible to store dynamic fields without the type
> indicator postfix. In our Solr environment, I would like to
> 1.) use dynamic fields ("data drivern collections" with no fixed fields
> specified in advance)
> 2.) be able to specify the field type, but without interacting with Solr
> schema API separately
> 3.) I do not want type indicator postfixes to appear in field names, since
> that would make queries more complicated to form.
>
> For example, I could imagine that when I add a document like this:
>
>
> CloudSolrClient solr  = ...
>
> SolrInputDocument document = new SolrInputDocument();
>
> document.addField("foobar_s", "theValue")
>
> solr.add("someCollection", document);
>
> Solr would recognize that the "foobar_s" field name contains a type
> indicator and it would create a field named "foobar" with the type of
> string.
>
> Is there any way to achieve this behavior (have Solr create the field with
> the requested type, but without the type indicator postfix)? If yes, how
> could I do that?
>
> Thanks,
> Peter

Re: Specifying dynamic field type without polluting actual field names with type indicators

[Steve Rowe]

Peter,

It’s an interesting idea.  Could you make a Solr JIRA?

I don’t know where the field type specification would go, but providing a 
mechanism to specify field type for previously non-existent fields, outside of 
the field names themselves, seems useful.

In the meantime, do you know about field aliasing?  

1. You can get results back that rename fields to whatever you want: see the 
section “Field Name Aliases” here: 
<https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters>.

2. On the query side, eDisMax can perform aliasing so that user-specified field 
names in queries get mapped to one or more indexed fields: look for “alias” in 
<https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser>.

--
Steve
www.lucidworks.com

> On May 19, 2016, at 4:43 AM, Horváth Péter Gergely 
>  wrote:
> 
> Hi Steve,
> 
> Yes, I know the schema API, however I do not want to specify the field type
> problematically for every single field.
> 
> I would like to be able to specify the field type when it is being added
> (similar to the name postfixes, but without affecting the field names).
> 
> Thanks,
> Peter
> 
> 
> 2016-05-17 17:08 GMT+02:00 Steve Rowe :
> 
>> Hi Peter,
>> 
>> Are you familiar with the Schema API?: <
>> https://cwiki.apache.org/confluence/display/solr/Schema+API>
>> 
>> You can use it to create fields, field types, etc. prior to ingesting your
>> data.
>> 
>> --
>> Steve
>> www.lucidworks.com
>> 
>>> On May 17, 2016, at 11:05 AM, Horváth Péter Gergely <
>> peter.gergely.horv...@gmail.com> wrote:
>>> 
>>> Hi All,
>>> 
>>> By default Solr allows you to define the type of a dynamic field by
>>> appending a post-fix to the name itself. E.g. creating a color_s field
>>> instructs Solr to create a string field. My understanding is that if we
>> do
>>> this, all queries must refer the post-fixed field name as well. So
>>> instead of a query like color:"red", we will have to write something like
>>> color_s:"red" -- and so on for other field types as well.
>>> 
>>> I am wondering if it is possible to specify the data type used for a
>> field
>>> in Solr 6.0.0, without having to modify the field name. (Or at least in a
>>> way that would allow us to use the original field name) Do you have any
>>> idea, how to achieve this? I am fine, if we have to specify the field
>> type
>>> during the insertion of a document, however, I do not want to keep using
>>> post-fixes while running queries...
>>> 
>>> Thanks,
>>> Peter
>> 
>> 

Creating dynamic field, but strip the type indicator postfix from the name

[Horváth Péter Gergely]

Hi Everyone,

I am wondering if it is possible to store dynamic fields without the type
indicator postfix. In our Solr environment, I would like to
1.) use dynamic fields ("data drivern collections" with no fixed fields
specified in advance)
2.) be able to specify the field type, but without interacting with Solr
schema API separately
3.) I do not want type indicator postfixes to appear in field names, since
that would make queries more complicated to form.

For example, I could imagine that when I add a document like this:


CloudSolrClient solr  = ...

SolrInputDocument document = new SolrInputDocument();

document.addField("foobar_s", "theValue")

solr.add("someCollection", document);

Solr would recognize that the "foobar_s" field name contains a type
indicator and it would create a field named "foobar" with the type of
string.

Is there any way to achieve this behavior (have Solr create the field with
the requested type, but without the type indicator postfix)? If yes, how
could I do that?

Thanks,
Peter

Re: Specifying dynamic field type without polluting actual field names with type indicators

[Horváth Péter Gergely]

Hi Steve,

Yes, I know the schema API, however I do not want to specify the field type
problematically for every single field.

I would like to be able to specify the field type when it is being added
(similar to the name postfixes, but without affecting the field names).

Thanks,
Peter


2016-05-17 17:08 GMT+02:00 Steve Rowe :

> Hi Peter,
>
> Are you familiar with the Schema API?: <
> https://cwiki.apache.org/confluence/display/solr/Schema+API>
>
> You can use it to create fields, field types, etc. prior to ingesting your
> data.
>
> --
> Steve
> www.lucidworks.com
>
> > On May 17, 2016, at 11:05 AM, Horváth Péter Gergely <
> peter.gergely.horv...@gmail.com> wrote:
> >
> > Hi All,
> >
> > By default Solr allows you to define the type of a dynamic field by
> > appending a post-fix to the name itself. E.g. creating a color_s field
> > instructs Solr to create a string field. My understanding is that if we
> do
> > this, all queries must refer the post-fixed field name as well. So
> > instead of a query like color:"red", we will have to write something like
> > color_s:"red" -- and so on for other field types as well.
> >
> > I am wondering if it is possible to specify the data type used for a
> field
> > in Solr 6.0.0, without having to modify the field name. (Or at least in a
> > way that would allow us to use the original field name) Do you have any
> > idea, how to achieve this? I am fine, if we have to specify the field
> type
> > during the insertion of a document, however, I do not want to keep using
> > post-fixes while running queries...
> >
> > Thanks,
> > Peter
>
>

Re: Specifying dynamic field type without polluting actual field names with type indicators

[Shawn Heisey]

On 5/17/2016 9:05 AM, Horváth Péter Gergely wrote:
> By default Solr allows you to define the type of a dynamic field by
> appending a post-fix to the name itself. E.g. creating a color_s field
> instructs Solr to create a string field. My understanding is that if we do
> this, all queries must refer the post-fixed field name as well. So
> instead of a query like color:"red", we will have to write something like
> color_s:"red" -- and so on for other field types as well.
>
> I am wondering if it is possible to specify the data type used for a field
> in Solr 6.0.0, without having to modify the field name. (Or at least in a
> way that would allow us to use the original field name) Do you have any
> idea, how to achieve this? I am fine, if we have to specify the field type
> during the insertion of a document, however, I do not want to keep using
> post-fixes while running queries...

Dynamic fields are just a different way to specify field names.  Instead
of explictly naming the field, you can put a * at the beginning or end
of the name and define an entire series of potential field names with
one declaration, and tie those names to a specific type.

You can't specify a field type when doing a query, only the field name. 
Solr looks up the field name in the schema to determine the type.  If
you want to change the type of a given field name, you need to change
your schema, restart Solr or reload the core, and quite possibly also
reindex.

Thanks,
Shawn

Re: Specifying dynamic field type without polluting actual field names with type indicators

[Steve Rowe]

Hi Peter,

Are you familiar with the Schema API?: 
<https://cwiki.apache.org/confluence/display/solr/Schema+API>

You can use it to create fields, field types, etc. prior to ingesting your data.

--
Steve
www.lucidworks.com

> On May 17, 2016, at 11:05 AM, Horváth Péter Gergely 
>  wrote:
> 
> Hi All,
> 
> By default Solr allows you to define the type of a dynamic field by
> appending a post-fix to the name itself. E.g. creating a color_s field
> instructs Solr to create a string field. My understanding is that if we do
> this, all queries must refer the post-fixed field name as well. So
> instead of a query like color:"red", we will have to write something like
> color_s:"red" -- and so on for other field types as well.
> 
> I am wondering if it is possible to specify the data type used for a field
> in Solr 6.0.0, without having to modify the field name. (Or at least in a
> way that would allow us to use the original field name) Do you have any
> idea, how to achieve this? I am fine, if we have to specify the field type
> during the insertion of a document, however, I do not want to keep using
> post-fixes while running queries...
> 
> Thanks,
> Peter

Specifying dynamic field type without polluting actual field names with type indicators

[Horváth Péter Gergely]

Hi All,

By default Solr allows you to define the type of a dynamic field by
appending a post-fix to the name itself. E.g. creating a color_s field
instructs Solr to create a string field. My understanding is that if we do
this, all queries must refer the post-fixed field name as well. So
instead of a query like color:"red", we will have to write something like
color_s:"red" -- and so on for other field types as well.

I am wondering if it is possible to specify the data type used for a field
in Solr 6.0.0, without having to modify the field name. (Or at least in a
way that would allow us to use the original field name) Do you have any
idea, how to achieve this? I am fine, if we have to specify the field type
during the insertion of a document, however, I do not want to keep using
post-fixes while running queries...

Thanks,
Peter

Specifying dynamic field type without polluting actual field names with type indicators

[Horváth Péter Gergely]

Hi All,

By default Solr allows you to define the type of a dynamic field by
appending a post-fix to the name itself. E.g. creating a color_s field
instructs Solr to create a string field. My understanding is that if we do
this, all queries must refer the post-fixed field name as well. So
instead of a query like color:"red", we will have to write something like
color_s:"red" -- and so on for other field types as well.

I am wondering if it is possible to specify the data type used for a field
in Solr 6.0.0, without having to modify the field name. (Or at least in a
way that would allow us to use the original field name) Do you have any
idea, how to achieve this? I am fine, if we have to specify the field type
during the insertion of a document, however, I do not want to keep using
post-fixes while running queries...

Thanks,
Peter

Re: Why can a dynamic field ONLY start or end with '*' but not both?

[Erick Erickson]

Starting and ending globs were never officially supported, at
least as far back as 3.6. They were never programmatically
enforced either apparently.

This is from the 3.6 schema.xml:

 

So  it was not really discussed that I know about relative to the JIRAs you
mentioned, more like you were using a feature that just happened to work
despite explicitly being unsupported.

Best,
Erick



On Tue, Nov 17, 2015 at 8:39 AM, Frank Greguska  wrote:
> Hello,
>
> Prior to the implementation of SOLR-3251
> <https://issues.apache.org/jira/browse/SOLR-4503>, it seems it was possible
> to create dynamic fields using multiple 'glob' characters.
>
> e.g. 
>
> Since this commit
> <https://github.com/apache/lucene-solr/commit/5cbf7f54c38924c19f9cdc347c729abfd7b17076#diff-ca2b07d8a1eaf45cdb5e73eba102ba38R423>,
> a constraint has been added that the name of a dynamic field must either
> start or end with '*', but not both. Was this decision discussed anywhere?
> It's causing a schema of mine to break after upgrading and I'm just curious
> if there was a technical reason for implementing this constraint.
>
> Thank you,
>
> Frank
>
> P.S. I've asked a related question
> <http://stackoverflow.com/q/33747996/953327> on StackOverflow.

Why can a dynamic field ONLY start or end with '*' but not both?

[Frank Greguska]

Hello,

Prior to the implementation of SOLR-3251
<https://issues.apache.org/jira/browse/SOLR-4503>, it seems it was possible
to create dynamic fields using multiple 'glob' characters.

e.g. 

Since this commit
<https://github.com/apache/lucene-solr/commit/5cbf7f54c38924c19f9cdc347c729abfd7b17076#diff-ca2b07d8a1eaf45cdb5e73eba102ba38R423>,
a constraint has been added that the name of a dynamic field must either
start or end with '*', but not both. Was this decision discussed anywhere?
It's causing a schema of mine to break after upgrading and I'm just curious
if there was a technical reason for implementing this constraint.

Thank you,

Frank

P.S. I've asked a related question
<http://stackoverflow.com/q/33747996/953327> on StackOverflow.

Re: solr update dynamic field generates multiValued error

[Aman Tandon]

Sure. thank you Upayavira

With Regards
Aman Tandon

On Mon, Sep 21, 2015 at 6:01 PM, Upayavira  wrote:

> You cannot do multi valued fields with LatLongType fields. Therefore, if
> that is a need, you will have to investigate RPT fields.
>
> I'm not sure how you do distance boosting there, so I'd suggest you ask
> that as a separate question with a new title.
>
> Upayavira
>
> On Mon, Sep 21, 2015, at 01:27 PM, Aman Tandon wrote:
> > We are using LatLonType to use the gradual boosting / distance based
> > boosting of search results.
> >
> > With Regards
> > Aman Tandon
> >
> > On Mon, Sep 21, 2015 at 5:39 PM, Upayavira  wrote:
> >
> > > Aman,
> > >
> > > I cannot promise to answer questions promptly - like most people on
> this
> > > list, we answer if/when we have a gap in our workload.
> > >
> > > The reason you are getting the non multiValued field error is because
> > > your latlon field does not have multiValued="true" enabled.
> > >
> > > However, the field type definition notes that this field type does not
> > > support multivalued fields, so you're not gonna get anywhere with that
> > > route.
> > >
> > > Have you tried the location_rpt type?
> > > (solr.SpatialRecursivePrefixTreeFieldType). This is a newer, and as I
> > > understand it, far more flexible field type - for example, you can
> index
> > > shapes into it as well as locations.
> > >
> > > I'd suggest you read this page, and pay particular attention to
> mentions
> > > of RPT:
> > >
> > > https://cwiki.apache.org/confluence/display/solr/Spatial+Search
> > >
> > > Upayavira
> > >
> > > On Mon, Sep 21, 2015, at 10:36 AM, Aman Tandon wrote:
> > > > Upayavira, please help
> > > >
> > > > With Regards
> > > > Aman Tandon
> > > >
> > > > On Mon, Sep 21, 2015 at 2:38 PM, Aman Tandon <
> amantandon...@gmail.com>
> > > > wrote:
> > > >
> > > > > Error is
> > > > >
> > > > > 
> > > > > 
> > > > > 400 > > > > name="QTime">28ERROR:
> > > > > [doc=9474144846] multiple values encountered for non multiValued
> field
> > > > > latlon_0_coordinate: [11.0183, 11.0183] > > > > name="code">400
> > > > > 
> > > > >
> > > > > And my configuration is
> > > > >
> > > > > 
> > > > > > > > >  stored="true" />
> > > > >
> > > > >  
> > > > >  > > > > subFieldSuffix="_coordinate"/>
> > > > >
> > > > >> > > > required="false" multiValued="false" />
> > > > >
> > > > >  how you know it is because of stored="true"?
> > > > >
> > > > > As Erick replied in the last mail thread,
> > > > > I'm not getting any multiple values in the _coordinate fields.
> > > However, I
> > > > > _do_ get the error if my dynamic *_coordinate field is set to
> > > > > stored="true".
> > > > >
> > > > > And stored="true" is mandatory for using the atomic updates.
> > > > >
> > > > > With Regards
> > > > > Aman Tandon
> > > > >
> > > > > On Mon, Sep 21, 2015 at 2:22 PM, Upayavira  wrote:
> > > > >
> > > > >> Can you show the error you are getting, and how you know it is
> because
> > > > >> of stored="true"?
> > > > >>
> > > > >> Upayavira
> > > > >>
> > > > >> On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
> > > > >> > Hi Erick,
> > > > >> >
> > > > >> > I am getting the same error because my dynamic field
> *_coordinate is
> > > > >> > stored="true".
> > > > >> > How can I get rid of this error?
> > > > >> >
> > > > >> > And I have to use the atomic update. Please help!!
> > > > >> >
> > > > >> > With Regards
> > > > >> > Aman Tandon
> > > > >> >
> > > > >> > On 

Re: solr update dynamic field generates multiValued error

[Upayavira]

You cannot do multi valued fields with LatLongType fields. Therefore, if
that is a need, you will have to investigate RPT fields.

I'm not sure how you do distance boosting there, so I'd suggest you ask
that as a separate question with a new title.

Upayavira

On Mon, Sep 21, 2015, at 01:27 PM, Aman Tandon wrote:
> We are using LatLonType to use the gradual boosting / distance based
> boosting of search results.
> 
> With Regards
> Aman Tandon
> 
> On Mon, Sep 21, 2015 at 5:39 PM, Upayavira  wrote:
> 
> > Aman,
> >
> > I cannot promise to answer questions promptly - like most people on this
> > list, we answer if/when we have a gap in our workload.
> >
> > The reason you are getting the non multiValued field error is because
> > your latlon field does not have multiValued="true" enabled.
> >
> > However, the field type definition notes that this field type does not
> > support multivalued fields, so you're not gonna get anywhere with that
> > route.
> >
> > Have you tried the location_rpt type?
> > (solr.SpatialRecursivePrefixTreeFieldType). This is a newer, and as I
> > understand it, far more flexible field type - for example, you can index
> > shapes into it as well as locations.
> >
> > I'd suggest you read this page, and pay particular attention to mentions
> > of RPT:
> >
> > https://cwiki.apache.org/confluence/display/solr/Spatial+Search
> >
> > Upayavira
> >
> > On Mon, Sep 21, 2015, at 10:36 AM, Aman Tandon wrote:
> > > Upayavira, please help
> > >
> > > With Regards
> > > Aman Tandon
> > >
> > > On Mon, Sep 21, 2015 at 2:38 PM, Aman Tandon 
> > > wrote:
> > >
> > > > Error is
> > > >
> > > > 
> > > > 
> > > > 400 > > > name="QTime">28ERROR:
> > > > [doc=9474144846] multiple values encountered for non multiValued field
> > > > latlon_0_coordinate: [11.0183, 11.0183] > > > name="code">400
> > > > 
> > > >
> > > > And my configuration is
> > > >
> > > > 
> > > > > > >  stored="true" />
> > > >
> > > >  
> > > >  > > > subFieldSuffix="_coordinate"/>
> > > >
> > > >> > > required="false" multiValued="false" />
> > > >
> > > >  how you know it is because of stored="true"?
> > > >
> > > > As Erick replied in the last mail thread,
> > > > I'm not getting any multiple values in the _coordinate fields.
> > However, I
> > > > _do_ get the error if my dynamic *_coordinate field is set to
> > > > stored="true".
> > > >
> > > > And stored="true" is mandatory for using the atomic updates.
> > > >
> > > > With Regards
> > > > Aman Tandon
> > > >
> > > > On Mon, Sep 21, 2015 at 2:22 PM, Upayavira  wrote:
> > > >
> > > >> Can you show the error you are getting, and how you know it is because
> > > >> of stored="true"?
> > > >>
> > > >> Upayavira
> > > >>
> > > >> On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
> > > >> > Hi Erick,
> > > >> >
> > > >> > I am getting the same error because my dynamic field *_coordinate is
> > > >> > stored="true".
> > > >> > How can I get rid of this error?
> > > >> >
> > > >> > And I have to use the atomic update. Please help!!
> > > >> >
> > > >> > With Regards
> > > >> > Aman Tandon
> > > >> >
> > > >> > On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa  > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hey Erick, i think that you were right, there was a mix in the
> > > >> schemas and
> > > >> > > that was generating the error on some of the documents.
> > > >> > >
> > > >> > > Thanks for the help guys!
> > > >> > >
> > > >> > >
> > > >> > > 2014-08-05 1:28 GMT-03:00 Erick Erickson  > >:
> > > >> > >
> > > >> > > > Hmmm, I jus

Re: solr update dynamic field generates multiValued error

[Aman Tandon]

We are using LatLonType to use the gradual boosting / distance based
boosting of search results.

With Regards
Aman Tandon

On Mon, Sep 21, 2015 at 5:39 PM, Upayavira  wrote:

> Aman,
>
> I cannot promise to answer questions promptly - like most people on this
> list, we answer if/when we have a gap in our workload.
>
> The reason you are getting the non multiValued field error is because
> your latlon field does not have multiValued="true" enabled.
>
> However, the field type definition notes that this field type does not
> support multivalued fields, so you're not gonna get anywhere with that
> route.
>
> Have you tried the location_rpt type?
> (solr.SpatialRecursivePrefixTreeFieldType). This is a newer, and as I
> understand it, far more flexible field type - for example, you can index
> shapes into it as well as locations.
>
> I'd suggest you read this page, and pay particular attention to mentions
> of RPT:
>
> https://cwiki.apache.org/confluence/display/solr/Spatial+Search
>
> Upayavira
>
> On Mon, Sep 21, 2015, at 10:36 AM, Aman Tandon wrote:
> > Upayavira, please help
> >
> > With Regards
> > Aman Tandon
> >
> > On Mon, Sep 21, 2015 at 2:38 PM, Aman Tandon 
> > wrote:
> >
> > > Error is
> > >
> > > 
> > > 
> > > 400 > > name="QTime">28ERROR:
> > > [doc=9474144846] multiple values encountered for non multiValued field
> > > latlon_0_coordinate: [11.0183, 11.0183] > > name="code">400
> > > 
> > >
> > > And my configuration is
> > >
> > > 
> > > > >  stored="true" />
> > >
> > >  
> > >  > > subFieldSuffix="_coordinate"/>
> > >
> > >> > required="false" multiValued="false" />
> > >
> > >  how you know it is because of stored="true"?
> > >
> > > As Erick replied in the last mail thread,
> > > I'm not getting any multiple values in the _coordinate fields.
> However, I
> > > _do_ get the error if my dynamic *_coordinate field is set to
> > > stored="true".
> > >
> > > And stored="true" is mandatory for using the atomic updates.
> > >
> > > With Regards
> > > Aman Tandon
> > >
> > > On Mon, Sep 21, 2015 at 2:22 PM, Upayavira  wrote:
> > >
> > >> Can you show the error you are getting, and how you know it is because
> > >> of stored="true"?
> > >>
> > >> Upayavira
> > >>
> > >> On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
> > >> > Hi Erick,
> > >> >
> > >> > I am getting the same error because my dynamic field *_coordinate is
> > >> > stored="true".
> > >> > How can I get rid of this error?
> > >> >
> > >> > And I have to use the atomic update. Please help!!
> > >> >
> > >> > With Regards
> > >> > Aman Tandon
> > >> >
> > >> > On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa  >
> > >> > wrote:
> > >> >
> > >> > > Hey Erick, i think that you were right, there was a mix in the
> > >> schemas and
> > >> > > that was generating the error on some of the documents.
> > >> > >
> > >> > > Thanks for the help guys!
> > >> > >
> > >> > >
> > >> > > 2014-08-05 1:28 GMT-03:00 Erick Erickson  >:
> > >> > >
> > >> > > > Hmmm, I jus tried this with a 4.x build and I can update the
> > >> document
> > >> > > > multiple times without a problem. I just indexed the standard
> > >> exampledocs
> > >> > > > and then updated a doc like this (vidcard.xml was the base):
> > >> > > >
> > >> > > > 
> > >> > > > 
> > >> > > >   EN7800GTX/2DHTV/256M
> > >> > > >
> > >> > > >   eoe changed this
> > >> puppy
> > >> > > > 
> > >> > > >   
> > >> > > > 
> > >> > > >
> > >> > > > I'm not getting any multiple values in the _coordinate fields.
> > >> However, I
> > >> > > > _do_ get the error if my dynamic *_coordinat

Re: solr update dynamic field generates multiValued error

[Upayavira]

Aman,

I cannot promise to answer questions promptly - like most people on this
list, we answer if/when we have a gap in our workload.

The reason you are getting the non multiValued field error is because
your latlon field does not have multiValued="true" enabled.

However, the field type definition notes that this field type does not
support multivalued fields, so you're not gonna get anywhere with that
route.

Have you tried the location_rpt type?
(solr.SpatialRecursivePrefixTreeFieldType). This is a newer, and as I
understand it, far more flexible field type - for example, you can index
shapes into it as well as locations.

I'd suggest you read this page, and pay particular attention to mentions
of RPT:

https://cwiki.apache.org/confluence/display/solr/Spatial+Search

Upayavira

On Mon, Sep 21, 2015, at 10:36 AM, Aman Tandon wrote:
> Upayavira, please help
> 
> With Regards
> Aman Tandon
> 
> On Mon, Sep 21, 2015 at 2:38 PM, Aman Tandon 
> wrote:
> 
> > Error is
> >
> > 
> > 
> > 400 > name="QTime">28ERROR:
> > [doc=9474144846] multiple values encountered for non multiValued field
> > latlon_0_coordinate: [11.0183, 11.0183] > name="code">400
> > 
> >
> > And my configuration is
> >
> > 
> > >  stored="true" />
> >
> >  
> >  > subFieldSuffix="_coordinate"/>
> >
> >> required="false" multiValued="false" />
> >
> >  how you know it is because of stored="true"?
> >
> > As Erick replied in the last mail thread,
> > I'm not getting any multiple values in the _coordinate fields. However, I
> > _do_ get the error if my dynamic *_coordinate field is set to
> > stored="true".
> >
> > And stored="true" is mandatory for using the atomic updates.
> >
> > With Regards
> > Aman Tandon
> >
> > On Mon, Sep 21, 2015 at 2:22 PM, Upayavira  wrote:
> >
> >> Can you show the error you are getting, and how you know it is because
> >> of stored="true"?
> >>
> >> Upayavira
> >>
> >> On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
> >> > Hi Erick,
> >> >
> >> > I am getting the same error because my dynamic field *_coordinate is
> >> > stored="true".
> >> > How can I get rid of this error?
> >> >
> >> > And I have to use the atomic update. Please help!!
> >> >
> >> > With Regards
> >> > Aman Tandon
> >> >
> >> > On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa 
> >> > wrote:
> >> >
> >> > > Hey Erick, i think that you were right, there was a mix in the
> >> schemas and
> >> > > that was generating the error on some of the documents.
> >> > >
> >> > > Thanks for the help guys!
> >> > >
> >> > >
> >> > > 2014-08-05 1:28 GMT-03:00 Erick Erickson :
> >> > >
> >> > > > Hmmm, I jus tried this with a 4.x build and I can update the
> >> document
> >> > > > multiple times without a problem. I just indexed the standard
> >> exampledocs
> >> > > > and then updated a doc like this (vidcard.xml was the base):
> >> > > >
> >> > > > 
> >> > > > 
> >> > > >   EN7800GTX/2DHTV/256M
> >> > > >
> >> > > >   eoe changed this
> >> puppy
> >> > > > 
> >> > > >   
> >> > > > 
> >> > > >
> >> > > > I'm not getting any multiple values in the _coordinate fields.
> >> However, I
> >> > > > _do_ get the error if my dynamic *_coordinate field is set to
> >> > > > stored="true".
> >> > > >
> >> > > > Did you perhaps change this at some point? Whenever I change the
> >> schema,
> >> > > I
> >> > > > try to 'rm -rf solr/collection/data' just to be sure I've purged all
> >> > > traces
> >> > > > of the former schema definition.
> >> > > >
> >> > > > Best,
> >> > > > Erick
> >> > > >
> >> > > >
> >> > > > On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa 
> >> > > wrote:
> >> > > >
> >> > > > > No

Re: solr update dynamic field generates multiValued error

[Aman Tandon]

Upayavira, please help

With Regards
Aman Tandon

On Mon, Sep 21, 2015 at 2:38 PM, Aman Tandon 
wrote:

> Error is
>
> 
> 
> 400 name="QTime">28ERROR:
> [doc=9474144846] multiple values encountered for non multiValued field
> latlon_0_coordinate: [11.0183, 11.0183] name="code">400
> 
>
> And my configuration is
>
> 
>  stored="true" />
>
>  
>  subFieldSuffix="_coordinate"/>
>
>required="false" multiValued="false" />
>
>  how you know it is because of stored="true"?
>
> As Erick replied in the last mail thread,
> I'm not getting any multiple values in the _coordinate fields. However, I
> _do_ get the error if my dynamic *_coordinate field is set to
> stored="true".
>
> And stored="true" is mandatory for using the atomic updates.
>
> With Regards
> Aman Tandon
>
> On Mon, Sep 21, 2015 at 2:22 PM, Upayavira  wrote:
>
>> Can you show the error you are getting, and how you know it is because
>> of stored="true"?
>>
>> Upayavira
>>
>> On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
>> > Hi Erick,
>> >
>> > I am getting the same error because my dynamic field *_coordinate is
>> > stored="true".
>> > How can I get rid of this error?
>> >
>> > And I have to use the atomic update. Please help!!
>> >
>> > With Regards
>> > Aman Tandon
>> >
>> > On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa 
>> > wrote:
>> >
>> > > Hey Erick, i think that you were right, there was a mix in the
>> schemas and
>> > > that was generating the error on some of the documents.
>> > >
>> > > Thanks for the help guys!
>> > >
>> > >
>> > > 2014-08-05 1:28 GMT-03:00 Erick Erickson :
>> > >
>> > > > Hmmm, I jus tried this with a 4.x build and I can update the
>> document
>> > > > multiple times without a problem. I just indexed the standard
>> exampledocs
>> > > > and then updated a doc like this (vidcard.xml was the base):
>> > > >
>> > > > 
>> > > > 
>> > > >   EN7800GTX/2DHTV/256M
>> > > >
>> > > >   eoe changed this
>> puppy
>> > > > 
>> > > >   
>> > > > 
>> > > >
>> > > > I'm not getting any multiple values in the _coordinate fields.
>> However, I
>> > > > _do_ get the error if my dynamic *_coordinate field is set to
>> > > > stored="true".
>> > > >
>> > > > Did you perhaps change this at some point? Whenever I change the
>> schema,
>> > > I
>> > > > try to 'rm -rf solr/collection/data' just to be sure I've purged all
>> > > traces
>> > > > of the former schema definition.
>> > > >
>> > > > Best,
>> > > > Erick
>> > > >
>> > > >
>> > > > On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa 
>> > > wrote:
>> > > >
>> > > > > No, they are not declarad explicitly.
>> > > > >
>> > > > > This is how they are created:
>> > > > >
>> > > > > > stored="true"/>
>> > > > >
>> > > > > > > > > >  stored="false"/>
>> > > > >
>> > > > > > > > > > subFieldSuffix="_coordinate"/>
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > 2014-08-04 22:28 GMT-03:00 Michael Ryan :
>> > > > >
>> > > > > > Are the latLong_0_coordinate and latLong_1_coordinate fields
>> > > populated
>> > > > > > using copyField? If so, this sounds like it could be
>> > > > > > https://issues.apache.org/jira/browse/SOLR-3502.
>> > > > > >
>> > > > > > -Michael
>> > > > > >
>> > > > > > -Original Message-
>> > > > > > From: Franco Giacosa [mailto:fgiac...@gmail.com]
>> > > > > > Sent: Monday, August 04, 2014 9:05 PM
>> > > > > > To: solr-user@lucene.apache.org
>> > > > > > Subject: solr update dynamic field gener

Re: solr update dynamic field generates multiValued error

[Aman Tandon]

Error is



40028ERROR:
[doc=9474144846] multiple values encountered for non multiValued field
latlon_0_coordinate: [11.0183, 11.0183]400


And my configuration is


   

 


  

 how you know it is because of stored="true"?

As Erick replied in the last mail thread,
I'm not getting any multiple values in the _coordinate fields. However, I
_do_ get the error if my dynamic *_coordinate field is set to stored="true".

And stored="true" is mandatory for using the atomic updates.

With Regards
Aman Tandon

On Mon, Sep 21, 2015 at 2:22 PM, Upayavira  wrote:

> Can you show the error you are getting, and how you know it is because
> of stored="true"?
>
> Upayavira
>
> On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
> > Hi Erick,
> >
> > I am getting the same error because my dynamic field *_coordinate is
> > stored="true".
> > How can I get rid of this error?
> >
> > And I have to use the atomic update. Please help!!
> >
> > With Regards
> > Aman Tandon
> >
> > On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa 
> > wrote:
> >
> > > Hey Erick, i think that you were right, there was a mix in the schemas
> and
> > > that was generating the error on some of the documents.
> > >
> > > Thanks for the help guys!
> > >
> > >
> > > 2014-08-05 1:28 GMT-03:00 Erick Erickson :
> > >
> > > > Hmmm, I jus tried this with a 4.x build and I can update the document
> > > > multiple times without a problem. I just indexed the standard
> exampledocs
> > > > and then updated a doc like this (vidcard.xml was the base):
> > > >
> > > > 
> > > > 
> > > >   EN7800GTX/2DHTV/256M
> > > >
> > > >   eoe changed this puppy
> > > > 
> > > >   
> > > > 
> > > >
> > > > I'm not getting any multiple values in the _coordinate fields.
> However, I
> > > > _do_ get the error if my dynamic *_coordinate field is set to
> > > > stored="true".
> > > >
> > > > Did you perhaps change this at some point? Whenever I change the
> schema,
> > > I
> > > > try to 'rm -rf solr/collection/data' just to be sure I've purged all
> > > traces
> > > > of the former schema definition.
> > > >
> > > > Best,
> > > > Erick
> > > >
> > > >
> > > > On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa 
> > > wrote:
> > > >
> > > > > No, they are not declarad explicitly.
> > > > >
> > > > > This is how they are created:
> > > > >
> > > > >  stored="true"/>
> > > > >
> > > > >  > > > >  stored="false"/>
> > > > >
> > > > >  > > > > subFieldSuffix="_coordinate"/>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2014-08-04 22:28 GMT-03:00 Michael Ryan :
> > > > >
> > > > > > Are the latLong_0_coordinate and latLong_1_coordinate fields
> > > populated
> > > > > > using copyField? If so, this sounds like it could be
> > > > > > https://issues.apache.org/jira/browse/SOLR-3502.
> > > > > >
> > > > > > -Michael
> > > > > >
> > > > > > -Original Message-
> > > > > > From: Franco Giacosa [mailto:fgiac...@gmail.com]
> > > > > > Sent: Monday, August 04, 2014 9:05 PM
> > > > > > To: solr-user@lucene.apache.org
> > > > > > Subject: solr update dynamic field generates multiValued error
> > > > > >
> > > > > > Hello everyone, this is my first time posting a question, so
> forgive
> > > me
> > > > > if
> > > > > > i'm missing something.
> > > > > >
> > > > > > This is my problem:
> > > > > >
> > > > > > I have a schema.xml that has the following latLong information
> > > > > >
> > > > > > The dynamicField generates 2 dynamic fields that have the lat
> and the
> > > > > long
> > > > > > (latLong_0_coordinate and latLong_1_coordinate)
> > > > > >
> > > > > > So for example a document will have
> > > > > >
> > > > > > "latLong_0_coordinate": 40.4114, "latLong_1_coordinate":
> -74.1031,
> > > > > > "latLong": "40.4114,-74.1031",
> > > > > >
> > > > > > Now when I try to update a document (i don't update the latLong
> > > field.
> > > > I
> > > > > > just update other parts of the document using atomic update) solr
> > > > > > re-creates the dynamicField and adds the same value again, like
> its
> > > > using
> > > > > > add instead of set. So when i do an update the fields of the doc
> look
> > > > > like
> > > > > > this
> > > > > >
> > > > > > "latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
> > > > > > [-74.1031,-74.1031] "latLong": "40.4114,-74.1031",
> > > > > >
> > > > > > So the dynamicFields now have 2 values, so the next time that I
> want
> > > to
> > > > > > update the document a schema error is throw because im trying to
> > > store
> > > > a
> > > > > > collection into a none multivalued field.
> > > > > >
> > > > > >
> > > > > > Thanks in advanced.
> > > > > >
> > > > >
> > > >
> > >
>

Re: solr update dynamic field generates multiValued error

[Upayavira]

Can you show the error you are getting, and how you know it is because
of stored="true"?

Upayavira

On Mon, Sep 21, 2015, at 09:30 AM, Aman Tandon wrote:
> Hi Erick,
> 
> I am getting the same error because my dynamic field *_coordinate is
> stored="true".
> How can I get rid of this error?
> 
> And I have to use the atomic update. Please help!!
> 
> With Regards
> Aman Tandon
> 
> On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa 
> wrote:
> 
> > Hey Erick, i think that you were right, there was a mix in the schemas and
> > that was generating the error on some of the documents.
> >
> > Thanks for the help guys!
> >
> >
> > 2014-08-05 1:28 GMT-03:00 Erick Erickson :
> >
> > > Hmmm, I jus tried this with a 4.x build and I can update the document
> > > multiple times without a problem. I just indexed the standard exampledocs
> > > and then updated a doc like this (vidcard.xml was the base):
> > >
> > > 
> > > 
> > >   EN7800GTX/2DHTV/256M
> > >
> > >   eoe changed this puppy
> > > 
> > >   
> > > 
> > >
> > > I'm not getting any multiple values in the _coordinate fields. However, I
> > > _do_ get the error if my dynamic *_coordinate field is set to
> > > stored="true".
> > >
> > > Did you perhaps change this at some point? Whenever I change the schema,
> > I
> > > try to 'rm -rf solr/collection/data' just to be sure I've purged all
> > traces
> > > of the former schema definition.
> > >
> > > Best,
> > > Erick
> > >
> > >
> > > On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa 
> > wrote:
> > >
> > > > No, they are not declarad explicitly.
> > > >
> > > > This is how they are created:
> > > >
> > > > 
> > > >
> > > >  > > >  stored="false"/>
> > > >
> > > >  > > > subFieldSuffix="_coordinate"/>
> > > >
> > > >
> > > >
> > > >
> > > > 2014-08-04 22:28 GMT-03:00 Michael Ryan :
> > > >
> > > > > Are the latLong_0_coordinate and latLong_1_coordinate fields
> > populated
> > > > > using copyField? If so, this sounds like it could be
> > > > > https://issues.apache.org/jira/browse/SOLR-3502.
> > > > >
> > > > > -Michael
> > > > >
> > > > > -Original Message-
> > > > > From: Franco Giacosa [mailto:fgiac...@gmail.com]
> > > > > Sent: Monday, August 04, 2014 9:05 PM
> > > > > To: solr-user@lucene.apache.org
> > > > > Subject: solr update dynamic field generates multiValued error
> > > > >
> > > > > Hello everyone, this is my first time posting a question, so forgive
> > me
> > > > if
> > > > > i'm missing something.
> > > > >
> > > > > This is my problem:
> > > > >
> > > > > I have a schema.xml that has the following latLong information
> > > > >
> > > > > The dynamicField generates 2 dynamic fields that have the lat and the
> > > > long
> > > > > (latLong_0_coordinate and latLong_1_coordinate)
> > > > >
> > > > > So for example a document will have
> > > > >
> > > > > "latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
> > > > > "latLong": "40.4114,-74.1031",
> > > > >
> > > > > Now when I try to update a document (i don't update the latLong
> > field.
> > > I
> > > > > just update other parts of the document using atomic update) solr
> > > > > re-creates the dynamicField and adds the same value again, like its
> > > using
> > > > > add instead of set. So when i do an update the fields of the doc look
> > > > like
> > > > > this
> > > > >
> > > > > "latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
> > > > > [-74.1031,-74.1031] "latLong": "40.4114,-74.1031",
> > > > >
> > > > > So the dynamicFields now have 2 values, so the next time that I want
> > to
> > > > > update the document a schema error is throw because im trying to
> > store
> > > a
> > > > > collection into a none multivalued field.
> > > > >
> > > > >
> > > > > Thanks in advanced.
> > > > >
> > > >
> > >
> >

Re: solr update dynamic field generates multiValued error

[Aman Tandon]

Hi Erick,

I am getting the same error because my dynamic field *_coordinate is
stored="true".
How can I get rid of this error?

And I have to use the atomic update. Please help!!

With Regards
Aman Tandon

On Tue, Aug 5, 2014 at 10:27 PM, Franco Giacosa  wrote:

> Hey Erick, i think that you were right, there was a mix in the schemas and
> that was generating the error on some of the documents.
>
> Thanks for the help guys!
>
>
> 2014-08-05 1:28 GMT-03:00 Erick Erickson :
>
> > Hmmm, I jus tried this with a 4.x build and I can update the document
> > multiple times without a problem. I just indexed the standard exampledocs
> > and then updated a doc like this (vidcard.xml was the base):
> >
> > 
> > 
> >   EN7800GTX/2DHTV/256M
> >
> >   eoe changed this puppy
> > 
> >   
> > 
> >
> > I'm not getting any multiple values in the _coordinate fields. However, I
> > _do_ get the error if my dynamic *_coordinate field is set to
> > stored="true".
> >
> > Did you perhaps change this at some point? Whenever I change the schema,
> I
> > try to 'rm -rf solr/collection/data' just to be sure I've purged all
> traces
> > of the former schema definition.
> >
> > Best,
> > Erick
> >
> >
> > On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa 
> wrote:
> >
> > > No, they are not declarad explicitly.
> > >
> > > This is how they are created:
> > >
> > > 
> > >
> > >  > >  stored="false"/>
> > >
> > >  > > subFieldSuffix="_coordinate"/>
> > >
> > >
> > >
> > >
> > > 2014-08-04 22:28 GMT-03:00 Michael Ryan :
> > >
> > > > Are the latLong_0_coordinate and latLong_1_coordinate fields
> populated
> > > > using copyField? If so, this sounds like it could be
> > > > https://issues.apache.org/jira/browse/SOLR-3502.
> > > >
> > > > -Michael
> > > >
> > > > -Original Message-
> > > > From: Franco Giacosa [mailto:fgiac...@gmail.com]
> > > > Sent: Monday, August 04, 2014 9:05 PM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: solr update dynamic field generates multiValued error
> > > >
> > > > Hello everyone, this is my first time posting a question, so forgive
> me
> > > if
> > > > i'm missing something.
> > > >
> > > > This is my problem:
> > > >
> > > > I have a schema.xml that has the following latLong information
> > > >
> > > > The dynamicField generates 2 dynamic fields that have the lat and the
> > > long
> > > > (latLong_0_coordinate and latLong_1_coordinate)
> > > >
> > > > So for example a document will have
> > > >
> > > > "latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
> > > > "latLong": "40.4114,-74.1031",
> > > >
> > > > Now when I try to update a document (i don't update the latLong
> field.
> > I
> > > > just update other parts of the document using atomic update) solr
> > > > re-creates the dynamicField and adds the same value again, like its
> > using
> > > > add instead of set. So when i do an update the fields of the doc look
> > > like
> > > > this
> > > >
> > > > "latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
> > > > [-74.1031,-74.1031] "latLong": "40.4114,-74.1031",
> > > >
> > > > So the dynamicFields now have 2 values, so the next time that I want
> to
> > > > update the document a schema error is throw because im trying to
> store
> > a
> > > > collection into a none multivalued field.
> > > >
> > > >
> > > > Thanks in advanced.
> > > >
> > >
> >
>

Re: Dynamic field rule plugin?

[Upayavira]

On Fri, Aug 28, 2015, at 11:09 PM, Hari Iyer wrote:
> Hi,
> 
> I am new to Solr and am trying to create dynamic field rules in my
> Schema. 
> 
> I would like to use file name suffix to indicate other properties besides
> the data type and multivalued as provided in the default schema. 
> 
> It appears that specifying this via a pattern leads to duplication as
> there
> are various combinations that need to be specified here. It would help to
> have code where I can build parts of the rule 
> 
> e.g. if suffix has '_s' then set stored=true
> 
> if suffix has '_m' then set multivalued=true
> 
> and so on
> 
>  
> 
> From the documentation and various implementation examples (drupal etc) I
> can only see them specifying all combinations.
> 
> Is there any way (plugin?) to incrementally build the rule?

Nope - you will need to have one for each combination, so _s, _sm, _m,
etc. Each field can only match a single field type, otherwise it'd be
impossible to work out which rules should be considered from the
different field types.

Upayavira

Dynamic field rule plugin?

[Hari Iyer]

Hi,

I am new to Solr and am trying to create dynamic field rules in my Schema. 

I would like to use file name suffix to indicate other properties besides
the data type and multivalued as provided in the default schema. 

It appears that specifying this via a pattern leads to duplication as there
are various combinations that need to be specified here. It would help to
have code where I can build parts of the rule 

e.g. if suffix has '_s' then set stored=true

if suffix has '_m' then set multivalued=true

and so on

 

>From the documentation and various implementation examples (drupal etc) I
can only see them specifying all combinations.

Is there any way (plugin?) to incrementally build the rule?

Thanks,

Hari

 

dynamic field value in ValueSource

[Mathijs Corten]

Hello,

At this moment i'm writing my own SOLR plugin with a custom 
ValueSourceParser and ValueSource, it's goal is to read a few fields 
from the document (For now).


I'm currently testing with the following fields:

multiValued="false" />
stored="true" multiValued="false" />


The problem is, when i try to read the dynamic field with the following 
code it does not return the correct value (it prints '`', should be 6):


public FunctionValues getValues(Map map, final AtomicReaderContext 
arc) throws IOException {
final BinaryDocValues valueDynamic = 
FieldCache.DEFAULT.getTerms(arc.reader(), "price_discount_1390980_6", 
false);

return new FunctionValues() {

@Override
public float floatVal(int doc) {
final BytesRef dynamic = valueDynamic.get(doc);
LOGGER.error(dynamic.utf8ToString()); // prints '`'
LOGGER.error(dynamic.toString()); // prints [60 8 0 0 0 6]
}
}
}

When i execute the query: price_discount_1390980_6:6, it gives a result 
so i know the value of the field should be 6.


When i try to read a normal field like active or roomids it works fine:
public FunctionValues getValues(Map map, final AtomicReaderContext 
arc) throws IOException {
final BinaryDocValues valueActive = 
FieldCache.DEFAULT.getTerms(arc.reader(), "active", false);
final BinaryDocValues valueActive = 
FieldCache.DEFAULT.getTerms(arc.reader(), "roomids", false);

return new FunctionValues() {

@Override
public float floatVal(int doc) {
final BytesRef active = valueActive.get(doc);
LOGGER.error(active.utf8ToString());
}
}
}

So my question is:
Does anyone know how to get/fix the dynamic field value? would be nice 
if it would just return 6 :P.


Hope to see an answer soon,

Mathijs

Re: Solr Dynamic Field Performance

[Erick Erickson]

@Bill:
Well, if you're creating 5,000 dynamic fields, whether from lots
of matches to a few patterns or 5,000 patterns,
 you deserve what you get ;). Notice I was careful to say:

bq:  Dynamic fields, once they are actually _in_ a document aren't any
different than statically defined fields

So while your search performance might suffer if you defined 5,000
dynamic fields,
it would have suffered just the same if you'd used 5,000 static fields.

There could conceivably be some indexing performance if you had, say, 5,000
dynamic field _patterns_, but I'm not very worried about that

@Saumitra:
That post has nothing to do with the fact that the fields are dynamic.
The writer
would have had the same problem with statically defined fields that had
the same characteristics. See Hoss' explanation for sparse fields.

Best,
Erick

On Mon, Sep 15, 2014 at 11:12 AM, Saumitra Srivastav
 wrote:
> Thanks for reply Erick. I found this article:
> http://e-mats.org/2011/01/solr-memory-usage-and-dynamic-fields/
> <http://e-mats.org/2011/01/solr-memory-usage-and-dynamic-fields/ >
>
> Just to confirm, the fieldCache doesn't store anything extra when we use
> dynamic fields and do sorting query on that...right?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Dynamic-Field-Performance-tp4158737p4158919.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Dynamic Field Performance

[Erick Erickson]

@Bill:
Well, if you're creating 5,000 dynamic fields, whether from lots
of matches to a few patterns or 5,000 patterns,
 you deserve what you get ;). Notice I was careful to say:

bq:  Dynamic fields, once they are actually _in_ a document aren't any
different than statically defined fields

So while your search performance might suffer if you defined 5,000
dynamic fields,
it would have suffered just the same if you'd used 5,000 static fields.

There could conceivably be some indexing performance if you had, say, 5,000
dynamic field _patterns_, but I'm not very worried about that

@Saumitra:
That post has nothing to do with the fact that the fields are dynamic.
The writer
would have had the same problem with statically defined fields that had
the same characteristics. See Hoss' explanation for sparse fields.

Best,
Erick

On Mon, Sep 15, 2014 at 11:12 AM, Saumitra Srivastav
 wrote:
> Thanks for reply Erick. I found this article:
> http://e-mats.org/2011/01/solr-memory-usage-and-dynamic-fields/
> <http://e-mats.org/2011/01/solr-memory-usage-and-dynamic-fields/ >
>
> Just to confirm, the fieldCache doesn't store anything extra when we use
> dynamic fields and do sorting query on that...right?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Dynamic-Field-Performance-tp4158737p4158919.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Dynamic Field Performance

[Chris Hostetter]

:   
: 
: Just to confirm, the fieldCache doesn't store anything extra when we use
: dynamic fields and do sorting query on that...right?

the way the FieldCache works, is that it has to "cache" a value for every 
document -- even if that document doesn't have a value in the indexed 
field that FieldCache was built from.

so if you have an index of 10K docs, and only 17 of those docs have a 
value in some field "my_really_sparse_int_field" then if you do a search 
and sort on "my_really_sparse_int_field" you're going to get a a 
FieldCache containing an array of 10K ints -- and all but 17 of them will 
be "0".

if you sort on 199 other fields like those, you're going to have 199 other 
arrays of 10K ints.


Instead of relying on the FieldCache, you should consider using DocValues, 
which -- unless i'm forgetting something -- should give you a lot better 
memory efficiencies if you have a lot of dynamicFields that are really 
sparse and you want to sort on them (at the expense of more disk space 
being used in your index).  But i haven't done extensive memory testng of 
this to prove it to myself.


-Hoss
http://www.lucidworks.com/

Re: Solr Dynamic Field Performance

[Saumitra Srivastav]

Thanks for reply Erick. I found this article:
http://e-mats.org/2011/01/solr-memory-usage-and-dynamic-fields/
<http://e-mats.org/2011/01/solr-memory-usage-and-dynamic-fields/ >  

Just to confirm, the fieldCache doesn't store anything extra when we use
dynamic fields and do sorting query on that...right?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Dynamic-Field-Performance-tp4158737p4158919.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Dynamic Field Performance

[Bill Bell]

How about perf if you dynamically create 5000 fields ?

Bill Bell
Sent from mobile


> On Sep 14, 2014, at 10:06 AM, Erick Erickson  wrote:
> 
> Dynamic fields, once they are actually _in_ a document, aren't any
> different than statically defined fields. Literally, there's no place
> in the search code that I know of that _ever_ has to check
> whether a field was dynamically or statically defined.
> 
> AFAIK, the only additional cost would be figuring out which pattern
> matched at index time, which is such a tiny portion of the cost of
> indexing that I doubt you could measure it.
> 
> Best,
> Erick
> 
> On Sun, Sep 14, 2014 at 7:58 AM, Saumitra Srivastav
>  wrote:
>> I have a collection with 200 fields and >300M docs running in cloud mode.
>> Each doc have around 20 fields. I now have a use case where I need to
>> replace these explicit fields with 6 dynamic fields. Each of these 200
>> fields will match one of the 6 dynamic field.
>> 
>> I am evaluating performance implications of switching to dynamicFields. I
>> have tested with a smaller dataset(5M docs) but didn't noticed any indexing
>> or query performance degradation.
>> 
>> Query on dynamic fields will either be faceting, range query or full text
>> search.
>> 
>> Are there any known performance issues with using dynamicFields instead of
>> explicit ones?
>> 
>> 
>> 
>> 
>> --
>> View this message in context: 
>> http://lucene.472066.n3.nabble.com/Solr-Dynamic-Field-Performance-tp4158737.html
>> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Dynamic Field Performance

[Erick Erickson]

Dynamic fields, once they are actually _in_ a document, aren't any
different than statically defined fields. Literally, there's no place
in the search code that I know of that _ever_ has to check
whether a field was dynamically or statically defined.

AFAIK, the only additional cost would be figuring out which pattern
matched at index time, which is such a tiny portion of the cost of
indexing that I doubt you could measure it.

Best,
Erick

On Sun, Sep 14, 2014 at 7:58 AM, Saumitra Srivastav
 wrote:
> I have a collection with 200 fields and >300M docs running in cloud mode.
> Each doc have around 20 fields. I now have a use case where I need to
> replace these explicit fields with 6 dynamic fields. Each of these 200
> fields will match one of the 6 dynamic field.
>
> I am evaluating performance implications of switching to dynamicFields. I
> have tested with a smaller dataset(5M docs) but didn't noticed any indexing
> or query performance degradation.
>
> Query on dynamic fields will either be faceting, range query or full text
> search.
>
> Are there any known performance issues with using dynamicFields instead of
> explicit ones?
>
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Dynamic-Field-Performance-tp4158737.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Solr Dynamic Field Performance

[Saumitra Srivastav]

I have a collection with 200 fields and >300M docs running in cloud mode.
Each doc have around 20 fields. I now have a use case where I need to
replace these explicit fields with 6 dynamic fields. Each of these 200
fields will match one of the 6 dynamic field.

I am evaluating performance implications of switching to dynamicFields. I
have tested with a smaller dataset(5M docs) but didn't noticed any indexing
or query performance degradation. 

Query on dynamic fields will either be faceting, range query or full text
search.

Are there any known performance issues with using dynamicFields instead of
explicit ones?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Dynamic-Field-Performance-tp4158737.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr update dynamic field generates multiValued error

[Franco Giacosa]

Hey Erick, i think that you were right, there was a mix in the schemas and
that was generating the error on some of the documents.

Thanks for the help guys!


2014-08-05 1:28 GMT-03:00 Erick Erickson :

> Hmmm, I jus tried this with a 4.x build and I can update the document
> multiple times without a problem. I just indexed the standard exampledocs
> and then updated a doc like this (vidcard.xml was the base):
>
> 
> 
>   EN7800GTX/2DHTV/256M
>
>   eoe changed this puppy
> 
>   
> 
>
> I'm not getting any multiple values in the _coordinate fields. However, I
> _do_ get the error if my dynamic *_coordinate field is set to
> stored="true".
>
> Did you perhaps change this at some point? Whenever I change the schema, I
> try to 'rm -rf solr/collection/data' just to be sure I've purged all traces
> of the former schema definition.
>
> Best,
> Erick
>
>
> On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa  wrote:
>
> > No, they are not declarad explicitly.
> >
> > This is how they are created:
> >
> > 
> >
> >  >  stored="false"/>
> >
> >  > subFieldSuffix="_coordinate"/>
> >
> >
> >
> >
> > 2014-08-04 22:28 GMT-03:00 Michael Ryan :
> >
> > > Are the latLong_0_coordinate and latLong_1_coordinate fields populated
> > > using copyField? If so, this sounds like it could be
> > > https://issues.apache.org/jira/browse/SOLR-3502.
> > >
> > > -Michael
> > >
> > > -Original Message-
> > > From: Franco Giacosa [mailto:fgiac...@gmail.com]
> > > Sent: Monday, August 04, 2014 9:05 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: solr update dynamic field generates multiValued error
> > >
> > > Hello everyone, this is my first time posting a question, so forgive me
> > if
> > > i'm missing something.
> > >
> > > This is my problem:
> > >
> > > I have a schema.xml that has the following latLong information
> > >
> > > The dynamicField generates 2 dynamic fields that have the lat and the
> > long
> > > (latLong_0_coordinate and latLong_1_coordinate)
> > >
> > > So for example a document will have
> > >
> > > "latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
> > > "latLong": "40.4114,-74.1031",
> > >
> > > Now when I try to update a document (i don't update the latLong field.
> I
> > > just update other parts of the document using atomic update) solr
> > > re-creates the dynamicField and adds the same value again, like its
> using
> > > add instead of set. So when i do an update the fields of the doc look
> > like
> > > this
> > >
> > > "latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
> > > [-74.1031,-74.1031] "latLong": "40.4114,-74.1031",
> > >
> > > So the dynamicFields now have 2 values, so the next time that I want to
> > > update the document a schema error is throw because im trying to store
> a
> > > collection into a none multivalued field.
> > >
> > >
> > > Thanks in advanced.
> > >
> >
>

Re: solr update dynamic field generates multiValued error

[Erick Erickson]

Hmmm, I jus tried this with a 4.x build and I can update the document
multiple times without a problem. I just indexed the standard exampledocs
and then updated a doc like this (vidcard.xml was the base):



  EN7800GTX/2DHTV/256M

  eoe changed this puppy

  


I'm not getting any multiple values in the _coordinate fields. However, I
_do_ get the error if my dynamic *_coordinate field is set to stored="true".

Did you perhaps change this at some point? Whenever I change the schema, I
try to 'rm -rf solr/collection/data' just to be sure I've purged all traces
of the former schema definition.

Best,
Erick


On Mon, Aug 4, 2014 at 7:04 PM, Franco Giacosa  wrote:

> No, they are not declarad explicitly.
>
> This is how they are created:
>
> 
>
>   stored="false"/>
>
>  subFieldSuffix="_coordinate"/>
>
>
>
>
> 2014-08-04 22:28 GMT-03:00 Michael Ryan :
>
> > Are the latLong_0_coordinate and latLong_1_coordinate fields populated
> > using copyField? If so, this sounds like it could be
> > https://issues.apache.org/jira/browse/SOLR-3502.
> >
> > -Michael
> >
> > -Original Message-
> > From: Franco Giacosa [mailto:fgiac...@gmail.com]
> > Sent: Monday, August 04, 2014 9:05 PM
> > To: solr-user@lucene.apache.org
> > Subject: solr update dynamic field generates multiValued error
> >
> > Hello everyone, this is my first time posting a question, so forgive me
> if
> > i'm missing something.
> >
> > This is my problem:
> >
> > I have a schema.xml that has the following latLong information
> >
> > The dynamicField generates 2 dynamic fields that have the lat and the
> long
> > (latLong_0_coordinate and latLong_1_coordinate)
> >
> > So for example a document will have
> >
> > "latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
> > "latLong": "40.4114,-74.1031",
> >
> > Now when I try to update a document (i don't update the latLong field. I
> > just update other parts of the document using atomic update) solr
> > re-creates the dynamicField and adds the same value again, like its using
> > add instead of set. So when i do an update the fields of the doc look
> like
> > this
> >
> > "latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
> > [-74.1031,-74.1031] "latLong": "40.4114,-74.1031",
> >
> > So the dynamicFields now have 2 values, so the next time that I want to
> > update the document a schema error is throw because im trying to store a
> > collection into a none multivalued field.
> >
> >
> > Thanks in advanced.
> >
>

Re: solr update dynamic field generates multiValued error

[Franco Giacosa]

No, they are not declarad explicitly.

This is how they are created:










2014-08-04 22:28 GMT-03:00 Michael Ryan :

> Are the latLong_0_coordinate and latLong_1_coordinate fields populated
> using copyField? If so, this sounds like it could be
> https://issues.apache.org/jira/browse/SOLR-3502.
>
> -Michael
>
> -Original Message-
> From: Franco Giacosa [mailto:fgiac...@gmail.com]
> Sent: Monday, August 04, 2014 9:05 PM
> To: solr-user@lucene.apache.org
> Subject: solr update dynamic field generates multiValued error
>
> Hello everyone, this is my first time posting a question, so forgive me if
> i'm missing something.
>
> This is my problem:
>
> I have a schema.xml that has the following latLong information
>
> The dynamicField generates 2 dynamic fields that have the lat and the long
> (latLong_0_coordinate and latLong_1_coordinate)
>
> So for example a document will have
>
> "latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
> "latLong": "40.4114,-74.1031",
>
> Now when I try to update a document (i don't update the latLong field. I
> just update other parts of the document using atomic update) solr
> re-creates the dynamicField and adds the same value again, like its using
> add instead of set. So when i do an update the fields of the doc look like
> this
>
> "latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
> [-74.1031,-74.1031] "latLong": "40.4114,-74.1031",
>
> So the dynamicFields now have 2 values, so the next time that I want to
> update the document a schema error is throw because im trying to store a
> collection into a none multivalued field.
>
>
> Thanks in advanced.
>

RE: solr update dynamic field generates multiValued error

[Michael Ryan]

Are the latLong_0_coordinate and latLong_1_coordinate fields populated using 
copyField? If so, this sounds like it could be 
https://issues.apache.org/jira/browse/SOLR-3502.

-Michael

-Original Message-
From: Franco Giacosa [mailto:fgiac...@gmail.com] 
Sent: Monday, August 04, 2014 9:05 PM
To: solr-user@lucene.apache.org
Subject: solr update dynamic field generates multiValued error

Hello everyone, this is my first time posting a question, so forgive me if i'm 
missing something.

This is my problem:

I have a schema.xml that has the following latLong information

The dynamicField generates 2 dynamic fields that have the lat and the long 
(latLong_0_coordinate and latLong_1_coordinate)

So for example a document will have

"latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
"latLong": "40.4114,-74.1031",

Now when I try to update a document (i don't update the latLong field. I just 
update other parts of the document using atomic update) solr re-creates the 
dynamicField and adds the same value again, like its using add instead of set. 
So when i do an update the fields of the doc look like this

"latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
[-74.1031,-74.1031] "latLong": "40.4114,-74.1031",

So the dynamicFields now have 2 values, so the next time that I want to update 
the document a schema error is throw because im trying to store a collection 
into a none multivalued field.


Thanks in advanced.

solr update dynamic field generates multiValued error

[Franco Giacosa]

Hello everyone, this is my first time posting a question, so forgive me if
i'm missing something.

This is my problem:

I have a schema.xml that has the following latLong information

The dynamicField generates 2 dynamic fields that have the lat and the long
(latLong_0_coordinate and latLong_1_coordinate)

So for example a document will have

"latLong_0_coordinate": 40.4114, "latLong_1_coordinate": -74.1031,
"latLong": "40.4114,-74.1031",

Now when I try to update a document (i don't update the latLong field. I
just update other parts of the document using atomic update) solr
re-creates the dynamicField and adds the same value again, like its using
add instead of set. So when i do an update the fields of the doc look like
this

"latLong_0_coordinate": [40.4114,40.4114] "latLong_1_coordinate":
[-74.1031,-74.1031] "latLong": "40.4114,-74.1031",

So the dynamicFields now have 2 values, so the next time that I want to
update the document a schema error is throw because im trying to store a
collection into a none multivalued field.


Thanks in advanced.

Re: Dynamic field doesnt work

[Anton Gavazuk]

Jack

thank you very much! So obvious - damn, Ive mixed my codebase with solr
shema xml :)

2014-07-03 14:23 GMT+02:00 Jack Krupansky :

> s/dynamic_field/dynamicField/

Re: Dynamic field doesnt work

[Alexandre Rafalovitch]

So, I was right about misspelling, but totally missed the one staring
me in the face :-)

Hopefully, that was all.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency


On Thu, Jul 3, 2014 at 7:23 PM, Jack Krupansky  wrote:
> s/dynamic_field/dynamicField/
>
> -- Jack Krupansky
>
> -Original Message- From: Alexandre Rafalovitch
> Sent: Thursday, July 3, 2014 5:45 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Dynamic field doesnt work
>
>
> I would say something is misspelt somewhere. Put a dynamic field
> called '*' and see what you end up getting in the schema. Obviously,
> on a separate test instance.
>
> Regards,
>   Alex.
> Personal website: http://www.outerthoughts.com/
> Current project: http://www.solr-start.com/ - Accelerating your Solr
> proficiency
>
>
> On Thu, Jul 3, 2014 at 4:11 PM, Anton Gavazuk 
> wrote:
>>
>> Hi dear SOLR community,
>>
>> I'm having a problem with a dynamic field in my schema - basically when
>> I'm
>> adding any document with key matching dynamic field definition I'm getting
>> back an error which says something like unknown field 'test_exattrs'
>>
>> The schema contains such definition
>> > stored="true"/>
>>
>> SOLR version 4.7.2
>>
>> What might be wrong with my setup?
>>
>> Thanks,
>> Anton
>
>

Re: Dynamic field doesnt work

[Jack Krupansky]

s/dynamic_field/dynamicField/

-- Jack Krupansky

-Original Message- 
From: Alexandre Rafalovitch

Sent: Thursday, July 3, 2014 5:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Dynamic field doesnt work

I would say something is misspelt somewhere. Put a dynamic field
called '*' and see what you end up getting in the schema. Obviously,
on a separate test instance.

Regards,
  Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr 
proficiency



On Thu, Jul 3, 2014 at 4:11 PM, Anton Gavazuk  
wrote:

Hi dear SOLR community,

I'm having a problem with a dynamic field in my schema - basically when 
I'm

adding any document with key matching dynamic field definition I'm getting
back an error which says something like unknown field 'test_exattrs'

The schema contains such definition
stored="true"/>


SOLR version 4.7.2

What might be wrong with my setup?

Thanks,
Anton 

Re: Dynamic field doesnt work

[Alexandre Rafalovitch]

I would say something is misspelt somewhere. Put a dynamic field
called '*' and see what you end up getting in the schema. Obviously,
on a separate test instance.

Regards,
   Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency


On Thu, Jul 3, 2014 at 4:11 PM, Anton Gavazuk  wrote:
> Hi dear SOLR community,
>
> I'm having a problem with a dynamic field in my schema - basically when I'm
> adding any document with key matching dynamic field definition I'm getting
> back an error which says something like unknown field 'test_exattrs'
>
> The schema contains such definition
> 
>
> SOLR version 4.7.2
>
> What might be wrong with my setup?
>
> Thanks,
> Anton

Dynamic field doesnt work

[Anton Gavazuk]

Hi dear SOLR community,

I'm having a problem with a dynamic field in my schema - basically when I'm
adding any document with key matching dynamic field definition I'm getting
back an error which says something like unknown field 'test_exattrs'

The schema contains such definition


SOLR version 4.7.2

What might be wrong with my setup?

Thanks,
Anton

Re: dynamic field assignments

[John Thorhauer]

Chris,

Thanks so much for the suggestion.  I will look into this approach.  It
looks very promising!

John


On Mon, May 5, 2014 at 9:50 PM, Chris Hostetter wrote:

>
> : My understanding is that DynamicField can do something like
> : FOO_BAR_TEXT_* but what I really need is *_TEXT_* as I might have
> : FOO_BAR_TEXT_1 but I also might have WIDGET_BAR_TEXT_2.  Both of those
> : field names need to map to a field type of 'fullText'.
>
> I'm pretty sure you can get what you are after with the new Manged Schema
> functionality...
>
> https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode
>
> https://cwiki.apache.org/confluence/display/solr/Managed+Schema+Definition+in+SolrConfig
>
> Assuming you have managed schema enabled in solrconfig.xml, and you define
> both of your fieldTypes using names like "text" and "select" then
> something like this should work in your processor chain...
>
>  
>.*_TEXT_.*
>text
>  
>  
>.*_SELECT_.*
>select
>  
>
>
> (Normally that processor is used once with multiple value->type mappings
> -- but in your case you don't care about the run-time value, just the run
> time field name regex (which should also be configurable according
> to the various FieldNameSelector rules...
>
>
> https://lucene.apache.org/solr/4_8_0/solr-core/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.html
>
> https://lucene.apache.org/solr/4_8_0/solr-core/org/apache/solr/update/processor/FieldMutatingUpdateProcessorFactory.html
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
John Thorhauer
Director/Remote Interfaces
Yakabod, Inc.
301-662-4554 x2105

Re: dynamic field assignments

[Chris Hostetter]

: My understanding is that DynamicField can do something like
: FOO_BAR_TEXT_* but what I really need is *_TEXT_* as I might have
: FOO_BAR_TEXT_1 but I also might have WIDGET_BAR_TEXT_2.  Both of those
: field names need to map to a field type of 'fullText'.

I'm pretty sure you can get what you are after with the new Manged Schema 
functionality...

https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode
https://cwiki.apache.org/confluence/display/solr/Managed+Schema+Definition+in+SolrConfig

Assuming you have managed schema enabled in solrconfig.xml, and you define 
both of your fieldTypes using names like "text" and "select" then 
something like this should work in your processor chain... 

 
   .*_TEXT_.*
   text
 
 
   .*_SELECT_.*
   select
 


(Normally that processor is used once with multiple value->type mappings 
-- but in your case you don't care about the run-time value, just the run 
time field name regex (which should also be configurable according 
to the various FieldNameSelector rules...

https://lucene.apache.org/solr/4_8_0/solr-core/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.html
https://lucene.apache.org/solr/4_8_0/solr-core/org/apache/solr/update/processor/FieldMutatingUpdateProcessorFactory.html


-Hoss
http://www.lucidworks.com/

Re: dynamic field assignments

[Jack Krupansky]

Solr only supports mapping of values to field names, not mapping to field 
types. Field names are then mapped to field types.


DynamicField only supports prefix OR suffix wildcard, not both in the same 
pattern.


In the future, please take care to design your data model with the features 
and limitations of Solr in mind, then you won't find yourself boxed into a 
corner like this.


In short, it sounds like you need to go back to the drawing board and 
redesign your data model.


Also, take a look at the schemaless and dynamic schema modes that have 
recently been added to Solr - these provide the tools to dynamically add 
fields to a schema.


See:
https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode

Dynamic field are a very powerful feature of Solr, but please don't treat 
them as a panacea for weak data modeling. Use them only in moderation.


-- Jack Krupansky

-Original Message- 
From: John Thorhauer

Sent: Friday, April 25, 2014 7:49 AM
To: solr-user@lucene.apache.org
Subject: Re: dynamic field assignments

Jack,

Thanks for your help.


Reading your last paragraph, how is that any different than exactly what
DynamicField actually does?


My understanding is that DynamicField can do something like
FOO_BAR_TEXT_* but what I really need is *_TEXT_* as I might have
FOO_BAR_TEXT_1 but I also might have WIDGET_BAR_TEXT_2.  Both of those
field names need to map to a field type of 'fullText'.


You say you want to change fields at "run time" - what is "run time"? When
exactly do your field names change?


What I mean is that when the document is fed to solr.  So on the
update process when the document is being indexed.  The document that
is being indexed may have fields that are unknown until the time of
indexing.  However, some of those fields will follow a predictable
naming patter as mentioned above.


You can always write an update request processor to do any manipulation of
field values at index time.


I see that Solr has this capability.  However, I dont think I need to
manipulate field values.  I need to map a field/value to a particular
fieldType for indexing. 

Re: dynamic field assignments

[John Thorhauer]

Jack,

Thanks for your help.

> Reading your last paragraph, how is that any different than exactly what
> DynamicField actually does?

My understanding is that DynamicField can do something like
FOO_BAR_TEXT_* but what I really need is *_TEXT_* as I might have
FOO_BAR_TEXT_1 but I also might have WIDGET_BAR_TEXT_2.  Both of those
field names need to map to a field type of 'fullText'.

> You say you want to change fields at "run time" - what is "run time"? When
> exactly do your field names change?

What I mean is that when the document is fed to solr.  So on the
update process when the document is being indexed.  The document that
is being indexed may have fields that are unknown until the time of
indexing.  However, some of those fields will follow a predictable
naming patter as mentioned above.

> You can always write an update request processor to do any manipulation of
> field values at index time.

I see that Solr has this capability.  However, I dont think I need to
manipulate field values.  I need to map a field/value to a particular
fieldType for indexing.

Re: dynamic field assignments

[Jack Krupansky]

Reading your last paragraph, how is that any different than exactly what 
DynamicField actually does?


You say you want to change fields at "run time" - what is "run time"? When 
exactly do your field names change? To be clear, field names do not change 
in Solr once the data is written to the index.


You can always write an update request processor to do any manipulation of 
field values at index time.


Have you had your data model reviewed by a professional Solr consultant to 
verify that it is indeed a reasonable approach? We can answer direct 
questions on this list, but it is not a substitute for professional review.


-- Jack Krupansky

-Original Message- 
From: John Thorhauer

Sent: Friday, April 25, 2014 7:10 AM
To: solr-user@lucene.apache.org
Subject: dynamic field assignments

I have a scenario where I would like dynamically assign incoming
document fields to two different solr schema fieldTypes.  One
fieldType will be an exact match fieldType while the other will be a
full text fieldType.  I know that I can use the dynamicField to assign
fields using the asterisk in a naming pattern.  However, I have a set
of incoming data in which the field names can change at run time.  The
fields will follow a predictable pattern but the pattern can not be
recognized using the dynamicField type.

So here is an example of the new types of field names that I need to
be able to process:

FOO_BAR_TEXT_1
FOO_BAR_TEXT_2
FOO_BAR_TEXT_3
FOO_BAR_TEXT_4

FOO_BAR_SELECT_1
FOO_BAR_SELECT_2
FOO_BAR_SELECT_3

So the above fields will not be defined in advance.  I need to map all
fields with the name FOO_BAR_SELECT_* to a fieldType of 'exactMatch'
and I need to map all of the fields with name FOO_BAR_TEXT_* to a
fieldType of full'text'.  I was hoping there might be a way of doing
this dynamically.  Does anyone have any ideas how to approach this?

Thanks,
John Thorhauer 

dynamic field assignments

[John Thorhauer]

I have a scenario where I would like dynamically assign incoming
document fields to two different solr schema fieldTypes.  One
fieldType will be an exact match fieldType while the other will be a
full text fieldType.  I know that I can use the dynamicField to assign
fields using the asterisk in a naming pattern.  However, I have a set
of incoming data in which the field names can change at run time.  The
fields will follow a predictable pattern but the pattern can not be
recognized using the dynamicField type.

So here is an example of the new types of field names that I need to
be able to process:

FOO_BAR_TEXT_1
FOO_BAR_TEXT_2
FOO_BAR_TEXT_3
FOO_BAR_TEXT_4

FOO_BAR_SELECT_1
FOO_BAR_SELECT_2
FOO_BAR_SELECT_3

So the above fields will not be defined in advance.  I need to map all
fields with the name FOO_BAR_SELECT_* to a fieldType of 'exactMatch'
and I need to map all of the fields with name FOO_BAR_TEXT_* to a
fieldType of full'text'.  I was hoping there might be a way of doing
this dynamically.  Does anyone have any ideas how to approach this?

Thanks,
John Thorhauer

Re: dynamic field question

[Otis Gospodnetic]

David,

Yes. Document grouping (aka  field collapsing) will help you here. It
should also allow you to create a better search experience on the front end
- it's often better to narrow down where in a large document a match is
than give users a large doc and say: we know the match is in here
somewhere, you just have to locate it now.

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Oct 9, 2013 6:21 AM, "Twomey, David"  wrote:

> OK.  Then the JSON returned would contain a lot of documents that are
> really sections. This would work fine for the use-case I mentioned but I
> also use the index for full-text search of the whole document.  Therefore,
> I would need to parse the result JSON in a way that combines the solr docs
> returned in to one virtual doc based on source document ID.
>
> Is that correct?
>
> On 10/9/13 6:15 AM, "Jack Krupansky"  wrote:
>
> >I'd suggest that each of your source document sections would be a
> >distinct
> >solr document. All of the sections could have a "source document ID"
> >field
> >to tie them together.
> >
> >Dynamic fields work best when used in moderation. Your use case seems
> >like
> >an excessive use of dynamic fields.
> >
> >-- Jack Krupansky
> >
> >-Original Message-
> >From: Twomey, David
> >Sent: Tuesday, October 08, 2013 6:59 PM
> >To: solr-user@lucene.apache.org
> >Subject: dynamic field question
> >
> >
> >I am having trouble trying to return a particular dynamic field only
> >instead
> >of all dynamic fields.
> >
> >Imagine I have a document with an unknown number of sections.  Each
> >section
> >can have a 'title' and a 'body'
> >
> >I have each section title and body as dynamic fields such as
> >section_title_*
> >and section_body_*
> >
> >Imagine that some documents contain a section that has a title="Appendix"
> >
> >I want a query that will find all docs with that section and return just
> >the
> >Appendix section.
> >
> >I don't know how to return just that one section though
> >
> >I can copyField my dynamic field section_title_* into a static field
> >called
> >section_titles and query that for docs that contain the Appendix
> >
> >But I don't know how to only return that one dynamic field
> >
> >?q=section_titles:Appendix&fl=section_body_*
> >
> >Any ideas?   I can't seem to put a conditional in the fl parameter
> >
> >
> >
>
>

Re: dynamic field question

[Twomey, David]

OK.  Then the JSON returned would contain a lot of documents that are
really sections. This would work fine for the use-case I mentioned but I
also use the index for full-text search of the whole document.  Therefore,
I would need to parse the result JSON in a way that combines the solr docs
returned in to one virtual doc based on source document ID.

Is that correct?

On 10/9/13 6:15 AM, "Jack Krupansky"  wrote:

>I'd suggest that each of your source document sections would be a
>distinct 
>solr document. All of the sections could have a "source document ID"
>field 
>to tie them together.
>
>Dynamic fields work best when used in moderation. Your use case seems
>like 
>an excessive use of dynamic fields.
>
>-- Jack Krupansky
>
>-Original Message-
>From: Twomey, David
>Sent: Tuesday, October 08, 2013 6:59 PM
>To: solr-user@lucene.apache.org
>Subject: dynamic field question
>
>
>I am having trouble trying to return a particular dynamic field only
>instead 
>of all dynamic fields.
>
>Imagine I have a document with an unknown number of sections.  Each
>section 
>can have a 'title' and a 'body'
>
>I have each section title and body as dynamic fields such as
>section_title_* 
>and section_body_*
>
>Imagine that some documents contain a section that has a title="Appendix"
>
>I want a query that will find all docs with that section and return just
>the 
>Appendix section.
>
>I don't know how to return just that one section though
>
>I can copyField my dynamic field section_title_* into a static field
>called 
>section_titles and query that for docs that contain the Appendix
>
>But I don't know how to only return that one dynamic field
>
>?q=section_titles:Appendix&fl=section_body_*
>
>Any ideas?   I can't seem to put a conditional in the fl parameter
>
>
>

Re: dynamic field question

[Aloke Ghoshal]

Hi David,

A separate Solr document for each section is a good option if you also need
to handle phrases, case, special characters, etc. within the title field.
How do you map them to dynamic fields?

E.g.: "Appendix for cities", "APPENDIX 1: Cities"

Regards,
Aloke


On Wed, Oct 9, 2013 at 9:45 AM, Jack Krupansky wrote:

> I'd suggest that each of your source document sections would be a distinct
> solr document. All of the sections could have a "source document ID" field
> to tie them together.
>
> Dynamic fields work best when used in moderation. Your use case seems like
> an excessive use of dynamic fields.
>
> -- Jack Krupansky
>
> -Original Message- From: Twomey, David
> Sent: Tuesday, October 08, 2013 6:59 PM
> To: solr-user@lucene.apache.org
> Subject: dynamic field question
>
>
>
> I am having trouble trying to return a particular dynamic field only
> instead of all dynamic fields.
>
> Imagine I have a document with an unknown number of sections.  Each
> section can have a 'title' and a 'body'
>
> I have each section title and body as dynamic fields such as
> section_title_* and section_body_*
>
> Imagine that some documents contain a section that has a title="Appendix"
>
> I want a query that will find all docs with that section and return just
> the Appendix section.
>
> I don't know how to return just that one section though
>
> I can copyField my dynamic field section_title_* into a static field
> called section_titles and query that for docs that contain the Appendix
>
> But I don't know how to only return that one dynamic field
>
> ?q=section_titles:Appendix&fl=**section_body_*
>
> Any ideas?   I can't seem to put a conditional in the fl parameter
>
>
>
>