Re: Able to search with indexed=false and docvalues=true
On Mon, 2018-11-19 at 22:19 -0500, Rahul Goswami wrote: > I am using SolrCloud 7.2.1. My understanding is that setting > docvalues=true would optimize faceting, grouping and sorting; but for > a field to be searchable it needs to be indexed=true. Erick explained the search thing, so I'll just note that faceting on a DocValues=true indexed=false field on a multi-shard index also has a performance penalty as the field will be slow-searched (using the DocValues) in the secondary fine-counting phase. - Toke Eskildsen, Royal Danish Library
Solr statistics
HI ,
I am using stats component to determine stats on two columns together.
stats query is failing with following exception -
*Invalid Number: num_useful *
*Query response : *
{
"responseHeader": {
"zkConnected": true,
"status": 400,
"QTime": 19,
"params": {
* "q": "cid:HT201222",*
"indent": [
"true",
"true"
],
"stats": "true",
"rows": [
"0",
"0"
],
"wt": "json",
"stats.field": "*{!func}termfreq('num_not_useful','num_useful')*"
}
},
"error": {
"metadata": [
"error-class",
"org.apache.solr.common.SolrException",
"root-error-class",
"org.apache.solr.common.SolrException"
],
"msg": "Invalid Number: num_useful",
"code": 400
}
}
Could you please help me in resolving the issue? Thanks.
Regards,
Anil
RE: Error:Missing Required Fields for Atomic Updates
Hi Rahul Router name is "compositeId". To let you know the issue is intermittent. Thanks Rajeswari -Original Message- From: Rahul Goswami [mailto:rahul196...@gmail.com] Sent: Tuesday, November 20, 2018 11:47 AM To: solr-user@lucene.apache.org Subject: Re: Error:Missing Required Fields for Atomic Updates What is the Router name for your collection? Is it "implicit" (You can know this from the "Overview" of you collection in the admin UI) ? If yes, what is the router.field parameter the collection was created with? Rahul On Mon, Nov 19, 2018 at 11:19 PM Rajeswari Kolluri < rajeswari.koll...@oracle.com> wrote: > > Hi Rahul > > Below is part of schema , entityid is my unique id field. Getting > exception missing required field for "category" during atomic updates. > > > entityid > required="true" multiValued="false" /> > required="false" multiValued="false" /> > stored="true" required="false" multiValued="false" /> > stored="true" required="false" multiValued="false" /> > stored="true" required="false" multiValued="false" /> > stored="true" required="false" multiValued="false" /> > stored="true" required="false" multiValued="false" /> > required="true" docValues="true" /> > required="false" multiValued="true" /> > > > > Thanks > Rajeswari > > -Original Message- > From: Rahul Goswami [mailto:rahul196...@gmail.com] > Sent: Tuesday, November 20, 2018 9:33 AM > To: solr-user@lucene.apache.org > Subject: Re: Error:Missing Required Fields for Atomic Updates > > What’s your update query? > > You need to provide the unique id field of the document you are updating. > > Rahul > > On Mon, Nov 19, 2018 at 10:58 PM Rajeswari Kolluri < > rajeswari.koll...@oracle.com> wrote: > > > Hi, > > > > > > > > > > > > Using Solr 7.5.0. While performing atomic updates on a document on > > Solr Cloud using SolJ getting exceptions "Missing Required Field". > > > > > > > > Please let me know the solution, would not want to update the > > required fields during atomic updates. > > > > > > > > > > > > Thanks > > > > Rajeswari > > >
solr is using TLS1.0
Hi All, I have enabled SSL for solr using steps mentioned over Lucene website .And though solr console URL is now secure(https) ,it is still using TLS v1.0. I have tried few things to force SSL to use TLS1.2 protocol ,but they have not worked for me . While trying to do same ,I have observed solr itself does not offer any solr property to specify cipher ,algorithm or TLS version . Following things have been tried : 1.key store /trust store for solr to enable SSL with different key algorithm ,etc combinations for the certificates 2.different solr versions for step 1(solr 5.x,6.x,7.x-we are using solr 5.3 currently) 3.using java version 1.8 and adding solr certificate in java keystore to enforce TLS1.2 4.various kind of keystores like JKS,PKCS12,etc Can anyone offer any suggestions on same ?I have not been able to find much about same niofficial site. Thanks & Regards, - Anchal Sharma
replication?command=restore not working
Hi I am working on backups. I have created a backup with below command: /http://dev/solr/XXX/replication?command=backup&name=XXXBackup&location=D:\Backup\/ all worked fine, files have been created. I wanted to restore the index from this backup with below command: /http://dev/solr/XXX/replication?command=restore&name=XXXBackup&location=D:\Backup\/ I am getting below response when I am checking the status of the restore command: / snapshot.XXXBackup success / All looks fine, but the index is empty, no records in the index. This Index is really small as currently only testing this feature. When I run the /replication?command=details/ command, I have noticed that the location of the index points not to the index but to the restore folder within the index folder, instead of /D:\Data\XXX/ it points to: /D:\Data\XXX\restore.snapshot.XXXBackup/ and Solr created /index.properties/ file which points to the /restore.snapshot.XXXBackup/ *Question 1: Am I doing something wrong?* I have copied all the files from /D:\Backup\snapshot.XXXBackup/ to the index location and all is fine, all records are there, so the backup worked fine, but the restore command not. *Question 2: My second question, is it "fine" to copy all the files from the backup folder into the index folder?* Thank you Damian -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr statistics
You are using a function query as stats.field and as seen here:
https://lucene.apache.org/solr/guide/7_4/function-queries.html the syntax
for termfreq is termfreq(field_name, value). You're using
termfreq('num_not_useful','num_useful'). It looks like num_useful is a
numeric (int, float) type in your schema and you are passing a string
('num_useful'). If that is the case then you need to pass a numeric value
(e.g., termfreq(num_not_useful, 99)). Does it make sense?
Edward
On Tue, Nov 20, 2018 at 6:59 AM Anil wrote:
> HI ,
>
> I am using stats component to determine stats on two columns together.
> stats query is failing with following exception -
> *Invalid Number: num_useful *
>
>
> *Query response : *
> {
> "responseHeader": {
> "zkConnected": true,
> "status": 400,
> "QTime": 19,
> "params": {
> * "q": "cid:HT201222",*
> "indent": [
> "true",
> "true"
> ],
> "stats": "true",
> "rows": [
> "0",
> "0"
> ],
> "wt": "json",
> "stats.field": "*{!func}termfreq('num_not_useful','num_useful')*"
> }
> },
> "error": {
> "metadata": [
> "error-class",
> "org.apache.solr.common.SolrException",
> "root-error-class",
> "org.apache.solr.common.SolrException"
> ],
> "msg": "Invalid Number: num_useful",
> "code": 400
> }
> }
>
> Could you please help me in resolving the issue? Thanks.
>
> Regards,
> Anil
>
Re: Solr statistics
Thanks Edward.
Can we find stats on two fields ? (eg - sum = sum of (userful+not useful)) ?
On Tue, 20 Nov 2018 at 16:14, Edward Ribeiro
wrote:
> You are using a function query as stats.field and as seen here:
> https://lucene.apache.org/solr/guide/7_4/function-queries.html the syntax
> for termfreq is termfreq(field_name, value). You're using
> termfreq('num_not_useful','num_useful'). It looks like num_useful is a
> numeric (int, float) type in your schema and you are passing a string
> ('num_useful'). If that is the case then you need to pass a numeric value
> (e.g., termfreq(num_not_useful, 99)). Does it make sense?
>
> Edward
>
> On Tue, Nov 20, 2018 at 6:59 AM Anil wrote:
>
> > HI ,
> >
> > I am using stats component to determine stats on two columns together.
> > stats query is failing with following exception -
> > *Invalid Number: num_useful *
> >
> >
> > *Query response : *
> > {
> > "responseHeader": {
> > "zkConnected": true,
> > "status": 400,
> > "QTime": 19,
> > "params": {
> > * "q": "cid:HT",*
> > "indent": [
> > "true",
> > "true"
> > ],
> > "stats": "true",
> > "rows": [
> > "0",
> > "0"
> > ],
> > "wt": "json",
> > "stats.field": "*{!func}termfreq('num_not_useful','num_useful')*"
> > }
> > },
> > "error": {
> > "metadata": [
> > "error-class",
> > "org.apache.solr.common.SolrException",
> > "root-error-class",
> > "org.apache.solr.common.SolrException"
> > ],
> > "msg": "Invalid Number: num_useful",
> > "code": 400
> > }
> > }
> >
> > Could you please help me in resolving the issue? Thanks.
> >
> > Regards,
> > Anil
> >
>
Re: Solr statistics
You can get stats on multiple fields, it's just a matter of specifying the
stats.field parameter multiple times:
stats.field=field1&stats.field=field2, etc.
Besides that Solr also has a built-in sum function so that you can sum the
values of two or more fields: sum(field1, field2, ...). You could try to
use that function in a stats field.
Edward
On Tue, Nov 20, 2018 at 9:24 AM Anil wrote:
>
> Thanks Edward.
>
> Can we find stats on two fields ? (eg - sum = sum of (userful+not
useful)) ?
>
> On Tue, 20 Nov 2018 at 16:14, Edward Ribeiro
> wrote:
>
> > You are using a function query as stats.field and as seen here:
> > https://lucene.apache.org/solr/guide/7_4/function-queries.html the
syntax
> > for termfreq is termfreq(field_name, value). You're using
> > termfreq('num_not_useful','num_useful'). It looks like num_useful is a
> > numeric (int, float) type in your schema and you are passing a string
> > ('num_useful'). If that is the case then you need to pass a numeric
value
> > (e.g., termfreq(num_not_useful, 99)). Does it make sense?
> >
> > Edward
> >
> > On Tue, Nov 20, 2018 at 6:59 AM Anil wrote:
> >
> > > HI ,
> > >
> > > I am using stats component to determine stats on two columns together.
> > > stats query is failing with following exception -
> > > *Invalid Number: num_useful *
> > >
> > >
> > > *Query response : *
> > > {
> > > "responseHeader": {
> > > "zkConnected": true,
> > > "status": 400,
> > > "QTime": 19,
> > > "params": {
> > > * "q": "cid:HT",*
> > > "indent": [
> > > "true",
> > > "true"
> > > ],
> > > "stats": "true",
> > > "rows": [
> > > "0",
> > > "0"
> > > ],
> > > "wt": "json",
> > > "stats.field":
"*{!func}termfreq('num_not_useful','num_useful')*"
> > > }
> > > },
> > > "error": {
> > > "metadata": [
> > > "error-class",
> > > "org.apache.solr.common.SolrException",
> > > "root-error-class",
> > > "org.apache.solr.common.SolrException"
> > > ],
> > > "msg": "Invalid Number: num_useful",
> > > "code": 400
> > > }
> > > }
> > >
> > > Could you please help me in resolving the issue? Thanks.
> > >
> > > Regards,
> > > Anil
> > >
> >
Re: Error:Missing Required Fields for Atomic Updates
On 11/19/2018 9:19 PM, Rajeswari Kolluri wrote: Below is part of schema , entityid is my unique id field. Getting exception missing required field for "category" during atomic updates. Your category field is not stored. And it's required. It does have docValues, but unless your Solr version and schema version are new enough, this definition will make the field unsuitable for atomic updates. What is your Solr version, and what version do you have declared in your schema? (newest schema version that I know about is 1.6) This is the information about what's needed for Atomic Updates to work: https://lucene.apache.org/solr/guide/7_5/updating-parts-of-documents.html#field-storage That documentation is not as specific as it needs to be when it says that the field can have docValues enabled to meet the requirements. You must also have useDocValuesAsStored enabled. This is enabled by default if your schema version is 1.6 or later ... which is the schema version that examples are set to in the later releases of Solr. But if your schema was designed for an earlier version of Solr, its version probably may be something earlier than 1.6. The minimum Solr version required to be able to use docValues as stored is 5.5.0. If you have a Solr version before 5.5.0, docValues will not work for atomic updates at all. Thanks, Shawn
solr 7.3.1 how to parse LatLonPointSpatialField in custom .jar
Hello,
We are using solr with a home made jar with a custom function.
function(0.1,1.0,43.8341851366,5.7818349,43.8342868634,5.7821059,latlng_pi)
where latlng_pi is a document field of type location
In solr 5.5.2, location was defined like this
and parsed in the jar like this (with fq being an instance of
org.apache.solr.search.FunctionQParser)
value = fp.parseValueSource()
In solr 7.3.1, we changed the definition to
because solr.LatLonType is now deprecated.
we now have an error
"A ValueSource isn't directly available from this field. Instead try a
query using the distance as the score."
from org.apache.solr.schema.AbstractSpatialFieldType
@Override
public ValueSource getValueSource(SchemaField field, QParser parser) {
//This is different from Solr 3 LatLonType's approach which uses the
MultiValueSource concept to directly expose
// the x & y pair of FieldCache value sources.
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
"A ValueSource isn't directly available from this field. Instead try a
query using the distance as the score.");
}
To correct this error, we tried to see how the value was parsed in
*GeoDistValueSourceParser,
but it seems to us (we are not java programmers) very hacky and complicated
and we would like to know if there is a simple solution to parse a
*LatLonPointSpatialField
in our jar.
Thanks,
Elisabeth
Solr cache clear
Hi all, Without restarting is it possible to clear the cache?
Re: Extracting important multi term phrases from the text
@David Sorry for late reply. The SKG query that I am using is actually
fairly basic in itself. For example,
{
> "queries":[
> "dataStoreId:\"123\"",
> "text:\"foo\""
> ],
> "compare":[
> {
> "type":"text_shingles",
> "limit":30,
> "discover_values":true
> }
> ]
> }
What I am expecting is that SKG will return words/phrases that are related
to the term "foo". I am filtering the text through StopWordFilter before
that. I have also found that specifying a good foreground can drastically
improve the results.
Good luck!
- Pratik
On Fri, Nov 16, 2018 at 11:15 AM Alexandre Rafalovitch
wrote:
> Good catch Pratik.
>
> It is in Javadoc, but not in the reference guide:
>
> https://lucene.apache.org/core/6_3_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilterFactory.html
> . I'll try to fix that later (SOLR-12996).
>
> Regards,
>Alex.
> On Fri, 16 Nov 2018 at 10:44, Pratik Patel wrote:
> >
> > @Markus @Walter, @Alexandre is right. The culprit was not StopWord
> Filter,
> > it was ShingleFilter. I could not find parameter filterToken in
> > documentation, is it a new addition? BTW, I tried that and it works.
> Thanks!
> > I still ended up using pattern replacement filter because I did not want
> > any single word string in that field.
> >
> > @David I am using SKG through the plugin. So it is a POST request with
> > query in body. I haven't yet upgraded to version 7.5.
> >
> > Thank you all for the help!
> >
> > Regards,
> > Pratik
> >
> > On Fri, Nov 16, 2018 at 8:36 AM David Hastings <
> hastings.recurs...@gmail.com>
> > wrote:
> >
> > > Which function of the SKG are you using? significantTerms?
> > >
> > > On Thu, Nov 15, 2018 at 7:09 PM Alexandre Rafalovitch <
> arafa...@gmail.com>
> > > wrote:
> > >
> > > > I think the underscore actually comes from the Shingles (parameter
> > > > fillerToken). Have you tried setting it to empty string?
> > > >
> > > > Regards,
> > > >Alex.
> > > > On Thu, 15 Nov 2018 at 17:16, Pratik Patel
> wrote:
> > > > >
> > > > > Hi Markus,
> > > > >
> > > > > Thanks for the reply. I tried using ShingleFilter and it seems to
> > > > > be working. However, I am hitting an issue when it is used with
> > > > > StopWordFilter. StopWordFilter leaves an underscore "_" for removed
> > > words
> > > > > and it kind of screws up the data in index.
> > > > >
> > > > > I tried setting enablePositionIncrements="false" for stop word
> filter
> > > but
> > > > > that parameter only works for lucene version 4.3 or earlier. Looks
> like
> > > > > it's an open issue in lucene
> > > > > https://issues.apache.org/jira/browse/LUCENE-4065
> > > > >
> > > > > For now, I am trying to find a workaround using
> > > > PatternReplaceFilterFactory.
> > > > >
> > > > > Regards,
> > > > > Pratik
> > > > >
> > > > > On Thu, Nov 15, 2018 at 4:15 PM Markus Jelsma <
> > > > markus.jel...@openindex.io>
> > > > > wrote:
> > > > >
> > > > > > Hello Pratik,
> > > > > >
> > > > > > We would use ShingleFilter for this indeed. If you only want
> > > > > > bigrams/shingles, don't forget to disable outputUnigrams and set
> both
> > > > > > shinle size limits to 2.
> > > > > >
> > > > > > Regards,
> > > > > > Markus
> > > > > >
> > > > > > -Original message-
> > > > > > > From:Pratik Patel
> > > > > > > Sent: Thursday 15th November 2018 17:00
> > > > > > > To: solr-user@lucene.apache.org
> > > > > > > Subject: Extracting important multi term phrases from the text
> > > > > > >
> > > > > > > Hello Everyone,
> > > > > > >
> > > > > > > Standard way of tokenizing in solr would divide the text by
> white
> > > > space
> > > > > > in
> > > > > > > solr.
> > > > > > >
> > > > > > > Is there a way by which we can index multi-term phrases like
> > > "Machine
> > > > > > > Learning" instead of "Machine", "Learning"?
> > > > > > > Is it possible to create a specific field type for such phrases
> > > > which has
> > > > > > > its own indexing pipeline? I am open to storing n-grams but
> these
> > > > n-grams
> > > > > > > would be across terms and not just one term? In other words, I
> > > don't
> > > > want
> > > > > > > to store n-grams of the term "machine", I want to store n-grams
> > > for a
> > > > > > > sentence like below.
> > > > > > >
> > > > > > > "I like machine learning" --> "I like", "like machine",
> "machine
> > > > > > learning"
> > > > > > > and so on.
> > > > > > >
> > > > > > > It seems like Shingle Filter (
> > > > > > >
> > > > > >
> > > >
> > >
> https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-ShingleFilter
> > > > > > )
> > > > > > > may be used for this. Is there a better alternative?
> > > > > > >
> > > > > > > I want to use this field as an input to Semantic Knowledge
> Graph.
> > > The
> > > > > > > plugin works great for words. But now I want to use it for
> phrases.
> > > > Any
> > > > > > > idea around this would be really helpful.
> > > > > > >
> > > > > > > Thanks a lot!
> > > > > > >
> > > > > > > - Pratik
> > >
Re: Solr cache clear
On 11/20/2018 9:25 AM, Rajdeep Sahoo wrote: Hi all, Without restarting is it possible to clear the cache? You'll need to clarify what cache you're talking about, but I think for the most part that if you reload the core (or collection if running SolrCloud) that all caches should be rebuilt empty. Thanks, Shawn
Solr Cloud configuration
I cant seem to find the documentation on how to actually edit the schema file myself, everything seems to lead me to using an API to add fields and stop words etc. this is more or less obnoxious, and the admin api for adding fields/field types is not exactly functional. is there a guide or something to let me know how to do it normally like in standalone solr?
Re: Solr Cloud configuration
Hi David, You can upload configuration to the zookeeper - it is nearly the same as the standaloneconfig. You can also edit the schema.xml in this file. At least I do it like this. Mit freundlichem Gruß / kind regards Wolfgang Freudenberger Pure Host IT-Services Münsterstr. 14 48341 Altenberge GERMANY Tel.: (+49) 25 71 - 99 20 170 Fax: (+49) 25 71 - 99 20 171 Umsatzsteuer ID DE259181123 Informieren Sie sich über unser gesamtes Leistungsspektrum unter www.pure-host.de Get our whole services at www.pure-host.de Am 20.11.2018 um 19:38 schrieb David Hastings: I cant seem to find the documentation on how to actually edit the schema file myself, everything seems to lead me to using an API to add fields and stop words etc. this is more or less obnoxious, and the admin api for adding fields/field types is not exactly functional. is there a guide or something to let me know how to do it normally like in standalone solr?
Re: Solr Cloud configuration
Thanks, researching that now, but this seems extremely annoying. wouldnt it just be easier if you could edit the config files raw from the admin UI? On Tue, Nov 20, 2018 at 1:41 PM Pure Host - Wolfgang Freudenberger < w.freudenber...@pure-host.de> wrote: > Hi David, > > > You can upload configuration to the zookeeper - it is nearly the same as > the standaloneconfig. > > You can also edit the schema.xml in this file. At least I do it like this. > > Mit freundlichem Gruß / kind regards > > Wolfgang Freudenberger > Pure Host IT-Services > Münsterstr. 14 > 48341 Altenberge > GERMANY > Tel.: (+49) 25 71 - 99 20 170 > Fax: (+49) 25 71 - 99 20 171 > > Umsatzsteuer ID DE259181123 > > Informieren Sie sich über unser gesamtes Leistungsspektrum unter > www.pure-host.de > Get our whole services at www.pure-host.de > > Am 20.11.2018 um 19:38 schrieb David Hastings: > > I cant seem to find the documentation on how to actually edit the schema > > file myself, everything seems to lead me to using an API to add fields > and > > stop words etc. this is more or less obnoxious, and the admin api for > > adding fields/field types is not exactly functional. is there a guide or > > something to let me know how to do it normally like in standalone solr? > > > >
Re: Solr Cloud configuration
David: Sure would. See https://issues.apache.org/jira/browse/SOLR-5287. Especially the bits about how allowing this leads to security vulnerabilities. You're not the first one who had this idea ;). Whether those security issues are still valid is another question I suppose. Best, Erick On Tue, Nov 20, 2018 at 11:01 AM David Hastings wrote: > > Thanks, researching that now, but this seems extremely annoying. wouldnt > it just be easier if you could edit the config files raw from the admin > UI? > > On Tue, Nov 20, 2018 at 1:41 PM Pure Host - Wolfgang Freudenberger < > w.freudenber...@pure-host.de> wrote: > > > Hi David, > > > > > > You can upload configuration to the zookeeper - it is nearly the same as > > the standaloneconfig. > > > > You can also edit the schema.xml in this file. At least I do it like this. > > > > Mit freundlichem Gruß / kind regards > > > > Wolfgang Freudenberger > > Pure Host IT-Services > > Münsterstr. 14 > > 48341 Altenberge > > GERMANY > > Tel.: (+49) 25 71 - 99 20 170 > > Fax: (+49) 25 71 - 99 20 171 > > > > Umsatzsteuer ID DE259181123 > > > > Informieren Sie sich über unser gesamtes Leistungsspektrum unter > > www.pure-host.de > > Get our whole services at www.pure-host.de > > > > Am 20.11.2018 um 19:38 schrieb David Hastings: > > > I cant seem to find the documentation on how to actually edit the schema > > > file myself, everything seems to lead me to using an API to add fields > > and > > > stop words etc. this is more or less obnoxious, and the admin api for > > > adding fields/field types is not exactly functional. is there a guide or > > > something to let me know how to do it normally like in standalone solr? > > > > > > >
Re: Solr Cloud configuration
Well considering that any access to the user interface by anyone can completely destroy entire collections/cores, I would think the security of the stop word file wouldnt be that important Thanks Erick, it seems the only reason I have any desire to use SolrCloud is the use of streaming expressions. I think thats the only benefit that more hardware cant solve. On Tue, Nov 20, 2018 at 2:17 PM Erick Erickson wrote: > David: > > Sure would. See https://issues.apache.org/jira/browse/SOLR-5287. > Especially the bits about how allowing this leads to security > vulnerabilities. You're not the first one who had this idea ;). > > Whether those security issues are still valid is another question I > suppose. > > Best, > Erick > On Tue, Nov 20, 2018 at 11:01 AM David Hastings > wrote: > > > > Thanks, researching that now, but this seems extremely annoying. wouldnt > > it just be easier if you could edit the config files raw from the admin > > UI? > > > > On Tue, Nov 20, 2018 at 1:41 PM Pure Host - Wolfgang Freudenberger < > > w.freudenber...@pure-host.de> wrote: > > > > > Hi David, > > > > > > > > > You can upload configuration to the zookeeper - it is nearly the same > as > > > the standaloneconfig. > > > > > > You can also edit the schema.xml in this file. At least I do it like > this. > > > > > > Mit freundlichem Gruß / kind regards > > > > > > Wolfgang Freudenberger > > > Pure Host IT-Services > > > Münsterstr. 14 > > > 48341 Altenberge > > > GERMANY > > > Tel.: (+49) 25 71 - 99 20 170 > > > Fax: (+49) 25 71 - 99 20 171 > > > > > > Umsatzsteuer ID DE259181123 > > > > > > Informieren Sie sich über unser gesamtes Leistungsspektrum unter > > > www.pure-host.de > > > Get our whole services at www.pure-host.de > > > > > > Am 20.11.2018 um 19:38 schrieb David Hastings: > > > > I cant seem to find the documentation on how to actually edit the > schema > > > > file myself, everything seems to lead me to using an API to add > fields > > > and > > > > stop words etc. this is more or less obnoxious, and the admin api > for > > > > adding fields/field types is not exactly functional. is there a > guide or > > > > something to let me know how to do it normally like in standalone > solr? > > > > > > > > > > >
Re: Solr cache clear
Why would you want to? This sounds like an XY problem, there's some problem you think would be cured by clearing the cache. What is that problem? Because I doubt this would do anything useful, pretty soon the caches would be filled up again and you'd be right back where you started and the real solution is to stop doing whatever you're doing that leads to whatever the real problem is. Maybe reducing the cache sizes. Best, Erick On Tue, Nov 20, 2018 at 9:05 AM Shawn Heisey wrote: > > On 11/20/2018 9:25 AM, Rajdeep Sahoo wrote: > > Hi all, > > Without restarting is it possible to clear the cache? > > You'll need to clarify what cache you're talking about, but I think for > the most part that if you reload the core (or collection if running > SolrCloud) that all caches should be rebuilt empty. > > Thanks, > Shawn >
Re: Solr Cloud configuration
David, One benefit of the way recommended in the reference guide is that it lets you use zookeeper upconfig/downconfig as deployment tools on a set of text files, which in turn allows you to manage your Solr configuration like any other bit of source code, e.g. with version control and, if your situation permits, things like branching and pull requests or other review mechanisms. In particular I have found the capacity to view diffs, have peers review, and the ease of deploying changes to test and staging environments before moving them into production is worth the effort all by itself. HTH, AC On Tue, Nov 20, 2018 at 2:22 PM David Hastings wrote: > Well considering that any access to the user interface by anyone can > completely destroy entire collections/cores, I would think the security of > the stop word file wouldnt be that important > Thanks Erick, it seems the only reason I have any desire to use SolrCloud > is the use of streaming expressions. I think thats the only benefit that > more hardware cant solve. > > On Tue, Nov 20, 2018 at 2:17 PM Erick Erickson > wrote: > > > David: > > > > Sure would. See https://issues.apache.org/jira/browse/SOLR-5287. > > Especially the bits about how allowing this leads to security > > vulnerabilities. You're not the first one who had this idea ;). > > > > Whether those security issues are still valid is another question I > > suppose. > > > > Best, > > Erick > > On Tue, Nov 20, 2018 at 11:01 AM David Hastings > > wrote: > > > > > > Thanks, researching that now, but this seems extremely annoying. > wouldnt > > > it just be easier if you could edit the config files raw from the admin > > > UI? > > > > > > On Tue, Nov 20, 2018 at 1:41 PM Pure Host - Wolfgang Freudenberger < > > > w.freudenber...@pure-host.de> wrote: > > > > > > > Hi David, > > > > > > > > > > > > You can upload configuration to the zookeeper - it is nearly the same > > as > > > > the standaloneconfig. > > > > > > > > You can also edit the schema.xml in this file. At least I do it like > > this. > > > > > > > > Mit freundlichem Gruß / kind regards > > > > > > > > Wolfgang Freudenberger > > > > Pure Host IT-Services > > > > Münsterstr. 14 > > > > 48341 Altenberge > > > > GERMANY > > > > Tel.: (+49) 25 71 - 99 20 170 > > > > Fax: (+49) 25 71 - 99 20 171 > > > > > > > > Umsatzsteuer ID DE259181123 > > > > > > > > Informieren Sie sich über unser gesamtes Leistungsspektrum unter > > > > www.pure-host.de > > > > Get our whole services at www.pure-host.de > > > > > > > > Am 20.11.2018 um 19:38 schrieb David Hastings: > > > > > I cant seem to find the documentation on how to actually edit the > > schema > > > > > file myself, everything seems to lead me to using an API to add > > fields > > > > and > > > > > stop words etc. this is more or less obnoxious, and the admin api > > for > > > > > adding fields/field types is not exactly functional. is there a > > guide or > > > > > something to let me know how to do it normally like in standalone > > solr? > > > > > > > > > > > > > > > >
Re: Solr cache clear
Disabling or reducing autowarming can help too, in addition to cache size reduction. Edward Em ter, 20 de nov de 2018 17:29, Erick Erickson Why would you want to? This sounds like an XY problem, there's some > problem you think would be cured by clearing the cache. What is > that problem? > > Because I doubt this would do anything useful, pretty soon the caches > would be filled up again and you'd be right back where you started and > the real solution is to stop doing whatever you're doing that leads to > whatever the real problem is. Maybe reducing the cache sizes. > > Best, > Erick > On Tue, Nov 20, 2018 at 9:05 AM Shawn Heisey wrote: > > > > On 11/20/2018 9:25 AM, Rajdeep Sahoo wrote: > > > Hi all, > > > Without restarting is it possible to clear the cache? > > > > You'll need to clarify what cache you're talking about, but I think for > > the most part that if you reload the core (or collection if running > > SolrCloud) that all caches should be rebuilt empty. > > > > Thanks, > > Shawn > > >
Re: Solr Cloud configuration
Hi David, Well, as a last resort you can resort to classic schema.xml if you are using standalone Solr and don't bother to give up schema API. Then you are back to manually editing conf/ files. See: https://lucene.apache.org/solr/guide/7_4/schema-factory-definition-in-solrconfig.html Best regards, Edward Em ter, 20 de nov de 2018 18:21, Adam Constabaris David, > > One benefit of the way recommended in the reference guide is that it lets > you use zookeeper upconfig/downconfig as deployment tools on a set of text > files, which in turn allows you to manage your Solr configuration like any > other bit of source code, e.g. with version control and, if your situation > permits, things like branching and pull requests or other review > mechanisms. > > In particular I have found the capacity to view diffs, have peers review, > and the ease of deploying changes to test and staging environments before > moving them into production is worth the effort all by itself. > > HTH, > > AC > > > > On Tue, Nov 20, 2018 at 2:22 PM David Hastings < > hastings.recurs...@gmail.com> > wrote: > > > Well considering that any access to the user interface by anyone can > > completely destroy entire collections/cores, I would think the security > of > > the stop word file wouldnt be that important > > Thanks Erick, it seems the only reason I have any desire to use SolrCloud > > is the use of streaming expressions. I think thats the only benefit that > > more hardware cant solve. > > > > On Tue, Nov 20, 2018 at 2:17 PM Erick Erickson > > wrote: > > > > > David: > > > > > > Sure would. See https://issues.apache.org/jira/browse/SOLR-5287. > > > Especially the bits about how allowing this leads to security > > > vulnerabilities. You're not the first one who had this idea ;). > > > > > > Whether those security issues are still valid is another question I > > > suppose. > > > > > > Best, > > > Erick > > > On Tue, Nov 20, 2018 at 11:01 AM David Hastings > > > wrote: > > > > > > > > Thanks, researching that now, but this seems extremely annoying. > > wouldnt > > > > it just be easier if you could edit the config files raw from the > admin > > > > UI? > > > > > > > > On Tue, Nov 20, 2018 at 1:41 PM Pure Host - Wolfgang Freudenberger < > > > > w.freudenber...@pure-host.de> wrote: > > > > > > > > > Hi David, > > > > > > > > > > > > > > > You can upload configuration to the zookeeper - it is nearly the > same > > > as > > > > > the standaloneconfig. > > > > > > > > > > You can also edit the schema.xml in this file. At least I do it > like > > > this. > > > > > > > > > > Mit freundlichem Gruß / kind regards > > > > > > > > > > Wolfgang Freudenberger > > > > > Pure Host IT-Services > > > > > Münsterstr. 14 > > > > > 48341 Altenberge > > > > > GERMANY > > > > > Tel.: (+49) 25 71 - 99 20 170 > > > > > Fax: (+49) 25 71 - 99 20 171 > > > > > > > > > > Umsatzsteuer ID DE259181123 > > > > > > > > > > Informieren Sie sich über unser gesamtes Leistungsspektrum unter > > > > > www.pure-host.de > > > > > Get our whole services at www.pure-host.de > > > > > > > > > > Am 20.11.2018 um 19:38 schrieb David Hastings: > > > > > > I cant seem to find the documentation on how to actually edit the > > > schema > > > > > > file myself, everything seems to lead me to using an API to add > > > fields > > > > > and > > > > > > stop words etc. this is more or less obnoxious, and the admin > api > > > for > > > > > > adding fields/field types is not exactly functional. is there a > > > guide or > > > > > > something to let me know how to do it normally like in standalone > > > solr? > > > > > > > > > > > > > > > > > > > > > >
Re: Solr Cloud configuration
But then I would lose the steaming expressions right? > On Nov 20, 2018, at 6:00 PM, Edward Ribeiro wrote: > > Hi David, > > Well, as a last resort you can resort to classic schema.xml if you are > using standalone Solr and don't bother to give up schema API. Then you are > back to manually editing conf/ files. See: > > https://lucene.apache.org/solr/guide/7_4/schema-factory-definition-in-solrconfig.html > > Best regards, > Edward > > > Em ter, 20 de nov de 2018 18:21, Adam Constabaris escreveu: > >> David, >> >> One benefit of the way recommended in the reference guide is that it lets >> you use zookeeper upconfig/downconfig as deployment tools on a set of text >> files, which in turn allows you to manage your Solr configuration like any >> other bit of source code, e.g. with version control and, if your situation >> permits, things like branching and pull requests or other review >> mechanisms. >> >> In particular I have found the capacity to view diffs, have peers review, >> and the ease of deploying changes to test and staging environments before >> moving them into production is worth the effort all by itself. >> >> HTH, >> >> AC >> >> >> >> On Tue, Nov 20, 2018 at 2:22 PM David Hastings < >> hastings.recurs...@gmail.com> >> wrote: >> >>> Well considering that any access to the user interface by anyone can >>> completely destroy entire collections/cores, I would think the security >> of >>> the stop word file wouldnt be that important >>> Thanks Erick, it seems the only reason I have any desire to use SolrCloud >>> is the use of streaming expressions. I think thats the only benefit that >>> more hardware cant solve. >>> >>> On Tue, Nov 20, 2018 at 2:17 PM Erick Erickson >>> wrote: >>> David: Sure would. See https://issues.apache.org/jira/browse/SOLR-5287. Especially the bits about how allowing this leads to security vulnerabilities. You're not the first one who had this idea ;). Whether those security issues are still valid is another question I suppose. Best, Erick On Tue, Nov 20, 2018 at 11:01 AM David Hastings wrote: > > Thanks, researching that now, but this seems extremely annoying. >>> wouldnt > it just be easier if you could edit the config files raw from the >> admin > UI? > > On Tue, Nov 20, 2018 at 1:41 PM Pure Host - Wolfgang Freudenberger < > w.freudenber...@pure-host.de> wrote: > >> Hi David, >> >> >> You can upload configuration to the zookeeper - it is nearly the >> same as >> the standaloneconfig. >> >> You can also edit the schema.xml in this file. At least I do it >> like this. >> >> Mit freundlichem Gruß / kind regards >> >> Wolfgang Freudenberger >> Pure Host IT-Services >> Münsterstr. 14 >> 48341 Altenberge >> GERMANY >> Tel.: (+49) 25 71 - 99 20 170 >> Fax: (+49) 25 71 - 99 20 171 >> >> Umsatzsteuer ID DE259181123 >> >> Informieren Sie sich über unser gesamtes Leistungsspektrum unter >> www.pure-host.de >> Get our whole services at www.pure-host.de >> >>> Am 20.11.2018 um 19:38 schrieb David Hastings: >>> I cant seem to find the documentation on how to actually edit the schema >>> file myself, everything seems to lead me to using an API to add fields >> and >>> stop words etc. this is more or less obnoxious, and the admin >> api for >>> adding fields/field types is not exactly functional. is there a guide or >>> something to let me know how to do it normally like in standalone solr? >>> >> >> >>> >>
Re: solr is using TLS1.0
On 11/20/2018 3:02 AM, Anchal Sharma2 wrote: I have enabled SSL for solr using steps mentioned over Lucene website .And though solr console URL is now secure(https) ,it is still using TLS v1.0. I have tried few things to force SSL to use TLS1.2 protocol ,but they have not worked for me . While trying to do same ,I have observed solr itself does not offer any solr property to specify cipher ,algorithm or TLS version . Following things have been tried : 1.key store /trust store for solr to enable SSL with different key algorithm ,etc combinations for the certificates 2.different solr versions for step 1(solr 5.x,6.x,7.x-we are using solr 5.3 currently) 3.using java version 1.8 and adding solr certificate in java keystore to enforce TLS1.2 Solr lets Java and Jetty handle TLS. Solr itself doesn't get involved except to provide information to other software. There are a whole lot of versions of Java 8, and at least three vendors for it. The big names are Oracle, IBM, and OpenJDK. What vendor and exact version of Java are you running? What OS is it on? Do you have the "unlimited JCE" addition installed in your Java and enabled? If your Java version is new enough, you won't need to mess with JCE. See this page: https://golb.hplar.ch/2017/10/JCE-policy-changes-in-Java-SE-8u151-and-8u152.html Solr 5.3 ships with Jetty 9.2.11, which is considered very outdated by the Jetty project -- released well over three years ago. From the perspective of the Solr project, version 5.3 is also very old -- two major versions behind what's current, and also released three years ago. Jetty 9.2 is up to 9.2.26. The current version is Jetty 9.4.14. The latest version of Solr (7.5.0) is shipping with Jetty 9.4.11. I think Jetty will likely be upgraded to the latest release for Solr 7.6.0. Have you made any changes to the Jetty config, particularly jetty-ssl.xml? One thing you might try, although I'll warn you that it may make no difference at all, is to remove the parts of that config file that exclude certain protocols and ciphers, letting Jetty decide for itself what it should use. Recent versions of Jetty and Java have very good defaults. I do not know whether Jetty 9.2.11 (included with Solr 5.3, as mentioned) has good defaults or not. Thanks, Shawn
Re: Able to search with indexed=false and docvalues=true
Erick and Toke, Thank you for the replies. I am surprised there already isn’t a JIRA for this. In my opinion, this should be an error condition on search or alternatively should simply be giving zero results. That would be a defined behavior as opposed to now, where the searches are not particularly functional for any industry size load anyway. Thanks, Rahul On Tue, Nov 20, 2018 at 3:37 AM Toke Eskildsen wrote: > On Mon, 2018-11-19 at 22:19 -0500, Rahul Goswami wrote: > > I am using SolrCloud 7.2.1. My understanding is that setting > > docvalues=true would optimize faceting, grouping and sorting; but for > > a field to be searchable it needs to be indexed=true. > > Erick explained the search thing, so I'll just note that faceting on a > DocValues=true indexed=false field on a multi-shard index also has a > performance penalty as the field will be slow-searched (using the > DocValues) in the secondary fine-counting phase. > > - Toke Eskildsen, Royal Danish Library > > >
RE: Error:Missing Required Fields for Atomic Updates
Hi Shawn, Schema version is 1.6 and Solr version is 7.5. While creating a document , I have provide the required information to field " category". Now, I would like to update document for other set of fields using Atomic Update but not category field. While performing atomic update ,exception caught with "Missing required field on category". Thanks Rajeswari -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Tuesday, November 20, 2018 8:38 PM To: solr-user@lucene.apache.org Subject: Re: Error:Missing Required Fields for Atomic Updates On 11/19/2018 9:19 PM, Rajeswari Kolluri wrote: > Below is part of schema , entityid is my unique id field. Getting > exception missing required field for "category" during atomic updates. > stored="false" required="true" docValues="true" /> Your category field is not stored. And it's required. It does have docValues, but unless your Solr version and schema version are new enough, this definition will make the field unsuitable for atomic updates. What is your Solr version, and what version do you have declared in your schema? (newest schema version that I know about is 1.6) This is the information about what's needed for Atomic Updates to work: https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F5_updating-2Dparts-2Dof-2Ddocuments.html-23field-2Dstorage&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HEBHcZdIeDHAvl73r3r2wMPqU2fuBMQEqkrJDiOkgU0&m=E0eYGosr2K2uq4yr-Kk3r1d49jU0cFLhakbRohIP0rQ&s=o3x6BbYYdEkVFyvQKF3ZIZlwV5iBh3J3lsskRzt--S8&e= That documentation is not as specific as it needs to be when it says that the field can have docValues enabled to meet the requirements. You must also have useDocValuesAsStored enabled. This is enabled by default if your schema version is 1.6 or later ... which is the schema version that examples are set to in the later releases of Solr. But if your schema was designed for an earlier version of Solr, its version probably may be something earlier than 1.6. The minimum Solr version required to be able to use docValues as stored is 5.5.0. If you have a Solr version before 5.5.0, docValues will not work for atomic updates at all. Thanks, Shawn
Re: Restrict search on term/phrase count in document.
Thanks for your replies. The requirement is basically to avoid documents which may have a match but with very less number of term or phrase in it. May a be 1/2 matches. The user is interested in those document which has matched term/phrase beyond a certain number. This can be a valid feature/requirement. Best, Modassar On Mon, Nov 19, 2018 at 10:55 PM Alessandro Benedetti wrote: > I agree with Alexandre, it seems suspicious. > Anyway, if you want to query for single term frequencies occurrence you > could make use of the function range query parser : > > > https://lucene.apache.org/solr/guide/6_6/other-parsers.html#OtherParsers-FunctionRangeQueryParser > > And the function: > > termfreq > Returns the number of times the term appears in the field for that > document. > termfreq(text,'memory') > > tf > Term frequency; returns the term frequency factor for the given term, using > the Similarity for the field. The tf-idf value increases proportionally to > the number of times a word appears in the document, but is offset by the > frequency of the word in the document, which helps to control for the fact > that some words are generally more common than others. See also idf. > tf(text,'solr') > > Cheers > > > > - > --- > Alessandro Benedetti > Search Consultant, R&D Software Engineer, Director > Sease Ltd. - www.sease.io > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Re: Able to search with indexed=false and docvalues=true
On 11/20/2018 8:18 PM, Rahul Goswami wrote: Erick and Toke, Thank you for the replies. I am surprised there already isn’t a JIRA for this. In my opinion, this should be an error condition on search or alternatively should simply be giving zero results. That would be a defined behavior as opposed to now, where the searches are not particularly functional for any industry size load anyway. It wouldn't be a good idea to turn that into an error condition, at least not in any 7.x version. There could be a lot of users out there who are unknowingly relying on that functionality, and would be very surprised to find their index doesn't work any more when they upgrade. It's slow, but maybe they have very small indexes. Maybe the error condition should be related to a new schema property, something like allowQueryOnDocValues. This would default to true with current schema versions and false in the next schema version, which I think is 1.7. Then a user could choose to allow them on a field-by-field basis, by reading documentation that outlines the severe performance disadvantages. Thanks, Shawn
Re: Error:Missing Required Fields for Atomic Updates
bq. While creating a document , I have provide the required information to field "category". But you did not store it, you have stored="false" for that field. Atomic updates require that all source fields are stored. What happens under the covers is that the stored data is read from the doc, your update is overlaid and the doc is re-indexed. And since category is not stored, it's not present in the new index and the update fails. That's just the way atomic updates work. That said, if you _do_ store the cat field and still have this problem a JIRA would be in order. Frankly I do not know if setting docValues="true" and usedocValuesAsStored="true" will work or not, have you tried? Best, Erick On Tue, Nov 20, 2018 at 8:06 PM Rajeswari Kolluri wrote: > > Hi Shawn, > > Schema version is 1.6 and Solr version is 7.5. > > While creating a document , I have provide the required information to field > " category". > > Now, I would like to update document for other set of fields using Atomic > Update but not category field. > > While performing atomic update ,exception caught with "Missing required field > on category". > > > Thanks > Rajeswari > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Tuesday, November 20, 2018 8:38 PM > To: solr-user@lucene.apache.org > Subject: Re: Error:Missing Required Fields for Atomic Updates > > On 11/19/2018 9:19 PM, Rajeswari Kolluri wrote: > > Below is part of schema , entityid is my unique id field. Getting > > exception missing required field for "category" during atomic updates. > > > > stored="false" required="true" docValues="true" /> > > Your category field is not stored. And it's required. It does have > docValues, but unless your Solr version and schema version are new enough, > this definition will make the field unsuitable for atomic updates. > > What is your Solr version, and what version do you have declared in your > schema? (newest schema version that I know about is 1.6) > > This is the information about what's needed for Atomic Updates to work: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F5_updating-2Dparts-2Dof-2Ddocuments.html-23field-2Dstorage&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HEBHcZdIeDHAvl73r3r2wMPqU2fuBMQEqkrJDiOkgU0&m=E0eYGosr2K2uq4yr-Kk3r1d49jU0cFLhakbRohIP0rQ&s=o3x6BbYYdEkVFyvQKF3ZIZlwV5iBh3J3lsskRzt--S8&e= > > That documentation is not as specific as it needs to be when it says that the > field can have docValues enabled to meet the requirements. You must also > have useDocValuesAsStored enabled. This is enabled by default if your schema > version is 1.6 or later ... which is the schema version that examples are set > to in the later releases of Solr. But if your schema was designed for an > earlier version of Solr, its version probably may be something earlier than > 1.6. > > The minimum Solr version required to be able to use docValues as stored is > 5.5.0. If you have a Solr version before 5.5.0, docValues will not work for > atomic updates at all. > > Thanks, > Shawn >
RE: Error:Missing Required Fields for Atomic Updates
Hi Erick As per the documentation - > https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org > _solr_guide_7-5F5_updating-2Dparts-2Dof-2Ddocuments.html-23field-2Dsto > rage&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HEBHcZdI > eDHAvl73r3r2wMPqU2fuBMQEqkrJDiOkgU0&m=E0eYGosr2K2uq4yr-Kk3r1d49jU0cFLh > akbRohIP0rQ&s=o3x6BbYYdEkVFyvQKF3ZIZlwV5iBh3J3lsskRzt--S8&e= For atomic updates to work either Stored or docValues is to be true. Current configuration satisfies this condition. " The core functionality of atomically updating a document requires that all fields in your schema must be configured as stored (stored="true") or docValues (docValues="true")" Anyways will give a try by setting stored =true and also setting usedocValuesAsStored="true" will update you soon. Thanks Rajeswari -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Wednesday, November 21, 2018 10:34 AM To: solr-user Subject: Re: Error:Missing Required Fields for Atomic Updates bq. While creating a document , I have provide the required information to field "category". But you did not store it, you have stored="false" for that field. Atomic updates require that all source fields are stored. What happens under the covers is that the stored data is read from the doc, your update is overlaid and the doc is re-indexed. And since category is not stored, it's not present in the new index and the update fails. That's just the way atomic updates work. That said, if you _do_ store the cat field and still have this problem a JIRA would be in order. Frankly I do not know if setting docValues="true" and usedocValuesAsStored="true" will work or not, have you tried? Best, Erick On Tue, Nov 20, 2018 at 8:06 PM Rajeswari Kolluri wrote: > > Hi Shawn, > > Schema version is 1.6 and Solr version is 7.5. > > While creating a document , I have provide the required information to field > " category". > > Now, I would like to update document for other set of fields using Atomic > Update but not category field. > > While performing atomic update ,exception caught with "Missing required field > on category". > > > Thanks > Rajeswari > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Tuesday, November 20, 2018 8:38 PM > To: solr-user@lucene.apache.org > Subject: Re: Error:Missing Required Fields for Atomic Updates > > On 11/19/2018 9:19 PM, Rajeswari Kolluri wrote: > > Below is part of schema , entityid is my unique id field. Getting > > exception missing required field for "category" during atomic updates. > > > > stored="false" required="true" docValues="true" /> > > Your category field is not stored. And it's required. It does have > docValues, but unless your Solr version and schema version are new enough, > this definition will make the field unsuitable for atomic updates. > > What is your Solr version, and what version do you have declared in > your schema? (newest schema version that I know about is 1.6) > > This is the information about what's needed for Atomic Updates to work: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org > _solr_guide_7-5F5_updating-2Dparts-2Dof-2Ddocuments.html-23field-2Dsto > rage&d=DwIDaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=HEBHcZdI > eDHAvl73r3r2wMPqU2fuBMQEqkrJDiOkgU0&m=E0eYGosr2K2uq4yr-Kk3r1d49jU0cFLh > akbRohIP0rQ&s=o3x6BbYYdEkVFyvQKF3ZIZlwV5iBh3J3lsskRzt--S8&e= > > That documentation is not as specific as it needs to be when it says that the > field can have docValues enabled to meet the requirements. You must also > have useDocValuesAsStored enabled. This is enabled by default if your schema > version is 1.6 or later ... which is the schema version that examples are set > to in the later releases of Solr. But if your schema was designed for an > earlier version of Solr, its version probably may be something earlier than > 1.6. > > The minimum Solr version required to be able to use docValues as stored is > 5.5.0. If you have a Solr version before 5.5.0, docValues will not work for > atomic updates at all. > > Thanks, > Shawn >
Re: Live publishing and solr performance optimization
Sharding can be one of the option. But what is the size of your documents? And which Solr version are you using? Regards, Edwin On Tue, 20 Nov 2018 at 01:40, Balanathagiri Ayyasamypalanivel < bala.cit...@gmail.com> wrote: > Hi, > We are in the process for live Publishing document in solr and the same > time we have to maintain the search performance. > > Total existing docs : 120 million > Expected data for live publishing : 1 million > > For every 1 hour, we will get 1m docs to publish in live to the hot solr > collection, can you please provide your suggestions on how effectively we > can do this. > > Regards, > Bala. >
Re: Error:Missing Required Fields for Atomic Updates
On 11/20/2018 9:07 PM, Rajeswari Kolluri wrote: Schema version is 1.6 and Solr version is 7.5. While creating a document , I have provide the required information to field " category". Now, I would like to update document for other set of fields using Atomic Update but not category field. While performing atomic update ,exception caught with "Missing required field on category". The schema version of 1.6 should make useDocValuesAsStored default to true for any field type that supports docValues.Usually, the "string" type is defined using the StrField class, which does support docValues. The TextField class does not support docValues. When you do a query for the document that you're trying to update and aren't restricting things with the fl parameter, is the category field shown in the search results? If not, then you won't be able to update that document without providing the category field. Thanks, Shawn
RE: Error:Missing Required Fields for Atomic Updates
Hi Shawn, When I query the document, category field is shown in search. Thanks Rajeswari -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Wednesday, November 21, 2018 11:38 AM To: solr-user@lucene.apache.org Subject: Re: Error:Missing Required Fields for Atomic Updates On 11/20/2018 9:07 PM, Rajeswari Kolluri wrote: > Schema version is 1.6 and Solr version is 7.5. > > While creating a document , I have provide the required information to field > " category". > > Now, I would like to update document for other set of fields using Atomic > Update but not category field. > > While performing atomic update ,exception caught with "Missing required field > on category". The schema version of 1.6 should make useDocValuesAsStored default to true for any field type that supports docValues.Usually, the "string" type is defined using the StrField class, which does support docValues. The TextField class does not support docValues. When you do a query for the document that you're trying to update and aren't restricting things with the fl parameter, is the category field shown in the search results? If not, then you won't be able to update that document without providing the category field. Thanks, Shawn
