Re: Is it impossible to update an index that is undergoing an optimize?
On 11/6/2015 8:39 PM, Yonik Seeley wrote:
> On Fri, Nov 6, 2015 at 10:20 PM, Shawn Heisey wrote:
>> Is there a decent API for getting uniqueKey?
> Not off the top of my head.
> I deeply regret making it configurable and not just using "id" ;-)
By poking around in the admin UI with Firebug, I found something that
will work for me to get the uniqueKey field name:
SolrQuery uniqueKeyQuery = new SolrQuery();
uniqueKeyQuery.setRequestHandler("/admin/luke");
uniqueKeyQuery.set("show", "schema");
QueryResponse rsp = client.query(coreName, uniqueKeyQuery);
String uniqueKey = (String)
rsp.getResponse().findRecursive("schema", "uniqueKeyField");
Thanks,
Shawn
Re: Is it impossible to update an index that is undergoing an optimize?
On Sat, Nov 7, 2015 at 9:09 AM, Yonik Seeley wrote: > On Fri, Nov 6, 2015 at 10:20 PM, Shawn Heisey wrote: > > Is there a decent API for getting uniqueKey? > > Not off the top of my head. > I deeply regret making it configurable and not just using "id" ;-) > Maybe this? https://cwiki.apache.org/confluence/display/solr/Schema+API#SchemaAPI-ListUniqueKey > > -Yonik >
Re: Is it impossible to update an index that is undergoing an optimize?
On Fri, Nov 6, 2015 at 10:20 PM, Shawn Heisey wrote: > Is there a decent API for getting uniqueKey? Not off the top of my head. I deeply regret making it configurable and not just using "id" ;-) -Yonik
Re: Is it impossible to update an index that is undergoing an optimize?
On 11/6/2015 6:18 PM, Yonik Seeley wrote: > On Wed, Nov 4, 2015 at 3:36 PM, Shawn Heisey wrote: >> The specific index update that fails during the optimize is the SolrJ >> deleteByQuery call. > > deleteByQuery may be the outlier here... we have to jump through extra > hoops internally because we don't know which documents it will affect. > Normal adds and deletes should proceed in parallel though. I'm not doing the delete query on the uniqueKey field. It's on a separate (but also unique) "delete id" field. Because I query each shard before I actually do the delete, I could retrieve the uniqueKey field and then issue a standard delete request with the IDs that I receive. I would prefer to be able to query Solr on some API to determine the uniqueKey field name, so I don't need to have that field name in my configuration, then use that info on the query. I would definitely prefer to not use the "files" api to retrieve schema.xml, because I would then need to parse XML. Is there a decent API for getting uniqueKey? Thanks, Shawn
Re: Is it impossible to update an index that is undergoing an optimize?
It is pretty handy, though. Great for expunging docs that are marked deleted or are expired. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 6, 2015, at 5:31 PM, Alexandre Rafalovitch wrote: > > Elasticsearch removed deleteByQuery from the core all together. > Definitely an outlier :-) > > Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: > http://www.solr-start.com/ > > > On 6 November 2015 at 20:18, Yonik Seeley wrote: >> On Wed, Nov 4, 2015 at 3:36 PM, Shawn Heisey wrote: >>> The specific index update that fails during the optimize is the SolrJ >>> deleteByQuery call. >> >> deleteByQuery may be the outlier here... we have to jump through extra >> hoops internally because we don't know which documents it will affect. >> Normal adds and deletes should proceed in parallel though. >> >> -Yonik
Re: Is it impossible to update an index that is undergoing an optimize?
Elasticsearch removed deleteByQuery from the core all together. Definitely an outlier :-) Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/ On 6 November 2015 at 20:18, Yonik Seeley wrote: > On Wed, Nov 4, 2015 at 3:36 PM, Shawn Heisey wrote: >> The specific index update that fails during the optimize is the SolrJ >> deleteByQuery call. > > deleteByQuery may be the outlier here... we have to jump through extra > hoops internally because we don't know which documents it will affect. > Normal adds and deletes should proceed in parallel though. > > -Yonik
Re: Is it impossible to update an index that is undergoing an optimize?
On Wed, Nov 4, 2015 at 3:36 PM, Shawn Heisey wrote: > The specific index update that fails during the optimize is the SolrJ > deleteByQuery call. deleteByQuery may be the outlier here... we have to jump through extra hoops internally because we don't know which documents it will affect. Normal adds and deletes should proceed in parallel though. -Yonik
Re: Is it impossible to update an index that is undergoing an optimize?
On 11/6/2015 2:23 PM, Pushkar Raste wrote: > I may be wrong but I think 'delete' and 'optimize' can not be executed > concurrently on a Lucene index It certainly is looking that way. After discussing it with Hoss on IRC, I tried a manual test where I started an optimize and then did some "add" requests from the admin UI. Those succeeded, but whenever my SolrJ code does a delete request on an index that is being optimized, that request blocks until either the optimize finishes or the SO_TIMEOUT (15 minutes) that I configured on HttpClient is reached. My SolrJ code always starts with any deletes that need to happen, then moves onto other changes like reinserts and adds. If delete and optimize cannot happen at the same time, is that a bug? This is happening on both 4.9.1 and 5.2.1. Thanks, Shawn
Re: Is it impossible to update an index that is undergoing an optimize?
I may be wrong but I think 'delete' and 'optimize' can not be executed concurrently on a Lucene index On 4 November 2015 at 15:36, Shawn Heisey wrote: > On 11/4/2015 1:17 PM, Yonik Seeley wrote: > > On Wed, Nov 4, 2015 at 3:06 PM, Shawn Heisey > wrote: > >> I had understood that since 4.0, Solr (Lucene) can continue to update an > >> index even while that index is optimizing. > > Yes, that should be the case. > > > >> I have discovered in the logs of my SolrJ index maintenance program that > >> this does not appear to actually be true. > > Hmmm, perhaps some other resource is getting exhausted, like number of > > background merges hit the limit? > > I hope it's a misconfiguration, not a bug. > > Below is my indexConfig. I have already increased maxMergeCount because > without that, full-import from MySQL will stop processing updates during > a large merge, and the pause is long enough that the JDBC connection > times out and closes. > > > > 35 > 35 > 105 > > > 1 > 6 > > 48 > false > > > The specific index update that fails during the optimize is the SolrJ > deleteByQuery call. > > Thanks, > Shawn > >
Re: Is it impossible to update an index that is undergoing an optimize?
On 11/4/2015 1:17 PM, Yonik Seeley wrote: > On Wed, Nov 4, 2015 at 3:06 PM, Shawn Heisey wrote: >> I had understood that since 4.0, Solr (Lucene) can continue to update an >> index even while that index is optimizing. > Yes, that should be the case. > >> I have discovered in the logs of my SolrJ index maintenance program that >> this does not appear to actually be true. > Hmmm, perhaps some other resource is getting exhausted, like number of > background merges hit the limit? I hope it's a misconfiguration, not a bug. Below is my indexConfig. I have already increased maxMergeCount because without that, full-import from MySQL will stop processing updates during a large merge, and the pause is long enough that the JDBC connection times out and closes. 35 35 105 1 6 48 false The specific index update that fails during the optimize is the SolrJ deleteByQuery call. Thanks, Shawn
Re: Is it impossible to update an index that is undergoing an optimize?
On Wed, Nov 4, 2015 at 3:06 PM, Shawn Heisey wrote: > I had understood that since 4.0, Solr (Lucene) can continue to update an > index even while that index is optimizing. Yes, that should be the case. > I have discovered in the logs of my SolrJ index maintenance program that > this does not appear to actually be true. Hmmm, perhaps some other resource is getting exhausted, like number of background merges hit the limit? -Yonik
Is it impossible to update an index that is undergoing an optimize?
I had understood that since 4.0, Solr (Lucene) can continue to update an index even while that index is optimizing. I have discovered in the logs of my SolrJ index maintenance program that this does not appear to actually be true. My dev index is running Solr 5.2.1 and my production indexes are 4.9.1. It's easy enough to fix my build program so no updates happen during my daily optimize, but I'm really hoping to keep it in the background. Have I possibly misconfigured something that turns off the ability to update an optimizing index? Thanks, Shawn
