Upayavira:
I am using solr 4.7 and yes I am using TieredMergePolicy

Erick:
All my boxes have SSD’s and there isn’t a big disparity between qTime and 
response time.
The performance hit on my end is because of the fragmented index files causing 
more disk seeks are you mentioned.
And I tried requesting fewer docs too but that did not help .



> On Jun 30, 2015, at 5:23 AM, Erick Erickson <erickerick...@gmail.com> wrote:
> 
> I've actually seen this happen right in front of my eyes "in the
> field". However, that was a very high-performance environment. My
> assumption was that fragmented index files were causing more disk
> seeks especially for the first-pass query response in distributed
> mode. So, if the problem is similar, it should go away if you test
> requesting fewer docs. Note: This is not a cure for your problem, but
> would be useful for identifying if it's similar to what I saw.
> 
> NOTE: the symptom was a significant disparity between the QTime (which
> does not measure assembling the document) and the response time. _If_
> that's the case and _if_ my theory that disk access is the culprit,
> then SOLR-5478 and SOLR-6810 should be a big help as they remove the
> first-pass decompression for distributed searches.
> 
> If that hypothesis has any validity, I'd expect you're running on
> spinning-disks rather than SSDs, is that so?
> 
> Best,
> Erick
> 
> On Tue, Jun 30, 2015 at 2:07 AM, Upayavira <u...@odoko.co.uk> wrote:
>> We need to work out why your performance is bad without optimise. What
>> version of Solr are you using? Can you confirm that your config is using
>> the TieredMergePolicy?
>> 
>> Upayavira
>> 
>> Oe, Jun 30, 2015, at 04:48 AM, Summer Shire wrote:
>>> Hi Upayavira and Erick,
>>> 
>>> There are two things we are talking about here.
>>> 
>>> First: Why am I optimizing? If I don’t our SEARCH (NOT INDEXING)
>>> performance is 100% worst.
>>> The problem lies in the number of total segments. We have to have max
>>> segments 1 or 2.
>>> I have done intensive performance related tests around number of
>>> segments, merge factor or changing the Merge policy.
>>> 
>>> Second: Solr does not perform better for me without an optimize. So now
>>> that I have to optimize the second issue
>>> is updating concurrently during an optimize. If I update when an optimize
>>> is happening the optimize takes 5 times as long as
>>> the normal optimize.
>>> 
>>> So is there any way other than creating a postOptimize hook and writing
>>> the status in a file and somehow making it available to the indexer.
>>> All of this just sounds traumatic :)
>>> 
>>> Thanks
>>> Summer
>>> 
>>> 
>>>> On Jun 29, 2015, at 5:40 AM, Erick Erickson <erickerick...@gmail.com> 
>>>> wrote:
>>>> 
>>>> Steven:
>>>> 
>>>> Yes, but....
>>>> 
>>>> First, here's Mike McCandles' excellent blog on segment merging:
>>>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>>>> 
>>>> I think the third animation is the TieredMergePolicy. In short, yes an
>>>> optimize will reclaim disk space. But as you update, this is done for
>>>> you anyway. About the only time optimizing is at all beneficial is
>>>> when you have a relatively static index. If you're continually
>>>> updating documents, and by that I mean replacing some existing
>>>> documents, then you'll immediately start generating "holes" in your
>>>> index.
>>>> 
>>>> And if you _do_ optimize, you wind up with a huge segment. And since
>>>> the default policy tries to merge segments of roughly the same size,
>>>> it accumulates deletes for quite a while before they merged away.
>>>> 
>>>> And if you don't update existing docs or delete docs, then there's no
>>>> wasted space anyway.
>>>> 
>>>> Summer:
>>>> 
>>>> First off, why do you care about not updating during optimizing?
>>>> There's no good reason you have to worry about that, you can freely
>>>> update while optimizing.
>>>> 
>>>> But frankly I have to agree with Upayavira that on the face of it
>>>> you're doing a lot of extra work. See above, but you optimize while
>>>> indexing, so immediately you're rather defeating the purpose.
>>>> Personally I'd only optimize relatively static indexes and, by
>>>> definition, you're index isn't static since the second process is just
>>>> waiting to modify it.
>>>> 
>>>> Best,
>>>> Erick
>>>> 
>>>> On Mon, Jun 29, 2015 at 8:15 AM, Steven White <swhite4...@gmail.com> wrote:
>>>>> Hi Upayavira,
>>>>> 
>>>>> This is news to me that we should not optimize and index.
>>>>> 
>>>>> What about disk space saving, isn't optimization to reclaim disk space or
>>>>> is Solr somehow does that?  Where can I read more about this?
>>>>> 
>>>>> I'm on Solr 5.1.0 (may switch to 5.2.1)
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Steve
>>>>> 
>>>>> On Mon, Jun 29, 2015 at 4:16 AM, Upayavira <u...@odoko.co.uk> wrote:
>>>>> 
>>>>>> I'm afraid I don't understand. You're saying that optimising is causing
>>>>>> performance issues?
>>>>>> 
>>>>>> Simple solution: DO NOT OPTIMIZE!
>>>>>> 
>>>>>> Optimisation is very badly named. What it does is squashes all segments
>>>>>> in your index into one segment, removing all deleted documents. It is
>>>>>> good to get rid of deletes - in that sense the index is "optimized".
>>>>>> However, future merges become very expensive. The best way to handle
>>>>>> this topic is to leave it to Lucene/Solr to do it for you. Pretend the
>>>>>> "optimize" option never existed.
>>>>>> 
>>>>>> This is, of course, assuming you are using something like Solr 3.5+.
>>>>>> 
>>>>>> Upayavira
>>>>>> 
>>>>>> On Mon, Jun 29, 2015, at 08:08 AM, Summer Shire wrote:
>>>>>>> 
>>>>>>> Have to cause of performance issues.
>>>>>>> Just want to know if there is a way to tap into the status.
>>>>>>> 
>>>>>>>> On Jun 28, 2015, at 11:37 PM, Upayavira <u...@odoko.co.uk> wrote:
>>>>>>>> 
>>>>>>>> Bigger question, why are you optimizing? Since 3.6 or so, it generally
>>>>>>>> hasn't been requires, even, is a bad thing.
>>>>>>>> 
>>>>>>>> Upayavira
>>>>>>>> 
>>>>>>>>> On Sun, Jun 28, 2015, at 09:37 PM, Summer Shire wrote:
>>>>>>>>> Hi All,
>>>>>>>>> 
>>>>>>>>> I have two indexers (Independent processes ) writing to a common solr
>>>>>>>>> core.
>>>>>>>>> If One indexer process issued an optimize on the core
>>>>>>>>> I want the second indexer to wait adding docs until the optimize has
>>>>>>>>> finished.
>>>>>>>>> 
>>>>>>>>> Are there ways I can do this programmatically?
>>>>>>>>> pinging the core when the optimize is happening is returning OK
>>>>>> because
>>>>>>>>> technically
>>>>>>>>> solr allows you to update when an optimize is happening.
>>>>>>>>> 
>>>>>>>>> any suggestions ?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Summer
>>>>>> 
>>> 

Reply via email to