Re: Solr off-heap FieldCache & HelioSearch

2016-06-03 Thread Phillip Peleshok
Funny you say that, as that's exactly what happened. Tried them a couple weeks ago and nothing. Going at them again and will see what happens. Yeah, we're in the same boat. We started with the profilers (Yourkit) to track down the causes. Mainly got hit in the field cache and ordinal maps

Re: Solr off-heap FieldCache & HelioSearch

2016-06-03 Thread Phillip Peleshok
Thank you for the info on this. Yeah, I should've raised this in the dev lists; sorry about that. Funny you mention that since I was trending in that direction as well. Then saw the off-heap stuff and thought it might have had an easy way out. I'd like to focus on the re-use scheme to be

Re: Solr off-heap FieldCache & HelioSearch

2016-06-03 Thread Jeff Wartes
For what it’s worth, I’d suggest you go into a conversation with Azul with a more explicit “I’m looking to buy” approach. I reached out to them with a more “I’m exploring my options” attitude, and never even got a trial. I get the impression their business model involves a fairly expensive (to

Re: Solr off-heap FieldCache & HelioSearch

2016-06-03 Thread Toke Eskildsen
On Thu, 2016-06-02 at 18:14 -0700, Erick Erickson wrote: > But memory is an ongoing struggle I'm afraid. With fear of going too far into devel-territory... There are several places in Solr where memory usage if far from optimal with high-cardinality data and where improvements can be made

Re: Solr off-heap FieldCache & HelioSearch

2016-06-02 Thread Phillip Peleshok
Fantastic! I'm sorry I couldn't find that JIRA before and for getting you to track it down. Yup, I noticed that for the docvalues with the ordinal map and I'm definitely leveraging all that but I'm hitting the terms limit now and that ends up pushing me over. I'll see about giving Zing/Azul a

Re: Solr off-heap FieldCache & HelioSearch

2016-06-02 Thread Erick Erickson
Basically it never reached consensus, see the discussion at: https://issues.apache.org/jira/browse/SOLR-6638 If you can afford it I've seen people with very good results using Zing/Azul, but that can be expensive. DocValues can help for fields you facet and sort on, those essentially move memory

Solr off-heap FieldCache & HelioSearch

2016-06-01 Thread Phillip Peleshok
Hey everyone, I've been using Solr for some time now and running into GC issues as most others have. Now I've exhausted all the traditional GC settings recommended by various individuals (ie Shawn Heisey, etc) but neither proved sufficient. The one solution that I've seen that proved useful is