Re: solr blocking and client timeout issue

2015-07-23 Thread Erick Erickson
B=4.0),segFN=segments_e9nk,generation=665696} >>>>> >>>>> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@ >>> >>> /opt/solr/solr/collection1/data/index >>>>> >>>>> lockFactory=org.apache.lucene.s

Re: solr blocking and client timeout issue

2015-07-23 Thread Jeremy Ashcraft
_ From: Shawn Heisey Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production

Re: solr blocking and client timeout issue

2015-07-21 Thread Jeremy Ashcraft
), 5cf034a9-d93a-4307-a367-02cb21fa8e35 (1507244513404387328), 816e3a04-9d0e-4587-a3ee-9f9e7b0c7d74 (1507244513405435904)],commit=} 0 50 could that be causing some of the problems? From: Shawn Heisey Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.or

Re: solr blocking and client timeout issue

2015-07-21 Thread Daniel Collins
_flush > >> INFO - 2015-07-20 12:26:21.026; > >> org.apache.solr.update.processor.LogUpdateProcessor; [collection1] > >> webapp=/solr path=/update params={omitHeader=false&wt=json} > >> {add=[8653ea29-a327-4a54-9b00-8468241f2d7c (150724451340333875

Re: solr blocking and client timeout issue

2015-07-20 Thread Erick Erickson
816e3a04-9d0e-4587-a3ee-9f9e7b0c7d74 (1507244513405435904)],commit=} 0 50 >> >> could that be causing some of the problems? >> >> >> From: Shawn Heisey >> Sent: Monday, July 20, 2015 11:44 AM >> To: solr-user@l

Re: solr blocking and client timeout issue

2015-07-20 Thread Jeremy Ashcraft
50 could that be causing some of the problems? From: Shawn Heisey Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to

Re: solr blocking and client timeout issue

2015-07-20 Thread Jeremy Ashcraft
problems? From: Shawn Heisey Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: > I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i >

Re: solr blocking and client timeout issue

2015-07-20 Thread Shawn Heisey
On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: > I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i > can get production upgraded tonight. > > still getting the big GC pauses this morning, even after applying the > GC tuning options. Everything was fine throughout the weekend. >

Re: solr blocking and client timeout issue

2015-07-20 Thread Jeremy Ashcraft
I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still getting the big GC pauses this morning, even after applying the GC tuning options. Everything was fine throughout the weekend. My biggest concern is that this instance had been r

Re: solr blocking and client timeout issue

2015-07-19 Thread Shawn Heisey
On 7/19/2015 12:46 AM, Jeremy Ashcraft wrote: > That did the trick. The GC tuning options also seems to be working, but > I guess we'll see when traffic ramps back up on monday. Thanks for all > your help! > > On 7/18/2015 8:16 AM, Shawn Heisey wrote: >> The first thing I'd try is removing the U

Re: solr blocking and client timeout issue

2015-07-18 Thread Jeremy Ashcraft
That did the trick. The GC tuning options also seems to be working, but I guess we'll see when traffic ramps back up on monday. Thanks for all your help! On 7/18/2015 8:16 AM, Shawn Heisey wrote: The first thing I'd try is removing the UseLargePages option and see if it goes away.

Re: solr blocking and client timeout issue

2015-07-18 Thread Jeremy Ashcraft
I will try that. traffic is pretty dead over the weekend, so I probably won't be able to tell if its effective or not until monday. thanks again! On 7/18/2015 8:16 AM, Shawn Heisey wrote: On 7/18/2015 12:42 AM, Jeremy Ashcraft wrote: I turned on GC logging and verified that its definitely be

Re: solr blocking and client timeout issue

2015-07-18 Thread Shawn Heisey
On 7/18/2015 12:42 AM, Jeremy Ashcraft wrote: > I turned on GC logging and verified that its definitely being caused by > a GC pause. I tried the tuning option from the article and get this > warning: > > OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory > (errno = 1). > > any r

Re: solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
I turned on GC logging and verified that its definitely being caused by a GC pause. I tried the tuning option from the article and get this warning: OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory (errno = 1). any recommendations on how to get rid of that warning, and sho

Re: solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
thanks! i will definitely try some of those and let you know how it turns out On 7/17/2015 4:14 PM, Shawn Heisey wrote: On 7/17/2015 4:39 PM, Jeremy Ashcraft wrote: Solr 4.4.0 /usr/bin/java -version java version "1.7.0_25" OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64) OpenJDK 64-Bi

Re: solr blocking and client timeout issue

2015-07-17 Thread Shawn Heisey
On 7/17/2015 4:39 PM, Jeremy Ashcraft wrote: > Solr 4.4.0 > > /usr/bin/java -version > java version "1.7.0_25" > OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64) > OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) > > CentOS6 > > /usr/bin/java -Xms1G -Xmx4G -Dsolr.solr.home=/opt/solr

Re: solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
M To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue Since there's basically no information about your setup here, my best guess is that you're seeing huge garbage collection pauses. There are two problems that can lead to long GC pauses. One is a super lar

Re: solr blocking and client timeout issue

2015-07-17 Thread Shawn Heisey
On 7/17/2015 4:02 PM, Jeremy Ashcraft wrote: > having an issue where solr becomes unresponsive for unknown reasons. > Client requests timeout for minutes at a time (sometimes only some > requests time out while others work fine). The logs don't reveal any > clues, other than just a big gap > y

solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
having an issue where solr becomes unresponsive for unknown reasons. Client requests timeout for minutes at a time (sometimes only some requests time out while others work fine). The logs don't reveal any clues, other than just a big gap example: INFO - 2015-07-17 14:39:57.195; org.apache