does 'passenger-status' show anything interesting? On Tue, Jan 17, 2017 at 11:27 AM, 'Jason B. Nance' via Foreman users < [email protected]> wrote:
> Hello all, > > This has happened again today on my lab server while publishing content > view versions and I have debug logging enabled in both Foreman and httpd, > but this is the only thing reported by foreman-tail when I attempt to hit > the web UI: > > ==> /var/log/httpd/foreman-ssl_error_ssl.log <== > [Tue Jan 17 10:06:41.022510 2017] [ssl:info] [pid 4712] [client > 172.16.246.240:33626] AH01964: Connection to child 37 established (server > katello.ipa.centric.lab:443) > [Tue Jan 17 10:06:41.022782 2017] [ssl:debug] [pid 4712] > ssl_engine_kernel.c(1886): [client 172.16.246.240:33626] AH02044: No > matching SSL virtual host for servername localhost found (using > default/first virtual host) > [Tue Jan 17 10:06:41.024377 2017] [ssl:debug] [pid 4712] > ssl_engine_kernel.c(1812): [client 172.16.246.240:33626] AH02041: > Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) > [Tue Jan 17 10:06:41.024696 2017] [ssl:info] [pid 4712] (70014)End of file > found: [client 172.16.246.240:33626] AH01991: SSL input filter read > failed. > [Tue Jan 17 10:06:41.024714 2017] [ssl:debug] [pid 4712] > ssl_engine_io.c(992): [client 172.16.246.240:33626] AH02001: Connection > closed to child 37 with standard shutdown (server > katello.ipa.centric.lab:443) > [Tue Jan 17 10:06:41.027909 2017] [ssl:info] [pid 6040] [client > 172.16.246.240:33628] AH01964: Connection to child 67 established (server > katello.ipa.centric.lab:443) > [Tue Jan 17 10:06:41.028336 2017] [ssl:debug] [pid 6040] > ssl_engine_kernel.c(1886): [client 172.16.246.240:33628] AH02044: No > matching SSL virtual host for servername localhost found (using > default/first virtual host) > [Tue Jan 17 10:06:41.030217 2017] [ssl:debug] [pid 6040] > ssl_engine_kernel.c(1812): [client 172.16.246.240:33628] AH02041: > Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) > [Tue Jan 17 10:06:41.031205 2017] [ssl:debug] [pid 6040] > ssl_engine_kernel.c(224): [client 172.16.246.240:33628] AH02034: Initial > (No.1) HTTPS request received for child 67 (server > katello.ipa.centric.lab:443) > [Tue Jan 17 10:06:41.031345 2017] [ssl:warn] [pid 6040] [client > 172.16.246.240:33628] AH02227: Failed to set r->user to > 'SSL_CLIENT_S_DN_CN' > [Tue Jan 17 10:06:41.031370 2017] [authz_core:debug] [pid 6040] > mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626: > authorization result of Require all granted: granted > [Tue Jan 17 10:06:41.031377 2017] [authz_core:debug] [pid 6040] > mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626: > authorization result of <RequireAny>: granted > [Tue Jan 17 10:06:41.031510 2017] [:debug] [pid 6040] > mod_intercept_form_submit.c(405): intercept_form_submit_init invoked > [Tue Jan 17 10:06:41.031520 2017] [:debug] [pid 6040] > mod_intercept_form_submit.c(407): skipping, no POST request > > There are currently 92 Apache children (max clients is set to 256). > > I'm guessing the above log messages don't really say anything. Does > anyone have any other ideas as to how to troubleshoot this? I pulled an > strace of all the httpd processes while trying to hit the site, but it's > like drinking through a fire hose. > > Thanks, > > j > > > ----- Original Message ----- > From: "Jason Nance" <[email protected]> > To: "Foreman Users" <[email protected]> > Sent: Monday, January 16, 2017 10:36:36 AM > Subject: Re: [foreman-users] Web Components Stop Responding > > Okay, this happened again today, but this time I was publishing a new > version of a content view. Apache stopped responding to any/all requests > and isn't logging anything, either. However, my 'set logging to debug' > stuff was reverted by the upgrade I did last week and I didn't realize it > (until now). *SMH* So I guess I'll restart and wait for the next time it > happens. > > Netstat says: > > # netstat -an | egrep '80|443' > tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN > tcp 1 0 172.16.246.31:54206 184.51.114.249:80 > CLOSE_WAIT > tcp 1 0 172.16.246.31:49617 207.99.69.162:80 > CLOSE_WAIT > tcp 1 0 172.16.246.31:49616 207.99.69.162:80 > CLOSE_WAIT > tcp 1 0 172.16.246.31:59602 207.99.69.162:80 > CLOSE_WAIT > tcp 0 0 127.0.0.1:59480 127.0.0.1:27017 > ESTABLISHED > tcp 1 0 172.16.246.31:54266 184.51.114.249:80 > CLOSE_WAIT > tcp 0 0 172.16.246.31:5671 172.16.246.31:56280 > ESTABLISHED > tcp 0 0 127.0.0.1:27017 127.0.0.1:59480 > ESTABLISHED > tcp 0 0 172.16.246.31:56280 172.16.246.31:5671 > ESTABLISHED > tcp 1 0 172.16.246.31:36114 184.51.114.242:80 > CLOSE_WAIT > tcp 0 0 172.16.246.31:53508 172.16.246.31:443 > ESTABLISHED > tcp 0 0 172.16.246.31:53514 172.16.246.31:443 > ESTABLISHED > tcp 1 0 172.16.246.31:59603 207.99.69.162:80 > CLOSE_WAIT > tcp 0 0 172.16.246.31:53324 172.16.246.31:443 > ESTABLISHED > tcp 0 0 172.16.246.31:53340 172.16.246.31:443 > ESTABLISHED > tcp6 0 0 :::80 :::* LISTEN > tcp6 0 0 :::8080 :::* LISTEN > tcp6 0 0 :::443 :::* LISTEN > tcp6 0 0 :::8443 :::* LISTEN > tcp6 0 0 127.0.0.1:8005 :::* LISTEN > tcp6 0 0 :::8009 :::* LISTEN > tcp6 32 0 172.16.246.31:443 172.16.246.31:53482 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53170 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.240:58842 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53476 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53488 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.240:58778 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53376 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.31:53324 > ESTABLISHED > tcp6 0 0 172.16.246.31:443 172.16.246.31:53340 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53510 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53256 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53146 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53502 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.240:58774 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53304 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53450 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53140 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53486 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53456 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53454 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53492 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53370 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53498 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53156 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.240:58848 > ESTABLISHED > tcp6 0 0 172.16.246.31:443 172.16.246.240:58854 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53472 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53506 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53528 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53344 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53518 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53500 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53466 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53504 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53364 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.240:58838 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53428 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53470 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53494 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53530 > CLOSE_WAIT > tcp6 32 0 172.16.246.31:443 172.16.246.31:53378 > CLOSE_WAIT > tcp6 0 0 172.16.246.31:443 172.16.246.31:53508 > ESTABLISHED > tcp6 0 0 172.16.246.31:443 172.16.246.31:53514 > ESTABLISHED > tcp6 32 0 172.16.246.31:443 172.16.246.31:53218 > CLOSE_WAIT > udp 0 0 0.0.0.0:51680 0.0.0.0:* > > > > > ----- Original Message ----- > From: "Jason Nance" <[email protected]> > To: "Foreman Users" <[email protected]> > Sent: Thursday, January 12, 2017 2:22:51 PM > Subject: Re: [foreman-users] Web Components Stop Responding > > Not sure what was going on. I typically have fewer than 10 hosts > connected to this lab install which was hanging. I updated it to > 3.2.2/1.13.3 on Monday and have debug logging on and haven't seen the issue > reappear. > > *shrug* > > > ----- Original Message ----- > From: "Stephen Benjamin" <[email protected]> > To: "Foreman Users" <[email protected]> > Sent: Tuesday, January 10, 2017 6:13:27 PM > Subject: Re: [foreman-users] Web Components Stop Responding > > ----- Original Message ----- > > From: "'Jason B. Nance' via Foreman users" <foreman-users@googlegroups. > com> > > To: "Foreman Users" <[email protected]> > > Sent: Friday, January 6, 2017 11:41:55 AM > > Subject: [foreman-users] Web Components Stop Responding > > > > Hello Everyone, > > > > On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions > as > > well) I run into the situation where occasionally the web interface and > API > > stop responding. Attempts to load any page, click on any > > button/link/whatever results in the browser just spinning and eventually > > timing out. The only log message printed is: > > > > ==> /var/log/httpd/foreman-ssl_error_ssl.log <== > > [Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client > > 172.16.246.240:56536] AH02227: Failed to set r->user to > > 'SSL_CLIENT_S_DN_CN' > > > > (I see this frequently under normal conditions) > > > > The system is not under heavy loads during this freeze. Load average is > > about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process > doing > > noticeable work is mongodb. > > > > Puppet runs on clients also fail during this time because > /etc/puppet/node.rb > > <fqdn> fails with: > > > > Could not send facts to Foreman: Net::ReadTimeout > > > > (no log messages are printed) > > > > A 'yum install' on a client is able to download the package, but then > hangs > > on "Uploading Package Profile" (and eventually times out and skips it). > > > > If I restart [tomcat and] httpd things start responding again. I'm not > sure > > if the tomcat restart matters, as I did it first and nothing changed but > > after I restarted httpd - which took much longer than normal - is when > > everything came back to life. There were about 55 connections to httpd > at > > the time, nearly all in CLOSE_WAIT. > > > > Anyone have ideas as to what may be happening? > > This sounds an awful lot like what we fixed in http://projects.theforeman. > org/issues/14023. > > Do you have a lot of puppet clients checking in? You could try to tweak > the passenger > settings as described here in the Red Hat BZ (will be the default in > Foreman 1.14), and > see if that helps: > > https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11 > > - Stephen > > > > > Thanks, > > > > j > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Foreman users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > Visit this group at https://groups.google.com/group/foreman-users. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
