Hello all, This has happened again today on my lab server while publishing content view versions and I have debug logging enabled in both Foreman and httpd, but this is the only thing reported by foreman-tail when I attempt to hit the web UI:
==> /var/log/httpd/foreman-ssl_error_ssl.log <== [Tue Jan 17 10:06:41.022510 2017] [ssl:info] [pid 4712] [client 172.16.246.240:33626] AH01964: Connection to child 37 established (server katello.ipa.centric.lab:443) [Tue Jan 17 10:06:41.022782 2017] [ssl:debug] [pid 4712] ssl_engine_kernel.c(1886): [client 172.16.246.240:33626] AH02044: No matching SSL virtual host for servername localhost found (using default/first virtual host) [Tue Jan 17 10:06:41.024377 2017] [ssl:debug] [pid 4712] ssl_engine_kernel.c(1812): [client 172.16.246.240:33626] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) [Tue Jan 17 10:06:41.024696 2017] [ssl:info] [pid 4712] (70014)End of file found: [client 172.16.246.240:33626] AH01991: SSL input filter read failed. [Tue Jan 17 10:06:41.024714 2017] [ssl:debug] [pid 4712] ssl_engine_io.c(992): [client 172.16.246.240:33626] AH02001: Connection closed to child 37 with standard shutdown (server katello.ipa.centric.lab:443) [Tue Jan 17 10:06:41.027909 2017] [ssl:info] [pid 6040] [client 172.16.246.240:33628] AH01964: Connection to child 67 established (server katello.ipa.centric.lab:443) [Tue Jan 17 10:06:41.028336 2017] [ssl:debug] [pid 6040] ssl_engine_kernel.c(1886): [client 172.16.246.240:33628] AH02044: No matching SSL virtual host for servername localhost found (using default/first virtual host) [Tue Jan 17 10:06:41.030217 2017] [ssl:debug] [pid 6040] ssl_engine_kernel.c(1812): [client 172.16.246.240:33628] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) [Tue Jan 17 10:06:41.031205 2017] [ssl:debug] [pid 6040] ssl_engine_kernel.c(224): [client 172.16.246.240:33628] AH02034: Initial (No.1) HTTPS request received for child 67 (server katello.ipa.centric.lab:443) [Tue Jan 17 10:06:41.031345 2017] [ssl:warn] [pid 6040] [client 172.16.246.240:33628] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN' [Tue Jan 17 10:06:41.031370 2017] [authz_core:debug] [pid 6040] mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626: authorization result of Require all granted: granted [Tue Jan 17 10:06:41.031377 2017] [authz_core:debug] [pid 6040] mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626: authorization result of <RequireAny>: granted [Tue Jan 17 10:06:41.031510 2017] [:debug] [pid 6040] mod_intercept_form_submit.c(405): intercept_form_submit_init invoked [Tue Jan 17 10:06:41.031520 2017] [:debug] [pid 6040] mod_intercept_form_submit.c(407): skipping, no POST request There are currently 92 Apache children (max clients is set to 256). I'm guessing the above log messages don't really say anything. Does anyone have any other ideas as to how to troubleshoot this? I pulled an strace of all the httpd processes while trying to hit the site, but it's like drinking through a fire hose. Thanks, j ----- Original Message ----- From: "Jason Nance" <[email protected]> To: "Foreman Users" <[email protected]> Sent: Monday, January 16, 2017 10:36:36 AM Subject: Re: [foreman-users] Web Components Stop Responding Okay, this happened again today, but this time I was publishing a new version of a content view. Apache stopped responding to any/all requests and isn't logging anything, either. However, my 'set logging to debug' stuff was reverted by the upgrade I did last week and I didn't realize it (until now). *SMH* So I guess I'll restart and wait for the next time it happens. Netstat says: # netstat -an | egrep '80|443' tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN tcp 1 0 172.16.246.31:54206 184.51.114.249:80 CLOSE_WAIT tcp 1 0 172.16.246.31:49617 207.99.69.162:80 CLOSE_WAIT tcp 1 0 172.16.246.31:49616 207.99.69.162:80 CLOSE_WAIT tcp 1 0 172.16.246.31:59602 207.99.69.162:80 CLOSE_WAIT tcp 0 0 127.0.0.1:59480 127.0.0.1:27017 ESTABLISHED tcp 1 0 172.16.246.31:54266 184.51.114.249:80 CLOSE_WAIT tcp 0 0 172.16.246.31:5671 172.16.246.31:56280 ESTABLISHED tcp 0 0 127.0.0.1:27017 127.0.0.1:59480 ESTABLISHED tcp 0 0 172.16.246.31:56280 172.16.246.31:5671 ESTABLISHED tcp 1 0 172.16.246.31:36114 184.51.114.242:80 CLOSE_WAIT tcp 0 0 172.16.246.31:53508 172.16.246.31:443 ESTABLISHED tcp 0 0 172.16.246.31:53514 172.16.246.31:443 ESTABLISHED tcp 1 0 172.16.246.31:59603 207.99.69.162:80 CLOSE_WAIT tcp 0 0 172.16.246.31:53324 172.16.246.31:443 ESTABLISHED tcp 0 0 172.16.246.31:53340 172.16.246.31:443 ESTABLISHED tcp6 0 0 :::80 :::* LISTEN tcp6 0 0 :::8080 :::* LISTEN tcp6 0 0 :::443 :::* LISTEN tcp6 0 0 :::8443 :::* LISTEN tcp6 0 0 127.0.0.1:8005 :::* LISTEN tcp6 0 0 :::8009 :::* LISTEN tcp6 32 0 172.16.246.31:443 172.16.246.31:53482 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53170 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.240:58842 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53476 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53488 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.240:58778 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53376 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.31:53324 ESTABLISHED tcp6 0 0 172.16.246.31:443 172.16.246.31:53340 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53510 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53256 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53146 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53502 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.240:58774 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53304 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53450 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53140 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53486 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53456 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53454 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53492 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53370 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53498 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53156 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.240:58848 ESTABLISHED tcp6 0 0 172.16.246.31:443 172.16.246.240:58854 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53472 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53506 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53528 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53344 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53518 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53500 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53466 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53504 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53364 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.240:58838 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53428 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53470 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53494 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53530 CLOSE_WAIT tcp6 32 0 172.16.246.31:443 172.16.246.31:53378 CLOSE_WAIT tcp6 0 0 172.16.246.31:443 172.16.246.31:53508 ESTABLISHED tcp6 0 0 172.16.246.31:443 172.16.246.31:53514 ESTABLISHED tcp6 32 0 172.16.246.31:443 172.16.246.31:53218 CLOSE_WAIT udp 0 0 0.0.0.0:51680 0.0.0.0:* ----- Original Message ----- From: "Jason Nance" <[email protected]> To: "Foreman Users" <[email protected]> Sent: Thursday, January 12, 2017 2:22:51 PM Subject: Re: [foreman-users] Web Components Stop Responding Not sure what was going on. I typically have fewer than 10 hosts connected to this lab install which was hanging. I updated it to 3.2.2/1.13.3 on Monday and have debug logging on and haven't seen the issue reappear. *shrug* ----- Original Message ----- From: "Stephen Benjamin" <[email protected]> To: "Foreman Users" <[email protected]> Sent: Tuesday, January 10, 2017 6:13:27 PM Subject: Re: [foreman-users] Web Components Stop Responding ----- Original Message ----- > From: "'Jason B. Nance' via Foreman users" <[email protected]> > To: "Foreman Users" <[email protected]> > Sent: Friday, January 6, 2017 11:41:55 AM > Subject: [foreman-users] Web Components Stop Responding > > Hello Everyone, > > On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as > well) I run into the situation where occasionally the web interface and API > stop responding. Attempts to load any page, click on any > button/link/whatever results in the browser just spinning and eventually > timing out. The only log message printed is: > > ==> /var/log/httpd/foreman-ssl_error_ssl.log <== > [Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client > 172.16.246.240:56536] AH02227: Failed to set r->user to > 'SSL_CLIENT_S_DN_CN' > > (I see this frequently under normal conditions) > > The system is not under heavy loads during this freeze. Load average is > about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing > noticeable work is mongodb. > > Puppet runs on clients also fail during this time because /etc/puppet/node.rb > <fqdn> fails with: > > Could not send facts to Foreman: Net::ReadTimeout > > (no log messages are printed) > > A 'yum install' on a client is able to download the package, but then hangs > on "Uploading Package Profile" (and eventually times out and skips it). > > If I restart [tomcat and] httpd things start responding again. I'm not sure > if the tomcat restart matters, as I did it first and nothing changed but > after I restarted httpd - which took much longer than normal - is when > everything came back to life. There were about 55 connections to httpd at > the time, nearly all in CLOSE_WAIT. > > Anyone have ideas as to what may be happening? This sounds an awful lot like what we fixed in http://projects.theforeman.org/issues/14023. Do you have a lot of puppet clients checking in? You could try to tweak the passenger settings as described here in the Red Hat BZ (will be the default in Foreman 1.14), and see if that helps: https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11 - Stephen > Thanks, > > j > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
