Hi Rob Sorry I missed the full question: What are the contents of /var/log/pki/pki-tomcat/ca ?
Could it be that the CA can't write its own logs? What does the latest catalina log show in the parent directory? /var/log/pki/pki-tomcat/ca was empty until I created /var/log/pki/pki-tomcat/ca/logs and /var/log/pki/pki-tomcat/ca/debug directories. I dont think the ca would have trouble writing its logs, the structure is all owned by pkiuser: drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca Now that I think about it, I do remember some issues with runaway logs filling up /var/log, and I deleted some directories, and recreated them, but I dont think pki-tomcat suffered then. On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <[email protected]> wrote: > > Hi Rob > > Today's catalina log file writes: > > WARNING: Exception processing realm > com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > at > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > at > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > at > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > at java.lang.Thread.run(Thread.java:748) > > Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase > backgroundProcess > WARNING: Exception processing realm > com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > at > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > at > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > at > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > at java.lang.Thread.run(Thread.java:748) > > Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase > backgroundProcess > WARNING: Exception processing realm > com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > at > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > at > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > at > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > at java.lang.Thread.run(Thread.java:748) > > Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase > backgroundProcess > WARNING: Exception processing realm > com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > at > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > at > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > at > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > at java.lang.Thread.run(Thread.java:748) > > On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <[email protected]> wrote: > > > > Sina Owolabi wrote: > > > Log directories on the server: > > > > > > /var/log/pki/pki-tomcat/ca/debug > > > /var/log/pki/pki-tomcat/ca/logs > > > /var/log/pki/server/upgrade/10.1.2 > > > /var/log/pki/server/upgrade/10.1.99 > > > /var/log/pki/server/upgrade/10.2.1 > > > /var/log/pki/server/upgrade/10.2.2 > > > /var/log/pki/server/upgrade/10.2.3 > > > /var/log/pki/server/upgrade/10.2.4 > > > /var/log/pki/server/upgrade/10.2.5 > > > /var/log/pki/server/upgrade/10.2.6 > > > /var/log/pki/server/upgrade/10.3.0 > > > /var/log/pki/server/upgrade/10.3.3 > > > /var/log/pki/server/upgrade/10.4.0 > > > /var/log/pki/server/upgrade/10.4.1 > > > /var/log/pki/server/upgrade/10.5.1 > > > > > > /var/log/pki/pki-tomcat/ca/debug > > > > You stated you had created this directory yourself. > > > > What are the contents of /var/log/pki/pki-tomcat/ca ? > > > > Could it be that the CA can't write its own logs? What does the latest > > catalina log show in the parent directory? > > > > rob > > > > > /var/log/pki/pki-tomcat/ca/logs > > > are both empty. > > > > > > On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <[email protected]> wrote: > > >> > > >> Sina Owolabi wrote: > > >>> Hi Florence > > >>> > > >>> and thanks for the help. > > >>> ipactl status: > > >>> [root@services ~]# ipactl status --ignore-service-failure; cat > > >>> Directory Service: RUNNING > > >>> krb5kdc Service: RUNNING > > >>> kadmin Service: RUNNING > > >>> named Service: RUNNING > > >>> httpd Service: RUNNING > > >>> ipa-custodia Service: RUNNING > > >>> ntpd Service: RUNNING > > >>> pki-tomcatd Service: STOPPED > > >>> ipa-otpd Service: RUNNING > > >>> ipa-dnskeysyncd Service: RUNNING > > >>> ipa: INFO: The ipactl command was successful > > >>> > > >>> > > >>> systemctl status -l [email protected]; cat > > >>> ? [email protected] - PKI Tomcat Server pki-tomcat > > >>> Loaded: loaded (/lib/systemd/system/[email protected]; enabled; > > >>> vendor preset: disabled) > > >>> Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago > > >>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, > > >>> status=0/SUCCESS) > > >>> Main PID: 1376 (java) > > >>> CGroup: > > >>> /system.slice/system-pki\x2dtomcatd.slice/[email protected] > > >>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java > > >>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath > > >>> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > > >>> -Dcatalina.base=/var/lib/pki/pki-tomcat > > >>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= > > >>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp > > >>> -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties > > >>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > > >>> -Djava.security.manager > > >>> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy > > >>> org.apache.catalina.startup.Bootstrap start > > >>> > > >>> systemctl status [email protected]: > > >>> > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception > > >>> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f > > >>> background process > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: > > >>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) > > >>> Mar 05 09:40:43 services.qrios.com server[1376]: at > > >>> java.lang.Thread.run(Thread.java:748) > > >> > > >> The logs will contain much more useful information. dogtag keeps > > >> changing the location of the logs and I forget exactly where it is in > > >> your version but it's somewhere in /var/log/pki*/pki*/ca/... > > >> > > >> The log may be named debug or debug-<date> > > >> > > >> Also look at the selftest log in the same directory. > > >> > > >> There are a LOT of red herrings in the dogtag logs so proceed with > > >> caution. > > >> > > >> You do not need to touch or create anything for this logging to take > > >> place. You should delete the directory you created. > > >> > > >> rob > > >> > > >> > > >>> > > >>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud <[email protected]> > > >>> wrote: > > >>>> > > >>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: > > >>>>> Hi! > > >>>>> > > >>>>> I tried to follow this solution for cert renewal for RHEL6: > > >>>>> https://access.redhat.com/solutions/643753 (Sorry, desperation is > > >>>>> setting in), but when I attempted Step 2, I got: > > >>>>> > > >>>> Hi, > > >>>> > > >>>> 1. this note was written for RHEL 6 but you said in your first e-mail > > >>>> that your server is running CentOS 7 with ipa 4.5.4. Please don't > > >>>> follow > > >>>> those instructions as they are not adapted to your deployment. > > >>>> The instructions for RHEL 7 are available at > > >>>> https://access.redhat.com/solutions/3357261. > > >>>> > > >>>> 2. In a previous e-mail, the output of getcert list | grep -i expires > > >>>> did not show any expired certificates, so I would not rush into wrong > > >>>> conclusions. We need to understand first why pki did not start. > > >>>> > > >>>> What is the output of: > > >>>> $ ipactl status > > >>>> $ systemctl status [email protected] > > >>>> > > >>>> flo > > >>>> > > >>>>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert > > >>>>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do > > >>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" > > >>>>> | grep -i after; done > > >>>>> auditSigningCert cert-pki-ca > > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>> certificate/key database is in an old, unsupported format. > > >>>>> ocspSigningCert cert-pki-ca > > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>> certificate/key database is in an old, unsupported format. > > >>>>> subsystemCert cert-pki-ca > > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>> certificate/key database is in an old, unsupported format. > > >>>>> Server-Cert cert-pki-ca > > >>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > >>>>> certificate/key database is in an old, unsupported format. > > >>>>> > > >>>>> Could this be the root of my problems? > > >>>>> And how can I convert them? > > >>>>> > > >>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <[email protected]> > > >>>>> wrote: > > >>>>>> > > >>>>>> Restarting ipa didnt create the logs. > > >>>>>> Please, what else can i do? > > >>>>>> > > >>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <[email protected]> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>> Hi! > > >>>>>>> > > >>>>>>> getcert list | grep -i expires > > >>>>>>> expires: 2019-04-13 12:08:20 UTC > > >>>>>>> expires: 2019-04-13 12:08:06 UTC > > >>>>>>> expires: 2019-04-13 12:07:50 UTC > > >>>>>>> expires: 2035-06-01 08:33:01 UTC > > >>>>>>> expires: 2019-04-13 12:07:41 UTC > > >>>>>>> expires: 2019-04-13 12:06:55 UTC > > >>>>>>> expires: 2019-05-05 12:06:41 UTC > > >>>>>>> expires: 2019-05-05 12:06:56 UTC > > >>>>>>> expires: 2020-01-17 19:56:03 UTC > > >>>>>>> > > >>>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am > > >>>>>>> creating one and running "ipactl restart". > > >>>>>>> > > >>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <[email protected]> > > >>>>>>> wrote: > > >>>>>>>> > > >>>>>>>> Sina Owolabi via FreeIPA-users wrote: > > >>>>>>>>> Hi! > > >>>>>>>>> > > >>>>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version > > >>>>>>>>> 4.5.4, > > >>>>>>>>> api version 2.228), with one master, and two replicas, and I > > >>>>>>>>> noticed > > >>>>>>>>> that pki-tomcatd no longer works on the master, after attempting a > > >>>>>>>>> reboot. > > >>>>>>>>> pki-tomcatd works fine on the slaves. > > >>>>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts > > >>>>>>>>> management, user passwords, etc), I receive responses like this: > > >>>>>>>>> > > >>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > > >>>>>>>>> communicate with CMS (Internal Server Error) > > >>>>>>>>> But on the replicas, functions work fine. > > >>>>>>>>> Please can someone guide me on how to fix this? > > >>>>>>>> > > >>>>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have > > >>>>>>>> some > > >>>>>>>> pointers. I'd look at selftests.log first. > > >>>>>>>> > > >>>>>>>> My guess is that some of the CA certificates have failed to renew. > > >>>>>>>> > > >>>>>>>> getcert list | grep -i expires > > >>>>>>>> > > >>>>>>>> rob > > >>>>> _______________________________________________ > > >>>>> FreeIPA-users mailing list -- [email protected] > > >>>>> To unsubscribe send an email to > > >>>>> [email protected] > > >>>>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > >>>>> List Guidelines: > > >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines > > >>>>> List Archives: > > >>>>> https://lists.fedorahosted.org/archives/list/[email protected] > > >>>>> > > >>>> > > >> > > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
