Sina Owolabi via FreeIPA-users wrote: > Hi Rob > > Sorry I missed the full question: > What are the contents of /var/log/pki/pki-tomcat/ca ? > > Could it be that the CA can't write its own logs? What does the latest > catalina log show in the parent directory? > > /var/log/pki/pki-tomcat/ca was empty until I created > /var/log/pki/pki-tomcat/ca/logs and > /var/log/pki/pki-tomcat/ca/debug directories. > I dont think the ca would have trouble writing its logs, the structure > is all owned by pkiuser: > drwxrwx---. 4 pkiuser pkiuser 4096 Nov 14 08:23 /var/log/pki/pki-tomcat/ca > > Now that I think about it, I do remember some issues with runaway logs > filling up /var/log, and > I deleted some directories, and recreated them, but I dont think > pki-tomcat suffered then.
Hard to know. If the process was already running at the time things may have appeared ok until it was restarted. debug is a log file, not a directory. My 4.4.4 install contains the following in /var/log/pki: drwxr-xr-x. 3 root root 21 Mar 30 2017 ./server drwxrwx---. 3 pkiuser pkiuser 12288 Mar 6 01:22 ./pki-tomcat drwxrwx---. 4 pkiuser pkiuser 4096 Feb 7 11:27 ./pki-tomcat/ca drwxrwx---. 2 pkiuser pkiuser 86 Dec 4 11:00 ./pki-tomcat/ca/archive drwxrwx---. 2 pkiuser pkiuser 84 Feb 7 11:27 ./pki-tomcat/ca/signedAudit Be sure to run restorecon -R on /var/log/pki to ensure the SELinux contexts are correct. rob > > On Tue, Mar 5, 2019 at 11:46 PM Sina Owolabi <[email protected]> wrote: >> >> Hi Rob >> >> Today's catalina log file writes: >> >> WARNING: Exception processing realm >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> at >> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> at >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> at >> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> at java.lang.Thread.run(Thread.java:748) >> >> Mar 05, 2019 11:44:19 PM org.apache.catalina.core.ContainerBase >> backgroundProcess >> WARNING: Exception processing realm >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> at >> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> at >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> at >> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> at java.lang.Thread.run(Thread.java:748) >> >> Mar 05, 2019 11:44:29 PM org.apache.catalina.core.ContainerBase >> backgroundProcess >> WARNING: Exception processing realm >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> at >> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> at >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> at >> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> at java.lang.Thread.run(Thread.java:748) >> >> Mar 05, 2019 11:44:39 PM org.apache.catalina.core.ContainerBase >> backgroundProcess >> WARNING: Exception processing realm >> com.netscape.cms.tomcat.ProxyRealm@2bfea12f background process >> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >> at >> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >> at >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >> at >> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >> at java.lang.Thread.run(Thread.java:748) >> >> On Tue, Mar 5, 2019 at 5:20 PM Rob Crittenden <[email protected]> wrote: >>> >>> Sina Owolabi wrote: >>>> Log directories on the server: >>>> >>>> /var/log/pki/pki-tomcat/ca/debug >>>> /var/log/pki/pki-tomcat/ca/logs >>>> /var/log/pki/server/upgrade/10.1.2 >>>> /var/log/pki/server/upgrade/10.1.99 >>>> /var/log/pki/server/upgrade/10.2.1 >>>> /var/log/pki/server/upgrade/10.2.2 >>>> /var/log/pki/server/upgrade/10.2.3 >>>> /var/log/pki/server/upgrade/10.2.4 >>>> /var/log/pki/server/upgrade/10.2.5 >>>> /var/log/pki/server/upgrade/10.2.6 >>>> /var/log/pki/server/upgrade/10.3.0 >>>> /var/log/pki/server/upgrade/10.3.3 >>>> /var/log/pki/server/upgrade/10.4.0 >>>> /var/log/pki/server/upgrade/10.4.1 >>>> /var/log/pki/server/upgrade/10.5.1 >>>> >>>> /var/log/pki/pki-tomcat/ca/debug >>> >>> You stated you had created this directory yourself. >>> >>> What are the contents of /var/log/pki/pki-tomcat/ca ? >>> >>> Could it be that the CA can't write its own logs? What does the latest >>> catalina log show in the parent directory? >>> >>> rob >>> >>>> /var/log/pki/pki-tomcat/ca/logs >>>> are both empty. >>>> >>>> On Tue, Mar 5, 2019 at 4:57 PM Rob Crittenden <[email protected]> wrote: >>>>> >>>>> Sina Owolabi wrote: >>>>>> Hi Florence >>>>>> >>>>>> and thanks for the help. >>>>>> ipactl status: >>>>>> [root@services ~]# ipactl status --ignore-service-failure; cat >>>>>> Directory Service: RUNNING >>>>>> krb5kdc Service: RUNNING >>>>>> kadmin Service: RUNNING >>>>>> named Service: RUNNING >>>>>> httpd Service: RUNNING >>>>>> ipa-custodia Service: RUNNING >>>>>> ntpd Service: RUNNING >>>>>> pki-tomcatd Service: STOPPED >>>>>> ipa-otpd Service: RUNNING >>>>>> ipa-dnskeysyncd Service: RUNNING >>>>>> ipa: INFO: The ipactl command was successful >>>>>> >>>>>> >>>>>> systemctl status -l [email protected]; cat >>>>>> ? [email protected] - PKI Tomcat Server pki-tomcat >>>>>> Loaded: loaded (/lib/systemd/system/[email protected]; enabled; >>>>>> vendor preset: disabled) >>>>>> Active: active (running) since Tue 2019-03-05 09:14:15 WAT; 26min ago >>>>>> Process: 1233 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, >>>>>> status=0/SUCCESS) >>>>>> Main PID: 1376 (java) >>>>>> CGroup: >>>>>> /system.slice/system-pki\x2dtomcatd.slice/[email protected] >>>>>> └─1376 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java >>>>>> -DRESTEASY_LIB=/usr/share/java/resteasy-base -classpath >>>>>> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar >>>>>> -Dcatalina.base=/var/lib/pki/pki-tomcat >>>>>> -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= >>>>>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp >>>>>> -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties >>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >>>>>> -Djava.security.manager >>>>>> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy >>>>>> org.apache.catalina.startup.Bootstrap start >>>>>> >>>>>> systemctl status [email protected]: >>>>>> >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: WARNING: Exception >>>>>> processing realm com.netscape.cms.tomcat.ProxyRealm@2bfea12f >>>>>> background process >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: >>>>>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) >>>>>> Mar 05 09:40:43 services.qrios.com server[1376]: at >>>>>> java.lang.Thread.run(Thread.java:748) >>>>> >>>>> The logs will contain much more useful information. dogtag keeps >>>>> changing the location of the logs and I forget exactly where it is in >>>>> your version but it's somewhere in /var/log/pki*/pki*/ca/... >>>>> >>>>> The log may be named debug or debug-<date> >>>>> >>>>> Also look at the selftest log in the same directory. >>>>> >>>>> There are a LOT of red herrings in the dogtag logs so proceed with >>>>> caution. >>>>> >>>>> You do not need to touch or create anything for this logging to take >>>>> place. You should delete the directory you created. >>>>> >>>>> rob >>>>> >>>>> >>>>>> >>>>>> On Tue, Mar 5, 2019 at 9:16 AM Florence Blanc-Renaud <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> On 3/5/19 8:44 AM, Sina Owolabi via FreeIPA-users wrote: >>>>>>>> Hi! >>>>>>>> >>>>>>>> I tried to follow this solution for cert renewal for RHEL6: >>>>>>>> https://access.redhat.com/solutions/643753 (Sorry, desperation is >>>>>>>> setting in), but when I attempted Step 2, I got: >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> 1. this note was written for RHEL 6 but you said in your first e-mail >>>>>>> that your server is running CentOS 7 with ipa 4.5.4. Please don't follow >>>>>>> those instructions as they are not adapted to your deployment. >>>>>>> The instructions for RHEL 7 are available at >>>>>>> https://access.redhat.com/solutions/3357261. >>>>>>> >>>>>>> 2. In a previous e-mail, the output of getcert list | grep -i expires >>>>>>> did not show any expired certificates, so I would not rush into wrong >>>>>>> conclusions. We need to understand first why pki did not start. >>>>>>> >>>>>>> What is the output of: >>>>>>> $ ipactl status >>>>>>> $ systemctl status [email protected] >>>>>>> >>>>>>> flo >>>>>>> >>>>>>>> # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert >>>>>>>> cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do >>>>>>>> echo $nickname; certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" >>>>>>>> | grep -i after; done >>>>>>>> auditSigningCert cert-pki-ca >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>>>>>> certificate/key database is in an old, unsupported format. >>>>>>>> ocspSigningCert cert-pki-ca >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>>>>>> certificate/key database is in an old, unsupported format. >>>>>>>> subsystemCert cert-pki-ca >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>>>>>> certificate/key database is in an old, unsupported format. >>>>>>>> Server-Cert cert-pki-ca >>>>>>>> certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The >>>>>>>> certificate/key database is in an old, unsupported format. >>>>>>>> >>>>>>>> Could this be the root of my problems? >>>>>>>> And how can I convert them? >>>>>>>> >>>>>>>> On Mon, Mar 4, 2019 at 9:08 PM Sina Owolabi <[email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Restarting ipa didnt create the logs. >>>>>>>>> Please, what else can i do? >>>>>>>>> >>>>>>>>> On Mon, Mar 4, 2019 at 8:47 PM Sina Owolabi <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi! >>>>>>>>>> >>>>>>>>>> getcert list | grep -i expires >>>>>>>>>> expires: 2019-04-13 12:08:20 UTC >>>>>>>>>> expires: 2019-04-13 12:08:06 UTC >>>>>>>>>> expires: 2019-04-13 12:07:50 UTC >>>>>>>>>> expires: 2035-06-01 08:33:01 UTC >>>>>>>>>> expires: 2019-04-13 12:07:41 UTC >>>>>>>>>> expires: 2019-04-13 12:06:55 UTC >>>>>>>>>> expires: 2019-05-05 12:06:41 UTC >>>>>>>>>> expires: 2019-05-05 12:06:56 UTC >>>>>>>>>> expires: 2020-01-17 19:56:03 UTC >>>>>>>>>> >>>>>>>>>> I didnt find a /var/log/pki/pki-tomcat/ca/debug directory, but I am >>>>>>>>>> creating one and running "ipactl restart". >>>>>>>>>> >>>>>>>>>> On Mon, Mar 4, 2019 at 8:10 PM Rob Crittenden <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Sina Owolabi via FreeIPA-users wrote: >>>>>>>>>>>> Hi! >>>>>>>>>>>> >>>>>>>>>>>> I am running a small IPA domain (CentOS 7 servers, ipa version >>>>>>>>>>>> 4.5.4, >>>>>>>>>>>> api version 2.228), with one master, and two replicas, and I >>>>>>>>>>>> noticed >>>>>>>>>>>> that pki-tomcatd no longer works on the master, after attempting a >>>>>>>>>>>> reboot. >>>>>>>>>>>> pki-tomcatd works fine on the slaves. >>>>>>>>>>>> I noticed if I try to run IPA functions (dns record removal, hosts >>>>>>>>>>>> management, user passwords, etc), I receive responses like this: >>>>>>>>>>>> >>>>>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>>>>>>> communicate with CMS (Internal Server Error) >>>>>>>>>>>> But on the replicas, functions work fine. >>>>>>>>>>>> Please can someone guide me on how to fix this? >>>>>>>>>>> >>>>>>>>>>> The CA log is in /var/log/pki/pki-tomcat/ca/debug. That may have >>>>>>>>>>> some >>>>>>>>>>> pointers. I'd look at selftests.log first. >>>>>>>>>>> >>>>>>>>>>> My guess is that some of the CA certificates have failed to renew. >>>>>>>>>>> >>>>>>>>>>> getcert list | grep -i expires >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>> _______________________________________________ >>>>>>>> FreeIPA-users mailing list -- [email protected] >>>>>>>> To unsubscribe send an email to >>>>>>>> [email protected] >>>>>>>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >>>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>>>> List Archives: >>>>>>>> https://lists.fedorahosted.org/archives/list/[email protected] >>>>>>>> >>>>>>> >>>>> >>> > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected]
