Re: [Freeipa-users] Trouble creating replica
Rich, 389-ds-base-1.2.11.5-1.fc17.x86_64. The box is a DL360G8. * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Wed, Feb 20, 2013 at 9:03 PM, Rich Megginson rmegg...@redhat.com wrote: On 02/20/2013 06:43 PM, Bret Wortman wrote: Mine was not. What platform? What version of 389-ds-base? — Bret Wortman On Wed, Feb 20, 2013 at 8:16 PM, Rich Megginson rmegg...@redhat.comwrote: On 02/20/2013 06:00 PM, KodaK wrote: On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.com wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. were these virtual machines? --Jason ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
On 02/21/2013 07:11 AM, Bret Wortman wrote: Rich, 389-ds-base-1.2.11.5-1.fc17.x86_64. The box is a DL360G8. https://fedorahosted.org/389/ticket/518 _ _ *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Wed, Feb 20, 2013 at 9:03 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 02/20/2013 06:43 PM, Bret Wortman wrote: Mine was not. What platform? What version of 389-ds-base? — Bret Wortman On Wed, Feb 20, 2013 at 8:16 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 02/20/2013 06:00 PM, KodaK wrote: On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wort...@damascusgrp.com wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. were these virtual machines? --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com mailto:Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
Thanks for the bug link. We let the developer we thought had messed things up out of the 4x4 cell we had stashed him in. He's still blinking from sunlight but the doctors tell us the facial twitching will stop in a month or two. * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Thu, Feb 21, 2013 at 10:54 AM, Rich Megginson rmegg...@redhat.comwrote: On 02/21/2013 07:11 AM, Bret Wortman wrote: Rich, 389-ds-base-1.2.11.5-1.fc17.x86_64. The box is a DL360G8. https://fedorahosted.org/389/ticket/518 * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Wed, Feb 20, 2013 at 9:03 PM, Rich Megginson rmegg...@redhat.comwrote: On 02/20/2013 06:43 PM, Bret Wortman wrote: Mine was not. What platform? What version of 389-ds-base? — Bret Wortman On Wed, Feb 20, 2013 at 8:16 PM, Rich Megginson rmegg...@redhat.comwrote: On 02/20/2013 06:00 PM, KodaK wrote: On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.com wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. were these virtual machines? --Jason ___ Freeipa-users mailing listFreeipa-users@redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
Digging further into my logs this morning, I've discovered that there's no new entries in /var/log/dirsrv/slapd-PKI-IPA since Feb 5 either. How can I tell why this isn't running? /var/log/dirsrv/slapd-MY-COM is getting updated and logged to, it's just the PKI piece that seems to be dead. Nothing in /etc/pki-ca has changed since last year, and the last updates to /var/lib/dirsrv/slapd-PKI-IPA/db or changelogs occurred on Feb 5. I just can't tell what that change was Would a key change or certificate change have affected this? Worst case, if I do something like this: # ipa-server-install -U --uninstall # ipa-server-install will I lose the hosts, policies users I already have configured? Does this stand a chance of getting me back up to where I can clone this box and get healthy again? * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Tue, Feb 19, 2013 at 2:01 PM, Bret Wortman bret.wort...@damascusgrp.comwrote: No, can't telnet to 7389 or 9444 either one: [root@ipamaster]# telnet oldmaster.my.com 7389 Trying 10.0.0.42... telnet: connect to address 10.0.0.42: COnnection refused [root@ipamaster]# I do note that I only have packages called dogtag-*-theme installed: [root@oldmaster]# yum list *dogtag* Loaded plugins: lnagpacks, presto, refresh-packagekit Installed Packages dogtag-pki-ca-theme.noarch 9.0.11-1.fc17 @fedora dogtag-pki-common-theme.noarch 9.0.11-1.fc17 @fedora Available Packages dogtag-pki.noarch 9.0.0-13.fc17 @fedora : I also noticed that, according to /var/log/pki-ca/catalina.out and /var/log/pki-ca/debug, this hasn't successfully run since 05-Feb. And no, I'm not sure what happened on that day to change things, but I'm trying to find out. (At least, I assume this logdir relates to dogtag) * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Tue, Feb 19, 2013 at 1:26 PM, Rob Crittenden rcrit...@redhat.comwrote: Natxo Asenjo wrote: On Tue, Feb 19, 2013 at 5:58 PM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wortman@**damascusgrp.combret.wort...@damascusgrp.com wrote: Digging a bit deeper, I found this in /var/log/pki-ca/catalina.out: : Could not connect to LDAP server host oldmaster.my.com http://oldmaster.my.com port 7389 Error netscape.ldap.LDAPException: failed to connect to server ldap://oldmaster.my.com:7389 http://oldmaster.my.com:7389 (91) This certainly appears to be a problem, but everyone's authenticating against oldmaster just fine. Thoughts, anyone? can you connect to that port (7389) on oldmaster.my.com http://oldmaster.my.com from the other replica? (try telnetting to the port: telnet oldmaster.my.com http://oldmaster.my.com 7389) 7389 is port in the 389-ds instance used by dogtag. Is the instance running on oldmaster? It isn't used for authentication which is why you aren't seeing problems with clients. rob __**_ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/**mailman/listinfo/freeipa-usershttps://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Wed, Feb 20, 2013 at 9:34 AM, Bret Wortman bret.wort...@damascusgrp.comwrote: I think this keeps coming back to the fact that ldap isn't listening on 7389 for some reason. When I try to *really* manually start pki-ca like this, it complains about ldap before dying: # sudo -u pkiuser -s /usr/lib/jvm/jre/bin/java -classpath :/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki-ca -Dcatalina.home=/usr/share/tomcat6 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat6/temp -Djava.util.logging.config.file=/var/lib/pki-ca/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start : : Could not connect to LDAP server host oldmaster.my.com port 7389 Error netscape.ldap.LDAPException: failed to connect to server ldap:// oldmaster.my.com:7389 (91) [root@oldmaster]# This bears out what I see in /var/log/pki-ca/catalina.out too. * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Wed, Feb 20, 2013 at 8:43 AM, Bret Wortman bret.wort...@damascusgrp.com wrote: On Wed, Feb 20, 2013 at 8:40 AM, Simo Sorce s...@redhat.com wrote: On Wed, 2013-02-20 at 08:08 -0500, Bret Wortman wrote: Digging further into my logs this morning, I've discovered that there's no new entries in /var/log/dirsrv/slapd-PKI-IPA since Feb 5 either. How can I tell why this isn't running? /var/log/dirsrv/slapd-MY-COM is getting updated and logged to, it's just the PKI piece that seems to be dead. Nothing in /etc/pki-ca has changed since last year, and the last updates to /var/lib/dirsrv/slapd-PKI-IPA/db or changelogs occurred on Feb 5. I just can't tell what that change was What error do you get if you try to start it ? [root@oldmaster]# pkicontrol start ca PKI-IPA PKI-IPA is an invalid 'pki-ca' instance [root@oldmaster]# Is there another, preferred way to start it? Would a key change or certificate change have affected this? An expired CA cert might cause the server to stop, but then you would see expired certs all over and also the main IPA instance would not start. Worst case, if I do something like this: # ipa-server-install -U --uninstall # ipa-server-install You will completely obliterate all your data. will I lose the hosts, policies users I already have configured? Does this stand a chance of getting me back up to where I can clone this box and get healthy again? Healthy will be, but with no data, don't do it. (and I suggest you make a full backup just in case) Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
On 02/20/2013 08:43 AM, Bret Wortman wrote: [root@oldmaster]# pkicontrol start ca PKI-IPA PKI-IPA is an invalid 'pki-ca' instance [root@oldmaster]# Is there another, preferred way to start it? pkiconsole is used to monitor/configure your instance, it's a GUI application. Perhaps it can also be used to start/stop instances but I've never seen it used that way and we don't use pkiconsole at all. Normally the pki-ca instance is controlled using the same service commands for any other daemon. Some of this has been in flux so the details may depend on your exact OS. If you don't provide a specific instance to start/stop then the service command will apply the action to all your instances, usaully this is fine as usaully you only have one instance. As for debugging what is going on. pki-ca is a tomcat instance. You need to locate it's log files under /var/log depending on the release it can be named slightly differently but it should be obvious. You need to understand how a tomcat instance starts, again this depends on the release. Early start up messages will be written to catalina.out, those are tomcat specific messages, if you have problems opening sockets (for instance bad certs) it should show up in this file. Once tomcat hands control over to the application (i.e. pki-ca) you will see messages in the debug file located under the /var/log/pki-ca (or whatever, depends on the release) directory. As I said it should be easy to find. Look in that file for obvious problems. HTH, I forget the exact version you're running on which OS. If the above is not specific enough we can get the dogtag folks to jump in. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
Bret Wortman wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! Glad you're up and running again. I'm curious, what version are you running? rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
I'm running 2.2.0-1.fc17.x86_64 And FWIW, the replica data file I was able to create after this just installed successfully on the new host. * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Wed, Feb 20, 2013 at 9:47 AM, Rob Crittenden rcrit...@redhat.com wrote: Bret Wortman wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.**ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! Glad you're up and running again. I'm curious, what version are you running? rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.comwrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
On 02/20/2013 06:00 PM, KodaK wrote: On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wort...@damascusgrp.com wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. were these virtual machines? --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
Mine was not. — Bret Wortman On Wed, Feb 20, 2013 at 8:16 PM, Rich Megginson rmegg...@redhat.com wrote: On 02/20/2013 06:00 PM, KodaK wrote: On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wort...@damascusgrp.com wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. were these virtual machines? --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
On 02/20/2013 06:43 PM, Bret Wortman wrote: Mine was not. What platform? What version of 389-ds-base? — Bret Wortman On Wed, Feb 20, 2013 at 8:16 PM, Rich Megginson rmegg...@redhat.com mailto:rmegg...@redhat.com wrote: On 02/20/2013 06:00 PM, KodaK wrote: On Wed, Feb 20, 2013 at 8:41 AM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wort...@damascusgrp.com wrote: Eureka! Someone had deleted the contents of /etc/dirsrv/slapd-PKI-IPA/dse.ldif. I replaced it from a saved copy and now everything's working as expected. Thanks everyone for your contributions, patience, and indulgence. And for a wonderful product! I wouldn't be too sure that someone deleted it. A couple of weeks ago I had a crash and half of my replicas had an empty dse.ldif. I think you and I may be hitting a bug. were these virtual machines? --Jason ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
On Tue, Feb 19, 2013 at 5:58 PM, Bret Wortman bret.wort...@damascusgrp.comwrote: Digging a bit deeper, I found this in /var/log/pki-ca/catalina.out: : Could not connect to LDAP server host oldmaster.my.com port 7389 Error netscape.ldap.LDAPException: failed to connect to server ldap:// oldmaster.my.com:7389 (91) This certainly appears to be a problem, but everyone's authenticating against oldmaster just fine. Thoughts, anyone? can you connect to that port (7389) on oldmaster.my.com from the other replica? (try telnetting to the port: telnet oldmaster.my.com 7389) ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
Natxo Asenjo wrote: On Tue, Feb 19, 2013 at 5:58 PM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wort...@damascusgrp.com wrote: Digging a bit deeper, I found this in /var/log/pki-ca/catalina.out: : Could not connect to LDAP server host oldmaster.my.com http://oldmaster.my.com port 7389 Error netscape.ldap.LDAPException: failed to connect to server ldap://oldmaster.my.com:7389 http://oldmaster.my.com:7389 (91) This certainly appears to be a problem, but everyone's authenticating against oldmaster just fine. Thoughts, anyone? can you connect to that port (7389) on oldmaster.my.com http://oldmaster.my.com from the other replica? (try telnetting to the port: telnet oldmaster.my.com http://oldmaster.my.com 7389) 7389 is port in the 389-ds instance used by dogtag. Is the instance running on oldmaster? It isn't used for authentication which is why you aren't seeing problems with clients. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Trouble creating replica
No, can't telnet to 7389 or 9444 either one: [root@ipamaster]# telnet oldmaster.my.com 7389 Trying 10.0.0.42... telnet: connect to address 10.0.0.42: COnnection refused [root@ipamaster]# I do note that I only have packages called dogtag-*-theme installed: [root@oldmaster]# yum list *dogtag* Loaded plugins: lnagpacks, presto, refresh-packagekit Installed Packages dogtag-pki-ca-theme.noarch 9.0.11-1.fc17 @fedora dogtag-pki-common-theme.noarch 9.0.11-1.fc17 @fedora Available Packages dogtag-pki.noarch 9.0.0-13.fc17 @fedora : I also noticed that, according to /var/log/pki-ca/catalina.out and /var/log/pki-ca/debug, this hasn't successfully run since 05-Feb. And no, I'm not sure what happened on that day to change things, but I'm trying to find out. (At least, I assume this logdir relates to dogtag) * * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman On Tue, Feb 19, 2013 at 1:26 PM, Rob Crittenden rcrit...@redhat.com wrote: Natxo Asenjo wrote: On Tue, Feb 19, 2013 at 5:58 PM, Bret Wortman bret.wort...@damascusgrp.com mailto:bret.wortman@**damascusgrp.combret.wort...@damascusgrp.com wrote: Digging a bit deeper, I found this in /var/log/pki-ca/catalina.out: : Could not connect to LDAP server host oldmaster.my.com http://oldmaster.my.com port 7389 Error netscape.ldap.LDAPException: failed to connect to server ldap://oldmaster.my.com:7389 http://oldmaster.my.com:7389 (91) This certainly appears to be a problem, but everyone's authenticating against oldmaster just fine. Thoughts, anyone? can you connect to that port (7389) on oldmaster.my.com http://oldmaster.my.com from the other replica? (try telnetting to the port: telnet oldmaster.my.com http://oldmaster.my.com 7389) 7389 is port in the 389-ds instance used by dogtag. Is the instance running on oldmaster? It isn't used for authentication which is why you aren't seeing problems with clients. rob __**_ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/**mailman/listinfo/freeipa-usershttps://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
