Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Alexander Bokovoy
On Mon, 14 Nov 2011, Dan Scott wrote:

 Hi,
 
 I've just upgraded a server from Fedora 15 to 16 and I'm having
 problems starting the dirsrv process:
 
 /var/log/messages
 Nov 14 09:38:27 fileserver1 ipactl[1351]: Failed to read data from
 Directory Service: Unknown error when retrieving list of services from
 LDAP: [Errno 2] No such file or directory
 Nov 14 09:38:27 fileserver1 ipactl[1351]: Shutting down
 Nov 14 09:38:27 fileserver1 ipactl[1351]: Starting Directory Service
 Nov 14 09:38:27 fileserver1 systemd[1]: ipa.service: main process
 exited, code=exited, status=1
 Nov 14 09:38:27 fileserver1 systemd[1]: Unit ipa.service entered failed state.
 
 The /var/log/dirsrv/slapd-EXAMPLE-COM/errors file contains no new
 entries since Friday 11th.
 
 Any ideas how I can get this fixed? How can I find out which 'file or
 directory' is missing?
Looks like LDAP socket is not yet available at the time we try to 
contact it. I think this was fixed in Fedora 16 package with this 
patch:
http://git.fedorahosted.org/git/?p=freeipa.git;a=commitdiff;h=5451328bc55fe964c61e7b87959310f9c6748cf8

Could you make sure 'systemctl start dirsrv.target' actually starts 
slapd for EXAMPLE-COM? If not, please show output of 

ls -l /etc/systemd/system/dirsrv.target.wants 

It may be that we would need to make a small upgrade script that 
re-installs proper systemd instances for dirsrv.target as those are 
produced during ipa-server-install and cannot be done automatically on 
upgrade without proper intervention yet.

Fedora 15 to Fedora 16 upgrade is a bit complicated due to change from 
System V to systemd.
-- 
/ Alexander Bokovoy

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Dan Scott
Hi,

On Mon, Nov 14, 2011 at 10:19, Alexander Bokovoy aboko...@redhat.com wrote:
 On Mon, 14 Nov 2011, Dan Scott wrote:

 Hi,

 I've just upgraded a server from Fedora 15 to 16 and I'm having
 problems starting the dirsrv process:

 /var/log/messages
 Nov 14 09:38:27 fileserver1 ipactl[1351]: Failed to read data from
 Directory Service: Unknown error when retrieving list of services from
 LDAP: [Errno 2] No such file or directory
 Nov 14 09:38:27 fileserver1 ipactl[1351]: Shutting down
 Nov 14 09:38:27 fileserver1 ipactl[1351]: Starting Directory Service
 Nov 14 09:38:27 fileserver1 systemd[1]: ipa.service: main process
 exited, code=exited, status=1
 Nov 14 09:38:27 fileserver1 systemd[1]: Unit ipa.service entered failed 
 state.

 The /var/log/dirsrv/slapd-EXAMPLE-COM/errors file contains no new
 entries since Friday 11th.

 Any ideas how I can get this fixed? How can I find out which 'file or
 directory' is missing?
 Looks like LDAP socket is not yet available at the time we try to
 contact it. I think this was fixed in Fedora 16 package with this
 patch:
 http://git.fedorahosted.org/git/?p=freeipa.git;a=commitdiff;h=5451328bc55fe964c61e7b87959310f9c6748cf8

 Could you make sure 'systemctl start dirsrv.target' actually starts
 slapd for EXAMPLE-COM? If not, please show output of

 ls -l /etc/systemd/system/dirsrv.target.wants

'systemctl start dirsrv.target' doesn't appear to do anything, nothing
shown on the command line and the logs don't change. The directory is
empty:

[root@fileserver1 schema]# ls -l /etc/systemd/system/dirsrv.target.wants/
total 0

 It may be that we would need to make a small upgrade script that
 re-installs proper systemd instances for dirsrv.target as those are
 produced during ipa-server-install and cannot be done automatically on
 upgrade without proper intervention yet.

Is this related to this:
https://fedoraproject.org/wiki/Common_F16_bugs#Upgrade_from_previous_releases_resets_the_enablement_status_of_services

Or is it to do with the dependencies of FreeIPA startup?

In any case, the process is still failing to start. Do I need to
create a link in dirsrv.target.wants to somewhere?

Thanks,

Dan

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Alexander Bokovoy
On Mon, 14 Nov 2011, Dan Scott wrote:
  Could you make sure 'systemctl start dirsrv.target' actually starts
  slapd for EXAMPLE-COM? If not, please show output of
 
  ls -l /etc/systemd/system/dirsrv.target.wants
 
 'systemctl start dirsrv.target' doesn't appear to do anything, nothing
 shown on the command line and the logs don't change. The directory is
 empty:
 
 [root@fileserver1 schema]# ls -l /etc/systemd/system/dirsrv.target.wants/
 total 0
Yes, as I expected (below).

  It may be that we would need to make a small upgrade script that
  re-installs proper systemd instances for dirsrv.target as those are
  produced during ipa-server-install and cannot be done automatically on
  upgrade without proper intervention yet.
 
 Is this related to this:
 https://fedoraproject.org/wiki/Common_F16_bugs#Upgrade_from_previous_releases_resets_the_enablement_status_of_services
 
 Or is it to do with the dependencies of FreeIPA startup?
It is mixture of those cases. systemd is more complicated and if in 
F15 we were able to get away via SystemV emulation, in F16 dirsrv migrated 
natively to systemd, managing instances through native systemd 
mechanism (dirsrv@EXAMPLE-COM.service as a service name, for 
example). 

This new mechanism is not accessible via SystemV emulation and we had 
to migrate to systemd as well -- which means ipa-server-install 
creates proper links and edits systemd service files as needed.

In addition, systemd does not really support our model of enabling 
services, as systemd is per-host while we need to replicate service 
state to multiple replicas. Thus, we do some of enable/disable/restart 
management in ipactl.

 In any case, the process is still failing to start. Do I need to
 create a link in dirsrv.target.wants to somewhere?
You need to do some steps like ipa-server-install does. I'm trying to 
get them separated in a small upgrade script but something like 
following needs to be done, completely untested, may eat your kitten, 
and realm/dirsrv instance names need to be replaced before running:

#! /usr/bin/python -E
from ipaserver.install.krbinstance import update_val_in_file
from ipapython import ipautil
from ipapython import services as ipaservices

# 1. Upgrade /etc/sysconfig/dirsrv for systemd
update_key_val_in_file(/etc/sysconfig/dirsrv, KRB5_KTNAME, 
/etc/dirsrv/ds.keytab)
update_key_val_in_file(/etc/sysconfig/dirsrv, export KRB5_KTNAME, 
/etc/dirsrv/ds.keytab)
# 2. Upgrade /etc/sysconfig/krb5kdc for systemd
replacevars = {'KRB5REALM':EXAMPLE.COM}
appendvars = {}
ipautil.config_replace_variables(/etc/sysconfig/krb5kdc,
replacevars=replacevars, appendvars=appendvars)
ipaservices.restore_context(/etc/sysconfig/krb5kdc)
# 3. Enable DS instances:
ipaservices.knownservices.dirsrv.enable(EXAMPLE-COM)
ipaservices.knownservices.dirsrv.enable(PKI-IPA)
# 4. Enable FreeIPA
ipaservices.knownservices.ipa.enable()
---

Note that these .enable() calls on Fedora 16 do much more than just 
'systemctl enable foo.service', they copy and modify service files, 
create symlinks and so on, all the dirty work required by systemd.
You may look at ipapython/platform/fedora16.py and systemd.py for 
details.
-- 
/ Alexander Bokovoy

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Dan Scott
Hi,

On Mon, Nov 14, 2011 at 13:06, Alexander Bokovoy aboko...@redhat.com wrote:
 On Mon, 14 Nov 2011, Dan Scott wrote:
 In any case, the process is still failing to start. Do I need to
 create a link in dirsrv.target.wants to somewhere?
 You need to do some steps like ipa-server-install does. I'm trying to
 get them separated in a small upgrade script but something like
 following needs to be done, completely untested, may eat your kitten,
 and realm/dirsrv instance names need to be replaced before running:
 
 #! /usr/bin/python -E
 from ipaserver.install.krbinstance import update_val_in_file
 from ipapython import ipautil
 from ipapython import services as ipaservices

 # 1. Upgrade /etc/sysconfig/dirsrv for systemd
 update_key_val_in_file(/etc/sysconfig/dirsrv, KRB5_KTNAME, 
 /etc/dirsrv/ds.keytab)
 update_key_val_in_file(/etc/sysconfig/dirsrv, export KRB5_KTNAME, 
 /etc/dirsrv/ds.keytab)
 # 2. Upgrade /etc/sysconfig/krb5kdc for systemd
 replacevars = {'KRB5REALM':EXAMPLE.COM}
 appendvars = {}
 ipautil.config_replace_variables(/etc/sysconfig/krb5kdc,
    replacevars=replacevars, appendvars=appendvars)
 ipaservices.restore_context(/etc/sysconfig/krb5kdc)
 # 3. Enable DS instances:
 ipaservices.knownservices.dirsrv.enable(EXAMPLE-COM)
 ipaservices.knownservices.dirsrv.enable(PKI-IPA)
 # 4. Enable FreeIPA
 ipaservices.knownservices.ipa.enable()
 ---

 Note that these .enable() calls on Fedora 16 do much more than just
 'systemctl enable foo.service', they copy and modify service files,
 create symlinks and so on, all the dirty work required by systemd.
 You may look at ipapython/platform/fedora16.py and systemd.py for
 details.

OK, looks like I'm getting there, but there's still a problem (I
replaced EXAMPLE-COM above and re-replaced it in the output below):

[root@fileserver1 ~]# ls -l /etc/systemd/system/dirsrv.target.wants
total 0
lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@EXAMPLE-COM.service -
/etc/systemd/system/dirsrv@.service
lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@PKI-IPA.service -
/etc/systemd/system/dirsrv@.service
[root@fileserver1 ~]# systemctl status dirsrv.service
dirsrv.service
  Loaded: error (Reason: No such file or directory)
  Active: inactive (dead)
[root@fileserver1 ~]#

My /var/log/dirsrv/slapd-EXAMPLE-COM/errors now contains:

[14/Nov/2011:14:55:16 -0500] set_krb5_creds - Could not get initial
credentials for principal [ldap/fileserver1.example@example.com]
in keytab [WRFILE:/etc/krb5.keytab]: 13 (Permission denied)
[14/Nov/2011:14:55:16 -0500] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (Credentials
cache file '/tmp/krb5cc_494' not found))
[14/Nov/2011:14:55:16 -0500] slapi_ldap_bind - Error: could not
perform interactive bind for id [] mech [GSSAPI]: error -2 (Local
error)

And the permissions on /etc/krb5.keytab:

[root@fileserver1 ~]# ls -Z /etc/krb5.keytab
-rw---. root root unconfined_u:object_r:krb5_keytab_t:s0 /etc/krb5.keytab

The permissions are the same on my other, replica, IPA server (which
is still Fedora 15). The other message above is correct:
/tmp/krb5cc_494 does not exist.

Thanks,

Dan

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Rich Megginson

On 11/14/2011 01:08 PM, Dan Scott wrote:

Hi,

On Mon, Nov 14, 2011 at 13:06, Alexander Bokovoyaboko...@redhat.com  wrote:

On Mon, 14 Nov 2011, Dan Scott wrote:

In any case, the process is still failing to start. Do I need to
create a link in dirsrv.target.wants to somewhere?

You need to do some steps like ipa-server-install does. I'm trying to
get them separated in a small upgrade script but something like
following needs to be done, completely untested, may eat your kitten,
and realm/dirsrv instance names need to be replaced before running:

#! /usr/bin/python -E
from ipaserver.install.krbinstance import update_val_in_file
from ipapython import ipautil
from ipapython import services as ipaservices

# 1. Upgrade /etc/sysconfig/dirsrv for systemd
update_key_val_in_file(/etc/sysconfig/dirsrv, KRB5_KTNAME, 
/etc/dirsrv/ds.keytab)
update_key_val_in_file(/etc/sysconfig/dirsrv, export KRB5_KTNAME, 
/etc/dirsrv/ds.keytab)
# 2. Upgrade /etc/sysconfig/krb5kdc for systemd
replacevars = {'KRB5REALM':EXAMPLE.COM}
appendvars = {}
ipautil.config_replace_variables(/etc/sysconfig/krb5kdc,
replacevars=replacevars, appendvars=appendvars)
ipaservices.restore_context(/etc/sysconfig/krb5kdc)
# 3. Enable DS instances:
ipaservices.knownservices.dirsrv.enable(EXAMPLE-COM)
ipaservices.knownservices.dirsrv.enable(PKI-IPA)
# 4. Enable FreeIPA
ipaservices.knownservices.ipa.enable()
---

Note that these .enable() calls on Fedora 16 do much more than just
'systemctl enable foo.service', they copy and modify service files,
create symlinks and so on, all the dirty work required by systemd.
You may look at ipapython/platform/fedora16.py and systemd.py for
details.

OK, looks like I'm getting there, but there's still a problem (I
replaced EXAMPLE-COM above and re-replaced it in the output below):

[root@fileserver1 ~]# ls -l /etc/systemd/system/dirsrv.target.wants
total 0
lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@EXAMPLE-COM.service -
/etc/systemd/system/dirsrv@.service
lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@PKI-IPA.service -
/etc/systemd/system/dirsrv@.service
[root@fileserver1 ~]# systemctl status dirsrv.service
dirsrv.service
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

Right - see http://directory.fedoraproject.org/wiki/Howto:systemd#FAQ

[root@fileserver1 ~]#

My /var/log/dirsrv/slapd-EXAMPLE-COM/errors now contains:

[14/Nov/2011:14:55:16 -0500] set_krb5_creds - Could not get initial
credentials for principal [ldap/fileserver1.example@example.com]
in keytab [WRFILE:/etc/krb5.keytab]: 13 (Permission denied)
[14/Nov/2011:14:55:16 -0500] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: error -2
(Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (Credentials
cache file '/tmp/krb5cc_494' not found))
[14/Nov/2011:14:55:16 -0500] slapi_ldap_bind - Error: could not
perform interactive bind for id [] mech [GSSAPI]: error -2 (Local
error)

And the permissions on /etc/krb5.keytab:

[root@fileserver1 ~]# ls -Z /etc/krb5.keytab
-rw---. root root unconfined_u:object_r:krb5_keytab_t:s0 /etc/krb5.keytab
Right - directory server usually runs as dirsrv:dirsrv not root:root - 
not sure what is responsible for ensuring the krb5.keytab is owned by 
the dirsrv user.

The permissions are the same on my other, replica, IPA server (which
is still Fedora 15). The other message above is correct:
/tmp/krb5cc_494 does not exist.

Thanks,

Dan

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Alexander Bokovoy
On Mon, 14 Nov 2011, Rich Megginson wrote:
 replaced EXAMPLE-COM above and re-replaced it in the output below):
 
 [root@fileserver1 ~]# ls -l /etc/systemd/system/dirsrv.target.wants
 total 0
 lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@EXAMPLE-COM.service -
 /etc/systemd/system/dirsrv@.service
 lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@PKI-IPA.service -
 /etc/systemd/system/dirsrv@.service
 [root@fileserver1 ~]# systemctl status dirsrv.service
 dirsrv.service
Loaded: error (Reason: No such file or directory)
Active: inactive (dead)
 Right - see http://directory.fedoraproject.org/wiki/Howto:systemd#FAQ
Yes, the target is dirsrv.target, not dirsrv.service, while instances 
are dirsrv@NAME.service. That is life.

systemctl start dirsrv.target

now would bring both instances up -- when you'll solve 
kerberos credentials access.

 [root@fileserver1 ~]#
 
 My /var/log/dirsrv/slapd-EXAMPLE-COM/errors now contains:
 
 [14/Nov/2011:14:55:16 -0500] set_krb5_creds - Could not get initial
 credentials for principal [ldap/fileserver1.example@example.com]
 in keytab [WRFILE:/etc/krb5.keytab]: 13 (Permission denied)
 [14/Nov/2011:14:55:16 -0500] slapd_ldap_sasl_interactive_bind - Error:
 could not perform interactive bind for id [] mech [GSSAPI]: error -2
 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
 GSS failure.  Minor code may provide more information (Credentials
 cache file '/tmp/krb5cc_494' not found))
 [14/Nov/2011:14:55:16 -0500] slapi_ldap_bind - Error: could not
 perform interactive bind for id [] mech [GSSAPI]: error -2 (Local
 error)
 
 And the permissions on /etc/krb5.keytab:
 
 [root@fileserver1 ~]# ls -Z /etc/krb5.keytab
 -rw---. root root unconfined_u:object_r:krb5_keytab_t:s0 /etc/krb5.keytab
 Right - directory server usually runs as dirsrv:dirsrv not root:root
 - not sure what is responsible for ensuring the krb5.keytab is owned
 by the dirsrv user.
It should be /etc/dirsrv/ds.keytab, not /etc/krb5.keytab. Could you 
please show your /etc/sysconfig/dirsrv? KRB5_KTNAME there should point 
to /etc/dirsrv/ds.keytab and as you have installation that worked 
before, the keytab should be in place already and with proper 
ownership (dirsrv:dirsrv).

Dan, could you please file a bug against freeipa in Fedora 16 to ask 
about upgrade from Fedora 15. I'll then work out the script and how to use 
it. I'm not sure it will be possible to use it in %post for upgrades 
but at least running it after yum upgrade would be possible.
-- 
/ Alexander Bokovoy

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users


Re: [Freeipa-users] Fedora 16 failing to start dirsrv process

2011-11-14 Thread Dan Scott
Hi,

On Mon, Nov 14, 2011 at 15:50, Alexander Bokovoy aboko...@redhat.com wrote:
 On Mon, 14 Nov 2011, Rich Megginson wrote:
 replaced EXAMPLE-COM above and re-replaced it in the output below):
 
 [root@fileserver1 ~]# ls -l /etc/systemd/system/dirsrv.target.wants
 total 0
 lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@EXAMPLE-COM.service -
 /etc/systemd/system/dirsrv@.service
 lrwxrwxrwx. 1 root root 35 Nov 14 14:49 dirsrv@PKI-IPA.service -
 /etc/systemd/system/dirsrv@.service
 [root@fileserver1 ~]# systemctl status dirsrv.service
 dirsrv.service
            Loaded: error (Reason: No such file or directory)
            Active: inactive (dead)
 Right - see http://directory.fedoraproject.org/wiki/Howto:systemd#FAQ
 Yes, the target is dirsrv.target, not dirsrv.service, while instances
 are dirsrv@NAME.service. That is life.

:) Nice and consistent with other 'services'. Do you know if it's
possible for 'systemctl status dirsrv.service' to return nothing,
instead of saying that it's dead? This would help reduce the
confusion.

 systemctl start dirsrv.target

 now would bring both instances up -- when you'll solve
 kerberos credentials access.

 [root@fileserver1 ~]#
 
 My /var/log/dirsrv/slapd-EXAMPLE-COM/errors now contains:
 
 [14/Nov/2011:14:55:16 -0500] set_krb5_creds - Could not get initial
 credentials for principal [ldap/fileserver1.example@example.com]
 in keytab [WRFILE:/etc/krb5.keytab]: 13 (Permission denied)
 [14/Nov/2011:14:55:16 -0500] slapd_ldap_sasl_interactive_bind - Error:
 could not perform interactive bind for id [] mech [GSSAPI]: error -2
 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
 GSS failure.  Minor code may provide more information (Credentials
 cache file '/tmp/krb5cc_494' not found))
 [14/Nov/2011:14:55:16 -0500] slapi_ldap_bind - Error: could not
 perform interactive bind for id [] mech [GSSAPI]: error -2 (Local
 error)
 
 And the permissions on /etc/krb5.keytab:
 
 [root@fileserver1 ~]# ls -Z /etc/krb5.keytab
 -rw---. root root unconfined_u:object_r:krb5_keytab_t:s0 
 /etc/krb5.keytab
 Right - directory server usually runs as dirsrv:dirsrv not root:root
 - not sure what is responsible for ensuring the krb5.keytab is owned
 by the dirsrv user.
 It should be /etc/dirsrv/ds.keytab, not /etc/krb5.keytab. Could you
 please show your /etc/sysconfig/dirsrv? KRB5_KTNAME there should point
 to /etc/dirsrv/ds.keytab and as you have installation that worked
 before, the keytab should be in place already and with proper
 ownership (dirsrv:dirsrv).

Thanks. I'd just figured this out and fixed my /etc/sysconfig/dirsrv
file. The two servers seem to be working and syncing now.

I've run into something else now though:

djscott@pc35:~$ ipa host-del pc60
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (Not Found)

Could this be related? Or should I start a new thread to try and solve it.

 Dan, could you please file a bug against freeipa in Fedora 16 to ask
 about upgrade from Fedora 15. I'll then work out the script and how to use
 it. I'm not sure it will be possible to use it in %post for upgrades
 but at least running it after yum upgrade would be possible.

Sure, will do.

Thanks,

Dan

___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users