Hi Dmitri
On Wed, Apr 1, 2015 at 2:23 PM, Dmitri Pal <d...@redhat.com> wrote: > On 04/01/2015 04:14 AM, Traiano Welcome wrote: >> >> Hi Martin >> >> Thanks for the response. Check results inline: >> >> >> On Wed, Apr 1, 2015 at 10:37 AM, Martin Babinsky <mbabi...@redhat.com> >> wrote: >>> >>> On 04/01/2015 09:20 AM, Traiano Welcome wrote: >>>> >>>> Some information from the dirsrv error log (sanitized: XYZ = realm): >>>> >>>> [01/Apr/2015:11:01:49 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 >>>> starting up >>>> [01/Apr/2015:11:01:49 +0300] schema-compat-plugin - warning: no >>>> entries set up under cn=computers, cn=compat,dc=idm,dc=local >>>> [01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password >>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>> should be added before the CoS Definition. >>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> cleanAllRUV task found, resuming the cleaning of rid(6)... >>>> [01/Apr/2015:11:01:49 +0300] - Skipping CoS Definition cn=Password >>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>> should be added before the CoS Definition. >>>> [01/Apr/2015:11:01:49 +0300] - slapd started. Listening on All >>>> Interfaces port 389 for LDAP requests >>>> [01/Apr/2015:11:01:49 +0300] - Listening on All Interfaces port 636 >>>> for LDAPS requests >>>> [01/Apr/2015:11:01:49 +0300] - Listening on >>>> /var/run/slapd-IDM-LOCAL.socket for LDAPI requests >>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:11:01:49 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>> Minor code may provide more information (No Kerberos credentials >>>> available)) >>>> [01/Apr/2015:11:01:49 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:11:01:49 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:11:01:49 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>> Minor code may provide more information (No Kerberos credentials >>>> available)) >>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - signaling operation >>>> threads >>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - waiting for 27 >>>> threads to terminate >>>> [01/Apr/2015:11:01:50 +0300] - slapd shutting down - closing down >>>> internal subsystems and plugins >>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Cleaning rid (6)... >>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Waiting to process all the updates from the deleted replica... >>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Waiting for all the replicas to be online... >>>> [01/Apr/2015:11:01:58 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Server shutting down. Process will resume at server startup >>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed >>>> out) >>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -1 (Can't contact LDAP server) >>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact >>>> LDAP server) () >>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>> Minor code may provide more information (No Kerberos credentials >>>> available)) >>>> errors >>>> [01/Apr/2015:11:02:09 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:11:02:09 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:11:02:09 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication >>>> bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): >>>> generic failure: GSSAPI Error: Unspecified GSS failure. Minor code >>>> may provide more information (No Kerberos credentials available)) >>>> [01/Apr/2015:11:02:09 +0300] - Waiting for 4 database threads to stop >>>> [01/Apr/2015:11:02:10 +0300] - All database threads now stopped >>>> [01/Apr/2015:11:02:10 +0300] - slapd stopped. >>>> [01/Apr/2015:10:15:39 +0300] - 389-Directory/1.3.1.6 B2014.160.2139 >>>> starting up >>>> [01/Apr/2015:10:15:39 +0300] schema-compat-plugin - warning: no >>>> entries set up under cn=computers, cn=compat,dc=idm,dc=local >>>> [01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password >>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>> should be added before the CoS Definition. >>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> cleanAllRUV task found, resuming the cleaning of rid(6)... >>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:10:15:39 +0300] - Skipping CoS Definition cn=Password >>>> Policy,cn=accounts,dc=idm,dc=local--no CoS Templates found, which >>>> should be added before the CoS Definition. >>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 2 (No such file or directory) >>>> [01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time >>>> skew (-2771 secs). Current seqnum=3 >>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meTokwtard-idm-slve.idm.local" (kwtard-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>> Minor code may provide more information (No Kerberos credentials >>>> available)) >>>> [01/Apr/2015:10:15:39 +0300] set_krb5_creds - Could not get initial >>>> credentials for principal [ldap/kwtpr-idm-mstr@] in keytab >>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not found) >>>> [01/Apr/2015:10:15:39 +0300] csngen_new_csn - Warning: too much time >>>> skew (-2770 secs). Current seqnum=1 >>>> [01/Apr/2015:10:15:39 +0300] - slapd started. Listening on All >>>> Interfaces port 389 for LDAP requests >>>> [01/Apr/2015:10:15:39 +0300] - Listening on All Interfaces port 636 >>>> for LDAPS requests >>>> [01/Apr/2015:10:15:39 +0300] - Listening on >>>> /var/run/slapd-IDM-LOCAL.socket for LDAPI requests >>>> [01/Apr/2015:10:15:39 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:10:15:39 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:10:15:39 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meToindpr-idm-slve.idm.local" (indpr-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>> Minor code may provide more information (No Kerberos credentials >>>> available)) >>>> [01/Apr/2015:10:15:40 +0300] csngen_new_csn - Warning: too much time >>>> skew (-2771 secs). Current seqnum=1 >>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - signaling operation >>>> threads >>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - waiting for 28 >>>> threads to terminate >>>> [01/Apr/2015:10:15:41 +0300] - slapd shutting down - closing down >>>> internal subsystems and plugins >>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Cleaning rid (6)... >>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Waiting to process all the updates from the deleted replica... >>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Waiting for all the replicas to be online... >>>> [01/Apr/2015:10:15:48 +0300] NSMMReplicationPlugin - CleanAllRUV Task: >>>> Server shutting down. Process will resume at server startup >>>> [01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -1 (Can't contact LDAP server) ((null)) errno 110 (Connection timed >>>> out) >>>> [01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -1 (Can't contact LDAP server) >>>> [01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meTokwtospr-idm-slve.idm.local" (kwtospr-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact >>>> LDAP server) () >>>> [01/Apr/2015:10:15:58 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:10:15:58 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:10:15:58 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meTokwtpr-idm-slve.idm.local" (kwtpr-idm-slve:389): >>>> Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) >>>> (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. >>>> Minor code may provide more information (No Kerberos credentials >>>> available)) >>>> [01/Apr/2015:10:15:59 +0300] slapd_ldap_sasl_interactive_bind - Error: >>>> could not perform interactive bind for id [] mech [GSSAPI]: LDAP error >>>> -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified >>>> GSS failure. Minor code may provide more information (No Kerberos >>>> credentials available)) errno 0 (Success) >>>> [01/Apr/2015:10:15:59 +0300] slapi_ldap_bind - Error: could not >>>> perform interactive bind for id [] authentication mechanism [GSSAPI]: >>>> error -2 (Local error) >>>> [01/Apr/2015:10:15:59 +0300] NSMMReplicationPlugin - >>>> agmt="cn=meToukpr-idm-slve.idm.local" (ukpr-idm-slve:389): Replication >>>> bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): >>>> generic failure: GSSAPI Error: Unspecified GSS failure. Minor code >>>> may provide more information (No Kerberos credentials available)) >>>> [01/Apr/2015:10:15:59 +0300] - Waiting for 4 database threads to stop >>>> [01/Apr/2015:10:16:00 +0300] - All database threads now stopped >>>> [01/Apr/2015:10:16:00 +0300] - slapd stopped. >>>> >>>> On Wed, Apr 1, 2015 at 9:56 AM, Traiano Welcome <trai...@gmail.com> >>>> wrote: >>>>> >>>>> Hi List >>>>> >>>>> I've just tried to restart my IPA services after recently adding a new >>>>> replica (0 configuration changes on the IPA server otherwise!), but >>>>> ipactl fails when starting up named: >>>>> >>>>> --- >>>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# ipactl start >>>>> Starting Directory Service >>>>> Starting krb5kdc Service >>>>> Starting kadmin Service >>>>> Starting named Service >>>>> Job for named.service failed. See 'systemctl status named.service' and >>>>> 'journalctl -xn' for details. >>>>> Failed to start named Service >>>>> Shutting down >>>>> Aborting ipactl >>>>> --- >>>>> >>>>> I then manual start named service and try again, but then smb service >>>>> fails: >>>>> >>>>> --- >>>>> [root@lolpr-xyz-mstr ~]# ipactl start >>>>> Existing service file detected! >>>>> Assuming stale, cleaning and proceeding >>>>> Starting Directory Service >>>>> Starting krb5kdc Service >>>>> Starting kadmin Service >>>>> Starting named Service >>>>> Starting ipa_memcached Service >>>>> Starting httpd Service >>>>> Starting pki-tomcatd Service >>>>> Starting smb Service >>>>> Job for smb.service failed. See 'systemctl status smb.service' and >>>>> 'journalctl -xn' for details. >>>>> Failed to start smb Service >>>>> Shutting down >>>>> Aborting ipactl >>>>> --- >>>>> >>>>> systemctl status shows the following output for smb.service: >>>>> >>>>> --- >>>>> [root@lolpr-xyz-mstr ~]# systemctl -l status smb.service >>>>> smb.service - Samba SMB Daemon >>>>> Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled) >>>>> Active: failed (Result: exit-code) since Wed 2015-04-01 09:21:10 >>>>> AST; 1min 14s ago >>>>> Process: 4662 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited, >>>>> status=1/FAILURE) >>>>> Main PID: 4662 (code=exited, status=1/FAILURE) >>>>> Status: "Starting process..." >>>>> CGroup: /system.slice/smb.service >>>>> >>>>> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI client step >>>>> 1 >>>>> Apr 01 09:21:09 lolpr-xyz-mstr.xyz.local smbd[4662]: GSSAPI Error: >>>>> Unspecified GSS failure. Minor code may provide more information >>>>> (Server ldap/lolpr-xyz-mstr@XYZ.LOCAL not found in Kerberos database) >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01 >>>>> 09:21:10.211028, 0] ipa_sam.c:4440(pdb_init_ipasam) >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: Failed to get base >>>>> DN. >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: [2015/04/01 >>>>> 09:21:10.211210, 0] >>>>> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name) >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local smbd[4662]: pdb backend >>>>> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not correctly >>>>> init (error was NT_STATUS_UNSUCCESSFUL) >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: main >>>>> process exited, code=exited, status=1/FAILURE >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start >>>>> Samba SMB Daemon. >>>>> Apr 01 09:21:10 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service >>>>> entered failed state. >>>>> Apr 01 09:21:12 lolpr-xyz-mstr.xyz.local systemd[1]: Stopped Samba SMB >>>>> Daemon. >>>>> --- >>>>> >>>>> >>>>> I manually try to start the smb service as follows, but can't (Of >>>>> course the directory service is not up, so there's a little catch22 >>>>> there and this many not mean much): >>>>> >>>>> >>>>> --- >>>>> >>>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# systemctl status smb.service >>>>> smb.service - Samba SMB Daemon >>>>> Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled) >>>>> Active: failed (Result: exit-code) since Wed 2015-04-01 09:50:38 >>>>> AST; >>>>> 57s ago >>>>> Process: 8089 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited, >>>>> status=1/FAILURE) >>>>> Main PID: 8089 (code=exited, status=1/FAILURE) >>>>> Status: "Starting process..." >>>>> >>>>> Apr 01 09:50:36 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error: >>>>> code=-1765328228, message=Cannot contact any KDC for realm 'XYZ.LOCAL' >>>>> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01 >>>>> 09:50:37.573772, 0] ipa_sam.c:4128(bind_callback_cleanup) >>>>> Apr 01 09:50:37 lolpr-xyz-mstr.xyz.local smbd[8089]: kerberos error: >>>>> code=-1765328228, message=Cannot contact any KDC for realm 'XYZ.LOCAL' >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01 >>>>> 09:50:38.574722, 0] ipa_sam.c:4440(pdb_init_ipasam) >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: Failed to get base >>>>> DN. >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: [2015/04/01 >>>>> 09:50:38.574903, 0] >>>>> ../source3/passdb/pdb_interface.c:178(make_pdb_method_name) >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local smbd[8089]: pdb backend >>>>> ipasam:ldapi://%2fvar%2frun%2fslapd-XYZ-LOCAL.socket did not correctly >>>>> init (error was NT_STATUS_UNSUCCESSFUL) >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: smb.service: main >>>>> process exited, code=exited, status=1/FAILURE >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start >>>>> Samba SMB Daemon. >>>>> Apr 01 09:50:38 lolpr-xyz-mstr.xyz.local systemd[1]: Unit smb.service >>>>> entered failed state. >>>>> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# >>>>> >>>>> --- >>>>> >>>>> Please could someone advise me on how to drill deeper into debugging >>>>> this issue to get ipactl to start ? >>>>> >>>>> NOTES: >>>>> >>>>> - This server is successfully in a Trust relationship with >>>>> ActiveDirectory. >>>>> - There are a number of replicas established which have been working >>>>> fine til this morning >>>>> - Another replica was added around the time of the failure using the >>>>> same steps as usual (not sure how this could be related) >>>>> >>>>> >>>>> Many thanks in advance, >>>>> Traiano >>>> >>>> >>> Hi Traiano, >>> >>> it seems like there is some problem with Kerberos keytab for DS service. >>> >>> Take a look at this guide: >>> >>> http://www.freeipa.org/page/Troubleshooting#Service_does_not_start >>> >>> and check whether there is something wrong with DS keytab and that the >>> service principal is set up correctly. >>> >> >> >> Walking through this pedantically: >> >> Service does not start: >> >> 1) See service log of the respective service for the exact error text. >> For example, the Directory Server stores the log in >> /var/log/dirsrv/slapd-REALM-NAME/errors >> >> check >> >> 2) Make sure that the server the service is running on has a fully >> qualified domain name >> >> --- >> [root@lolpr-xyz-mstr ~]# hostname >> lolpr-xyz-mstr.xyz.local >> [root@lolpr-xyz-mstr ~]# host `hostname` >> lolpr-xyz-mstr.xyz.local has address 172.16.100.68 >> [root@lolpr-xyz-mstr ~]# host 172.16.100.68 >> 68.100.16.172.in-addr.arpa domain name pointer lolpr-xyz-mstr.xyz.local. >> [root@lolpr-xyz-mstr ~]# >> --- >> >> 3) See what keys are in the keytab used for authentication of the service, >> e.g.: >> # klist -kt /etc/dirsrv/ds.keytab >> >> >> --- >> [root@lolpr-xyz-mstr slapd-XYZ-LOCAL]# klist -kt /etc/dirsrv/ds.keytab >> Keytab name: FILE:/etc/dirsrv/ds.keytab >> KVNO Timestamp Principal >> ---- ------------------- >> ------------------------------------------------------ >> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local@XYZ.LOCAL >> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local@XYZ.LOCAL >> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local@XYZ.LOCAL >> 2 11/06/2014 13:13:06 ldap/lolpr-xyz-mstr.xyz.local@XYZ.LOCAL >> --- >> >> 4) Make sure that the stored principals match the system FQDN system name >> >> check: >> >> --- >> [root@lolpr-xyz-mstr ~]# host lolpr-xyz-mstr.xyz.local >> lolpr-xyz-mstr.xyz.local has address 172.16.100.68 >> [root@lolpr-xyz-mstr ~]# >> --- >> >> 5) Make sure that the version of the keys (KVNO) stored in the keytab >> and in the FreeIPA server match: >> $ kvno ldap/ipa.example....@example.com >> >> >> check ... This is unusual: >> >> --- >> [root@lolpr-xyz-mstr ~]# kvno ldap/lolpr-xyz-mstr.xyz.local@XYZ.LOCAL >> kvno: Credentials cache keyring 'persistent:0:0' not found while >> getting client principal name >> --- >> >> Now, when I look at my krb5.conf, I see the file has had a recent >> change ... yet, I'm sure this file was never edited: Does the >> krb5.conf below look correct for a standard IPA primary server?: >> >> --- >> [root@lolpr-xyz-mstr ~]# ls -l /etc/krb5.conf >> -rw-r--r-- 1 root root 811 Apr 1 11:01 /etc/krb5.conf >> --- >> >> >> --- >> [root@lolpr-xyz-mstr ~]# cat /etc/krb5.conf >> includedir /var/lib/sss/pubconf/krb5.include.d/ >> >> [logging] >> default = FILE:/var/log/krb5libs.log >> kdc = FILE:/var/log/krb5kdc.log >> admin_server = FILE:/var/log/kadmind.log >> >> [libdefaults] >> default_realm = XYZ.LOCAL >> dns_lookup_realm = false >> dns_lookup_kdc = true >> rdns = false >> ticket_lifetime = 24h >> forwardable = yes >> default_ccache_name = KEYRING:persistent:%{uid} >> >> [realms] >> XYZ.LOCAL = { >> kdc = lolpr-xyz-mstr.xyz.local:88 >> master_kdc = lolpr-xyz-mstr.xyz.local:88 >> admin_server = lolpr-xyz-mstr.xyz.local:749 >> default_domain = xyz.local >> pkinit_anchors = FILE:/etc/ipa/ca.crt >> auth_to_local = >> RULE:[1:$1@$0](^.*@WINDOM.LOCAL$)s/@WINDOM.LOCAL/@windom.local/ >> auth_to_local = DEFAULT >> } >> >> [domain_realm] >> .xyz.local = XYZ.LOCAL >> xyz.local = XYZ.LOCAL >> >> [dbmodules] >> XYZ.LOCAL = { >> db_library = ipadb.so >> } >> --- > > > > I do not see any glaring problems in this file. > This seems to be 4.1 bits. IPA 3.3 on CentOS release 7.0.1406 (Core) > There is definitely something wrong with the Kerberos part though. > And the fact that you can't access credential cache is pointing to a > problem. Yes. Trying to start the krb5kdc service manually: --- job for krb5kdc.service failed. See 'systemctl status krb5kdc.service' and 'journalctl -xn' for details. --- Checking the krb5kdc.service status: --- [root@lolpr-xyz-mstr log]# systemctl status krb5kdc.service krb5kdc.service - Kerberos 5 KDC Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled) Active: failed (Result: exit-code) since Wed 2015-04-01 14:42:15 AST; 7s ago Process: 3884 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=1/FAILURE) Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Starting Kerberos 5 KDC... Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local krb5kdc[3884]: krb5kdc: cannot initialize realm XYZ.LOCAL - see log file for details Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: krb5kdc.service: control process exited, code=exited status=1 Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Failed to start Kerberos 5 KDC. Apr 01 14:42:15 lolpr-xyz-mstr.xyz.local systemd[1]: Unit krb5kdc.service entered failed state. --- Checking the logs: --- [root@lolpr-xyz-mstr log]# cat krb5kdc.log krb5kdc: Server error - while fetching master key K/M for realm XYZ.LOCAL --- > Do you see any selinux denials? Selinux has been disabled for months. I see this is still so in selinux conf: SELINUX=disabled > If the file was touched may be it was touched by recent update or > installation of some other package on the system. > The update/install might have set wrong context on the cred cache causing > problems like this. I've been careful to disable all external repos on the system since installation, so I'm only using packages on the original installation iso. It's a hermetically sealed system from the package point of view: [root@lolpr-xyz-mstr yum.repos.d]# ls -l total 4 -rw-r--r--. 1 root root 133 Nov 5 19:06 CentOS-Local.repo [root@lolpr-xyz-mstr yum.repos.d]# [root@lolpr-xyz-mstr yum.repos.d]# [root@lolpr-xyz-mstr yum.repos.d]# cat CentOS-Local.repo [LocalRepo] name=Local Repository baseurl=file:///repo enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 [root@lolpr-xyz-mstr yum.repos.d]# > > Anything interesting in the KDC log? > This looks like a clue: krb5kdc: Server error - while fetching master key K/M for realm XYZ.LOCAL ... But I'm not sure how to interpret this usefully ... > >> >> 6) Make sure that there are no DNS Issues and both forward and reverse >> DNS records of the are OK and match the system name and the stored >> principal keys >> >> check. DNS works. >> >> 7) Make sure that the system time difference on the host and FreeIPA >> server is not greater than 5 minutes >> >> They're one and the same in this case. >> >>> -- >>> Martin^3 Babinsky >> >> Thanks, >> Traiano >> > > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager IdM portfolio > Red Hat, Inc. > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project