Hello Flo, These are dirsrv logs provided by Customer around the time the idm02 is trying to become a replica
[06/Feb/2018:14:16:51.366390515 +0100] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [06/Feb/2018:14:16:52.913985123 +0100] - INFO - NSMMReplicationPlugin - repl5_tot_run - Beginning total update of replica "agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389)". [06/Feb/2018:14:17:03.114240744 +0100] - ERR - NSMMReplicationPlugin - repl5_tot_log_operation_failure - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): Received error -1 (Can't contact LDAP server): for total update operation [06/Feb/2018:14:17:03.125980541 +0100] - ERR - NSMMReplicationPlugin - release_replica - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): Unable to send endReplication extended operation (Can't contact LDAP server) [06/Feb/2018:14:17:03.127141614 +0100] - ERR - NSMMReplicationPlugin - repl5_tot_run - Total update failed for replica "agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389)", error (-11) [06/Feb/2018:14:17:03.166744429 +0100] - INFO - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): Replication bind with GSSAPI auth resumed [06/Feb/2018:14:17:04.351562786 +0100] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [06/Feb/2018:14:17:07.363975126 +0100] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [06/Feb/2018:14:17:10.387198653 +0100] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [06/Feb/2018:14:17:13.407233861 +0100] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToidm02.example.com <http://metoidm02.example.com/>" (idm02:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. And from idm02 06/Feb/2018:14:16:47.437206765 +0100] - NOTICE - ldbm_back_start - found 8010184k physical memory [06/Feb/2018:14:16:47.438565816 +0100] - NOTICE - ldbm_back_start - found 7207724k available [06/Feb/2018:14:16:47.439875019 +0100] - NOTICE - ldbm_back_start - cache autosizing: db cache: 320407k [06/Feb/2018:14:16:47.441028688 +0100] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (1 total): 524288k [06/Feb/2018:14:16:47.447478512 +0100] - NOTICE - ldbm_back_start - total cache size: 816125836 B; [06/Feb/2018:14:16:47.845657947 +0100] - INFO - slapd_daemon - slapd started. Listening on All Interfaces port 389 for LDAP requests [06/Feb/2018:14:16:47.847733508 +0100] - INFO - slapd_daemon - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests [06/Feb/2018:14:16:49.016885851 +0100] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToidm01.example.com <http://metoidm01.example.com/>" (idm01:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [06/Feb/2018:14:16:51.521915177 +0100] - ERR - ipa-topology-plugin - ipa_topo_be_state_changebackend userRoot is going offline; inactivate plugin [06/Feb/2018:14:16:51.523393396 +0100] - NOTICE - NSMMReplicationPlugin - multimaster_be_state_change - Replica dc=example,dc=com is going offline; disabling replication [06/Feb/2018:14:16:51.638698476 +0100] - INFO - dblayer_instance_start - Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [06/Feb/2018:14:16:53.546904089 +0100] - ERR - sasl_io_start_packet - SASL encrypted packet length exceeds maximum allowed limit (length=6765193, limit=2097152). Change the nsslapd-maxsasliosize attribute in cn=config to increase limit. [06/Feb/2018:14:16:53.814614772 +0100] - ERR - factory_destructor - ERROR bulk import abandoned [06/Feb/2018:14:16:53.851487751 +0100] - ERR - import_run_pass - import userRoot: Thread monitoring returned: -23 [06/Feb/2018:14:16:53.852810886 +0100] - ERR - import_main_offline - import userRoot: Aborting all Import threads... [06/Feb/2018:14:17:02.979086957 +0100] - ERR - import_main_offline - import userRoot: Import threads aborted. [06/Feb/2018:14:17:02.982961132 +0100] - INFO - import_main_offline - import userRoot: Closing files... [06/Feb/2018:14:17:03.092290649 +0100] - ERR - import_main_offline - import userRoot: Import failed. [06/Feb/2018:14:17:03.110305211 +0100] - ERR - process_bulk_import_op - NULL target sdn [06/Feb/2018:14:17:04.354545913 +0100] - ERR - NSMMReplicationPlugin - replica_replace_ruv_tombstone - Failed to update replication update vector for replica dc=example,dc=com: LDAP error Command on IPA server idm01.example.com <http://idm01.example.com/> [root@idm01 ~]# ldapsearch -xLLL -D "cn=directory manager" -W -b "dc=example,dc=com" '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' Enter LDAP Password: dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDN: krbprincipalname=ldap/idm02.example.com@EXAMPLE,cn=ser vices,cn=accounts,dc=example,dc=com nsDS5ReplicaId: 4 nsDS5ReplicaName: abd8ec06-40d511e5-8b849572-73def7f6 nsDS5ReplicaRoot: dc=example,dc=com nsDS5ReplicaType: 3 nsState:: BAAAAAAAAACIq3paAAAAAAAAAAAAAAAAmQAAAAAAAAADAAAAAAAAAA== nsds5ReplicaLegacyConsumer: off nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc, dc=example,dc=com nsds5replicabinddngroupcheckinterval: 60 objectClass: nsds5replica objectClass: top objectClass: extensibleobject nsds50ruv: {replicageneration} 5a70773d000000040000 nsds50ruv: {replica 4 ldap://idm01.example.com:389} 5a70779c000000040000 5a7aa c21000200040000 nsds50ruv: {replica 11 ldap://idm02.example.com:389} nsds5agmtmaxcsn: dc=example,dc=com;meToidm02.example.com;idm02.example.com <http://metoidm02.example.com%3Bidm02.example.com/>; 389;unavailable nsruvReplicaLastModified: {replica 4 ldap://idm01.example.com:389} 5a7aab88 nsruvReplicaLastModified: {replica 11 ldap://idm02.example.com:389} 00000000 nsds5ReplicaChangeCount: 1460 nsds5replicareapactive: 0 # ipa-replica-manage list-ruv Directory Manager password: Replica Update Vectors: idm01.example.com <http://idm01.example.com/>:389: 4 idm02.example.com <http://idm02.example.com/>:389: 11 Certificate Server Replica Update Vectors: No CS-RUVs found. On 03/13/2018 12:53 PM, Florence Blanc-Renaud wrote: > On 03/12/2018 06:09 PM, Amit wrote: >> Hello Flo, PFA replica-install log. > Hi, sorry if I was not clear, but I meant 389-ds access logs, located > in /var/log/dirsrv/slapd-DOMxxx/access. The ones from the master and > the soon-to-be-replica may provide more information. The customer may > also try ipa-replica-install with the -d option, which will add debug > information to the ipareplica-install.log file. Flo >> Thanks On 03/12/2018 01:59 PM, Florence Blanc-Renaud wrote: >>> On 03/10/2018 12:07 PM, Amit via FreeIPA-devel wrote: >>>> Ping!! On 03/09/2018 02:08 PM, Amit wrote: >>>>> Hello, Any thoughts would be helpful. Thanks On 03/07/2018 02:57 >>>>> PM, Amit wrote: >>>>>> Hello, This is scenario in customer env. Customer is using fresh >>>>>> machine to install replica. * IPA-Server * # >>>>>> ipa-server-install --no-ntp //Success *IPA >>>>>> Replica* # ipa-replica-install --principal admin >>>>>> --admin-password <secret> --setup-ca DEBUG Traceback (most >>>>>> recent call last): File >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", >>>>>> line 504, in start_creation run_step(full_msg, method) >>>>>> File >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", >>>>>> line 494, in run_step method() File >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", >>>>>> line 439, in __setup_replica cacert=self.ca_file) File >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>> line 1666, in setup_promote_replication raise >>>>>> RuntimeError("Failed to start replication") RuntimeError: >>>>>> Failed to start replication 2018-02-06T06:56:48Z DEBUG >>>>>> [error] RuntimeError: Failed to start replication >>>>>> 2018-02-06T06:56:48Z DEBUG Destroyed connection >>>>>> context.ldap2_113870544 2018-02-06T06:56:48Z DEBUG Backing >>>>>> up system configuration file '/etc/ipa/default.conf' >>>>>> 2018-02-06T06:56:48Z DEBUG Saving Index File to >>>>>> '/var/lib/ipa/sysrestore/sysrestore.index' >>>>>> 2018-02-06T06:56:48Z DEBUG File >>>>>> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line >>>>>> 172, in execute return_value = self.run() While I cannot >>>>>> repro in my local lab >>> Hi Amit, without any logs it is difficult to tell what could go >>> wrong. The part of code that is failing is doing 2 tasks: - starts >>> the replication by performing a LDAP modification on the replication >>> agreement (dn: >>> cn=meTo$master,cn=replica,cn=dc\3Ddomain\2Cdc\3Dcom,cn=mapping >>> tree,cn=config) in order to set the attribute >>> nsds5BeginReplicaRefresh=start - checks the replication status by >>> reading the replication agreement status (attributes >>> nsds5BeginReplicaRefresh, nsds5replicaUpdateInProgress, >>> nsds5ReplicaLastInitStatus, nsds5ReplicaLastInitStart and >>> nsds5ReplicaLastInitEnd). So if you have 389-ds access logs, you can >>> start by checking if the mod was successful. Then check the >>> replication status. Flo >> -- Thanks Amit Kumar !!If you stumble, get back up. What happened >> yesterday, no longer matters. Today is another day to move closer to >> your GOAL!! -- Thanks Amit Kumar !!If you stumble, get back up. What happened yesterday, no longer matters. Today is another day to move closer to your GOAL!!
_______________________________________________ FreeIPA-devel mailing list -- freeipa-devel@lists.fedorahosted.org To unsubscribe send an email to freeipa-devel-le...@lists.fedorahosted.org
