RE: [ActiveDir] Replication Problems and Tombstoned Objects

[WATSON, BEN]

Title: RE: [ActiveDir] Replication Problems and Tombstoned Objects

Correction, 10 domain controllers in 9 sites.   From: WATSON, BEN Sent: Friday, September 22, 2006 10:58 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects   Basic info and troubleshooting I've done to gather symptom information...   We are running a single forest, single domain Windows 2000 environment (I know, I know, I'm in the process of getting this ugpraded to Win2k3 R2) with 9 domain controllers and 8 sites.  Three of the sites are hub sites, and each hub site has 2 spoke sites.  Our main hub site has 2 domain controllers, and all other remote sites have a single domain controller.   The replication issues are actually affecting an entire site, unfortunately our main hub site (the one with 2 domain controllers).  Oddly enough, it's not Domain Controller specific, the problem is actually site specific, and even more specifically, it's only affecting replication traffic OUTBOUND from the site.  Inbound replication traffic works fine as well as replication between the two domain controllers inside the site.  At first, I thought the domain controller that was acting as a Bridgehead for our site was having issues, so I forced the other domain controller in the site to be the preferred bridgehead server, deleted all the connection objects, and allowed the KCC to recreate the connection objects.  It did this properly.  I then attempted to force replication to take place, and the same symptoms still persisted even though it was a completely different domain controller attempting to perform the intersite replication.   Here are the results of performing a, "REPADMIN /REPLADMIN /BYSRC /BYDEST /SORT:DELTA" command. Appsig-AV and Appsig-AD are the two domain controllers in the problem site.  Appsig-AD was the original DC that began showing problems in the site, and Appsig-AV is the domain controller I switched over to test intersite replication using a different DC.   Replication Summary Start Time: 2006-09-22 21:59:43 Beginning data collection for replication summary, this may take awhile:   ............. Source DC           largest delta  fails/total  %%  error  APPSIG-MDOPC              14m:06s    0 /  18    0  APPSIG-LAOPC              10m:09s    0 /  12    0  APPSIG-TXOPC              09m:52s    0 /   3    0  APPSIG-OCOPC              09m:52s    0 /   3    0  APPSIG-OROPC              02m:48s    0 /   6    0  APPSIG-UTOPC              02m:46s    0 /   6    0  APPSIG-DCOPC              02m:08s    0 /   3    0  APPSIG-VAOPC              02m:08s    0 /   3    0  APPSIG-AV           (unknown)        4 /  15   26  (8442) The replication system encountered an internal error.  APPSIG-AD           (unknown)        4 /  15   26  (8442) The replication system encountered an internal error. Destination DC    largest delta    fails/total  %%  error  APPSIG-VAOPC              14m:12s    0 /   3    0  APPSIG-TXOPC              10m:12s    0 /   3    0  APPSIG-DCOPC              07m:42s    0 /   3    0  APPSIG-OCOPC              07m:07s    0 /   3    0  APPSIG-AD                 04m:33s    0 /   3    0  APPSIG-AV                 02m:50s    0 /  15    0  APPSIG-LAOPC        (unknown)        2 /  15   13  (8442) The replication system encountered an internal error.  APPSIG-UTOPC        (unknown)        2 /   9   22  (8442) The replication system encountered an internal error.  APPSIG-MDOPC        (unknown)        2 /  21    9  (8442) The replication system encountered an internal error.  APPSIG-OROPC        (unknown)        2 /   9   22  (8442) The replication system encountered an internal error.   Now on to event log errors and warnings in the Directory Service event log.   Oddly enough, the domain controlllers in the problem site show no real errors or warnings to speak of.  However, the domain controllers that have direct site connections to this site have plenty of errors when trying to replicate from these sites.  I'm showing 4 errors/warnings when replication is attempted.  Here are the errors/events after making the registry changes Steve suggested.   Event ID: 1173 - Category: Interneal Processing - Type: Warning Internal event: Exception e0010002 has occurred with parameters 8442 and 20b4 (Internal ID 3050bdc). Event ID: 1084 - Category: Replication - Type: Error Replication error: The directory replication agent (DRA) couldn't update object CN="InfowebAccessDEL:e9888888-616b-4944-bbe1-c8265cf4cc89",CN=Deleted Objects,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) on this system with changes which have been received from source server e928ad23-039d-4dbd-b214-f88b4ae54819._msdcs.appsig.com. An error occurred during the application of the changes to the directory database on this system. The error message is: The replication system encountered an internal error. The directory will try to update the object later on the next replication cycle. Synchronization of this server with the source is effectively blocked until the update problem is corrected. If this condition appears to be related to a resource shortage, please stop and restart this Windows Domain Controller. If this condition is an internal error, a database error, or an object relationship or constraint error, manual intervention will be required to correct the database and allow the update to proceed. It is valuable to note that the problem is caused by the fact that the change on the remote system cannot be applied locally. Manually updating the objects on the local system in not recommended. Instead, on the source system (which has the changes already), try to reverse or back out the change. Then, on the next replication cycle, observe whether the change can now be applied locally. The record data is the status code. Event ID: 1085 - Category: Replication - Type: Warning Replication warning: The directory replication agent (DRA) couldn't synchronize partition DC=appsig,DC=com with partition on directory server b04a1a6f-dae6-4795-bb91-9805f458c9d5._msdcs.appsig.com. The error was: The replication system encountered an internal error. Please verify that the address can be resolved with DNS, and that it is reachable via the transport. If this error persists, the KCC will reconfigure the links around this server. The record data is the status code. Event ID: 1061 - Category: Replication - Type: Warning Internal error: The directory replication agent (DRA) call returned error 8442. That's all of it.  If you need me to get any further information, let me know and I'll get it immediately. Thank you for your help! ~Ben     From: [EMAIL PROTECTED] on behalf of Steve Linehan Sent: Fri 9/22/2006 8:34 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects You could also turn up additional logging which would give more details as to what the internal error is.  I would suggest starting with the following: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics 1. Locate the "5 Replication Events" value under the above key. 2. On the Edit menu, click DWORD, type 4, and then click OK. 3. Locate the "9 Internal Processing" value under the same key. 4. On the Edit menu, click DWORD, type 1, and then click OK. After you do this post the full event text for the error and any additional replication or internal processing errors.  I would expect to get back an Exception value with parameters and an internal id.  These can be used to determine what is causing the problem.  To answer your original question the tombstoned object will only be removed once the tombstone lifetime is reached and garbage collection has run.  I would not recommend changing the tombstone lifetime to correct this as it is forest wide and can lead to more serious problems than you currently have.  We should be able to determine the cause of the internal error and correct it without taking such risky and drastic measures. Thanks, -Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Vinnie Cardona Sent: Friday, September 22, 2006 9:53 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects What event id are you seeing associate with this error? Vinnie Cardona Systems Administrator Ernest Health, Inc Information Technology Dept 505.798.6472 -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of WATSON, BEN Sent: Friday, September 22, 2006 6:18 PM To: ActiveDir@mail.activedir.org Subject: [ActiveDir] Replication Problems and Tombstoned Objects Our forest is currently experiencing some replication issues.  The common error we have been receiving has revolved around a single object. To summarize, how do you permanently delete Active Directory objects? More specifically, how do you remove an object that is already tombstoned?  Here is why I need to do this, here is the full error... ------- Replication error: The directory replication agent (DRA) couldn't update object CN=InfowebAccess,OU=InfowebGroups,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) on this system with changes which have been received from source server e928ad23-039d-4dbd-b214-f88b4ae54819._msdcs.appsig.com. An error occurred during the application of the changes to the directory database on this system.  The error message is:  The replication system encountered an internal error.  The directory will try to update the object later on the next replication cycle. Synchronization of this server with the source is effectively blocked until the update problem is corrected.  If this condition appears to be related to a resource shortage, please stop and restart this Windows Domain Controller.  If this condition is an internal error, a database error, or an object relationship or constraint error, manual intervention will be required to correct the database and allow the update to proceed.  It is valuable to note that the problem is caused by the fact that the change on the remote system cannot be applied locally. Manually updating the objects on the local system in not recommended. Instead, on the source system (which has the changes already), try to reverse or back out the change. Then, on the next replication cycle, observe whether the change can now be applied locally.  The record data is the status code. ------- After I deleted this object, I continue to get the same error, except it now references the deleted (tombstoned) object as a roadblock. ------- Replication error: The directory replication agent (DRA) couldn't update object CN="InfowebAccess DEL:e9888888-616b-4944-bbe1-c8265cf4cc89",CN=Deleted Objects,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) etc...  (same as error above) ------- What would be the proper method to permanently remove a tombstoned object?  If I'm following the error messages, then removing the object permanently should (hopefully) resolve the issues. Thanks, ~Ben List info   : http://www.activedir.org/List.aspx List FAQ    : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info   : http://www.activedir.org/List.aspx List FAQ    : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info   : http://www.activedir.org/List.aspx List FAQ    : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx