When you say "test site" is that an isolated test site as in isolated from the rest of the forest? Or ??
You through longhorn schema in there because (?)
This just sounds like there was more tweaking and adding and configuring then you've been led to believe. I get the strong sense that there's more to this than you're able to see at the moment.
It might be a good idea to open a case with MS support asap cause I think that more information is going to have to be collected.
Al
On 9/23/06, WATSON, BEN <[EMAIL PROTECTED]> wrote:
Sorry, I keep re-reading the e-mail and realize there was information I failed to give you.
>From what I understand of how the schema extension was added, it was added manually simply through adsiedit and by entering in the requested information as adsiedit asked for it during the objectclass creation and the information supplied was the information given to us by the developer of the application.
Thanks,
~Ben
________________________________
From: [EMAIL PROTECTED] on behalf of Steve Linehan
Sent: Sat 9/23/2006 1:13 AM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
Actually looking at this further you will probably find that the schemas are in sync, i.e. the groupofURLs object class is defined across all of the servers. I say that because the error you would have gotten if it did not exist on the target would have been either schema mismatch or ERROR_DS_OBJ_CLASS_NOT_DEFINED. So what I suspect is that groupofURLs is not defined properly or is being referenced incorrectly. Can you dump the schema entry for this class from one of your servers snd post it? Also if you have the LDIF file that was used to update the schema that includes the definition of this object class that would be great as well. What I do not understand is how you have an object defined this way as I would have expected us to block creation of the object if this class is not defined/referenced properly. Any information on how the schema was modified and how these objects were created would be helpful. The fix will likely be to remove the groupofurls objectclass from the object but you need to determine how you got to this point so that it does not occur again.
Thanks,
-Steve
________________________________________
From: [EMAIL PROTECTED] On Behalf Of Steve Linehan
Sent: Saturday, September 23, 2006 2:54 AM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
Ben,
It would appear that the schema was modified on the source servers but does not match on the destination servers. I am not aware of a default objectclass called groupofURLs. Is this something that you modified recently? Can you dump the definition of this objectclass from a schema on the source and verify that the schema on the target does not match? Can you also send me a repadmin /showreps /v from a source and target. It would appear that you have a schema modification gone bad. Can you also search and see if you have any other objects on the source DC that have that objectclass listed?
Thanks,
-Steve
________________________________
From: [EMAIL PROTECTED] On Behalf Of WATSON, BEN
Sent: Saturday, September 23, 2006 2:28 AM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
Hi Steve,
First off, thanks for all your help, you are always incredibly helpful.
HereâEUR(tm)s the output you requested from the source server.
dn:CN=InfowebAccess\0ADEL:e9888888-616b-4944-bbe1-c8265cf4cc89,CN=Deleted Objects,DC=appsig,DC=com
>objectClass: top
>objectClass: groupOfURLs
>objectClass: group
I should note though that this object NEVER replicated to other sites. So the only output I can give you is from the source DC. At least on the surface, this object seems to be the source of the replication issues.
Thanks again,
~Ben
From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf Of Steve Linehan
Sent: Friday, September 22, 2006 11:49 PM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
Can you dump the objectclass attribute on the deleted object mentioned in the error on one of the source servers and a destination server? The second error code in the internal error event log seems to indicate that the objectclass is being updated with a value that is not a subclass.
C:\tools\err\Err>err 20b4
# for hex 0x20b4 / decimal 8372 :
ERROR_DS_OBJ_CLASS_NOT_SUBCLASS winerror.h
# The specified class is not a subclass.
# 1 matches found for "20b4"
Thanks,
-Steve
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of WATSON, BEN
Sent: Saturday, September 23, 2006 1:03 AM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
Correction, 10 domain controllers in 9 sites.
From: WATSON, BEN
Sent: Friday, September 22, 2006 10:58 PM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
Basic info and troubleshooting I've done to gather symptom information...
We are running a single forest, single domain Windows 2000 environment (I know, I know, I'm in the process of getting this ugpraded to Win2k3 R2) with 9 domain controllers and 8 sites. Three of the sites are hub sites, and each hub site has 2 spoke sites. Our main hub site has 2 domain controllers, and all other remote sites have a single domain controller.
The replication issues are actually affecting an entire site, unfortunately our main hub site (the one with 2 domain controllers). Oddly enough, it's not Domain Controller specific, the problem is actually site specific, and even more specifically, it's only affecting replication traffic OUTBOUND from the site. Inbound replication traffic works fine as well as replication between the two domain controllers inside the site. At first, I thought the domain controller that was acting as a Bridgehead for our site was having issues, so I forced the other domain controller in the site to be the preferred bridgehead server, deleted all the connection objects, and allowed the KCC to recreate the connection objects. It did this properly. I then attempted to force replication to take place, and the same symptoms still persisted even though it was a completely different domain controller attempting to perform the intersite replication.
Here are the results of performing a, "REPADMIN /REPLADMIN /BYSRC /BYDEST /SORT:DELTA" command.
Appsig-AV and Appsig-AD are the two domain controllers in the problem site. Appsig-AD was the original DC that began showing problems in the site, and Appsig-AV is the domain controller I switched over to test intersite replication using a different DC.
Replication Summary Start Time: 2006-09-22 21:59:43
Beginning data collection for replication summary, this may take awhile:
.............
Source DC largest delta fails/total %% error
APPSIG-MDOPC 14m:06s 0 / 18 0
APPSIG-LAOPC 10m:09s 0 / 12 0
APPSIG-TXOPC 09m:52s 0 / 3 0
APPSIG-OCOPC 09m:52s 0 / 3 0
APPSIG-OROPC 02m:48s 0 / 6 0
APPSIG-UTOPC 02m:46s 0 / 6 0
APPSIG-DCOPC 02m:08s 0 / 3 0
APPSIG-VAOPC 02m:08s 0 / 3 0
APPSIG-AV (unknown) 4 / 15 26 (8442) The replication system encountered an internal error.
APPSIG-AD (unknown) 4 / 15 26 (8442) The replication system encountered an internal error.
Destination DC largest delta fails/total %% error
APPSIG-VAOPC 14m:12s 0 / 3 0
APPSIG-TXOPC 10m:12s 0 / 3 0
APPSIG-DCOPC 07m:42s 0 / 3 0
APPSIG-OCOPC 07m:07s 0 / 3 0
APPSIG-AD 04m:33s 0 / 3 0
APPSIG-AV 02m:50s 0 / 15 0
APPSIG-LAOPC (unknown) 2 / 15 13 (8442) The replication system encountered an internal error.
APPSIG-UTOPC (unknown) 2 / 9 22 (8442) The replication system encountered an internal error.
APPSIG-MDOPC (unknown) 2 / 21 9 (8442) The replication system encountered an internal error.
APPSIG-OROPC (unknown) 2 / 9 22 (8442) The replication system encountered an internal error.
Now on to event log errors and warnings in the Directory Service event log.
Oddly enough, the domain controlllers in the problem site show no real errors or warnings to speak of. However, the domain controllers that have direct site connections to this site have plenty of errors when trying to replicate from these sites. I'm showing 4 errors/warnings when replication is attempted. Here are the errors/events after making the registry changes Steve suggested.
Event ID: 1173 - Category: Interneal Processing - Type: Warning
Internal event: Exception e0010002 has occurred with parameters 8442 and 20b4 (Internal ID 3050bdc).
Event ID: 1084 - Category: Replication - Type: Error
Replication error: The directory replication agent (DRA) couldn't update object CN="InfowebAccessDEL:e9888888-616b-4944-bbe1-c8265cf4cc89",CN=Deleted Objects,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) on this system with changes which have been received from source server e928ad23-039d-4dbd-b214-f88b4ae54819._msdcs.appsig.com. An error occurred during the application of the changes to the directory database on this system.
The error message is:
The replication system encountered an internal error.
The directory will try to update the object later on the next replication cycle. Synchronization of this server with the source is effectively blocked until the update problem is corrected.
If this condition appears to be related to a resource shortage, please stop and restart this Windows Domain Controller.
If this condition is an internal error, a database error, or an object relationship or constraint error, manual intervention will be required to correct the database and allow the update to proceed. It is valuable to note that the problem is caused by the fact that the change on the remote system cannot be applied locally. Manually updating the objects on the local system in not recommended. Instead, on the source system (which has the changes already), try to reverse or back out the change. Then, on the next replication cycle, observe whether the change can now be applied locally.
The record data is the status code.
Event ID: 1085 - Category: Replication - Type: Warning
Replication warning: The directory replication agent (DRA) couldn't synchronize partition DC=appsig,DC=com with partition on directory server b04a1a6f-dae6-4795-bb91-9805f458c9d5._msdcs.appsig.com.
The error was:
The replication system encountered an internal error.
Please verify that the address can be resolved with DNS, and that it is reachable via the transport. If this error persists, the KCC will reconfigure the links around this server.
The record data is the status code.
Event ID: 1061 - Category: Replication - Type: Warning
Internal error: The directory replication agent (DRA) call returned error 8442.
That's all of it. If you need me to get any further information, let me know and I'll get it immediately.
Thank you for your help!
~Ben
________________________________
From: [EMAIL PROTECTED] on behalf of Steve Linehan
Sent: Fri 9/22/2006 8:34 PM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
You could also turn up additional logging which would give more details as to what the internal error is. I would suggest starting with the following:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics
1. Locate the "5 Replication Events" value under the above key.
2. On the Edit menu, click DWORD, type 4, and then click OK.
3. Locate the "9 Internal Processing" value under the same key.
4. On the Edit menu, click DWORD, type 1, and then click OK.
After you do this post the full event text for the error and any additional replication or internal processing errors. I would expect to get back an Exception value with parameters and an internal id. These can be used to determine what is causing the problem. To answer your original question the tombstoned object will only be removed once the tombstone lifetime is reached and garbage collection has run. I would not recommend changing the tombstone lifetime to correct this as it is forest wide and can lead to more serious problems than you currently have. We should be able to determine the cause of the internal error and correct it without taking such risky and drastic measures.
Thanks,
-Steve
-----Original Message-----
From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf Of Vinnie Cardona
Sent: Friday, September 22, 2006 9:53 PM
To: [email protected]
Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects
What event id are you seeing associate with this error?
Vinnie Cardona
Systems Administrator
Ernest Health, Inc
Information Technology Dept
505.798.6472
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] ] On Behalf Of WATSON, BEN
Sent: Friday, September 22, 2006 6:18 PM
To: [email protected]
Subject: [ActiveDir] Replication Problems and Tombstoned Objects
Our forest is currently experiencing some replication issues. The
common error we have been receiving has revolved around a single object.
To summarize, how do you permanently delete Active Directory objects?
More specifically, how do you remove an object that is already
tombstoned? Here is why I need to do this, here is the full error...
-------
Replication error: The directory replication agent (DRA) couldn't update
object CN=InfowebAccess,OU=InfowebGroups,DC=appsig,DC=com (GUID
e9888888-616b-4944-bbe1-c8265cf4cc89) on this system with changes which
have been received from source server
e928ad23-039d-4dbd-b214-f88b4ae54819._msdcs.appsig.com. An error
occurred during the application of the changes to the directory database
on this system.
The error message is:
The replication system encountered an internal error.
The directory will try to update the object later on the next
replication cycle. Synchronization of this server with the source is
effectively blocked until the update problem is corrected.
If this condition appears to be related to a resource shortage, please
stop and restart this Windows Domain Controller.
If this condition is an internal error, a database error, or an object
relationship or constraint error, manual intervention will be required
to correct the database and allow the update to proceed. It is valuable
to note that the problem is caused by the fact that the change on the
remote system cannot be applied locally. Manually updating the objects
on the local system in not recommended. Instead, on the source system
(which has the changes already), try to reverse or back out the change.
Then, on the next replication cycle, observe whether the change can now
be applied locally.
The record data is the status code.
-------
After I deleted this object, I continue to get the same error, except it
now references the deleted (tombstoned) object as a roadblock.
-------
Replication error: The directory replication agent (DRA) couldn't update
object CN="InfowebAccess
DEL:e9888888-616b-4944-bbe1-c8265cf4cc89",CN=Deleted
Objects,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89)
etc... (same as error above)
-------
What would be the proper method to permanently remove a tombstoned
object? If I'm following the error messages, then removing the object
permanently should (hopefully) resolve the issues.
Thanks,
~Ben
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ml/threads.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ml/threads.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ml/threads.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ml/threads.aspx
List info : http://www.activedir.org/List.aspx
List FAQ : http://www.activedir.org/ListFAQ.aspx
List archive: http://www.activedir.org/ml/threads.aspx
