The branch, master has been updated via f77bfed s4:drsuapi: try to behave more like windows for usn order (bug #9508) via 16aef75 s4:drsuapi: make use of LDB_TYPESAFE_QSORT() and pass getnc_state via 88833b0 s4:drsuapi: make sure we report the meta data from the cycle start (bug #9508) via 1f89d64 s4:drsuapi: check the source_dsa_invocation_id (bug #9508) via 91f7f2c s4:drsuapi: make sure we never return the same highwatermark twice in a replication cycle (bug #9508) via 7e511b5 s4:drsuapi: add drsuapi_DsReplicaHighWaterMark_cmp() via 02de5b1 s4:drsuapi: always use the current uptodateness_vector via 025c6d6 s4:drsuapi: avoid a ldb_dn_copy() and use talloc_move() instead via 30be17b s4:drsuapi: remove unused 'highest_usn' from drsuapi_getncchanges_state via 551bb2c s4:drsuapi: move struct drsuapi_getncchanges_state to the top of getncchanges.c via 2e9b064 s4:dsdb/drepl: update the source_dsa_obj/invocation_id in repsFrom via e7a26d0 s4:dsdb/common: use 01.01.1970 as last_sync_success for our entry in the uptodatevector via 81fa179 s4:dsdb/common: use LDB_SEQ_HIGHEST_SEQ for our entry in the uptodatevector via 5ecbc89 s4:dsdb/repl_meta_data: don't merge highwatermark and uptodatevector (bug #9508) via ad43bb6 s4:dsdb/repl_meta_data: also update the last_sync_success in replUpToDateVector via 634f8cf s4:dsdb/repl_meta_data: store the last results and timestamps in the repsFrom via a37f46a s4:dsdb/repl_meta_data: always treat the highwatermark as opaque (bug #9508) via 257ae54 s4:scripting/python: always treat the highwatermark as opaque (bug #9508) from e2e2d72 selftest/flapping: more samba4.rpc.samr.large-dc.two subtests are flakey
http://gitweb.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit f77bfed088b93f3ed0f00d0c172ad495c6c2b09b Author: Stefan Metzmacher <me...@samba.org> Date: Sat Dec 15 10:18:08 2012 +0100 s4:drsuapi: try to behave more like windows for usn order (bug #9508) We don't behave completely like a Windows server, but it's much more identical than before. The partition head is always the first object followed by the rest sorted by uSNChanged. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Autobuild-User(master): Stefan Metzmacher <me...@samba.org> Autobuild-Date(master): Tue Jan 1 21:09:42 CET 2013 on sn-devel-104 commit 16aef75c4f83c114206aa7637fedc9c2c2486877 Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 15:16:28 2012 +0100 s4:drsuapi: make use of LDB_TYPESAFE_QSORT() and pass getnc_state Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 88833b089a90e8f685d15b508f2e4615afb3a16f Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 14:59:20 2012 +0100 s4:drsuapi: make sure we report the meta data from the cycle start (bug #9508) We should build the final highwatermark and uptodatevector of a replication cycle at the start of the cycle. Before we search for the currently missing objects. Otherwise we risk that some objects get lost. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 1f89d641d09ef983f6a5055bb75099dc0ce57aa8 Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 13:40:33 2012 +0100 s4:drsuapi: check the source_dsa_invocation_id (bug #9508) The given highwatermark is only valid relative to the specified source_dsa_invocation_id. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 91f7f2c04fd00e281b0755a331ca632a4905e3b5 Author: Stefan Metzmacher <me...@samba.org> Date: Mon Dec 17 11:30:26 2012 +0100 s4:drsuapi: make sure we never return the same highwatermark twice in a replication cycle (bug #9508) If the highwatermark given by the client is not the one we expect, we need to start a new replication cycle. Otherwise the destination dsa skips objects and linked attribute values. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 7e511b58318cef1b325a8191685ee156a7fc0cb7 Author: Stefan Metzmacher <me...@samba.org> Date: Mon Dec 17 11:13:43 2012 +0100 s4:drsuapi: add drsuapi_DsReplicaHighWaterMark_cmp() Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 02de5b140cfe6ea31e0686e5f0ff726a22153020 Author: Stefan Metzmacher <me...@samba.org> Date: Mon Dec 17 16:34:25 2012 +0100 s4:drsuapi: always use the current uptodateness_vector Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 025c6d62f3c1b0f760aaacb7b3960135319031da Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 12:44:43 2012 +0100 s4:drsuapi: avoid a ldb_dn_copy() and use talloc_move() instead Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 30be17bc5d6b3cf2ee0aef6663af78b153b2ab9a Author: Stefan Metzmacher <me...@samba.org> Date: Mon Dec 17 13:48:01 2012 +0100 s4:drsuapi: remove unused 'highest_usn' from drsuapi_getncchanges_state Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 551bb2ccea6a1d82dbe0d4a21c19a8d8bd13ccbc Author: Stefan Metzmacher <me...@samba.org> Date: Mon Dec 17 14:08:56 2012 +0100 s4:drsuapi: move struct drsuapi_getncchanges_state to the top of getncchanges.c Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 2e9b06412b09163d4b851135ef509d73bb6d61fc Author: Stefan Metzmacher <me...@samba.org> Date: Wed Dec 19 17:31:28 2012 +0100 s4:dsdb/drepl: update the source_dsa_obj/invocation_id in repsFrom The highwatermark is relative to the source_dsa_invocation_id. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit e7a26d02413005294180a1d9cd4c90d4ac4d9733 Author: Stefan Metzmacher <me...@samba.org> Date: Wed Dec 19 17:33:13 2012 +0100 s4:dsdb/common: use 01.01.1970 as last_sync_success for our entry in the uptodatevector This matches a Windows 2008R2 and 2012 server. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 81fa179b155a62f2f652fbb1fc4978c9f6eb5462 Author: Stefan Metzmacher <me...@samba.org> Date: Wed Dec 19 12:47:43 2012 +0100 s4:dsdb/common: use LDB_SEQ_HIGHEST_SEQ for our entry in the uptodatevector We should use the global highestCommittedUSN, not the per partition value. This matches a Windows 2008R2 and 2012 server. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 5ecbc892b5226d3d31da2c62ae5261a8d8a73072 Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 14:46:23 2012 +0100 s4:dsdb/repl_meta_data: don't merge highwatermark and uptodatevector (bug #9508) We should not do any magic regarding the highwatermark we got from the source dsa. We need to treat it as opaque and not try to be smart and merge it into the uptodatevector. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit ad43bb6086a7dbf48b405d0372ae85d2244384d9 Author: Stefan Metzmacher <me...@samba.org> Date: Thu Dec 20 15:46:05 2012 +0100 s4:dsdb/repl_meta_data: also update the last_sync_success in replUpToDateVector This matches Windows 2008R2 and Windows 2012. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 634f8cf7c43bd60507d842d35cf46c0017e34dce Author: Stefan Metzmacher <me...@samba.org> Date: Wed Dec 19 17:29:04 2012 +0100 s4:dsdb/repl_meta_data: store the last results and timestamps in the repsFrom Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit a37f46a9a83a03157276485eb583649b36fb6ee1 Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 14:46:23 2012 +0100 s4:dsdb/repl_meta_data: always treat the highwatermark as opaque (bug #9508) Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 257ae5443631e645842cfcc9c1cedce6c41d5afa Author: Stefan Metzmacher <me...@samba.org> Date: Tue Dec 18 14:46:23 2012 +0100 s4:scripting/python: always treat the highwatermark as opaque (bug #9508) Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> ----------------------------------------------------------------------- Summary of changes: source4/dsdb/common/util.c | 11 +- source4/dsdb/repl/drepl_out_helpers.c | 4 + source4/dsdb/samdb/ldb_modules/repl_meta_data.c | 49 +----- source4/rpc_server/drsuapi/getncchanges.c | 229 +++++++++++++++++------ source4/scripting/devel/getncchanges | 2 +- source4/scripting/devel/repl_cleartext_pwd.py | 2 +- source4/scripting/python/samba/drs_utils.py | 2 +- 7 files changed, 185 insertions(+), 114 deletions(-) Changeset truncated at 500 lines: diff --git a/source4/dsdb/common/util.c b/source4/dsdb/common/util.c index 4543003..2b96bd4 100644 --- a/source4/dsdb/common/util.c +++ b/source4/dsdb/common/util.c @@ -3487,9 +3487,10 @@ int dsdb_load_udv_v2(struct ldb_context *samdb, struct ldb_dn *dn, TALLOC_CTX *m const struct ldb_val *ouv_value; unsigned int i; int ret; - uint64_t highest_usn; + uint64_t highest_usn = 0; const struct GUID *our_invocation_id; - struct timeval now = timeval_current(); + static const struct timeval tv1970; + NTTIME nt1970 = timeval_to_nttime(&tv1970); ret = ldb_search(samdb, mem_ctx, &r, dn, LDB_SCOPE_BASE, attrs, NULL); if (ret != LDB_SUCCESS) { @@ -3530,7 +3531,7 @@ int dsdb_load_udv_v2(struct ldb_context *samdb, struct ldb_dn *dn, TALLOC_CTX *m return ldb_operr(samdb); } - ret = dsdb_load_partition_usn(samdb, dn, &highest_usn, NULL); + ret = ldb_sequence_number(samdb, LDB_SEQ_HIGHEST_SEQ, &highest_usn); if (ret != LDB_SUCCESS) { /* nothing to add - this can happen after a vampire */ TYPESAFE_QSORT(*cursors, *count, drsuapi_DsReplicaCursor2_compare); @@ -3540,7 +3541,7 @@ int dsdb_load_udv_v2(struct ldb_context *samdb, struct ldb_dn *dn, TALLOC_CTX *m for (i=0; i<*count; i++) { if (GUID_equal(our_invocation_id, &(*cursors)[i].source_dsa_invocation_id)) { (*cursors)[i].highest_usn = highest_usn; - (*cursors)[i].last_sync_success = timeval_to_nttime(&now); + (*cursors)[i].last_sync_success = nt1970; TYPESAFE_QSORT(*cursors, *count, drsuapi_DsReplicaCursor2_compare); return LDB_SUCCESS; } @@ -3553,7 +3554,7 @@ int dsdb_load_udv_v2(struct ldb_context *samdb, struct ldb_dn *dn, TALLOC_CTX *m (*cursors)[*count].source_dsa_invocation_id = *our_invocation_id; (*cursors)[*count].highest_usn = highest_usn; - (*cursors)[*count].last_sync_success = timeval_to_nttime(&now); + (*cursors)[*count].last_sync_success = nt1970; (*count)++; TYPESAFE_QSORT(*cursors, *count, drsuapi_DsReplicaCursor2_compare); diff --git a/source4/dsdb/repl/drepl_out_helpers.c b/source4/dsdb/repl/drepl_out_helpers.c index 16825d4..57205a8 100644 --- a/source4/dsdb/repl/drepl_out_helpers.c +++ b/source4/dsdb/repl/drepl_out_helpers.c @@ -627,6 +627,8 @@ static void dreplsrv_op_pull_source_apply_changes_trigger(struct tevent_req *req first_object = ctr1->first_object; linked_attributes_count = 0; linked_attributes = NULL; + rf1.source_dsa_obj_guid = ctr1->source_dsa_guid; + rf1.source_dsa_invocation_id = ctr1->source_dsa_invocation_id; rf1.highwatermark = ctr1->new_highwatermark; uptodateness_vector = NULL; /* TODO: map it */ more_data = ctr1->more_data; @@ -637,6 +639,8 @@ static void dreplsrv_op_pull_source_apply_changes_trigger(struct tevent_req *req first_object = ctr6->first_object; linked_attributes_count = ctr6->linked_attributes_count; linked_attributes = ctr6->linked_attributes; + rf1.source_dsa_obj_guid = ctr6->source_dsa_guid; + rf1.source_dsa_invocation_id = ctr6->source_dsa_invocation_id; rf1.highwatermark = ctr6->new_highwatermark; uptodateness_vector = ctr6->uptodateness_vector; more_data = ctr6->more_data; diff --git a/source4/dsdb/samdb/ldb_modules/repl_meta_data.c b/source4/dsdb/samdb/ldb_modules/repl_meta_data.c index 3ac1e6a..30b2a42 100644 --- a/source4/dsdb/samdb/ldb_modules/repl_meta_data.c +++ b/source4/dsdb/samdb/ldb_modules/repl_meta_data.c @@ -4529,7 +4529,7 @@ static int replmd_replicated_uptodate_modify(struct replmd_replicated_request *a * * plus optional values from our old vector and the one from the source_dsa */ - nuv.ctr.ctr2.count = 1 + ouv.ctr.ctr2.count; + nuv.ctr.ctr2.count = ouv.ctr.ctr2.count; if (ruv) nuv.ctr.ctr2.count += ruv->count; nuv.ctr.ctr2.cursors = talloc_array(ar, struct drsuapi_DsReplicaCursor2, @@ -4563,12 +4563,8 @@ static int replmd_replicated_uptodate_modify(struct replmd_replicated_request *a found = true; - /* - * we update only the highest_usn and not the latest_sync_success time, - * because the last success stands for direct replication - */ if (ruv->cursors[i].highest_usn > nuv.ctr.ctr2.cursors[j].highest_usn) { - nuv.ctr.ctr2.cursors[j].highest_usn = ruv->cursors[i].highest_usn; + nuv.ctr.ctr2.cursors[j] = ruv->cursors[i]; } break; } @@ -4581,43 +4577,6 @@ static int replmd_replicated_uptodate_modify(struct replmd_replicated_request *a } /* - * merge in the current highwatermark for the source_dsa - */ - found = false; - for (j=0; j < ni; j++) { - if (!GUID_equal(&ar->objs->source_dsa->source_dsa_invocation_id, - &nuv.ctr.ctr2.cursors[j].source_dsa_invocation_id)) { - continue; - } - - found = true; - - /* - * here we update the highest_usn and last_sync_success time - * because we're directly replicating from the source_dsa - * - * and use the tmp_highest_usn because this is what we have just applied - * to our ldb - */ - nuv.ctr.ctr2.cursors[j].highest_usn = ar->objs->source_dsa->highwatermark.tmp_highest_usn; - nuv.ctr.ctr2.cursors[j].last_sync_success = now; - break; - } - if (!found) { - /* - * here we update the highest_usn and last_sync_success time - * because we're directly replicating from the source_dsa - * - * and use the tmp_highest_usn because this is what we have just applied - * to our ldb - */ - nuv.ctr.ctr2.cursors[ni].source_dsa_invocation_id= ar->objs->source_dsa->source_dsa_invocation_id; - nuv.ctr.ctr2.cursors[ni].highest_usn = ar->objs->source_dsa->highwatermark.tmp_highest_usn; - nuv.ctr.ctr2.cursors[ni].last_sync_success = now; - ni++; - } - - /* * finally correct the size of the cursors array */ nuv.ctr.ctr2.count = ni; @@ -4652,7 +4611,9 @@ static int replmd_replicated_uptodate_modify(struct replmd_replicated_request *a ZERO_STRUCT(nrf); nrf.version = 1; nrf.ctr.ctr1 = *ar->objs->source_dsa; - nrf.ctr.ctr1.highwatermark.highest_usn = nrf.ctr.ctr1.highwatermark.tmp_highest_usn; + nrf.ctr.ctr1.last_attempt = now; + nrf.ctr.ctr1.last_success = now; + nrf.ctr.ctr1.result_last_attempt = WERR_OK; /* * first see if we already have a repsFrom value for the current source dsa diff --git a/source4/rpc_server/drsuapi/getncchanges.c b/source4/rpc_server/drsuapi/getncchanges.c index 09406d6..c3fd000 100644 --- a/source4/rpc_server/drsuapi/getncchanges.c +++ b/source4/rpc_server/drsuapi/getncchanges.c @@ -37,6 +37,45 @@ #include "auth/session.h" #include "dsdb/common/util.h" +/* state of a partially completed getncchanges call */ +struct drsuapi_getncchanges_state { + struct GUID *guids; + uint32_t num_records; + uint32_t num_processed; + struct ldb_dn *ncRoot_dn; + bool is_schema_nc; + uint64_t min_usn; + uint64_t max_usn; + struct drsuapi_DsReplicaHighWaterMark last_hwm; + struct ldb_dn *last_dn; + struct drsuapi_DsReplicaHighWaterMark final_hwm; + struct drsuapi_DsReplicaCursor2CtrEx *final_udv; + struct drsuapi_DsReplicaLinkedAttribute *la_list; + uint32_t la_count; + bool la_sorted; + uint32_t la_idx; +}; + +static int drsuapi_DsReplicaHighWaterMark_cmp(const struct drsuapi_DsReplicaHighWaterMark *h1, + const struct drsuapi_DsReplicaHighWaterMark *h2) +{ + if (h1->highest_usn < h2->highest_usn) { + return -1; + } else if (h1->highest_usn > h2->highest_usn) { + return 1; + } else if (h1->tmp_highest_usn < h2->tmp_highest_usn) { + return -1; + } else if (h1->tmp_highest_usn > h2->tmp_highest_usn) { + return 1; + } else if (h1->reserved_usn < h2->reserved_usn) { + return -1; + } else if (h1->reserved_usn > h2->reserved_usn) { + return 1; + } + + return 0; +} + /* build a DsReplicaObjectIdentifier from a ldb msg */ @@ -647,8 +686,9 @@ struct drsuapi_changed_objects { /* sort the objects we send by tree order */ -static int site_res_cmp_parent_order(struct drsuapi_changed_objects *m1, - struct drsuapi_changed_objects *m2) +static int site_res_cmp_anc_order(struct drsuapi_changed_objects *m1, + struct drsuapi_changed_objects *m2, + struct drsuapi_getncchanges_state *getnc_state) { return ldb_dn_compare(m2->dn, m1->dn); } @@ -656,23 +696,31 @@ static int site_res_cmp_parent_order(struct drsuapi_changed_objects *m1, /* sort the objects we send first by uSNChanged */ -static int site_res_cmp_dn_usn_order(struct drsuapi_changed_objects *m1, - struct drsuapi_changed_objects *m2) +static int site_res_cmp_usn_order(struct drsuapi_changed_objects *m1, + struct drsuapi_changed_objects *m2, + struct drsuapi_getncchanges_state *getnc_state) { - unsigned usnchanged1, usnchanged2; - unsigned cn1, cn2; + int ret; - cn1 = ldb_dn_get_comp_num(m1->dn); - cn2 = ldb_dn_get_comp_num(m2->dn); - if (cn1 != cn2) { - return cn1 > cn2 ? 1 : -1; + ret = ldb_dn_compare(getnc_state->ncRoot_dn, m1->dn); + if (ret == 0) { + return -1; } - usnchanged1 = m1->usn; - usnchanged2 = m2->usn; - if (usnchanged1 == usnchanged2) { - return 0; + + ret = ldb_dn_compare(getnc_state->ncRoot_dn, m2->dn); + if (ret == 0) { + return 1; + } + + if (m1->usn == m2->usn) { + return ldb_dn_compare(m2->dn, m1->dn); + } + + if (m1->usn < m2->usn) { + return -1; } - return usnchanged1 > usnchanged2 ? 1 : -1; + + return 1; } @@ -1147,23 +1195,6 @@ static WERROR getncchanges_change_master(struct drsuapi_bind_state *b_state, return WERR_OK; } -/* state of a partially completed getncchanges call */ -struct drsuapi_getncchanges_state { - struct GUID *guids; - uint32_t num_records; - uint32_t num_processed; - struct ldb_dn *ncRoot_dn; - bool is_schema_nc; - uint64_t min_usn; - uint64_t highest_usn; - struct ldb_dn *last_dn; - struct drsuapi_DsReplicaLinkedAttribute *la_list; - uint32_t la_count; - bool la_sorted; - uint32_t la_idx; - struct drsuapi_DsReplicaCursorCtrEx *uptodateness_vector; -}; - /* see if this getncchanges request includes a request to reveal secret information */ @@ -1464,12 +1495,15 @@ WERROR dcesrv_drsuapi_DsGetNCChanges(struct dcesrv_call_state *dce_call, TALLOC_ time_t start = time(NULL); bool max_wait_reached = false; bool has_get_all_changes = false; + struct GUID invocation_id; DCESRV_PULL_HANDLE_WERR(h, r->in.bind_handle, DRSUAPI_BIND_HANDLE); b_state = h->data; sam_ctx = b_state->sam_ctx_system?b_state->sam_ctx_system:b_state->sam_ctx; + invocation_id = *(samdb_ntds_invocation_id(sam_ctx)); + *r->out.level_out = 6; /* TODO: linked attributes*/ r->out.ctr->ctr6.linked_attributes_count = 0; @@ -1587,6 +1621,18 @@ allowed: req10->uptodateness_vector = NULL; } + if (GUID_all_zero(&req10->source_dsa_invocation_id)) { + req10->source_dsa_invocation_id = invocation_id; + } + + if (!GUID_equal(&req10->source_dsa_invocation_id, &invocation_id)) { + /* + * The given highwatermark is only valid relative to the + * specified source_dsa_invocation_id. + */ + ZERO_STRUCT(req10->highwatermark); + } + getnc_state = b_state->getncchanges_state; /* see if a previous replication has been abandoned */ @@ -1602,6 +1648,20 @@ allowed: } } + if (getnc_state) { + ret = drsuapi_DsReplicaHighWaterMark_cmp(&getnc_state->last_hwm, + &req10->highwatermark); + if (ret != 0) { + DEBUG(0,(__location__ ": DsGetNCChanges 2nd replication " + "on DN %s %s highwatermark (last_dn %s)\n", + ldb_dn_get_linearized(getnc_state->ncRoot_dn), + (ret > 0) ? "older" : "newer", + ldb_dn_get_linearized(getnc_state->last_dn))); + talloc_free(getnc_state); + getnc_state = NULL; + } + } + if (getnc_state == NULL) { getnc_state = talloc_zero(b_state, struct drsuapi_getncchanges_state); if (getnc_state == NULL) { @@ -1692,6 +1752,18 @@ allowed: extra_filter = lpcfg_parm_string(dce_call->conn->dce_ctx->lp_ctx, NULL, "drs", "object filter"); getnc_state->min_usn = req10->highwatermark.highest_usn; + getnc_state->max_usn = getnc_state->min_usn; + + getnc_state->final_udv = talloc_zero(getnc_state, + struct drsuapi_DsReplicaCursor2CtrEx); + if (getnc_state->final_udv == NULL) { + return WERR_NOMEM; + } + werr = get_nc_changes_udv(sam_ctx, getnc_state->ncRoot_dn, + getnc_state->final_udv); + if (!W_ERROR_IS_OK(werr)) { + return werr; + } if (req10->extended_op == DRSUAPI_EXOP_NONE) { werr = getncchanges_collect_objects(b_state, mem_ctx, req10, @@ -1719,24 +1791,22 @@ allowed: changes[i].dn = search_res->msgs[i]->dn; changes[i].guid = samdb_result_guid(search_res->msgs[i], "objectGUID"); changes[i].usn = ldb_msg_find_attr_as_uint64(search_res->msgs[i], "uSNChanged", 0); + + if (changes[i].usn > getnc_state->max_usn) { + getnc_state->max_usn = changes[i].usn; + } } if (req10->replica_flags & DRSUAPI_DRS_GET_ANC) { - TYPESAFE_QSORT(changes, - getnc_state->num_records, - site_res_cmp_parent_order); + LDB_TYPESAFE_QSORT(changes, + getnc_state->num_records, + getnc_state, + site_res_cmp_anc_order); } else { - TYPESAFE_QSORT(changes, - getnc_state->num_records, - site_res_cmp_dn_usn_order); - } - - getnc_state->uptodateness_vector = talloc_steal(getnc_state, req10->uptodateness_vector); - if (getnc_state->uptodateness_vector) { - /* make sure its sorted */ - TYPESAFE_QSORT(getnc_state->uptodateness_vector->cursors, - getnc_state->uptodateness_vector->count, - drsuapi_DsReplicaCursor_compare); + LDB_TYPESAFE_QSORT(changes, + getnc_state->num_records, + getnc_state, + site_res_cmp_usn_order); } for (i=0; i < getnc_state->num_records; i++) { @@ -1748,10 +1818,21 @@ allowed: } } + getnc_state->final_hwm.tmp_highest_usn = getnc_state->max_usn; + getnc_state->final_hwm.reserved_usn = 0; + getnc_state->final_hwm.highest_usn = getnc_state->max_usn; + talloc_free(search_res); talloc_free(changes); } + if (req10->uptodateness_vector) { + /* make sure its sorted */ + TYPESAFE_QSORT(req10->uptodateness_vector->cursors, + req10->uptodateness_vector->count, + drsuapi_DsReplicaCursor_compare); + } + /* Prefix mapping */ schema = dsdb_get_schema(sam_ctx, mem_ctx); if (!schema) { @@ -1853,7 +1934,7 @@ allowed: schema, &session_key, getnc_state->min_usn, req10->replica_flags, req10->partial_attribute_set, - getnc_state->uptodateness_vector, + req10->uptodateness_vector, req10->extended_op, max_wait_reached); if (!W_ERROR_IS_OK(werr)) { @@ -1867,17 +1948,27 @@ allowed: msg, &getnc_state->la_list, &getnc_state->la_count, - getnc_state->uptodateness_vector); + req10->uptodateness_vector); if (!W_ERROR_IS_OK(werr)) { return werr; } uSN = ldb_msg_find_attr_as_int(msg, "uSNChanged", -1); + if (uSN > getnc_state->max_usn) { + /* + * Only report the max_usn we had at the start + * of the replication cycle. + * + * If this object has changed lately we better + * let the destination dsa refetch the change. + * This is better than the risk of loosing some + * objects or linked attributes. + */ + uSN = 0; + } if (uSN > r->out.ctr->ctr6.new_highwatermark.tmp_highest_usn) { r->out.ctr->ctr6.new_highwatermark.tmp_highest_usn = uSN; - } - if (uSN > getnc_state->highest_usn) { - getnc_state->highest_usn = uSN; + r->out.ctr->ctr6.new_highwatermark.reserved_usn = 0; } if (obj->meta_data_ctr == NULL) { @@ -1893,11 +1984,11 @@ allowed: *currentObject = obj; currentObject = &obj->next_object; - talloc_free(getnc_state->last_dn); - getnc_state->last_dn = ldb_dn_copy(getnc_state, msg->dn); - DEBUG(8,(__location__ ": replicating object %s\n", ldb_dn_get_linearized(msg->dn))); + talloc_free(getnc_state->last_dn); + getnc_state->last_dn = talloc_move(getnc_state, &msg->dn); + talloc_free(msg_res); talloc_free(msg_dn); } @@ -1977,17 +2068,31 @@ allowed: if (!r->out.ctr->ctr6.more_data) { talloc_steal(mem_ctx, getnc_state->la_list); - r->out.ctr->ctr6.uptodateness_vector = talloc(mem_ctx, struct drsuapi_DsReplicaCursor2CtrEx); - r->out.ctr->ctr6.new_highwatermark.highest_usn = r->out.ctr->ctr6.new_highwatermark.tmp_highest_usn; - - werr = get_nc_changes_udv(sam_ctx, getnc_state->ncRoot_dn, - r->out.ctr->ctr6.uptodateness_vector); - if (!W_ERROR_IS_OK(werr)) { - return werr; - } + r->out.ctr->ctr6.new_highwatermark = getnc_state->final_hwm; + r->out.ctr->ctr6.uptodateness_vector = talloc_move(mem_ctx, + &getnc_state->final_udv); talloc_free(getnc_state); b_state->getncchanges_state = NULL; + } else { + ret = drsuapi_DsReplicaHighWaterMark_cmp(&r->out.ctr->ctr6.old_highwatermark, + &r->out.ctr->ctr6.new_highwatermark); + if (ret == 0) { + /* + * We need to make sure that we never return the + * same highwatermark within the same replication + * cycle more than once. Otherwise we cannot detect + * when the client uses an unexptected highwatermark. + * + * This is a HACK which is needed because our + * object ordering is wrong and set tmp_highest_usn + * to a value that is higher than what we already -- Samba Shared Repository