[SCM] Samba Shared Repository - branch master updated

2018-10-30 Thread Douglas Bagnall
The branch, master has been updated
   via  0c91024 netcmd: Add 'samba-tool group stats' command
   via  ca570bd netcmd: Include num-members in 'samba-tool group list 
--verbose'
   via  a9a65ad selftest: Specify different DB backends for restored 
testenvs
   via  5a57378 netcmd: Add backend-store option to domain backup/rename 
cmds
   via  4a8a0ab traffic_replay: logger was ignoring smb.conf log-level
   via  4943473 traffic_replay: Change print() to use logger()
   via  b40daca traffic_replay: Generate users faster by writing to local DB
  from  aeef8b4 dsdb group audit tests: log_membership_changes extra tests

https://git.samba.org/?p=samba.git;a=shortlog;h=master


- Log -
commit 0c910245fca70948a33eda99c9bc198d8b34675f
Author: Tim Beale 
Date:   Thu Oct 18 17:08:32 2018 +1300

netcmd: Add 'samba-tool group stats' command

With large domains it's hard to get an idea of how many groups there
are, and how many users are in each group, on average. However, this
could have a big impact on whether a problem can be reproduced or not.

This patch dumps out some summary information so that you can get a
quick idea of how big the groups are.

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

Autobuild-User(master): Douglas Bagnall 
Autobuild-Date(master): Wed Oct 31 03:40:41 CET 2018 on sn-devel-144

commit ca570bd4827aa8f61ceb137fbe748ac2f7929c44
Author: Tim Beale 
Date:   Thu Oct 18 16:59:24 2018 +1300

netcmd: Include num-members in 'samba-tool group list --verbose'

This adds an easy way for users to see (via samba-tool) how many members
are in various groups, without querying the members for each individual
group.

For example, you could pipe this output to grep to check for groups with
zero or one members (i.e. historic groups that may no longer make
sense).

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

commit a9a65adb023bde5567152101a7244968284b57d9
Author: Tim Beale 
Date:   Fri Oct 26 13:21:48 2018 +1300

selftest: Specify different DB backends for restored testenvs

Vary the DB backend that we use for the renamed DCs. The labdc and
renamedc are fairly similar, so let's have each of them use a different
backend.

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

commit 5a57378d342e690c8736877fb481ef34b27d4ea4
Author: Tim Beale 
Date:   Thu Oct 25 09:03:53 2018 +1300

netcmd: Add backend-store option to domain backup/rename cmds

Currently the online/rename backup files always use the default backend
(TDB) and there is no way to change this.

This patch adds the backend-store option to the backup commands so that
you can create a backup with an MDB backend, if needed.

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

commit 4a8a0ab32be3372888b02a4f0632246e483aa491
Author: Tim Beale 
Date:   Tue Oct 23 10:19:38 2018 +1300

traffic_replay: logger was ignoring smb.conf log-level

We were trying to access the debug-level (in python C bindings) before
the smb.conf had been loaded and actually set the debug-level. So it
would default to zero, regardless of what was in the smb.conf.

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

commit 4943473102883245fb2c286dfebe8a685d5b2026
Author: Tim Beale 
Date:   Wed Oct 17 11:26:28 2018 +1300

traffic_replay: Change print() to use logger()

This reduces noise, so the messages only come out if you specify
--debug.

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

commit b40daca6d4c886f2cea5efce38a5e945410d65d8
Author: Tim Beale 
Date:   Thu Oct 11 14:47:28 2018 +1300

traffic_replay: Generate users faster by writing to local DB

We can create user accounts much faster if the LDB connection talks
directly to the local sam.ldb file rather than going via LDAP. This
patch allows the 'host' argument to the tool to be a .ldb file (e.g.
"/usr/local/samba/private/sam.ldb") instead of a server name/IP.

In most cases, the traffic_replay tool wants to run on a remote device
(because the point of it is to send traffic to the DC). However, the
--generate-users-only is one case where the tool can be run locally,
directly on the test DC. (The traffic_replay user generation is handy
for standalone testing, because it also handles assigning group
memberships to the generated user accounts).

Note that you also need to use '--option="ldb:nosync = true"' to get
the improvement in performance.

Signed-off-by: Tim Beale 
Reviewed-by: Douglas Bagnall 

---

Summary of changes:
 docs-xml/manpages/samba-tool.8.xml |   5 ++
 python/samba/emulate/traffic.py|   

[SCM] NSS Wrapper Repository - branch master updated

2018-10-30 Thread Andreas Schneider
The branch, master has been updated
   via  70fb58a nwrap: Don't do a deep bind if we run with libasan
   via  4e67f44 tests: Check pointers of get(pw|gr)(nam|uid|gid)_r functions
   via  e171db5 nwrap: fix nwrap_module_getgrent_r
   via  c1bd13a nwrap: fix nwrap_module_getgrgid_r
   via  6bde6d5 nwrap: fix nwrap_module_getgrnam_r
   via  d3ad297 nwrap: fix nwrap_module_getpwent_r
   via  9e0b9d6 nwrap: fix nwrap_module_getpwuid_r
   via  433ab5e nwrap: fix nwrap_module_getpwnam_r
  from  2d57b2a ctest: Move to new dashboard

https://git.samba.org/?p=nss_wrapper.git;a=shortlog;h=master


- Log -
commit 70fb58a114e07a5a9766ad0efefb7aaa6b0e4cb8
Author: Andreas Schneider 
Date:   Sat Sep 1 19:41:46 2018 +0200

nwrap: Don't do a deep bind if we run with libasan

Signed-off-by: Andreas Schneider 
Reviewed-by: Jeremy Allison 

commit 4e67f442f5b2b8fd67e40bc27afefb73f41f7345
Author: Andreas Schneider 
Date:   Tue Oct 30 11:04:22 2018 +0100

tests: Check pointers of get(pw|gr)(nam|uid|gid)_r functions

Signed-off-by: Andreas Schneider 
Reviewed-by: Jeremy Allison 

commit e171db5578ecc25eec8979e20924a5eabcb9ea85
Author: Ralph Wuerthner 
Date:   Fri Oct 19 15:24:21 2018 +0200

nwrap: fix nwrap_module_getgrent_r

On success *grdstp must point to grdst, on error return NULL instead.

Signed-off-by: Ralph Wuerthner 
Reviewed-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
Reviewed-by: Andreas Schneider 

commit c1bd13a582fb885ea2e8385fec5b5eea94a7e2fc
Author: Ralph Wuerthner 
Date:   Fri Oct 19 15:22:23 2018 +0200

nwrap: fix nwrap_module_getgrgid_r

On success *grdstp must point to grdst, on error return NULL instead.

Signed-off-by: Ralph Wuerthner 
Reviewed-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
Reviewed-by: Andreas Schneider 

commit 6bde6d573508d11ecdeb549fb70e0f07beeb4027
Author: Ralph Wuerthner 
Date:   Fri Oct 19 15:19:28 2018 +0200

nwrap: fix nwrap_module_getgrnam_r

On success *grdstp must point to grdst, on error return NULL instead.

Signed-off-by: Ralph Wuerthner 
Reviewed-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
Reviewed-by: Andreas Schneider 

commit d3ad297a9420c7a752d7262a9870e16a29ab0a82
Author: Ralph Wuerthner 
Date:   Fri Oct 19 12:19:17 2018 +0200

nwrap: fix nwrap_module_getpwent_r

On success *pwdstp must point to pwdst, on error return NULL instead.

Signed-off-by: Ralph Wuerthner 
Reviewed-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
Reviewed-by: Andreas Schneider 

commit 9e0b9d6d4872db507b65989f975b8e145cf81e8b
Author: Ralph Wuerthner 
Date:   Fri Oct 19 12:15:53 2018 +0200

nwrap: fix nwrap_module_getpwuid_r

On success *pwdstp must point to pwdst, on error return NULL instead.

Signed-off-by: Ralph Wuerthner 
Reviewed-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
Reviewed-by: Andreas Schneider 

commit 433ab5ed5766150bf0786ba6c48e701205cd1a1c
Author: Ralph Wuerthner 
Date:   Fri Oct 19 11:27:04 2018 +0200

nwrap: fix nwrap_module_getpwnam_r

On success *pwdstp must point to pwdst, on error return NULL instead.

Signed-off-by: Ralph Wuerthner 
Reviewed-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 
Reviewed-by: Andreas Schneider 

---

Summary of changes:
 src/nss_wrapper.c | 30 +-
 tests/testsuite.c |  4 
 2 files changed, 25 insertions(+), 9 deletions(-)


Changeset truncated at 500 lines:

diff --git a/src/nss_wrapper.c b/src/nss_wrapper.c
index dd0848a..dd16fe6 100644
--- a/src/nss_wrapper.c
+++ b/src/nss_wrapper.c
@@ -846,7 +846,15 @@ static void *nwrap_load_lib_handle(enum nwrap_lib lib)
int i;
 
 #ifdef RTLD_DEEPBIND
-   flags |= RTLD_DEEPBIND;
+   const char *env = getenv("LD_PRELOAD");
+
+   /* Don't do a deepbind if we run with libasan */
+   if (env != NULL && strlen(env) < 1024) {
+   const char *p = strstr(env, "libasan.so");
+   if (p == NULL) {
+   flags |= RTLD_DEEPBIND;
+   }
+   }
 #endif
 
switch (lib) {
@@ -3846,9 +3854,7 @@ static int nwrap_module_getpwnam_r(struct nwrap_backend 
*b,
 {
int ret;
 
-   (void) b; /* unused */
-   (void) pwdst; /* unused */
-   (void) pwdstp; /* unused */
+   *pwdstp = NULL;
 
if (!b->fns->_nss_getpwnam_r) {
return NSS_STATUS_NOTFOUND;
@@ -3857,6 +3863,7 @@ static int nwrap_module_getpwnam_r(struct nwrap_backend 
*b,
ret = b->fns->_nss_getpwnam_r(name, pwdst, buf, buflen, );
switch (ret) {
case NSS_STATUS_SUCCESS:
+   *pwdstp = pwdst;
return 0;
case NSS_STATUS_NOTFOUND:
if 

[SCM] Samba Shared Repository - branch master updated

2018-10-30 Thread Stefan Metzmacher
The branch, master has been updated
   via  aeef8b4 dsdb group audit tests: log_membership_changes extra tests
   via  c952fc1 dsdb group audit tests: check_version improve diagnostics
   via  e2970887 dsdb group audit tests: check_timestamp improve diagnostics
   via  8420a4d dsdb group audit: align dn_compare with memcmp
   via  eeb4089 dsdb group_audit: Test to replicate BUG 13664
  from  852e1db dsdb: Add comments explaining the limitations of our 
current backlink behaviour

https://git.samba.org/?p=samba.git;a=shortlog;h=master


- Log -
commit aeef8b41fa03a32859f824f4a09560ad83bd2b50
Author: Gary Lockyer 
Date:   Thu Oct 25 10:52:55 2018 +1300

dsdb group audit tests: log_membership_changes extra tests

Add extra tests to ensure better test coverage of log_membership_changes

Signed-off-by: Gary Lockyer 
Reviewed-by: Stefan Metzmacher 

Autobuild-User(master): Stefan Metzmacher 
Autobuild-Date(master): Tue Oct 30 20:20:26 CET 2018 on sn-devel-144

commit c952fc1273397c04fddf177bcd809551d6324bdd
Author: Gary Lockyer 
Date:   Thu Oct 25 14:38:31 2018 +1300

dsdb group audit tests: check_version improve diagnostics

Change check_version to display the expected, actual along with the
line and name of the failing test, rather than the line in check_version

Signed-off-by: Gary Lockyer 
Reviewed-by: Stefan Metzmacher 

commit e2970887140d558c6359fd9b3f8c2a4c26d2cf35
Author: Gary Lockyer 
Date:   Thu Oct 25 13:28:09 2018 +1300

dsdb group audit tests: check_timestamp improve diagnostics

Change check_timestamp to display the expected, actual along with the
line and name of the failing test, rather than the line in
check_timestamp.

Signed-off-by: Gary Lockyer 
Reviewed-by: Stefan Metzmacher 

commit 8420a4d0fddd71af608635a707ef20f37fa9b627
Author: Gary Lockyer 
Date:   Thu Oct 25 10:52:27 2018 +1300

dsdb group audit: align dn_compare with memcmp

Rename the parameter names and adjust the  return codes from dn_compare
so that:
dn_compare(a, b) =>

LESS_THAN means a is less than b.
GREATER_THAN means a is greater than b.

Thanks to metze for suggesting the correct semantics for dn_compare

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13664

Signed-off-by: Gary Lockyer 
Reviewed-by: Stefan Metzmacher 

commit eeb4089dafc45277d8af19073ef9348451c1836a
Author: Gary Lockyer 
Date:   Tue Oct 23 17:14:34 2018 +1300

dsdb group_audit: Test to replicate BUG 13664

The group audit code incorrectly logs member additions and deletions.

Thanks to metze for the debugging that isolated the issue, and for
suggesting the fix to dn_compare.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13664

Signed-off-by: Gary Lockyer 
Reviewed-by: Stefan Metzmacher 

---

Summary of changes:
 source4/dsdb/samdb/ldb_modules/group_audit.c   |  31 +-
 .../samdb/ldb_modules/tests/test_group_audit.c | 716 -
 2 files changed, 718 insertions(+), 29 deletions(-)


Changeset truncated at 500 lines:

diff --git a/source4/dsdb/samdb/ldb_modules/group_audit.c 
b/source4/dsdb/samdb/ldb_modules/group_audit.c
index 1c74805..47b6943 100644
--- a/source4/dsdb/samdb/ldb_modules/group_audit.c
+++ b/source4/dsdb/samdb/ldb_modules/group_audit.c
@@ -311,35 +311,36 @@ enum dn_compare_result {
GREATER_THAN
 };
 /*
- * @brief compare parsed_dns
+ * @brief compare parsed_dn, using GUID ordering
  *
- * Compare two parsed_dn structures, parsing the entries if necessary.
+ * Compare two parsed_dn structures, using GUID ordering.
  * To avoid the overhead of parsing the DN's this function does a binary
- * compare first. Only parsing the DN's they are not equal at a binary level.
+ * compare first. The DN's tre only parsed if they are not equal at a binary
+ * level.
  *
  * @param ctx talloc context that will own the parsed dsdb_dn
  * @param ldb ldb_context
- * @param old_val The old value
- * @param new_val The old value
+ * @param dn1 The first dn
+ * @param dn2 The second dn
  *
  * @return BINARY_EQUAL values are equal at a binary level
  * EQUALDN's are equal but the meta data is different
- * LESS_THANold value < new value
- * GREATER_THAN old value > new value
+ * LESS_THANdn1's GUID is less than dn2's GUID
+ * GREATER_THAN dn1's GUID is greater than  dn2's GUID
  *
  */
 static enum dn_compare_result dn_compare(
TALLOC_CTX *mem_ctx,
struct ldb_context *ldb,
-   struct parsed_dn *old_val,
-   struct parsed_dn *new_val) {
+   struct parsed_dn *dn1,
+   struct parsed_dn *dn2) {
 
int res = 0;
 
/*
 * Do a binary compare first to avoid unnecessary parsing

[SCM] Socket Wrapper Repository - branch master updated

2018-10-30 Thread Andreas Schneider
The branch, master has been updated
   via  916ce9c cmake: Don't use string PREPEND
  from  32df6eb swrap: Fix coverity issues

https://git.samba.org/?p=socket_wrapper.git;a=shortlog;h=master


- Log -
commit 916ce9c66576f4de24dfe774fe9ab094187c1462
Author: Andreas Schneider 
Date:   Wed Oct 24 08:44:19 2018 +0200

cmake: Don't use string PREPEND

This is only supported in cmake >= 3.10.

Signed-off-by: Andreas Schneider 
Reviewed-by: Stefan Metzmacher 

---

Summary of changes:
 tests/CMakeLists.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


Changeset truncated at 500 lines:

diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 3578f76..b43d03c 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -83,8 +83,8 @@ function(ADD_CMOCKA_TEST_ENVIRONMENT _TEST_NAME)
 if (OSX)
 set(TORTURE_ENVIRONMENT 
"DYLD_FORCE_FLAT_NAMESPACE=1;DYLD_INSERT_LIBRARIES=${SOCKET_WRAPPER_LOCATION}")
 else ()
-string(REPLACE ";" ":" TORTURE_ENVIRONMENT "${PRELOAD_LIBRARIES}")
-string(PREPEND TORTURE_ENVIRONMENT "LD_PRELOAD=")
+string(REPLACE ";" ":" _TMP_ENV "${PRELOAD_LIBRARIES}")
+set(TORTURE_ENVIRONMENT "LD_PRELOAD=${_TMP_ENV}")
 endif()
 
 set_property(TEST


-- 
Socket Wrapper Repository



[SCM] Samba Shared Repository - branch master updated

2018-10-30 Thread Andrew Bartlett
The branch, master has been updated
   via  852e1db dsdb: Add comments explaining the limitations of our 
current backlink behaviour
   via  7a36cb3 s4:samldb: internally use extended dns while changing the 
primaryGroupID field
   via  0386307 s4:repl_meta_data: add support for 
DSDB_CONTROL_DBCHECK_FIX_LINK_DN_SID
   via  1ef145d s4:repl_meta_data: pass down struct 
replmd_replicated_request to replmd_modify_la_replace()
   via  738b52e s4:repl_meta_data: pass down struct 
replmd_replicated_request to replmd_modify_la_delete()
   via  70a306d s4:repl_meta_data: add missing 
 to a DEBUG message in replmd_modify_la_add()
   via  42e69a8 s4:repl_meta_data: pass down struct 
replmd_replicated_request to replmd_modify_la_add()
   via  966c7fe s4:repl_meta_data: pass down struct 
replmd_replicated_request to replmd_modify_handle_linked_attribs()
   via  f81771c blackbox/dbcheck-links: Test broken links with missing 
 on linked attributes
   via  a801799 dbchecker: Fix missing  on linked attributes
   via  c5c99b5 dbchecker: improve verbose output of do_modify()
   via  bb9c9e4 s4:dsdb: add DSDB_CONTROL_DBCHECK_FIX_LINK_DN_SID oid
   via  364ed53 testprogs/blackbox: add samba4.blackbox.test_primary_group 
test
   via  60131b4 s4:dsdb: fix comment on 
DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME
   via  0189f23 schema_samba4.ldif: add allocation of 
DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME
  from  500a729 tdb: Make record deletion circular-chain safe

https://git.samba.org/?p=samba.git;a=shortlog;h=master


- Log -
commit 852e1db12b0afa04a738c03bb2609c084fe96a7f
Author: Andrew Bartlett 
Date:   Tue Oct 30 15:56:43 2018 +1300

dsdb: Add comments explaining the limitations of our current backlink 
behaviour

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Andrew Bartlett 
Reviewed-by: Tim Beale 

Autobuild-User(master): Andrew Bartlett 
Autobuild-Date(master): Tue Oct 30 10:32:51 CET 2018 on sn-devel-144

commit 7a36cb30b716d56b84e894851c1a18e9eb3a0964
Author: Stefan Metzmacher 
Date:   Fri Aug 24 15:33:49 2018 +0200

s4:samldb: internally use extended dns while changing the primaryGroupID 
field

This is important, otherwise we'll loose the  component of the
linked attribute.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit 0386307e34097f5d9233c970983c7306d1705a87
Author: Stefan Metzmacher 
Date:   Fri Oct 12 15:56:18 2018 +0200

s4:repl_meta_data: add support for DSDB_CONTROL_DBCHECK_FIX_LINK_DN_SID

This will be used by dbcheck in the next commits.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit 1ef145d9d72d847055f6aba8a0070b3e1cfdabbc
Author: Stefan Metzmacher 
Date:   Fri Oct 12 18:43:25 2018 +0200

s4:repl_meta_data: pass down struct replmd_replicated_request to 
replmd_modify_la_replace()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit 738b52eb0856c8fcdbb8589e8061bcc14700c23a
Author: Stefan Metzmacher 
Date:   Fri Oct 12 18:43:25 2018 +0200

s4:repl_meta_data: pass down struct replmd_replicated_request to 
replmd_modify_la_delete()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit 70a306d0bd6806d1fd00d45e3d8cc70c73d09f79
Author: Stefan Metzmacher 
Date:   Fri Oct 12 19:34:08 2018 +0200

s4:repl_meta_data: add missing \n to a DEBUG message in 
replmd_modify_la_add()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit 42e69a86ca583e3cb20c63b9c6930b4b3425485d
Author: Stefan Metzmacher 
Date:   Fri Oct 12 18:43:25 2018 +0200

s4:repl_meta_data: pass down struct replmd_replicated_request to 
replmd_modify_la_add()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit 966c7febaf0245516481bde924ea6cd738eeb78b
Author: Stefan Metzmacher 
Date:   Fri Oct 12 18:43:25 2018 +0200

s4:repl_meta_data: pass down struct replmd_replicated_request to 
replmd_modify_handle_linked_attribs()

This will simplify further changes.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13418

Signed-off-by: Stefan Metzmacher 
Reviewed-by: Andrew Bartlett 

commit f81771c8593327e058b9cb4330d7e77083df3ea9
Author: Stefan Metzmacher 
Date:   Fri Oct 12 15:56:18 2018 +0200


autobuild[sn-devel-144]: intermittent test failure detected

2018-10-30 Thread autobuild
The autobuild test system (on sn-devel-144) has detected an intermittent 
failing test in 
the current master tree.

The autobuild log of the failure is available here:

   
https://git.samba.org/autobuild.flakey.sn-devel-144/2018-10-30-0735/flakey.log

The failure seems to be in the "samba" suite, whose build logs are available 
here:

   
https://git.samba.org/autobuild.flakey.sn-devel-144/2018-10-30-0735/samba.stderr
   
https://git.samba.org/autobuild.flakey.sn-devel-144/2018-10-30-0735/samba.stdout
  
The top commit at the time of the failure was:

commit 500a729a5566a1c8afc36e6f3e73910b13c8b9ad
Author: Volker Lendecke 
Date:   Wed Oct 24 14:31:34 2018 +0200

tdb: Make record deletion circular-chain safe

Before this patch we had 3 loops walking a hash chain to delete
records:

tdb_do_delete() to find the predecessor of the record that was to be
deleted. tdb_count_dead(), the name says it all and tdb_purge_dead()
to give back all dead records from a chain to the freelist.

This patch introduces tdb_trim_dead that walks a hash chain just
once. While it does so it counts the number of dead records, and all
records beyond tdb->max_dead_records are moved to the freelist.

Normal record deletion now works by always marking a record as dead in
step 1 and then calling tdb_trim_dead. This is made safe against
circular chains by doing the slow chain walk only in the case when we
did not delete a dead record during our walk.

It changes our dynamics a bit:

When deleting a record with non-zero max_dead_records, now we always
leave that number of records around when deleting, doing a blocking
lock on the freelist when we found too many dead records.

Previously when exceeding max_dead_records we wiped all dead records
to start accumulating them from scratch, assuming we could lock the
freelist in a nonblocking fashion.

The net effect for an uncontended freelist is the same: In
tdb_allocate() we still completely hand over all dead records to the
freelist when we could lock it, it just happens later than without
this patch.

This means for a lightly loaded system we will potentially leave more
dead records around in databases like locking.tdb. However, on a
heavily loaded system we become more predictable: If the freelist is
so heavily contended that across many deletes we can't get hold of it,
previously we accumulated more dead records than max_dead_records
would allow. This is a really lowlevel tradeoff that is likely hard to
measure, but to me becoming more deterministic without sacrificing too
much parallelism (we keep more dead records around) is worth trying.

Signed-off-by: Volker Lendecke 
Reviewed-by: Jeremy Allison 

Autobuild-User(master): Jeremy Allison 
Autobuild-Date(master): Tue Oct 30 02:48:38 CET 2018 on sn-devel-144

and the last 50 lines of the stdout log were:

[751(4033)/850 at 53m5s] samba.tests.blackbox.traffic_replay(ad_dc_ntvfs)
[752(4039)/850 at 53m15s] samba.tests.blackbox.traffic_learner(ad_dc_ntvfs)
[753(4041)/850 at 53m15s] samba.tests.blackbox.traffic_summary(ad_dc_ntvfs)
[754(4041)/850 at 53m15s] samba.tests.blackbox.smbcontrol(ad_dc_ntvfs:local)
[755(4043)/850 at 53m17s] samba.tests.blackbox.smbcontrol(promoted_dc:local)
[756(4045)/850 at 53m18s] samba4.ldap.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[757(4084)/850 at 53m30s] 
samba4.tokengroups.krb5.python(ad_dc_ntvfs)(ad_dc_ntvfs:local)
[758(4094)/850 at 53m38s] 
samba4.tokengroups.ntlm.python(ad_dc_ntvfs)(ad_dc_ntvfs:local)
[759(4103)/850 at 53m47s] samba4.sam.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[760(4120)/850 at 53m50s] 
samba4.user_account_control.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[761(4131)/850 at 54m4s] samba4.schemaInfo.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[762(4133)/850 at 54m6s] samba.tests.dsdb_schema_attributes(ad_dc_ntvfs:local)
[763(4139)/850 at 54m21s] 
samba4.urgent_replication.python(ad_dc_ntvfs)(ad_dc_ntvfs:local)
[764(4147)/850 at 54m25s] samba4.ldap.dirsync.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[765(4171)/850 at 55m55s] samba4.ldap.match_rules.python(ad_dc_ntvfs)
[766(4205)/850 at 56m13s] 
samba4.ldap.notification.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[767(4208)/850 at 56m22s] samba4.ldap.sites.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[768(4219)/850 at 56m28s] samba4.ldap.sort.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[769(4226)/850 at 56m33s] samba4.ldap.vlv.python(ad_dc_ntvfs)(ad_dc_ntvfs)
[770(4241)/850 at 1h29s] 
samba4.ldap.linked_attributes.python(ad_dc_ntvfs)(ad_dc_ntvfs:local)
[771(4261)/850 at 1h33s] samba4.ldap.rodc.python(rodc)(rodc)
[772(4267)/850 at 1h37s] samba4.ldap.rodc_rwdc.python(rodc)(rodc:local)
[773(4287)/850 at 1h8m10s] samba4.drs.replica_sync_rodc.python(rodc)(rodc:local)
[774(4289)/850 at 1h8m18s] samba4.ldap.passwordsettings.python(ad_dc_ntvfs)
[775(4301)/850 at 1h8m57s] samba4.ldap_schema.python(ad_dc_ntvfs)(ad_dc_ntvfs)
GSS client