The branch, master has been updated via b1740f3bafc CI: split out "samba-ad-dc-ntvfs[-py2]" test targets via 09c4e78fefd dsdb/tests/vlv: use only one toplevel dn that is correctly cleaned up via 162219de1df blackbox/dbcheck.sh: fix dbcheck_fix_one_way_links cleanup via 80d293ca0c4 selftest: improve debugging in dns_hub.py via 55cce815876 selftest: Make dns_hub socket timeout match DNS_REQUEST_TIMEOUT from 2e8daeb2bb3 docs-xml/smbdotconf: document export of SAMBA_CPS_{ACCOUNT,USER_PRINCIPAL,FULL}_NAME for check password script
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit b1740f3bafc2d460c137ac9f03514d57138d2f7a Author: Ralph Boehme <s...@samba.org> Date: Wed Jan 23 09:43:33 2019 +0100 CI: split out "samba-ad-dc-ntvfs[-py2]" test targets Many AD tests currently use the "samba" target. Split out a new target "samba-ad-dc-ntvfs" and have all tests that use the "ad_dc_ntvfs" env use the new target. This should greatly speed up the runtime for the "samba" target and avoid swapping. This reduces the total CI time by ~ 55%, I got an autobuild and a gitlab pipeline finished in just ~ 100 mins! Pair-Programmed-With: Stefan Metzmacher <me...@samba.org> Signed-off-by: Ralph Boehme <s...@samba.org> Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> Autobuild-User(master): Ralph Böhme <s...@samba.org> Autobuild-Date(master): Mon Feb 11 14:10:12 CET 2019 on sn-devel-144 commit 09c4e78fefd55d33f9fc2d3c3e1ba092a8cc490a Author: Stefan Metzmacher <me...@samba.org> Date: Fri Feb 8 10:57:13 2019 +0100 dsdb/tests/vlv: use only one toplevel dn that is correctly cleaned up Before "OU=vlvtestou2,%s" % (self.base_dn) was left behind after the test. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Ralph Boehme <s...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 162219de1df60573d0e42b73162fd5d086a812ad Author: Stefan Metzmacher <me...@samba.org> Date: Fri Feb 8 00:19:56 2019 +0100 blackbox/dbcheck.sh: fix dbcheck_fix_one_way_links cleanup Commit 35bfc62a31c9ad73449594ddd48f76f50e0abade changed dbcheck to not regard old one-way-links as errors. At that time the relavant trigger changed from fix_all_string_dn_component_mismatch to fix_all_old_dn_string_component_mismatch. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Ralph Boehme <s...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 80d293ca0c40b63251e0dd1f059129a24b5c241f Author: Stefan Metzmacher <me...@samba.org> Date: Wed Jan 23 09:34:40 2019 +0100 selftest: improve debugging in dns_hub.py We only print debug messages when the response is delayed by more than 2 seconds. Signed-off-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Ralph Boehme <s...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> commit 55cce815876a235f9a5e6015a96d1984bcf3a4ed Author: Tim Beale <timbe...@catalyst.net.nz> Date: Mon Feb 4 09:28:07 2019 +1300 selftest: Make dns_hub socket timeout match DNS_REQUEST_TIMEOUT I was hitting the recv_packet = s.recv(2048, 0) exception because the socket timeout was reached. We've seen it before, but it seemed more common after changing the default process-model to prefork. This patch makes the socket timeout used by the python code consistent with the C code. Signed-off-by: Tim Beale <timbe...@catalyst.net.nz> Reviewed-by: Stefan Metzmacher <me...@samba.org> Reviewed-by: Ralph Boehme <s...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> ----------------------------------------------------------------------- Summary of changes: .gitlab-ci.yml | 10 +++++++++ script/autobuild.py | 26 +++++++++++++++++----- selftest/target/dns_hub.py | 47 +++++++++++++++++++++++++++++++++------- source4/dsdb/tests/python/vlv.py | 18 +++++++-------- testprogs/blackbox/dbcheck.sh | 2 +- 5 files changed, 79 insertions(+), 24 deletions(-) Changeset truncated at 500 lines: diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 908c29ec9d9..0729599e1b8 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -118,3 +118,13 @@ build_samba_buildpy2_only: script: - python script/autobuild.py samba-buildpy2-only --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase +build_samba_ad_dc_ntvfs: + <<: *shared_template + script: + # this one takes about 100 mins to finish + - script/autobuild.py samba-ad-dc-ntvfs --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase + +build_samba_ad_dc_ntvfs_py2: + <<: *shared_template + script: + - script/autobuild.py samba-ad-dc-ntvfs-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase diff --git a/script/autobuild.py b/script/autobuild.py index 00f0d2202a3..b1bd9d1686c 100755 --- a/script/autobuild.py +++ b/script/autobuild.py @@ -49,6 +49,8 @@ builddirs = { "samba-none-env": ".", "samba-ad-dc": ".", "samba-ad-dc-py2": ".", + "samba-ad-dc-ntvfs": ".", + "samba-ad-dc-ntvfs-py2": ".", "samba-ad-dc-2": ".", "samba-ad-dc-2-py2": ".", "samba-ad-dc-backup": ".", @@ -93,8 +95,10 @@ tasks = { ("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"), ("clean", "make clean", "text/plain")], - # We have 'test' before 'install' because, 'test' should work without 'install (runs ad_dc_ntvfs and all the other envs)' - "samba": [("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), + # We have 'test' before 'install' because, 'test' should work without 'install (runs all the other envs)' + "samba": [ + ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), + ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), ("test", "make test FAIL_IMMEDIATELY=1 " "TESTS='${PY3_ONLY}" @@ -102,6 +106,7 @@ tasks = { "--exclude-env=nt4_dc " "--exclude-env=nt4_member " "--exclude-env=ad_dc " + "--exclude-env=ad_dc_ntvfs " "--exclude-env=ad_dc_no_nss " "--exclude-env=fl2003dc " "--exclude-env=fl2008r2dc " @@ -123,7 +128,6 @@ tasks = { ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"), ("clean", "make clean", "text/plain")], - # We split out this so the isolated nt4_dc tests do not wait for ad_dc or ad_dc_ntvfs tests (which are long) "samba-nt4": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), ("configure", "./configure.developer --without-ads --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), @@ -134,7 +138,6 @@ tasks = { ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"), ("clean", "make clean", "text/plain")], - # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long) "samba-fileserver": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), ("configure", "./configure.developer --without-ad-dc --without-ldap --without-ads --without-json --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), @@ -143,7 +146,6 @@ tasks = { "--include-env=fileserver'", "text/plain"), ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")], - # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long) "samba-ad-dc": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), @@ -157,7 +159,6 @@ tasks = { "--include-env=ad_member_idmap_ad'", "text/plain"), ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")], - # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long) "samba-ad-dc-2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), ("make", "make -j", "text/plain"), @@ -171,6 +172,19 @@ tasks = { "text/plain"), ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")], + # We split out the ad_dc_ntvfs tests (which are long) so other test do not wait + # This is currently the longest task, so we don't randomly delay it. + "samba-ad-dc-ntvfs": [ + ("random-sleep", "script/random-sleep.sh 1 1", "text/plain"), + ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"), + ("make", "make -j", "text/plain"), + ("test", "make test FAIL_IMMEDIATELY=1 " + "TESTS='${PY3_ONLY}" + "--include-env=ad_dc_ntvfs " + "'", + "text/plain"), + ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")], + # run the backup/restore testenvs separately as they're fairly standalone # (and CI seems to max out at ~8 different DCs running at once) "samba-ad-dc-backup": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"), diff --git a/selftest/target/dns_hub.py b/selftest/target/dns_hub.py index d2d1f39e752..2cfc8d076aa 100755 --- a/selftest/target/dns_hub.py +++ b/selftest/target/dns_hub.py @@ -24,6 +24,7 @@ import threading import sys import select import socket +import time from samba.dcerpc import dns import samba.ndr as ndr @@ -34,15 +35,27 @@ else: import socketserver sserver = socketserver +DNS_REQUEST_TIMEOUT = 10 + class DnsHandler(sserver.BaseRequestHandler): + dns_qtype_strings = dict((v, k) for k, v in vars(dns).items() if k.startswith('DNS_QTYPE_')) + def dns_qtype_string(self, qtype): + "Return a readable qtype code" + return self.dns_qtype_strings[qtype] + + dns_rcode_strings = dict((v, k) for k, v in vars(dns).items() if k.startswith('DNS_RCODE_')) + def dns_rcode_string(self, rcode): + "Return a readable error code" + return self.dns_rcode_strings[rcode] + def dns_transaction_udp(self, packet, host): "send a DNS query and read the reply" s = None try: send_packet = ndr.ndr_pack(packet) s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0) - s.settimeout(5) + s.settimeout(DNS_REQUEST_TIMEOUT) s.connect((host, 53)) s.sendall(send_packet, 0) recv_packet = s.recv(2048, 0) @@ -63,6 +76,13 @@ class DnsHandler(sserver.BaseRequestHandler): return 'ignore' if lname.endswith('dsfsdfs'): return 'fail' + if lname.endswith("torture1", 0, len(lname)-2): + # CATCH TORTURE100, TORTURE101, ... + return 'torture' + if lname.endswith('_none_.example.com'): + return 'torture' + if lname.endswith('torturedom.samba.example.com'): + return 'torture' if lname.endswith('adnonssdom.samba.example.com'): return '127.0.0.17' if lname.endswith('adnontlmdom.samba.example.com'): @@ -90,6 +110,7 @@ class DnsHandler(sserver.BaseRequestHandler): return None def handle(self): + start = time.monotonic() data, sock = self.request query = ndr.ndr_unpack(dns.name_packet, data) name = query.questions[0].name @@ -100,13 +121,13 @@ class DnsHandler(sserver.BaseRequestHandler): return elif forwarder is 'fail': pass - elif forwarder is not None: - response = self.dns_transaction_udp(query, forwarder) - else: + elif forwarder in ['torture', None]: response = query response.operation |= dns.DNS_FLAG_REPLY response.operation |= dns.DNS_FLAG_RECURSION_AVAIL response.operation |= dns.DNS_RCODE_NXDOMAIN + else: + response = self.dns_transaction_udp(query, forwarder) if response is None: response = query @@ -116,14 +137,24 @@ class DnsHandler(sserver.BaseRequestHandler): send_packet = ndr.ndr_pack(response) - print("dns_hub: sending %s to address %s for name %s\n" % - (forwarder, self.client_address, name)) + end = time.monotonic() + tdiff = end - start + errcode = response.operation & dns.DNS_RCODE + if tdiff > (DNS_REQUEST_TIMEOUT/5): + debug = True + else: + debug = False + if debug: + print("dns_hub: forwarder[%s] client[%s] name[%s][%s] %s response.operation[0x%x] tdiff[%s]\n" % + (forwarder, self.client_address, name, + self.dns_qtype_string(query.questions[0].question_type), + self.dns_rcode_string(errcode), response.operation, tdiff)) try: sock.sendto(send_packet, self.client_address) except socket.error as err: - print("Error sending %s to address %s for name %s: %s\n" % - (forwarder, self.client_address, name, err)) + print("dns_hub: Error sending response to client[%s] for name[%s] tdiff[%s]: %s\n" % + (self.client_address, name, tdiff, err)) class server_thread(threading.Thread): diff --git a/source4/dsdb/tests/python/vlv.py b/source4/dsdb/tests/python/vlv.py index 8550a38e287..29cc1718c78 100644 --- a/source4/dsdb/tests/python/vlv.py +++ b/source4/dsdb/tests/python/vlv.py @@ -153,12 +153,16 @@ class TestsWithUserOU(samba.tests.TestCase): session_info=system_session(lp), lp=lp) self.base_dn = self.ldb.domain_dn() - self.ou = "ou=vlv,%s" % self.base_dn + self.tree_dn = "ou=vlvtesttree,%s" % self.base_dn + self.ou = "ou=vlvou,%s" % self.tree_dn if opts.delete_in_setup: try: - self.ldb.delete(self.ou, ['tree_delete:1']) + self.ldb.delete(self.tree_dn, ['tree_delete:1']) except ldb.LdbError as e: - print("tried deleting %s, got error %s" % (self.ou, e)) + print("tried deleting %s, got error %s" % (self.tree_dn, e)) + self.ldb.add({ + "dn": self.tree_dn, + "objectclass": "organizationalUnit"}) self.ldb.add({ "dn": self.ou, "objectclass": "organizationalUnit"}) @@ -191,7 +195,7 @@ class TestsWithUserOU(samba.tests.TestCase): def tearDown(self): super(TestsWithUserOU, self).tearDown() if not opts.delete_in_setup: - self.ldb.delete(self.ou, ['tree_delete:1']) + self.ldb.delete(self.tree_dn, ['tree_delete:1']) class VLVTests(TestsWithUserOU): @@ -1368,11 +1372,7 @@ class PagedResultsTests(TestsWithUserOU): def test_paged_modify_object_scope(self): expr = "(objectClass=*)" - ou2 = "OU=vlvtestou2,%s" % (self.base_dn) - try: - self.ldb.delete(ou2, ['tree_delete:1']) - except ldb.LdbError: - pass + ou2 = "OU=vlvtestou2,%s" % (self.tree_dn) self.ldb.add({"dn": ou2, "objectclass": "organizationalUnit"}) # Do a separate, full search to get all results diff --git a/testprogs/blackbox/dbcheck.sh b/testprogs/blackbox/dbcheck.sh index 5691426d01b..caf89ca402b 100755 --- a/testprogs/blackbox/dbcheck.sh +++ b/testprogs/blackbox/dbcheck.sh @@ -19,7 +19,7 @@ dbcheck() { # This list of attributes can be freely extended dbcheck_fix_one_way_links() { - $PYTHON $BINDIR/samba-tool dbcheck --quiet --fix --yes fix_all_string_dn_component_mismatch --attrs="lastKnownParent defaultObjectCategory fromServer rIDSetReferences" --cross-ncs $ARGS + $PYTHON $BINDIR/samba-tool dbcheck --quiet --fix --yes fix_all_old_dn_string_component_mismatch --attrs="lastKnownParent defaultObjectCategory fromServer rIDSetReferences" --cross-ncs $ARGS } # This list of attributes can be freely extended -- Samba Shared Repository