The branch, master has been updated via dcc657a selftest: Add assertion that we actually fix the replPropertyMetaData sort order via 5504502 selftest: Add in steps to re-create this database via a6957ba Update release-4-1-0rc3 to include data using schema modifications via 6122aca ldb: create a cache of known wellknown objects instead of continously searching in the db via c049106 dbcheck: Use set() operations to make dbcheck more efficient via fb88f9c dbcheck: Try to avoid duplicate searches via 2ff9b17 dbcheck: Add additional tests for the attributeID list via 2766bad dbcheck: Add explict tests for unknown and unsorted attributeID values via e3cf25b pidl: Assert that python arrays will not overflow the C array via bed29f3 pydsdb: Allow the full range of uint32_t values for attributeID via 336d411 python/tests: Add tests for integer overflow handling via 5206ccd pidl: Change PIDL to correctly use and validate python integer types via 3faa7dc python: Use an unsigned integer for buf_size, not -1 via 4ef468e dnsserver: Remove incorrect and not required include of ldb_private.h from 617bc3f winbind: Remove "have_idmap_config" from winbindd_domain
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit dcc657a2219498beac8bde6cbf999ee7cf6acae8 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Aug 3 13:50:08 2015 +1200 selftest: Add assertion that we actually fix the replPropertyMetaData sort order This ensures that the dbcheck rule fixes the sort order (and only fixes the sort order). BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> Autobuild-User(master): Stefan Metzmacher <me...@samba.org> Autobuild-Date(master): Tue Aug 25 02:45:58 CEST 2015 on sn-devel-104 commit 5504502aa68f4901f52dc2e8f7ee8b3a9c74546e Author: Andrew Bartlett <abart...@samba.org> Date: Mon Aug 3 11:25:02 2015 +1200 selftest: Add in steps to re-create this database This may assist if this needs to be changed again BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit a6957ba5da3130994d2f35328aea6f4cd604147a Author: Andrew Bartlett <abart...@samba.org> Date: Mon Aug 3 11:24:10 2015 +1200 Update release-4-1-0rc3 to include data using schema modifications This allows us to know that the previous patches are correct. BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 6122acad0f1a7bc23b6f58862c16968e13da979d Author: Matthieu Patou <m...@matws.net> Date: Mon May 25 09:17:55 2015 -0700 ldb: create a cache of known wellknown objects instead of continously searching in the db Profiling on dbcheck have shown that we spend 10% of the time looking for wellknown objects. BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Change-Id: I13ed58e8062d1b7b6179d17b0e7e56f943572c6c Signed-off-by: Matthieu Patou <m...@matws.net> Reviewed-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit c049106bf8267b30a9242d2d574661291cced780 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Jul 27 15:11:56 2015 +1200 dbcheck: Use set() operations to make dbcheck more efficient BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit fb88f9cbd969267aaffa021724cf34087c653ba8 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Jul 27 15:44:56 2015 +1200 dbcheck: Try to avoid duplicate searches BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 2ff9b171378922e68205d5f0eefd2231607b5b89 Author: Andrew Bartlett <abart...@samba.org> Date: Tue Jul 28 16:11:54 2015 +1200 dbcheck: Add additional tests for the attributeID list BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 2766bad5ef1e1949746c059c29f179ddae476239 Author: Andrew Bartlett <abart...@samba.org> Date: Thu Jul 23 16:01:14 2015 +1200 dbcheck: Add explict tests for unknown and unsorted attributeID values Unknown attributeID values would cause an exception previously, and unsorted attributes cause a failure to replicate with Samba 4.2. In commit 61b978872fe86906611f64430b2608f5e7ea7ad8 we started to sort these values correctly, but previous versions of Samba did not sort them correctly (we sorted high-bit-set values as negative), and then after 9c9df40220234cba973e84b4985d90da1334a1d1 we stoped accepting these. To ensure we are allowed to make this unusual change to the replPropertyMetaData, a new OID is allocated and checked for in repl_meta_data.c BUG: https://bugzilla.samba.org/show_bug.cgi?id=10973 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit e3cf25bdc04795f884fe2a4ce0605d5175b45e93 Author: Andrew Bartlett <abart...@samba.org> Date: Thu Jul 30 14:28:48 2015 +1200 pidl: Assert that python arrays will not overflow the C array We do not write network services in Python, so this is not a security issue, but would cause a crash or other odd behaviour if the length was changed BUG: https://bugzilla.samba.org/show_bug.cgi?id=11430 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit bed29f3c92861fb2b83ad03a5c759061e1c31eba Author: Andrew Bartlett <abart...@samba.org> Date: Tue Jul 28 14:29:25 2015 +1200 pydsdb: Allow the full range of uint32_t values for attributeID The high bit may be set in these integers, so we need an unsigned int to store it in BUG: https://bugzilla.samba.org/show_bug.cgi?id=11429 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 336d41155e94303d0e1fa0530874539339439fed Author: Andrew Bartlett <abart...@samba.org> Date: Thu Jul 30 14:29:54 2015 +1200 python/tests: Add tests for integer overflow handling This also documents an issue with our python bindings and lists, as changes to integers in a list of integers are not preserved BUG: https://bugzilla.samba.org/show_bug.cgi?id=11429 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 5206ccd002b015a16952d21bd2fa441ff2e39eab Author: Andrew Bartlett <abart...@samba.org> Date: Mon Jul 27 10:57:43 2015 +1200 pidl: Change PIDL to correctly use and validate python integer types In particular, it is critical that we use unsigned integers of sufficient size in python for unsigned C integers, and it is critical that we check for overflow at both the python and C level. Otherwise, we may both represent and sort these incorrectly, in particular when sorting attributeID values from DRSUAPI which are represented as an signed enum in C and a uint32_t in IDL, but which often has the high bit set (in schema extensions). BUG: https://bugzilla.samba.org/show_bug.cgi?id=11429 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 3faa7dc5a7d39cfdfc908a51e38772fda801c1a3 Author: Andrew Bartlett <abart...@samba.org> Date: Wed Jul 29 15:25:09 2015 +1200 python: Use an unsigned integer for buf_size, not -1 This will fail once our python bindings correctly check value ranges BUG: https://bugzilla.samba.org/show_bug.cgi?id=11429 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> commit 4ef468eecdbff27347280060636afe27ce3a0b87 Author: Andrew Bartlett <abart...@samba.org> Date: Mon Aug 3 13:33:40 2015 +1200 dnsserver: Remove incorrect and not required include of ldb_private.h Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Stefan Metzmacher <me...@samba.org> ----------------------------------------------------------------------- Summary of changes: pidl/lib/Parse/Pidl/Samba4/Python.pm | 160 +- python/samba/dbchecker.py | 116 +- python/samba/samdb.py | 15 +- python/samba/tests/dcerpc/integer.py | 203 + python/samba/tests/dcerpc/sam.py | 2 +- python/samba/tests/dsdb.py | 24 + selftest/knownfail | 5 + selftest/tests.py | 1 + source4/dsdb/pydsdb.c | 4 +- source4/dsdb/samdb/ldb_modules/repl_meta_data.c | 31 +- source4/dsdb/samdb/samdb.h | 6 + source4/rpc_server/dnsserver/dcerpc_dnsserver.c | 1 - ...xpected-replpropertymetadata-after-dbcheck.ldif | 122 + ...pected-replpropertymetadata-before-dbcheck.ldif | 122 + .../provisions/release-4-1-0rc3/private/dns.keytab | Bin 1037 -> 1037 bytes ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 16082 ++++++------- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 22532 +++++++++--------- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 364 +- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 198 +- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 4 +- .../private/dns/sam.ldb.d/metadata.tdb.dump | 4 +- .../release-4-1-0rc3/private/dns/sam.ldb.dump | 4 +- .../release-4-1-0rc3/private/hklm.ldb.dump | 2 +- .../release-4-1-0rc3/private/idmap.ldb.dump | 36 +- .../provisions/release-4-1-0rc3/private/named.conf | 6 +- .../provisions/release-4-1-0rc3/private/named.txt | 8 +- .../release-4-1-0rc3/private/privilege.ldb.dump | 2 +- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 16154 ++++++------- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 22932 +++++++++---------- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 364 +- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 198 +- ...DRELEASE-4-1-0RC3,DC%3DSAMBA,DC%3DCORP.ldb.dump | 2798 +-- .../private/sam.ldb.d/metadata.tdb.dump | 4 +- .../release-4-1-0rc3/private/sam.ldb.dump | 8 +- .../release-4-1-0rc3/private/secrets.keytab | Bin 1482 -> 1482 bytes .../release-4-1-0rc3/private/secrets.ldb.dump | 10 +- .../release-4-1-0rc3/private/secrets.tdb.dump | 4 +- .../release-4-1-0rc3/private/share.ldb.dump | 2 +- .../private/smbd.tmp/msg/names.tdb.dump | 24 +- .../release-4-1-0rc3/steps-to-reproduce.txt | 30 + .../provisions/release-4-1-0rc3/sudo.schema.2.ldif | 52 + .../provisions/release-4-1-0rc3/sudo.schema.ldif | 235 + .../provisions/release-4-1-0rc3/sudoers-mod.ldif | 4 + .../provisions/release-4-1-0rc3/sudoers.ldif | 13 + source4/setup/schema_samba4.ldif | 1 + testprogs/blackbox/dbcheck-oldrelease.sh | 47 +- 46 files changed, 41678 insertions(+), 41256 deletions(-) create mode 100644 python/samba/tests/dcerpc/integer.py create mode 100644 source4/selftest/provisions/release-4-1-0rc3/expected-replpropertymetadata-after-dbcheck.ldif create mode 100644 source4/selftest/provisions/release-4-1-0rc3/expected-replpropertymetadata-before-dbcheck.ldif create mode 100644 source4/selftest/provisions/release-4-1-0rc3/steps-to-reproduce.txt create mode 100644 source4/selftest/provisions/release-4-1-0rc3/sudo.schema.2.ldif create mode 100644 source4/selftest/provisions/release-4-1-0rc3/sudo.schema.ldif create mode 100644 source4/selftest/provisions/release-4-1-0rc3/sudoers-mod.ldif create mode 100644 source4/selftest/provisions/release-4-1-0rc3/sudoers.ldif Changeset truncated at 500 lines: diff --git a/pidl/lib/Parse/Pidl/Samba4/Python.pm b/pidl/lib/Parse/Pidl/Samba4/Python.pm index ff68e7a..ad9ff88 100644 --- a/pidl/lib/Parse/Pidl/Samba4/Python.pm +++ b/pidl/lib/Parse/Pidl/Samba4/Python.pm @@ -10,7 +10,7 @@ use Exporter; use strict; use Parse::Pidl qw(warning fatal error); -use Parse::Pidl::Typelist qw(hasType resolveType getType mapTypeName expandAlias); +use Parse::Pidl::Typelist qw(hasType resolveType getType mapTypeName expandAlias bitmap_type_fn enum_type_fn); use Parse::Pidl::Util qw(has_property ParseExpr unmake_str); use Parse::Pidl::NDR qw(GetPrevLevel GetNextLevel ContainsDeferred ContainsPipe is_charset_array); use Parse::Pidl::CUtil qw(get_value_of get_pointer_to); @@ -953,14 +953,58 @@ sub ConvertObjectFromPythonData($$$$$$;$) $actual_ctype = $actual_ctype->{DATA}; } - if ($actual_ctype->{TYPE} eq "ENUM" or $actual_ctype->{TYPE} eq "BITMAP") { + # We need to cover ENUMs, BITMAPS and SCALAR values here, as + # all could otherwise be assigned invalid integer values + my $ctype_alias = ""; + my $uint_max = ""; + if ($actual_ctype->{TYPE} eq "ENUM") { + # Importantly, ENUM values are unsigned in pidl, and + # typically map to uint32 + $ctype_alias = enum_type_fn($actual_ctype); + } elsif ($actual_ctype->{TYPE} eq "BITMAP") { + $ctype_alias = bitmap_type_fn($actual_ctype); + } elsif ($actual_ctype->{TYPE} eq "SCALAR") { + $ctype_alias = expandAlias($actual_ctype->{NAME}); + } + + # This is the unsigned Python Integer -> C integer validation + # case. The signed case is below. + if ($ctype_alias =~ /^(uint[0-9]*|hyper|udlong|udlongr + |NTTIME_hyper|NTTIME|NTTIME_1sec + |uid_t|gid_t)$/x) { + $self->pidl("{"); + $self->indent; + $self->pidl("const unsigned long long uint_max = (sizeof($target) == 8) ? UINT64_MAX : (unsigned long long)((1ULL << (sizeof($target) * 8)) - 1);"); $self->pidl("if (PyLong_Check($cvar)) {"); $self->indent; - $self->pidl("$target = PyLong_AsLongLong($cvar);"); + $self->pidl("unsigned long long test_var;"); + $self->pidl("test_var = PyLong_AsUnsignedLongLong($cvar);"); + $self->pidl("if (PyErr_Occurred() != NULL) {"); + $self->indent; + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->pidl("if (test_var > uint_max) {"); + $self->indent; + $self->pidl("PyErr_Format(PyExc_OverflowError, \"Expected type %s or %s within range 0 - %llu, got %llu\",\\"); + $self->pidl(" PyInt_Type.tp_name, PyLong_Type.tp_name, uint_max, test_var);"); + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->pidl("$target = test_var;"); $self->deindent; $self->pidl("} else if (PyInt_Check($cvar)) {"); $self->indent; - $self->pidl("$target = PyInt_AsLong($cvar);"); + $self->pidl("long test_var;"); + $self->pidl("test_var = PyInt_AsLong($cvar);"); + $self->pidl("if (test_var < 0 || test_var > uint_max) {"); + $self->indent; + $self->pidl("PyErr_Format(PyExc_OverflowError, \"Expected type %s or %s within range 0 - %llu, got %ld\",\\"); + $self->pidl(" PyInt_Type.tp_name, PyLong_Type.tp_name, uint_max, test_var);"); + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->pidl("$target = test_var;"); $self->deindent; $self->pidl("} else {"); $self->indent; @@ -969,34 +1013,61 @@ sub ConvertObjectFromPythonData($$$$$$;$) $self->pidl($fail); $self->deindent; $self->pidl("}"); + $self->deindent; + $self->pidl("}"); return; } - if ($actual_ctype->{TYPE} eq "SCALAR" ) { - if (expandAlias($actual_ctype->{NAME}) =~ /^(u?int64|hyper|dlong|udlong|udlongr - |NTTIME_hyper|NTTIME|NTTIME_1sec - |uid_t|gid_t)$/x) { - $self->pidl("if (PyLong_Check($cvar)) {"); - $self->indent; - $self->pidl("$target = PyLong_AsLongLong($cvar);"); - $self->deindent; - $self->pidl("} else if (PyInt_Check($cvar)) {"); - $self->indent; - $self->pidl("$target = PyInt_AsLong($cvar);"); - $self->deindent; - $self->pidl("} else {"); - $self->indent; - $self->pidl("PyErr_Format(PyExc_TypeError, \"Expected type %s or %s\",\\"); - $self->pidl(" PyInt_Type.tp_name, PyLong_Type.tp_name);"); - $self->pidl($fail); - $self->deindent; - $self->pidl("}"); - return; - } - if (expandAlias($actual_ctype->{NAME}) =~ /^(char|u?int[0-9]*|time_t)$/) { - $self->pidl("PY_CHECK_TYPE(&PyInt_Type, $cvar, $fail);"); - $self->pidl("$target = PyInt_AsLong($cvar);"); - return; - } + + # Confirm the signed python integer fits in the C type + # correctly. It is subtly different from the unsigned case + # above, so while it looks like a duplicate, it is not + # actually a duplicate. + if ($ctype_alias =~ /^(dlong|char|int[0-9]*|time_t)$/x) { + $self->pidl("{"); + $self->indent; + $self->pidl("const long long int_max = (long long)((1ULL << (sizeof($target) * 8 - 1)) - 1);"); + $self->pidl("const long long int_min = -int_max - 1;"); + $self->pidl("if (PyLong_Check($cvar)) {"); + $self->indent; + $self->pidl("long long test_var;"); + $self->pidl("test_var = PyLong_AsLongLong($cvar);"); + $self->pidl("if (PyErr_Occurred() != NULL) {"); + $self->indent; + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->pidl("if (test_var < int_min || test_var > int_max) {"); + $self->indent; + $self->pidl("PyErr_Format(PyExc_OverflowError, \"Expected type %s or %s within range %lld - %lld, got %lld\",\\"); + $self->pidl(" PyInt_Type.tp_name, PyLong_Type.tp_name, int_min, int_max, test_var);"); + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->pidl("$target = test_var;"); + $self->deindent; + $self->pidl("} else if (PyInt_Check($cvar)) {"); + $self->indent; + $self->pidl("long test_var;"); + $self->pidl("test_var = PyInt_AsLong($cvar);"); + $self->pidl("if (test_var < int_min || test_var > int_max) {"); + $self->indent; + $self->pidl("PyErr_Format(PyExc_OverflowError, \"Expected type %s or %s within range %lld - %lld, got %ld\",\\"); + $self->pidl(" PyInt_Type.tp_name, PyLong_Type.tp_name, int_min, int_max, test_var);"); + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->pidl("$target = test_var;"); + $self->deindent; + $self->pidl("} else {"); + $self->indent; + $self->pidl("PyErr_Format(PyExc_TypeError, \"Expected type %s or %s\",\\"); + $self->pidl(" PyInt_Type.tp_name, PyLong_Type.tp_name);"); + $self->pidl($fail); + $self->deindent; + $self->pidl("}"); + $self->deindent; + $self->pidl("}"); + return; } if ($actual_ctype->{TYPE} eq "STRUCT" or $actual_ctype->{TYPE} eq "INTERFACE") { @@ -1152,6 +1223,13 @@ sub ConvertObjectFromPythonLevel($$$$$$$$) $self->pidl("$var_name = talloc_array_ptrtype($mem_ctx, $var_name, PyList_GET_SIZE($py_var));"); $self->pidl("if (!$var_name) { $fail; }"); $self->pidl("talloc_set_name_const($var_name, \"ARRAY: $var_name\");"); + } else { + $self->pidl("if (ARRAY_SIZE($var_name) != PyList_GET_SIZE($py_var)) {"); + $self->indent; + $self->pidl("PyErr_Format(PyExc_TypeError, \"Expected list of type %s, length %zu, got %zd\", Py_TYPE($py_var)->tp_name, ARRAY_SIZE($var_name), PyList_GET_SIZE($py_var));"); + $self->pidl("$fail"); + $self->deindent; + $self->pidl("}"); } $self->pidl("for ($counter = 0; $counter < PyList_GET_SIZE($py_var); $counter++) {"); $self->indent; @@ -1202,17 +1280,29 @@ sub ConvertScalarToPython($$$) $ctypename = expandAlias($ctypename); if ($ctypename =~ /^(int64|dlong)$/) { - return "PyLong_FromLongLong($cvar)"; + return "($cvar > LONG_MAX || $cvar < LONG_MIN) ? PyLong_FromLongLong($cvar) : PyInt_FromLong($cvar)"; } - if ($ctypename =~ /^(uint64|hyper|udlong|udlongr|NTTIME_hyper|NTTIME|NTTIME_1sec|uid_t|gid_t)$/) { - return "PyLong_FromUnsignedLongLong($cvar)"; + if ($ctypename =~ /^(uint64|hyper|NTTIME_hyper|NTTIME|NTTIME_1sec|udlong|udlongr|uid_t|gid_t)$/) { + return "$cvar > LONG_MAX ? PyLong_FromUnsignedLongLong($cvar) : PyInt_FromLong($cvar)"; } - if ($ctypename =~ /^(char|u?int[0-9]*|time_t)$/) { + if ($ctypename =~ /^(char|int|int8|int16|int32|time_t)$/) { return "PyInt_FromLong($cvar)"; } + # Needed to ensure unsigned values in a 32 or 16 bit enum is + # cast correctly to a uint32_t, not sign extended to a a + # possibly 64 bit unsigned long. (enums are signed in C, + # unsigned in NDR) + if ($ctypename =~ /^(uint32|uint3264)$/) { + return "(uint32_t)$cvar > LONG_MAX ? PyLong_FromUnsignedLong((uint32_t)$cvar) : PyInt_FromLong((uint32_t)$cvar)"; + } + + if ($ctypename =~ /^(uint|uint8|uint16|uint1632)$/) { + return "PyInt_FromLong((uint16_t)$cvar)"; + } + if ($ctypename eq "DATA_BLOB") { return "PyString_FromStringAndSize((char *)($cvar).data, ($cvar).length)"; } @@ -1487,7 +1577,7 @@ sub Parse($$$$$) my $py_obj; my ($ctype, $cvar) = @{$h->{'val'}}; if ($cvar =~ /^[0-9]+$/ or $cvar =~ /^0x[0-9a-fA-F]+$/) { - $py_obj = "PyInt_FromLong($cvar)"; + $py_obj = "$cvar > LONG_MAX ? PyLong_FromUnsignedLongLong($cvar) : PyInt_FromLong($cvar)"; } elsif ($cvar =~ /^".*"$/) { $py_obj = "PyString_FromString($cvar)"; } else { diff --git a/python/samba/dbchecker.py b/python/samba/dbchecker.py index 74e9678..4fb9d12 100644 --- a/python/samba/dbchecker.py +++ b/python/samba/dbchecker.py @@ -24,6 +24,7 @@ from base64 import b64decode from samba import dsdb from samba import common from samba.dcerpc import misc +from samba.dcerpc import drsuapi from samba.ndr import ndr_unpack, ndr_pack from samba.dcerpc import drsblobs from samba.common import dsdb_Dn @@ -63,6 +64,7 @@ class dbcheck(object): self.move_to_lost_and_found = False self.fix_instancetype = False self.fix_replmetadata_zero_invocationid = False + self.fix_replmetadata_unsorted_attid = False self.fix_deleted_deleted_objects = False self.fix_dn = False self.fix_base64_userparameters = False @@ -80,6 +82,8 @@ class dbcheck(object): self.wellknown_sds = get_wellknown_sds(self.samdb) self.fix_all_missing_objectclass = False + self.dn_set = set() + self.name_map = {} try: res = samdb.search(base="CN=DnsAdmins,CN=Users,%s" % samdb.domain_dn(), scope=ldb.SCOPE_BASE, @@ -125,12 +129,12 @@ class dbcheck(object): def check_database(self, DN=None, scope=ldb.SCOPE_SUBTREE, controls=[], attrs=['*']): '''perform a database check, returning the number of errors found''' - res = self.samdb.search(base=DN, scope=scope, attrs=['dn'], controls=controls) self.report('Checking %u objects' % len(res)) error_count = 0 for object in res: + self.dn_set.add(str(object.dn)) error_count += self.check_object(object.dn, attrs=attrs) if DN is None: @@ -698,18 +702,21 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) return 0 def process_metadata(self, val): - '''Read metadata properties and list attributes in it''' + '''Read metadata properties and list attributes in it. + raises KeyError if the attid is unknown.''' - list_att = [] + set_att = set() + list_attid = [] repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob, str(val)) obj = repl.ctr for o in repl.ctr.array: att = self.samdb_schema.get_lDAPDisplayName_by_attid(o.attid) - list_att.append(att.lower()) + set_att.add(att.lower()) + list_attid.append(o.attid) - return list_att + return (set_att, list_attid) def fix_metadata(self, dn, attr): @@ -989,11 +996,52 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) nmsg = ldb.Message() nmsg.dn = dn nmsg[attr] = ldb.MessageElement(replBlob, ldb.FLAG_MOD_REPLACE, attr) - if self.do_modify(nmsg, ["local_oid:1.3.6.1.4.1.7165.4.3.14:0"], + if self.do_modify(nmsg, ["local_oid:%s:0" % dsdb.DSDB_CONTROL_DBCHECK_MODIFY_RO_REPLICA, + "local_oid:1.3.6.1.4.1.7165.4.3.14:0"], "Failed to fix attribute %s" % attr): self.report("Fixed attribute '%s' of '%s'\n" % (attr, dn)) + def err_replmetadata_unknown_attid(self, dn, attr, repl_meta_data): + repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob, + str(repl_meta_data)) + ctr = repl.ctr + for o in ctr.array: + # Search for an invalid attid + try: + att = self.samdb_schema.get_lDAPDisplayName_by_attid(o.attid) + except KeyError: + self.report('ERROR: attributeID 0X%0X is not known in our schema, not fixing %s on %s\n' % (o.attid, attr, dn)) + return + + + def err_replmetadata_unsorted_attid(self, dn, attr, repl_meta_data): + repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob, + str(repl_meta_data)) + ctr = repl.ctr + found = False + + self.report('ERROR: unsorted attributeID values in %s on %s\n' % (attr, dn)) + if not self.confirm_all('Fix %s on %s by sorting the attribute list?' + % (attr, dn), 'fix_replmetadata_unsorted_attid'): + self.report('Not fixing %s on %s\n' % (attr, dn)) + return + + # Sort the array, except for the last element + ctr.array[:-1] = sorted(ctr.array[:-1], key=lambda o: o.attid) + + replBlob = ndr_pack(repl) + + nmsg = ldb.Message() + nmsg.dn = dn + nmsg[attr] = ldb.MessageElement(replBlob, ldb.FLAG_MOD_REPLACE, attr) + if self.do_modify(nmsg, ["local_oid:%s:0" % dsdb.DSDB_CONTROL_DBCHECK_MODIFY_RO_REPLICA, + "local_oid:1.3.6.1.4.1.7165.4.3.14:0", + "local_oid:1.3.6.1.4.1.7165.4.3.25:0"], + "Failed to fix attribute %s" % attr): + self.report("Fixed attribute '%s' of '%s'\n" % (attr, dn)) + + def is_deleted_deleted_objects(self, obj): faulty = False if "description" not in obj: @@ -1125,8 +1173,8 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) return 1 obj = res[0] error_count = 0 - list_attrs_from_md = [] - list_attrs_seen = [] + set_attrs_from_md = set() + set_attrs_seen = set() got_repl_property_meta_data = False got_objectclass = False @@ -1181,7 +1229,31 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) # We don't continue, as we may also have other fixes for this attribute # based on what other attributes we see. - list_attrs_from_md = self.process_metadata(obj[attrname]) + try: + (set_attrs_from_md, list_attid_from_md) = self.process_metadata(obj[attrname]) + except KeyError: + error_count += 1 + self.err_replmetadata_unknown_attid(dn, attrname, obj[attrname]) + continue + + if sorted(list_attid_from_md[:-1]) != list_attid_from_md[:-1]: + error_count += 1 + self.err_replmetadata_unsorted_attid(dn, attrname, obj[attrname]) + else: + # Here we check that the first attid is 0 + # (objectClass) and that the last on is the RDN + # from the DN. + rdn_attid = self.samdb_schema.get_attid_from_lDAPDisplayName(dn.get_rdn_name()) + if list_attid_from_md[-1] != rdn_attid: + error_count += 1 + self.report("ERROR: Not fixing incorrect final attributeID in '%s' on '%s', it should match the RDN %s" % + (attrname, str(dn), dn.get_rdn_name())) + + if list_attid_from_md[0] != 0: + error_count += 1 + self.report("ERROR: Not fixing incorrect inital attributeID in '%s' on '%s', it should be objectClass" % + (attrname, str(dn))) + got_repl_property_meta_data = True continue @@ -1274,7 +1346,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) if (not flag & dsdb.DS_FLAG_ATTR_NOT_REPLICATED and not flag & dsdb.DS_FLAG_ATTR_IS_CONSTRUCTED and not self.samdb_schema.get_linkId_from_lDAPDisplayName(attrname)): - list_attrs_seen.append(str(attrname).lower()) + set_attrs_seen.add(str(attrname).lower()) if syntax_oid in [ dsdb.DSDB_SYNTAX_BINARY_DN, dsdb.DSDB_SYNTAX_OR_NAME, dsdb.DSDB_SYNTAX_STRING_DN, ldb.SYNTAX_DN ]: @@ -1345,17 +1417,17 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) else: self.report("Not fixing isDeleted originating_change_time on '%s'" % str(dn)) - for att in list_attrs_seen: - if not att in list_attrs_from_md: - if show_dn: - self.report("On object %s" % dn) - show_dn = False - error_count += 1 - self.report("ERROR: Attribute %s not present in replication metadata" % att) - if not self.confirm_all("Fix missing replPropertyMetaData element '%s'" % att, 'fix_all_metadata'): - self.report("Not fixing missing replPropertyMetaData element '%s'" % att) - continue - self.fix_metadata(dn, att) + + for att in set_attrs_seen.difference(set_attrs_from_md): + if show_dn: + self.report("On object %s" % dn) + show_dn = False + error_count += 1 + self.report("ERROR: Attribute %s not present in replication metadata" % att) + if not self.confirm_all("Fix missing replPropertyMetaData element '%s'" % att, 'fix_all_metadata'): + self.report("Not fixing missing replPropertyMetaData element '%s'" % att) + continue + self.fix_metadata(dn, att) if self.is_fsmo_role(dn): if "fSMORoleOwner" not in obj and ("*" in attrs or "fsmoroleowner" in map(str.lower, attrs)): @@ -1363,7 +1435,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base))) error_count += 1 try: - if dn != self.samdb.get_root_basedn(): + if dn != self.samdb.get_root_basedn() and str(dn.parent()) not in self.dn_set: res = self.samdb.search(base=dn.parent(), scope=ldb.SCOPE_BASE, controls=["show_recycled:1", "show_deleted:1"]) except ldb.LdbError, (enum, estr): diff --git a/python/samba/samdb.py b/python/samba/samdb.py index e3a6292..817fbdb 100644 --- a/python/samba/samdb.py +++ b/python/samba/samdb.py @@ -39,6 +39,7 @@ class SamDB(samba.Ldb): """The SAM database.""" hash_oid_name = {} + hash_well_known = {} def __init__(self, url=None, lp=None, modules_dir=None, session_info=None, credentials=None, flags=0, options=None, global_schema=True, @@ -793,7 +794,19 @@ accountExpires: %u return dsdb._dsdb_get_nc_root(self, dn) def get_wellknown_dn(self, nc_root, wkguid): - return dsdb._dsdb_get_wellknown_dn(self, nc_root, wkguid) + h_nc = self.hash_well_known.get(str(nc_root)) + dn = None + if h_nc is not None: + dn = h_nc.get(wkguid) + if dn is None: + dn = dsdb._dsdb_get_wellknown_dn(self, nc_root, wkguid) + if dn is None: + return dn + if h_nc is None: + self.hash_well_known[str(nc_root)] = {} + h_nc = self.hash_well_known[str(nc_root)] + h_nc[wkguid] = dn + return dn def set_minPwdAge(self, value): m = ldb.Message() diff --git a/python/samba/tests/dcerpc/integer.py b/python/samba/tests/dcerpc/integer.py new file mode 100644 index 0000000..1a392ee --- /dev/null +++ b/python/samba/tests/dcerpc/integer.py @@ -0,0 +1,203 @@ +# Unix SMB/CIFS implementation. +# Copyright (C) Andrew Bartlett <abart...@samba.org> 2015 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +"""Tests for integer handling in PIDL generated bindings samba.dcerpc.*""" + +from samba.dcerpc import server_id, misc, srvsvc +import samba.tests + +class IntegerTests(samba.tests.TestCase): + + def test_uint32_into_hyper(self): + s = server_id.server_id() + s.unique_id = server_id.NONCLUSTER_VNN -- Samba Shared Repository